Lytro computational photography redefines the snapshot

By capturing the entire light field, instead of a plane of light brought into focus, Lytro opens up new ways to capture and manipulate photographs. Could this be photography’s next disruptive technology?

A well-funded startup based on years of research is close to introducing a consumer product that promises to transform how people take and use photographs. Later this year Lytro will start selling a light field camera that captures all the light rays in a scene. The massive amount of data it captures allows images to be focused after the picture is taken. The technology has the potential to redefine both still imagery and video, in both 2D and 3D.

light field image 1 — This image from Lytro illustrates the concept of the light field, which fully defines how a scene appears. It is the amount of light traveling in every direction through every point in space, all the light rays in a scene. Conventional cameras cannot record the light field. (Source: Lytro)

The technology is called computational photography and is based on recent research in light field science. In regular digital or film photography, a plane of light is brought into focus by a lens and captured by a sensor or by film. Lytro captures the entire light field including the vector of every light pixel. The sensor is the invention of Dr. Ren Ng, who did his Ph.D. work at Stanford on light field science, winning the ACM Dissertation Award in 2007. Ng’s research focused on miniaturizing the input from more than 100 cameras plugged in to a supercomputer. Lytro plans to introduce its first product, a consumer camera, in time for Christmas this year.

Once the light field is captured, Lytro software allows multiple versions of the image to be computed from the same captured light field data. Using an image of a person in the foreground and a building in the background as an example, the viewer can bring the person into focus, or the building, or anything else captured in the photograph. The light field sensor captures the color, intensity and direction of every light ray. The result is a photo that more closely imitates how the human eye sees, where every part of the field of view is in focus when it becomes the subject of attention. The output from the first generation Lytro camera will not be 3D geometry plus color, but multiple 2D perspective images in color, computed on request.

Two specific elements make Lytro’s approach to photographic special: One, instead of capturing a plane of light, they capture a field of light; every pixel has a light vector; two, Lytro uses software algorithms, i.e. computational photography, to compute multiple images from the same captured data.

light field image 2 — Recording light fields requires an innovative, new kind of sensor called a light field sensor. The light field sensor captures the color, intensity and vector direction of the rays of light. This directional information is completely lost with traditional camera sensors, which simply add up all the light rays and record them as a single amount of light. (Source: Lytro)

Tight-lipped operation

A request for information beyond what Lytro has released to the press and to eager potential customers was mostly met with a “we aren’t commenting” reply. Lytro won’t discuss the vendors for any of the components in the camera, what manufacturer will be building the cameras, the price of the first camera, or even how much memory the Lytro camera 1.0 will carry.

Lytro was willing to offer a bit of “informed speculation” about the potentially disruptive nature of its technology. “Given that we capture so much information in a scene, the sky is the limit on what we can do,” says Kira Wampler, vice president of marketing. “Eventually we could have a 3D holographic camera that ‘sees’ like your brain does. This is possible because the software is built around 3D vector graphics. Using the full light field, Lytro cameras will allow the user to easily switch between 2D and 3D views or shift the perspective of the scene.”

Video is also “definitely possible with light field technology,” Wampler added. “At this point Lytro is focused on bringing the first still consumer light field camera to the market, but there are plans for video on the long-term roadmap.”

There are other ways Lytro’s new camera could be a disruptive technology, once photographers get their hands on it:

There is no need for auto-focus
It works in all kinds of light conditions
It is fast
All the image work is done in post-processing
Images are immediately in perspective view, what many CAD users would call 2-1/2D.

The revolutionary possibilities

Lytro’s computational photography method separates image capture from image processing. Even if Lytro were not also capturing the complete light field, this one difference could revolutionize photography. But the added information from the light field opens new possibilities for both still imagery and video.

The idea of computational photography is not new; similar technology was used to correct the errors in the Hubble telescope’s lens a few years ago. Adobe is also working on light field photography; they demonstrated a multi-lens camera earlier this year that looks like the eye of a house fly blown up to poster size.

The Lytro differences—what makes it practical to sell as a consumer product—are the new ability to put enormous computing power inside a handheld camera, and what appears to be breakthrough research from Ng and his team.

light field image 3 — This is a static image illustrating how two views can be created from the same light field data. (Source: Lytro)

Click on different parts of the image below to see the results of a Lytro photo. (Credit: Eric Chen of Lytro: Wushu master Eric Dull in Golden Gate Park, San Francisco, California.)