Going deep—A look at the stereo 3D in Avatar: The Way of Water

Director James Cameron and his tech team at Lightstorm Entertainment worked with one of the industry’s top VFX studios, Weta FX, to redefine visual effects and stereo 3D for the Oscar-winning film Avatar: The Way of Water. Using the latest technical advancements, the teams immersed audiences in the compelling world of Pandora. In celebration of National 3-D Day, we dive deep into the work of this stereo 3D film.

Today, March 21, 2023, is National 3-D Day, founded by 3-D Space in 2020 to recognize the history and resurgence of 3D technology. It is celebrated on the third day of the third full week of the third month, naturally. Last year, we at Jon Peddie Research celebrated 3-D Day with an article that took our audience through the extensive history of stereo 3D. This year, we are marking the occasion by featuring the work of Oscar-winning Avatar: The Way of Water, which has established a new pinnacle in stereo 3D filmmaking.

(Source: National Day Calendar)

Our guide on this journey is Sam Cole, associate visual effects supervisor at Weta FX, who takes us behind the scenes of how stereo 3D was used to immerse viewers into the unique world of Avatar: The Way of Water.

From the outset, Weta FX partnered with Lightstorm Entertainment on the film, collaborating on new technology and techniques to fully realize director James Cameron’s vision. In addition to creating the flora and fauna of Pandora, Weta forged new developments for water in the sequel—no easy feat by any means.

Avatar: The Way of Water picks up a decade after the first film, as Jake Sully is living peacefully with his new family on Pandora. That is until humans return in a second attempt to colonize it and, in the process, destroy it. Meanwhile, Jake and Neytiri relocate to live among the Metkayina, who live along the Pandora shores that are teeming with aquatic life. But, they cannot escape the fight on their doorstep.

The evolution of stereo 3D

Avatar: The Way of Water once again drew people back to the theater, providing them with a sophisticated stereo 3D experience. Stereo 3D has evolved greatly over the past decades, especially since we last visited Pandora. Since then, it is apparent that neither Cameron and Lightstorm nor Weta have rested on their laurels as they sought to present something innovative and new.

As Cole points out, the stereo in the Avatar sequel is more immersive than anything previous, and that’s because filmmakers including Weta have much more experience with the genre and are able to make more enlightened and informed choices that draw the audience in, provide a better sense of depth, “and essentially make the cinema disappear and place people in that world.”

Laser projection and brighter projection, along with the increased quality of 3D glasses, have improved the stereo picture to where every seat in a theater provides good stereo viewing. In fact, increasing the brightness level has a tremendous impact on the viewing experience. “Films mastered for high dynamic range and projected using the latest technologies look amazing in stereo 3D—bright, clear, and with amazing color.” Avatar: The Way of Water is a perfect example.

In terms of the visual effects, Cole notes that the software, pipelines, and processes are more advanced, more mature, and faster today than they were even just a few years ago. This is important, explains Cole, because the more decisions an artist can make per minute, the better the image becomes, generally.

The Way of Water, in stereo 3D

The Avatar sequel used native stereo as opposed to 3D conversion done after the fact, so the stereo workflow and processes were meticulously planned from day one, and the shots were designed to work in stereo. Therefore, there were very few curveballs throughout the filmmaking process, says Cole.

In fact, a live feed of stereo from the set was constantly monitored by Geoff Burdick, senior VP of production services and technology at Lightstorm, who maintained steady contact with Cameron and Maria Battle-Campbell, first assistant director, who were on set watching a separate feed at projection size. They were seeing it directly and making stereo adjustments, calling the shots over the radio as the film was being shot, so there was no guesswork involved. “It makes a big, big difference,” says Cole.

He adds that using finely calibrated cameras and robust rigs are also important to getting a good result.

With stereo, things that artists could get away with in mono are no longer acceptable; cheats typically done in traditional flat compositing or flat VFX will not work. Tolerances for all the camera tracking and matchmoving had to be at sub-pixel level. Eyelines had to be exact. A new performance-driven eyeline system allowed live-action actors to naturally interact with 9-foot-tall CG Na’vi characters on stage using a computer-controlled cable system with a screen and speakers. The performance of the CG characters was displayed at the appropriate location and height on set, depending on the action in the shot, allowing live-action performers to make spontaneous and accurate acting choices in the moment.

With native stereo, a consistent volume must be maintained between the photography and the CG in order for the stereo world to work correctly. To accomplish this, Weta developed a real-time depth compositing system that provides a pixel-perfect, real-time composite in-camera, without a greenscreen, accurately positioning the live-action and CG elements together and producing a very close approximation of the final shot while still on set.

As Cole points out, the water generated a lot of spray, mist, and so forth atop the water surface, which also had to be treated as a volume. “It’s a very tricky shot integration to take a plate of mist and water being thrown in the air, and then insert a CG character right into the middle of it,” says Cole.

Additionally, a first-of-its-kind underwater performance system was developed at Lightstorm that used discrete volumes to capture the precise movements of actors underwater and when they transitioned above the surface, while having a greater capability to capture body and facial motion from up to 23 actors at a time.

Of course, CG water is never easy under any condition, and with stereo, it is doubly difficult. With water being so important in this film, Weta rebuilt its entire simulation approach to enable a new level of realism and interaction above and below the water’s surface.

To help provide a sense of scale as the characters are underwater, artists generated marine snow as a very subtle barrier between the viewer and the characters. Also, the debris had to naturally part in the water as the camera housing pushed through the volume. “That interplay between the plankton and the simulation really gives the impression of being in a big volume of water,” Cole notes.

Furthermore, the water surface is partially reflective, and in stereo, each eye can get a slightly different image, and that can reduce immersion. To rectify this, the Weta team wrote tools that would enable them to reduce or tune the stereo specifically in just the brightest parts of the reflection, making it more immersive.

And that was not the only specialized tool required for the VFX work on this stereo film. There were many.

Why stereo 3D?

A film like Avatar: The Way of Water requires a tremendous amount of technical innovation—in terms of the overall VFX and to accommodate the cutting-edge stereo 3D. Some of the work has been described already, but there is so much more. The Weta team came up with a new facial system called APFS that utilizes a neural network to help establish a muscle-based activation model, measuring the expansion and contraction of the performer’s underlying facial muscle system.

According to Weta, a total of 18.5PB of data was stored for this film—which is 18.5´ the amount used on the original film. In terms of the work required, this film had 10 VFX supervisors and nine animation supervisors; typically, a Weta project would have one or two of each.

So, with stereo 3D requiring so much additional work and development, why bother? “Stereo is the closest way that you have to taking somebody’s eyes and putting them on Pandora. It’s the most immersive and the truest to life—it’s how we see,” says Cole. “If you want to immerse somebody in something, you wouldn’t leave off half of one of the senses. Our stereovision is so ingrained in how we see things, you really want the walls of the theater to disappear and to be drawn into the world. That’s what films do, try to make you believe that you are there, in the middle of it all. Incorporating your stereo sense just furthers your immersion into the film.”

Why, then, aren’t there more stereo films being made? Cole offers a reasonable explanation: Because the process is difficult to do. They are a lot more time consuming to make, they require extra equipment and extra teams to manage the equipment, and more. Yet, Cole believes the appetite for stereo exists—as long as the content is compelling. “You got to make mind-blowing content,” he says, adding that the stereo has to be sympathetic to the story that it’s serving.

And that is certainly the case with Avatar: The Way of Water.