When it comes to digital characters, CG artists and animators have looked for solutions to help them create lifelike performances for realistic-looking characters.

DI4D was founded in January 2003 to commercialize research carried out at the University of Glasgow into stereo photogrammetry, a method of deriving 3D data (or 3D scans) from just two camera views of the same scene. At that time, laser scanning was the state of the art in 3D surface capture, but it was not ideal for capturing humans because they had to remain completely still for several seconds while they were scanned.

Alternatively, DI4D began using digital still cameras and photogrammetry, revolutionizing human facial capture because it is able to capture 3D shape and appearance (texture) simultaneously in a single flash. The company’s vision was to bring realistic virtual characters to life using photogrammetry to capture high-fidelity data directly from real-life humans. Today, photogrammetry has become the dominant method for capturing 3D facial shape and likeness.

That led DI4D to pioneer 4D surface capture, which extends 3D surface capture into the fourth dimension of time. By using photogrammetry with synchronized video cameras, it is possible to capture time sequences of 3D scans of a deforming surface, such as a human face, while it is talking or performing. 

However, even a few seconds of this type of “raw” 3D scan per frame data is huge in size, and with a different mesh topology per frame, the results are difficult to use for animation purposes. But DI4D has developed an optical flow-based tracking solution that enables 4D capture without the need for physical markers or special makeup, and without the need for a complex facial animation rig.

DI4D has developed two different systems for 4D capture. The first is the DI4D Pro system, which captures the highest-fidelity 4D data possible for a single-seated actor, allowing busy actors to be captured with minimal inconvenience. DI4D Pro is often used to capture high-fidelity expression data for facial rig creation or performance data for VFX face replacement shots, the company says.

The company’s second 4D capture system is the DI4D HMC, a wireless, helmet-mounted stereo video camera system that an actor wears, allowing them to act and move around freely while their facial performance is recorded. A streamlined version of DI4D’s 4D pipeline is then used to process the video data acquired by the stereo cameras to produce high-fidelity facial animation. A key benefit of HMC capture is that it allows facial performance data to be acquired from multiple actors simultaneously with body motion capture and audio recording, a combination that is known as full-performance capture (PCAP). PCAP allows more natural performances to be captured than other approaches that require each character’s facial, body, or vocal performance to be captured separately and then “Frankensteined” together.

Whichever system is used to acquire the data, DI4D’s proprietary 4D processing pipeline reproduces a dense 3D scan of the actor’s face in every frame of the captured performance. This means that the shape of the actor’s face does not need to be inferred from sparse data, such as markers or prominent facial features, as required by other facial capture solutions, explains CEO and DI4D co-founder Colin Urquhart, a 30-year industry veteran and innovator in the field of facial capture and animation. As a result, the company is able to produce facial animation that faithfully captures the subtlety and intricacies of an actor’s performance, delivering lifelike digital doubles and virtual human characters.

The two approaches to 4D capture offer different advantages and disadvantages: DI4D Pro produces the highest fidelity data but only for a single-seated actor, while DI4D HMC allows multiple actors to be captured simultaneously while moving around freely but resulting in lower fidelity data. To this end, DI4D has developed a hybrid facial animation solution called Pure4D that is designed to combine the benefits of both approaches to 4D capture. It combines 4D facial performance data captured using DI4D HMC with higher fidelity 4D expression data of the same actor captured with the DI4D Pro system. This results in facial animation with the fidelity and nuance of the DI4D Pro system but with performance that faithfully matches that captured with the DI4D HMC system.

Pure4D also uses machine learning to reduce the amount of manual cleanup that is required to create the best quality 4D data.

As the demand for more and more realistic digital human characters continues to grow, generating truly believable facial animation with traditional tools is becoming increasingly challenging. 4D facial capture for digital doubles will become an effective tool for animators to deliver high-quality storytelling experiences.