Going places in a headset.

Finnish-based Varjo has added realtime chroma-keying and object tracking to its XR-1 Developer Edition headset. Chroma-keying, also known as green-screening, has been used extensively in film making and broadcasting. Varjo has added that capability to its toolkit and is claiming to be the first company to do so for mixed reality devices. With object tracking, says the company, users can anchor any virtual objects to the real world using printable visual markers.

Those two features (chroma-key and object track) allow seamless integration of virtual and real worlds, and provide interactivity with photorealistic virtual content. Varjo says developers can achieve pixel perfect accuracy and occlusion inside mixed reality.

The ability to use Chroma-keying to combine the real and the digital adds verisimilitude to mixed reality applications. Users will really “be” there. (Source: Varjo)

Chroma-keying could be useful in professional workflows where aligning virtual content accurately with the physical world is crucial. Users could define parts of reality, identify them with color and replace them with virtual models or scenery, and, says the company, do that without heavy development costs. With chroma-key, virtual content also occludes perfectly with real-world objects or hands, allowing intuitive interactions. Varjo says, using their object tracking and visual markers, professionals can make virtual objects appear exactly where they want them in their surroundings. Example use cases include:

• Conduct training and simulation: A pilot can sit in a replica of a plane or helicopter cockpit and be able to look outside and see oneself flying in an ultra-immersive visual scenery, while operating physical cockpit controllers for realistic training. Chroma-keying also enables multi-user training scenarios.
• Design the products of tomorrow: An automotive designer can sit in a car and replace parts of the interior with designs that are not yet built in reality with the click of a button. Designers can also collaborate in an immersive mixed reality space, interacting with virtual models and making changes to them in realtime, or virtually ‘dress’ 3D prints to look like material-finished products.
• Run academic, clinical, and commercial research: Researchers can conduct studies inside life-like mixed reality, simultaneously combining virtual and real world elements into the research environment. The subjects can hold virtual products or instruments in their hands and interact with them.

Chroma-keying and object tracking are available in early access to all users of the XR-1 Developer Edition headset. The XR-1, says the company, is the only device capable of streaming high-resolution video of the reality to the user without any observable latency, enabling tweaking of every single pixel that the user sees inside the headset in photorealistic fidelity.

What do we think?

Varjo has jumped out front as a leading MR HMD supplier. The quality of the video mixing while wearing the HMD is the best we have seen. Adding chroma-keying and marker tracking is a logical and needed extension. Getting chroma-keying to work accurately with a moving device like an HMD is no trivial matter and we’re anxious to get a demonstration of this new capability.