New mixed reality features for professional applications put the personal back in computing. Virtual displays and experiential computing might replace desktops for designers and engineers.

I have reported several times on Finnish XR start-up Varjo, since I first recognized the potential of their concept of “foveated resolution.” I reported most recently a few weeks ago from AWE Europe in Munich (Varjo Pushes The Envelope on VR Performance). So, I wasn’t sure whether to accept their invitation to a brief meeting in London as they started out on a press tour in Europe and the US. As I was the first meeting on that day, it turns out that I was the first outside the company and its partners to see a really exciting new development using their XR1 headset.

The company has developed a new 2D/3D interface, which is code-named Varjo Workspace. The company describes it as a dimensional interface and it’s designed for Microsoft Windows and 3D software applications. The new Varjo Workspace enables users to switch between real, virtual, and mixed reality modes and stay within the world of the headset. It enables the use of virtual monitors that float in front of the viewer’s face and ends the awkwardness of working in a 3D environment like Unity, Unreal Engine, or 3ds Max and switching back to a 2D interface application like Word, or web browser.

The demo started out by putting me in the latest headset, which has just been made available as an early development product. As I was strapped into the headset, dual 4K Windows displays were immediately mapped onto the view of the real world coming from the headset’s cameras. The headset is a mixed reality device, which relies on high-quality cameras to map the outside world onto the display.

Varjo has done a lot of work to create a very fast video mapping pipeline which they say takes just 12 ms to map the video to the screen: 6 ms is taken in exposure time for the camera capture, 6 ms for processing, mapping, and synchronizing to the display.

The image is so responsive that I was able to handwrite my notes on the experience while wearing the headset—something I could not imagine with any other mixed or VR systems that I have tried.

The company has developed technology to allow the mapping of virtual Windows displays into the view of the real world in the headset. That’s great as the technology, combined with the high-resolution central area of the display, allows the operation of any 2D app. Meaning, it’s possible for people to work within 2D and 3D worlds.

In the second demonstration, a 3D car was rendered as though it was in a showroom. However, the Unity interface could be used to simply drag and drop items from the Unity menu into the 3D space in a very intuitive way. I quickly found myself dropping items into the scene that accurately scaled to the way they would look in ‘the real world.’

The next demonstration was of mixed reality. A couple of simple spheres were shown. The Varjo technology allows the lighting from the room to be analyzed and exploited so that a virtual reflection of my own hand could be mapped onto one of the spheres in realtime—very impressive.

Next up was a demonstration of a planetary simulation. The idea here was to show how virtual objects in the mixed reality world could be used to accurately occlude the background, which was being captured by cameras. It also showed how sophisticated lighting effects—such as a glow around the sun, added to the experience. It seems to me that the possibility of occluding the view of the world behind virtual objects is a key advantage of mixed reality.

The Pièce de Résistance

However, the really (and I mean really) impressive demo was the next one. First, I was instructed to look to my right where there was a door—like one between an aircraft cabin and the cockpit. I got up out of my chair and moved into the cockpit area of an airliner that was modeled down to the smallest detail. I sat in the captain’s chair and was able to inspect the different control systems. The headset gave enough resolution that I was able to read the detail on the radar screen. Impressive indeed! All the while, real-world items could have been mapped into the simulation (at I/Itsec, the system was being shown with physical controls mapped in).

When I turned around to look back, I could see right through the doorway, of course. It really was very impressive.

I don’t remember ever giving a spontaneous round of applause at a demo before!

The mixture of virtual and real-world images has been used by Volvo to design car interiors. The models can be shown in the glasses and mapped to the position of the real dashboard, so a current car can be driven around while a designer can sit in the car and see how the interior looks.

Now, the headset is heavy. It’s not cheap, it’s tethered, and I only saw a demonstration, but it really shows how mixed reality can have a dramatic effect on 3D design, simulation, and content creation.

Varjo originally captured my interest because of its innovative display technology, but the company has strengths with its own gaze tracking technology, very good and fast video integration, and systems integration with software.

What do we think?

Varjo has several considerable breakthroughs that set it apart from the current generation of mixed reality display companies. The company’s use of foveated resolution tricks has been expedient and has gotten it over the considerable hump presented by headset technology that cannot produce the required image quality. Foveated resolution delivers the highest possible resolution to a focused area of the display. Varjo has coupled foveated resolution with its eye-tracking technology to deliver the high resolution where the viewer is looking. The company’s ability to use high-quality video cameras as the headset’s “eyes” and map what they see to the display fast enough that the wearer isn’t aware of the process is another significant differentiation. Finally, the Varjo Workspace brings the company’s strengths together in an interface that melds 2D, 3D, reality, and virtual reality enabling users to work comfortably in a headset.

This article originally appeared in Display Daily, an on-line and free blog on all things display. We’ve added JPR’s take in the ‘What do we think?’ section.