The advanced AR hand tracking technology no one is using

PerceptiveIO showed off impressive AR tracking technology last year, but the company has gone dark in the meantime.

A recent article from Next Reality highlighted new AR technology capable of low-latency hand tracking. The team behind the tech is a company called PerceptiveIO, a San Francisco based startup made up partly of former Microsoft engineers. There are familiar names here, including Jeff Han whose company Perceptive Pixel was acquired by Microsoft in 2012, and Sharam Izadi a lead researcher at Microsoft who pioneered teleportation.

The company demonstrated its new technology at Siggraph Asia 2017, showing how their system could accurately recognize hand gestures at a rate of up to 1,000 frames per second.

The company describes the technology as “articulated distance fields for ultra-fast hand tracking.” According to its Siggraph presentation, “the key contribution is an articulated distance function that, for any pose, yields a closed form calculation of both the distance to the detailed surface geometry and the necessary derivatives to perform gradient based optimization.”

Okay, so that’s a lot to take in, but basically they created an algorithm capable of ultra-fast hand tracking by using a specialized model for hand geometry. The high frame rate even allows the system to capture subtle hand movements such as crossed fingers or a thumb poking through a fist.

The technology also has no problem dealing with high speed gestures from both hands. The company employs a fully convolutional network which predicts whether a pixel belongs to the left or right hand by using probability maps. The system can also look at your hand and to a certain extent predict how it’s going to move next based on its position, posture, and direction of movement.

The technology allows for ultra-fast tracking of both hands as they interact. (Source: Siggraph Asia 2017)

Both the accuracy and speed of movement, as well as the interaction between the two hands are based on recent scientific research into detailed hand geometry, partly conducted while some of the team members were still working at Microsoft.

Unfortunately we can’t tell you much more since it appears the company’s website has gone dark. The website’s squarespace registration had earlier expired; however, the address ( still functioned until it was cut off this week around Wednesday. All that remains of the startup is its business registration, LinkedIn page, and a sparse YouTube channel. It has been speculated that this is due to a dispute between the company and Microsoft regarding ownership of technologies; however, neither party has confirmed or denied this as of publication.

Whatever may have happened, the technologies showcased promises a much higher fidelity future for AR tracking. Anyone who has used the full body tracker on the Kinect, or even the specialized Leap Motion hand tracker can attest to the difficulty these devices have in capturing high speed or precise movements. Hopefully we will see a version of this tech in future AR/VR devices, because it would be a shame to see its potential untapped.

Check out the video below for an in-depth look at PerceptiveIO’s tracking system.

A detailed look at the high speed hand tracking system showcased at Siggraph Asia 2017. (Source: ‘Research in Science and Technology’ YouTube channel)