HPI demonstrates new AR/MR haptic feedback technology with Microsoft’s HoloLens

New research shows how haptic feedback can be achieved in augmented and mixed reality.

Last month we reported on new VR-based haptic feedback technology that may become a central part of future virtual reality (VR) packages. Now, a project run by the Hasso Plattner Institute (HPI) from the University of Potsdam is attempting to bring similar functionalities to augmented (AR) and mixed reality (MR) devices.

Haptic feedback refers to the use of touch in user interfaces. Simple examples would be the “rumble” or DualShock controllers for the PlayStation, or even the act of using a touchscreen to unlock your smartphone. Traditionally almost all of our technologies have relied at least partially on haptic feedback; driving a car, typing on a keyboard, manipulating a computer mouse, and so forth.

**An AR/MR haptic solution would require users to be able to interact with both physical and virtual objects simultaneously. (Source: Hasso Plattner Institute)**

This is why creating haptic technologies for new mixed reality solutions is so difficult. If the object you’re interacting with is virtual, how do you properly simulate the act of ‘touching’ it or understanding its weight and textile sensation? Additionally, with an AR headset users may have to come into contact with both virtual and physical objects, so a fully virtual solution as those employed in VR won’t cut it.

The solution that HPI came up with involves the use of electronic muscle stimulation (EMS) working alongside the Microsoft HoloLens AR headset. EMS has been widely used by medical practitioners and sports scientists, and refers to the stimulation of particular muscles (causing them to contract) using electric impulses. EMS can be used to stimulate muscles based on virtual input and thus create the sensation of touching and feeling the weight of a virtual object.

**EMS devices can be worn relatively discretely and are lightweight compared to other force feedback systems. (Source: Hasso Plattner Institute)**

The institute used EMS to add force feedback to a variety of simulations created to demonstrate the technology. This includes a game where the user has to balance a virtual marble on a physical tray, with their system simulating the weight of the marble as it moves around the tray. In another game, players could load and fire a virtual catapult, while another presented an “escape room” with a puzzle that had to be solved by using both virtual and physical props.

While commercial implementations of such systems probably won’t happen too soon, it’s exciting to see the continuing potential which AR/VR technologies are ushering in. We hope to hear more from the team, as well as any potential HoloLens 2 or 3 collaborations.

Check out the video below for a closer look at what the team’s haptic technologies can do.