Tricking the brain: New dimensions of haptics in virtual reality

André Zenner with the “Drag:on” controller, which consists of two flamenco fans. © Oliver Dietze/DFKI

How can virtual reality (VR) be experienced haptically, i.e., through the sense of touch? This is one of the fundamental questions that modern VR research is investigating. Computer scientist André Zenner, who is based in Saarbrücken, Germany, has come a significant step closer to answering this question in his doctoral thesis – by inventing new devices and developing software-based techniques inspired by human perception. He has now been awarded the prestigious “Best Dissertation Award” at the world’s leading VR conference.

The award-winning work is about how physical props (technical term: “proxies”) can be used to make objects in virtual environments tangible. “Of course, you can’t have a proxy for every virtual object, then the approach wouldn’t be scalable. In my dissertation, I, therefore, thought about what devices could look like that could be used to simulate the physical properties of several different virtual objects as effectively as possible,” explains André Zenner, who completed his doctorate at the Saarbrücken Graduate School of Computer Science at Saarland University and is now conducting research at Saarland University and the German Research Center for Artificial Intelligence.

This resulted in the prototypes for two special VR controllers, “Shifty” and “Drag:on”. VR controllers are devices that can be held in the user’s hand to control or manipulate objects in virtual reality using tracking technology.

“Shifty” is a tubular controller in which a movable weight is installed. The weight can be moved along the lengthwise axis by a motor, changing the center of gravity and inertia of the rod. “In combination with corresponding visualizations in virtual reality, Shifty can be used to create the illusion that a virtual object is getting longer or heavier,” explains André Zenner. In experiments, he was able to prove that objects are perceived as lighter or smaller when the weight is close to the user’s hand and that, coupled with the corresponding visual input; they are perceived as longer and heavier the further the weight in the rod moves away from the user. “This is mainly due to changes in the inertia of the controller, as the overall weight does not change,” explains André Zenner. The research and development department of gaming giant Sony is already experimenting with this concept and cites André Zenner’s work in the development of new VR controllers.

The second controller, “Drag:on,” consists of two flamenco fans that can be unfolded using servomotors, thus increasing the air resistance of the controller. This means that the further the fans are unfolded, the more force the user has to exert to move the controller through the air. “Coupled with the right visual stimuli, Drag:on can be used to create the impression that the user is holding a small shovel or a large paddle, for example, or that they are pushing a heavy trolley or are twisting a knob that is difficult to turn,” explains André Zenner. Both controllers are basic research and so-called “proof of concepts.” This means that the prototypes can be used to show in user experiments that different controller states can improve the perception of different VR objects. Still, specific products using this technology are not yet available on the market.

With the controllers, the Saarbrücken-based computer scientist first addressed the so-called ‘similarity problem.’ The aim here is to ensure that virtual and real objects feel as similar as possible. In the second part of his work, he dealt with the so-called ‘colocation problem,’ i.e., the question of how the proxy can be spatially located in real life where the user sees it in virtual reality. This is particularly challenging as the controllers act as proxies for different virtual objects. Consequently, the user must be given the illusion that they are reaching for various objects, although in reality, they will always grasp the same proxy.

To achieve this, the researcher made use of the already established method of “hand redirection.” As the name suggests, this involves redirecting the movement of the hand in virtual reality so that the user thinks they are reaching to the left, for example, even though they are actually stretching their hand forward. “We conducted experiments to investigate the point at which users realize that their hand has been redirected. Our results showed that this point was reached quickly, so we thought about how we could better conceal the hand redirection,” says André Zenner. The solution: he tricked the brain by only redirecting the hand when the brain was blind to visual changes – namely during blinking. Together with a student under his supervision, he developed the appropriate software and used the eye trackers built into many VR headsets. In control studies, the team was then able to show that their new controllers, in combination with hand redirection algorithms, led to more convincing VR perceptions than previously possible.

This research achievement has now been recognized internationally. The work entitled “Advancing Proxy-Based Haptic Feedback in Virtual Reality” was awarded the “IEEE VGTC Virtual Reality Best Dissertation Award” in mid-March at the “IEEE Conference on Virtual Reality and 3D User Interfaces”, which took place this year in Orlando, Florida. According to the IEEE, the award is presented each year to the author of the most outstanding dissertation defended in the previous two calendar years in the research fields of virtual and augmented reality. A total of 16 dissertations were submitted to the 2024 edition of the conference and reviewed by an international committee of experts. The thesis was supervised by Prof. Dr. Antonio Krüger, CEO of the German Research Center for Artificial Intelligence and Head of the Ubiquitous Media Lab at Saarland University.

Original Publication: 
Zenner, A. (2022). Advancing Proxy-Based Haptic Feedback in Virtual Reality. Universität des Saarlandes. doi:10.22028/D291-37879 (https://dx.doi.org/10.22028/D291-37879)

Further information:

https://ieeevr.org/vgtc/VR_best_dissertation/index.html

https://www.youtube.com/watch?v=1l0wKk6q_ss (Video “Shifty”)

https://www.youtube.com/watch?v=Wf1Ls4M2afg (Video “Drag:on”)

https://www.youtube.com/watch?v=-NuaQnvyV_8 (Video “Blink-Suppressed Hand Redirection”)

Questions answered by:
Dr.-Ing. André Zenner
Saarland University & German Research Center for Artificial Intelligence
Phone: +49 681 85775 5117
Mail: Andre.Zenner@dfki.de

Background German Research Center for Artifical Intelligence:
The German Research Center for Artificial Intelligence (DFKI) has operated as a non-profit, Public-Private-Partnership (PPP) since 1988. Today, it maintains sites in Kaiserslautern, Saarbrücken, Bremen, Niedersachsen, laboratories in Berlin, Darmstadt and Lübeck as well as branches in Trier. DFKI combines scientific excellence and commercially-oriented value creation with social awareness and is recognized as a major “Center of Excellence” by the international scientific community. In the field of artificial intelligence, DFKI has focused on the goal of human-centric AI for more than 35 years. Research is committed to essential, future-oriented areas of application and socially relevant topics. Currently, with a staff of about 1,560 employees from more than 76 countries, DFKI is developing the innovative software technologies of tomorrow.

Editor:
Philipp Zapf-Schramm
Saarland Informatics Campus
Phone: +49/681/302-70741
Mail: pzapf(at)cs.uni-saarland.de

Background Saarland Informatics Campus:
900 scientists (including 400 PhD students) and approx. 2500 students from more than 80 nations make the Saarland Informatics Campus (SIC) one of the leading locations for computer science in Germany and Europe. Four world-renowned research institutes, namely the German Research Center for Artificial Intelligence (DFKI), the Max Planck Institute for Informatics, the Max Planck Institute for Software Systems, and the Center for Bioinformatics along with Saarland University and its three departments and 24 degree programs, together cover the entire spectrum of computer science.