This Sony proto VR controller gets smarter as you use it

A video shared by Sony Interactive Entertainment shows a prototype controller that tracks the entire human hand. This controller was the subject of research done on overcoming the limits presented by gesture control systems and traditional controllers. With this new system, users would be able to utilize their hands and fingers without breaking the illusion that all of their digits are, indeed, being tracked in real time.

This full hand tracking used machine learning and capacitive sensors to break past the point at which previous trackers failed. This controller doesn't pretend like users aren't going to be interacting with handheld tools – the controller is integral in traversing a VR universe in which one is still stuck in a limited environment at home.

Instead, as the research paper suggests, they're using computer vision and well established ways and means of interacting with digital elements in the VR space. "We have introduced 2.5D representation to apply convolutional neural network methods on a capacitive image of the curved surface, and two types of network architectures based on recent achievements in the computer vision field were evaluated with our dataset," said the paper.

To create this new controller system, they created a prototype using 62 individual electrodes. They recorded training datasets of the many ways in which the hand generally moves in games using an optical tracking system. Together, these systems allow hand tracking and the virtual representation of hands in a game to grow better and better as the machine is fed more information.

"We also implemented real-time interactive applications using the prototype and demonstrated the possibility of intuitive interaction using fingers in VR applications," said the researchers in the paper. Above you'll see a demonstration video created by the authors of the paper, as noted below.

You can read the full paper this week courtesy of the ACM DL Digital Library. The paper goes by the name 'Evaluation of Machine Learning Techniques for Hand Pose Estimation on Handheld Device with Proximity Sensor' and can be found with code DOI: 10.1145/3313831.3376712. This paper was authored by Kazuyuki Arimatsu, Sony Interactive Entertainment Inc., Japan, and Hideki Mori, Sony Interactive Entertainment Inc., Japan.