Facebook Reality Labs talks hand tracking technology

Facebook Reality Labs (FRL) works on many projects intending to make interacting with technology more approachable. Technology FRL is working on includes VR headsets and AR glasses. Hand tracking technology is particularly important to virtual reality allowing the tech to be more immersive with a "come as you are" approach to human interaction.The work being performed at FRL aims to build more natural devices and began with Touch controllers. These controllers were designed to be comfortable to hold and had lots of sensors inside. Touch controllers delivered life-like hand presence and made basic VR interactions feel more like the real thing.

FRL is working towards eliminating the need for hand controllers and instead sensing the bare human hand. The lab began working on hand tracking for VR over five years ago and at the time no one never shipped consumer-quality, controller-free hand tracking capability. That meant the technology had to be created from scratch.

The video below follows the development process showing prototypes of some of the devices that led to the final product. FRL is currently exploring new experimental forms of text input, a common and critical task for productivity. FRL has proposed a novel method to enable touch typing on any flat surface without a keyboard using hand motion from a marker-based hand tracking system. That system inputs and decodes the motion directly into the text the user intended to type.

Researchers note that the technology is still in the early research phase but illustrates the potential of hand tracking for activity scenarios. Researchers had to make sense of erratic typing patterns and adopted statistical decoding techniques from automatic speech recognition using a motion model to predict keystrokes from hand motion. Using the method, typists averaged 73 words per minute with a 2.4 percent uncorrected error rate using their hands, a flat surface, and nothing else. Typists achieved similar speed and accuracy using the new system compared to when they used an actual keyboard.