Facebook is developing a way to type using VR on tabletops
Hand tracking and VR, typing away with no keyboard
Scott SteinEditor at Large
I started with CNET reviewing laptops in 2009. Now I explore wearable tech, VR/AR, tablets, gaming and future/emerging trends in our changing world. Other obsessions include magic, immersive theater, puzzles, board games, cooking, improv and the New York Jets. My background includes an MFA in theater which I apply to thinking about immersive experiences of the future.
ExpertiseVR and AR, gaming, metaverse technologies, wearable tech, tabletsCredentials
Nearly 20 years writing about tech, and over a decade reviewing wearable tech, VR, and AR products and apps
Doing office work in VR isn't easy. It isn't just the lack of software plugins for VR and most productivity apps, since VR headsets can already transform into wearable monitors. The keyboard part is what gets really frustrating. Facebook's working on a way to bypass physical keyboards altogether with hand tracking.
Facebook Reality Labs' latest research project, being presented at the Symposium on User Interface and Software Technology, demonstrates hand-typing on a desk at up to 73 words per minute. The project isn't ready for Oculus Quest VR headsets yet, which already have hand tracking but lack the support for typing input. The research project looks to be using extra tracking sensors right now, along with specialized gloves studded with motion-capture markers. But the idea is to eventually evolve the Oculus VR hand tracking to enable typing, just like it does hand tracking now.
"To support touch typing without a physical keyboard -- and without the benefit of haptic feedback from individual keys -- the team had to make sense of erratic typing patterns," Facebook's post says of the research project. Facebook's typing AI is adapted from speech recognition, but searching for particular keystroke motions amid the general finger movement. It works predictively to anticipate letters as they're being typed. The research so far is producing typing at "73 words per minute with a 2.4% uncorrected error rate using their hands, a flat surface, and nothing else," Facebook's post claims.
Facebook's already supporting certain VR-optimized physical keyboards with Oculus Quest VR by the end of the year, but keyboard-free typing could be a way for VR and eventually smart glasses to work without any accessories at all.