Facebook is developing a way to type using VR on tabletops

Hand tracking and VR, typing away with no keyboard

Scott Stein Editor at Large
I started with CNET reviewing laptops in 2009. Now I explore wearable tech, VR/AR, tablets, gaming and future/emerging trends in our changing world. Other obsessions include magic, immersive theater, puzzles, board games, cooking, improv and the New York Jets. My background includes an MFA in theater which I apply to thinking about immersive experiences of the future.
Expertise VR and AR, gaming, metaverse technologies, wearable tech, tablets Credentials
  • Nearly 20 years writing about tech, and over a decade reviewing wearable tech, VR, and AR products and apps
Scott Stein
2 min read

Facebook Reality Labs' VR typing research is working, but with specialized marker-gloves for now. Eventually it could be on headsets.


Doing office work in VR isn't easy. It isn't just the lack of software plugins for VR and most productivity apps, since VR headsets can already transform into wearable monitors. The keyboard part is what gets really frustrating. Facebook's working on a way to bypass physical keyboards altogether with hand tracking.

Facebook Reality Labs' latest research project, being presented at the Symposium on User Interface and Software Technology, demonstrates hand-typing on a desk at up to 73 words per minute. The project isn't ready for Oculus Quest VR headsets yet, which already have hand tracking but lack the support for typing input. The research project looks to be using extra tracking sensors right now, along with specialized gloves studded with motion-capture markers. But the idea is to eventually evolve the Oculus VR hand tracking to enable typing, just like it does hand tracking now.


Facebook's motion model sees the keystrokes as they're happening in 3D space.


"To support touch typing without a physical keyboard -- and without the benefit of haptic feedback from individual keys -- the team had to make sense of erratic typing patterns," Facebook's post says of the research project. Facebook's typing AI is adapted from speech recognition, but searching for particular keystroke motions amid the general finger movement. It works predictively to anticipate letters as they're being typed. The research so far is producing typing at "73 words per minute with a 2.4% uncorrected error rate using their hands, a flat surface, and nothing else," Facebook's post claims.

Facebook's already supporting certain VR-optimized physical keyboards with Oculus Quest VR by the end of the year, but keyboard-free typing could be a way for VR and eventually smart glasses to work without any accessories at all.