How Facebook AR glasses could become real, thanks to Oculus VR tech

Facebook's Joel Hesch talks about how Oculus Insight tracking is the present and future for VR/AR.

Scott Stein Editor at Large
I started with CNET reviewing laptops in 2009. Now I explore wearable tech, VR/AR, tablets, gaming and future/emerging trends in our changing world. Other obsessions include magic, immersive theater, puzzles, board games, cooking, improv and the New York Jets. My background includes an MFA in theater which I apply to thinking about immersive experiences of the future.
Expertise VR and AR | Gaming | Metaverse technologies | Wearable tech | Tablets Credentials
  • Nearly 20 years writing about tech, and over a decade reviewing wearable tech, VR, and AR products and apps
Scott Stein
4 min read

The Oculus Quest, and its built-in tracking system, may point towards AR glasses next.

Sarah Tew/CNET

Facebook's Oculus has seen a lot of changes lately, with departures by key executives from the mobile VR team, Nate Mitchell and Max Cohen. But Facebook's standalone VR headset released this spring, Oculus Quest, remains the best VR hardware achievement this year... and may indicate where Facebook could be heading next in AR.

Facebook is now talking openly about how Oculus Insight is the technology that will drive future headsets as they move from virtual reality to augmented reality. Insight is the computer vision and AI-based tracking in the Rift S and Quest headsets, which uses in-headset cameras, motion sensors and infrared LEDs in the controllers to map real spaces into VR and make everything feel smooth and fluid. Facebook's not the only company to use this type of tracking in VR, but it's becoming clear that Facebook also sees this tech as the stepping-stone to AR, too, and will be part of what makes Facebook's long-promised AR glasses tech work.

Facebook engineering manager Joel Hesch, who's led development of Oculus Insight, talked to CNET about where the tracking tech could end up.

A mix of VR and actual reality

There are times when Oculus Quest feels like a gateway to a blend of virtual and real, especially when setting up a virtual playspace boundary and stepping into a play-world bubble: The Oculus Insight camera-based tracking is good enough to feel like you're painting boundaries in the real world. The tracking could be used for larger-scale experiences next.

Hesch referred to last year's Oculus Connect 5 conference as an indicator, referring to an Arena Scale Tracking demo last year where players ran around a giant arena with obstacles, playing a Laser Tag-like version of the VR shooter game Dead and Buried. "We had real world objects, these boxes you could hide behind, aligned with virtual world objects. That type of experience gives a flavor of some of the things we're considering, how do we think about mixing real world and virtual world content, and how do we think about multi-user experiences, with more than one person in the space."

Watch this: We took Oculus Quest on vacation

It remains to be seen whether Facebook will enable Oculus Quest to be more customizable for companies making location based experiences, or LBEs (theme park-like VR attractions like The Void). (The "option's not off the table" for makers of experiences to customize Oculus Insight tracking, says Hesch, but admits the priority is working for "as many cases as possible.") Some companies, like MANUVR, are already finding a way to blend Oculus Quest with larger arenas using Microsoft's Azure spatial anchors. Facebook has already improved the tracking on Oculus Rift S and Quest via a system update. Hesch says, "we're very committed to a long road map of improvements."

A gateway to a Facebook AR headset

Facebook's blog post says, "the future for this technology is in all-day wearable AR glasses that are spatially aware. This will require running SLAM (simultaneous location and mapping) with even greater constraints, including further reducing latencies and cutting power consumption down to as little as 2 percent of what's needed for SLAM on an HMD (head-mounted display)." But AR glasses could also mean changing the inputs from Oculus Quest's physical controllers to something else.

Hesch didn't say whether future Facebook AR headsets would also use controllers like Oculus Rift S and Quest, but he says there's a possibility that Oculus VR or AR could eventually recognize hand gestures, too: "In theory, you have a set of four cameras on the device, so if you look at academic literature, there are a number of different computer vision, perception applications that can be done with normal camera data. Not speaking about anything specific on the roadmap, but you can use your imagination with what are the different possibilities that come to bear when you have camera data available."

With AR headsets, inputs could end up being more hand-based, much like what Microsoft HoloLens and Magic Leap already incorporate. "If you think about input in general for virtual and augmented computing experiences," Hesch says, "what should that look like? We really center around trying to identify what makes the most sense for users in the uses cases they're tackling... and we expect that to evolve over time. We're very committed to looking at the full space of options for input tracking and understanding what we can bring to bear in products that will make the user experience the best thing possible."


Facebook's Portal tech already uses AR elements, but doesn't intersect with Oculus Insight tech... yet.

James Martin/CNET

Will Facebook Portal and other devices intersect?

The spatial tracking that Oculus Insight allows won't just be VR headset-oriented: Hesch says elements of that computer vision tech are already being used in Facebook's phone-based AR. "For what we do in AR on mobile today and what we're doing on Quest and Rift S, a lot of those common building blocks like the Insight system are actually running across platforms, and we see that as a real opportunity to leverage our investments as we look at new form factors and new products we can bring to market." Hesch zeroes on spatial AI in particular: "We think spatial AI is something that's going to be so foundationally important to all the platforms."

Facebook Portal, the videoconferencing tablet that's part of Facebook's AR/VR division, doesn't use Oculus Insight technology yet. But, Hesch suggests "if you look at the cross-platform intersection between wearable devices and a base station device, there's definitely potentials for collaboration/overlap." 

All other announcements on where Oculus is headed next will likely come at Oculus Connect 6 in late September, which is only a month away. We'll see how much more of Facebook's AR plans are unveiled there.

Oculus Quest is fully mobile VR

See all photos