Mark Zuckerberg sees the future of AR inside VR like Oculus Quest

In an exclusive interview, Zuckerberg explains the path from VR to AR, and why Facebook's doubling down on its popular Quest headset.
James Martin/CNET

I am led down a staircase by a small team of Facebook communications people, into a high-ceilinged, wide-open office in Facebook's Building MPK 21 in the heart of Silicon Valley. Open desks cut across the space. The air is calm. The room is silent. At one end of a long open-office desk, back to me, sits someone who looks like Mark Zuckerberg

In fact, it is Mark Zuckerberg.

We head into a glass-walled meeting room, and I take my seat on a sofa in Zuckerberg's private conference room. The CEO of Facebook walks in, shakes my hand and sits down on a chair a few feet away from me. Zuckerberg looks more relaxed in person than I expect. He reminds me of friends I've known who've led startups. I expected something unusual. Instead, our talk feels disarmingly normal. I don't remember what shirt he wore. It seemed nondescript and comfortable.

You'll have to take my word for all of this, because Facebook didn't let me take any photos or videos of our conversation -- even after I offered to post it all on Facebook.

I'm here to listen to Zuckerberg's pitch for Oculus, the virtual reality company he bought in 2014 for a then eye-popping $2 billion as part of his plan to expand Facebook's social media reach into the interactive virtual world and help fulfill his mission of "helping people come together." Oculus is promising "a new chapter in AR and VR" this year, months after the VR company's original founders exited along with a string of other high-profile execs whose shops had been bought by Facebook. Zuckerberg gets philosophical within the first minute of our 30-minute conversation: "There's a lot of questions that people have about where's this all going, why is Facebook doing it."

I expected news of a futuristic augmented reality headset, similar to the Microsoft's HoloLens 2 and Magic Leap gear I've tried out. Those visits had a familiar routine: get invited to headquarters, tour the premises and experience something new. My trip to Facebook on Monday, two days ahead of Oculus Connect, followed a similar pattern. But without the new-product part.

Instead of wowing us with a new AR headset, Zuckerberg used his keynote address at the developer conference Wednesday morning to talk about VR, and in particular, the far more affordable VR hardware Oculus already sells. Because, while Facebook doesn't have an AR headset ready yet, most people don't even know what AR is, let alone want to buy a $2,000-plus headset. It's hard enough getting people to try VR. Which is why the standalone $399 Oculus Quest might still be Facebook's most accessible immersive product for at least the next year.

Read more: Mark Zuckerberg isn't giving up on virtual reality (Q&A)

The Oculus Quest is "the best thing I've tried this year," I wrote in my review. CNET awarded it our prestigious Editors' Choice award because it's an affordable, wire-free VR headset that's accessible to everyone.  

Zuckerberg's team told me the Oculus Quest is getting key updates in the next year: It'll work with PCs to play content on Oculus' PC-connected Rift platform using a USB-C cable, using new technology called Oculus Link. I got a firsthand try at hand tracking, which is coming "early next year" and will enable people to try VR without learning how to use the standard controllers. And a new social platform in beta next year, called Facebook Horizon, is aiming to be the next attempt at bringing multiplayer experiences into Oculus, with games and creative tools.

facebook-f8-2019-9665

Mark Zuckerberg on stage Wednesday at Oculus Connect.

Angela Lang/CNET

Zuckerberg is pretty clear that Facebook's VR vision is still coming into focus, even after all these years. He gives me the standard caveats about the company not being where it needs to be, and how he knows the headsets have to get smaller, more comfortable and easier to use. But he also tells me VR is just the opening salvo in Facebook's longer-term ambitions, and that it's the bridge to a future where we'll communicate using augmented reality, which blends computer-generated images with the real world.

He's also clear that Facebook is the company -- the only company -- that should be leading the move into a future in which we work, play and interact inside these virtual and augmented worlds. And then he takes a thinly veiled dig at rivals Apple, Google, Microsoft and maybe even Magic Leap. 

"It'll probably turn out pretty different if we're helping to shape this versus any of the other companies that might work on it, who I think are more inclined to push a model that's more just around 'here's your app, here's your content, I'm gonna pull it from a store,'" he says. "We want to help shape the next computing platform to be more about interacting with people and not just apps and tasks."

As for when that's all going to happen -- the move from VR to AR, from niche market to mass consumer usage -- Zuckerberg, who's already five years in with Oculus, tells me it's going to take more time to get to a place where "people will kind of expect excellence from the work that we do." 

"I don't think it's a 2020 thing. But hopefully it's not a 2030 thing." And even then, Zuckerberg sees VR and AR still living together. "If you think about how we use screens, phones are the ones we bring with us, but half of our time with screens is TVs. I think VR is TV and AR is phones."

Aside from talking about larger-scale visions of the future, my time at Facebook was spent demoing the Oculus Quest's hand tracking, a straightforward and practical software addition that allows my hands to directly plunge into virtual worlds without needing a controller. While it may not sound exciting, it's probably the most helpful step forward toward putting Oculus' impressive but still off-putting VR goggles into more hands.

facebook-oculus-connect-6-vr-headset-0378

Making VR fists on the Oculus Quest in Facebook's demo room.

James Martin/CNET

The Oculus Quest can see my fingers now

In a well-lit room in Facebook's Building MPK 21.8, a team of engineers hands me a familiar-looking Oculus Quest. Controller-free hand tracking is coming in early 2020 to Facebook's standalone VR headset, powered by the four outward-facing black-and-white cameras on the front that also track full room motion and controller movement. I'm giving it a test drive.

Once the first demo is launched -- called "Elixir" and made by developer Magnopus and Oculus Studios -- I see my hands as two disembodied glove shapes, floating in front of me. I can move each finger. I can give a thumbs-up, point or make a peace sign. It's almost like holding the Oculus Quest's versatile and button-filled Oculus Touch controllers, which also simulate hand motion but can be difficult for newcomers to figure out. This time, it's just my hands. I stare at my virtual fingers. It's mesmerizing.

I'm in a cartoon-like magical laboratory. I dip my hands into a vat of yellow dragon vomit and my hands become yellow and goopy. I walk over to a fiery panel and place my hands there, and they burst into flames. I can pinch my fiery fingers, popping floating bugs. Blue toxic ooze gives me giant octopus hands, and I stare at my floppy, sucker-covered tentacle fingers.

It all really works, but I do notice a lag between my movements and the action that doesn't happen when you're using Oculus Quest controllers. Facebook's engineers say the Quest controllers are still the best way to play games, but hand controls will be for those who find the controllers confusing or want a more casual way to interact. Or, more importantly, for employee training apps for businesses.

Zuckerberg told me earlier that the company's aim with hand tracking is part of its pursuit of "presence." What this means may not even be a headset as we know it in the future. Zuckerberg sees presence in AR as different from what's available in AR today. "I think people ship different products that they will call AR, that don't have the ability to actually deliver a person, here, that you feel is with you. You might get notifications, you might get some glasses products that could be pretty useful and could do different things. But I think there's still some basic technology things that need to get worked on, that we're focused on."

facebook-oculus-connect-6-vr-headset-0490

Fine motor controls, like pinching, are a little harder to pull off.

James Martin/CNET

My second demo is more pedestrian. It's a water damage insurance claim inspection simulation for Farmers Insurance. I walk around a virtual kitchen looking for wet spots: a moldy area near the fridge, a drip under the sink. At each spot, I extend my hand and find a pop-up button, which I can tap to indicate whether I thought an area was damaged or not.  

Hand tracking has been around in both AR and VR: I've seen my hands mapped into virtual worlds at immersive Tribeca art installations and in location-based VR attractions like The Void. Expensive AR headsets, such as Microsoft's $3,500 HoloLens 2, rely on hand tracking instead of controllers. Facebook has acquired VR hand-tracking technology companies, including Nimble VR in 2014 and Pebbles Interfaces in 2015, and has promised breakthroughs in hand tracking on its headsets for a while. But this is the first time I've seen hand tracking working on an Oculus headset. The hand-tracking computer vision tech, developed within Facebook's Reality Labs, can recognize 20 landmarks or points per hand.

Facebook plans to offer the hand-tracking update along with a software development kit for app makers next year, and hand tracking will work in Oculus' Home software and in the VR OS, which power basic browsing. When controlling farther-off things, like a pointer or a remote control, you'll pinch your fingers together and navigate like an air mouse. Your hands could be a remote for browsing the web, playing videos or exploring 3D maps. I may prefer the excellent Oculus Touch controllers when I'm playing games like Beat Saber, but if I were casually connecting to watch a video, or even using an app like a virtual art museum, I'd probably prefer using my hands.

oculus-connect-2019-9892

Zuckerberg speaking Wednesday at Oculus Connect.

Angela Lang/CNET

Zuckerberg agrees, telling me he doesn't expect everyone to ditch their controllers. "There are things people want a controller for, because there are things that you can only do when you get feedback. But I think the main way people are going to want to interact with it is hands."

The hand-tracking demo isn't perfect, though. Aside from the lag, there's a lack of precision when trying to manipulate smaller objects, and the demo required a better lit room than the Oculus Quest's cameras rely on for room tracking. Certain features like recognizing hand-on-hand occlusion (recognizing when a hand is overlapping with another hand, like Apple is starting to do with objects in AR) aren't ready yet either, but are being worked on, according to Facebook's Jenny Spurlock and Robert Wang, who led hand-tracking development on the Oculus Quest.

Oculus has already made it clear the camera-based Oculus Insight tracking tech in Facebook's latest VR headsets is the beginning of what will be needed in a future Facebook AR headset. Maybe recognizing other objects in a room using those cameras -- and bringing more real objects into VR -- is next.

"One of the advantages of using the Quest's regular 2D monochrome cameras is that we have these advantages of size, weight, power and cost," Wang says. "It's a harder computer vision problem, but it keeps us in a pretty good cost boundary." 

"VR is such a new technology, we'd rather reduce friction than add weight to the headset," Spurlock adds.

wingstrikers1

Facebook Horizon, a new social VR experience launching in beta next year, focuses on lots of games and creative tools.

Facebook

Still trying to crack the social world

The weird thing about Oculus is how mostly nonsocial it is today. Facebook, the world's largest social network with over 2 billion active monthly users, still hasn't cracked making Oculus social. Apps like Rooms, Spaces and Venues have tried to bring people together to share experiences in real time in VR, but Facebook's support of these apps has been fragmented by platform. 

A new social VR platform, Facebook Horizon, launching in beta next year, looks to be a second (or third or fourth?) take on Facebook's social virtual universe. The company says it will be key for turning VR into a "critical mass" device. The number of people in the US using VR is expected to rise to 57.1 million in 2021 from about 43 million users today, according to eMarketer. When it comes to AR, eMarketer expects 85 million people in the US will use augmented reality tech at least once a month by 2021 -- up from 68.7 million in 2019.

Horizon will be a creative tool, according to Zuckerberg, allowing meetups as well as game- and experience-making. Zuckerberg hints at Oculus Quest users being able to play laser tag or ping-pong, or build other games on the fly. Horizon sounds a bit like Facebook's larger-scale version of existing social apps such as Altspace, Rec Room and VRChat. Whether it's any more successful at getting people to connect remains to be seen, but it could be a space where Facebook's recent work on building more advanced virtual avatars gets tested.

14-oculus-quest

The Oculus Quest and its controllers. The hardware went on sale earlier this year.

Sarah Tew/CNET

The Quest is also the new Rift

Facebook also announced Oculus Link, an update arriving in November that can tether the standalone Quest headset to a PC, playing Rift content off your computer via a single USB-C cable. Though I wasn't given a demo of Oculus Link, it sounds like the best of both mobile and PC worlds, since the $399 Quest costs the same as the less versatile PC-connected Oculus Rift S headset. 

Zuckerberg explains that while Oculus Link works with nearly any USB-C cable (USB-C 3.1 cables and newer), Oculus will sell a longer, higher-bandwidth cable that will charge the Quest at the same time.

What Facebook isn't ready to do is open up software development on the Quest, which has a library of apps that have been more heavily curated than the more open-to-developers Rift. Zuckerberg suggests that part of that involves the challenges of getting software to run well (he mentions "thermal envelopes"), and defers to Oculus Link as the way to get VR content that may not make it to the Quest. 

Get smaller, get cheaper (and make more of them)

Zuckerberg still won't share specific lifetime Oculus VR sales numbers, but says that "people have spent more than $100 million in VR content," on the Oculus store -- 20% of which has come from the 5-month-old Oculus Quest. The Quest, he says, is "selling as fast as we are making them." 

"There are operational constraints on how quickly this can grow at this point that we need to make sure we work through," he adds. "And we will. It's just the demand is a little more than we'd thought."

VR still isn't drawing enough of a crowd to make high-budget AAA games and apps sustainable without Oculus support, though. Zuckerberg still sees a drive for critical mass to be important. Getting more people to want to buy a VR headset needs to happen, too. 

"I don't think the general public is sitting there waiting for the next Oculus device, the way they're waiting for the next iPhone," says Victoria Petrock, a principal analyst at eMarketer who focuses on emerging technology.

oculus-connect-2019-9850
Angela Lang/CNET

Privacy questions, TBD

While Zuckerberg is focused on when VR and AR will mature, many Facebook watchers, including the developers he needs to convince to create VR and AR experiences for consumers and businesses, are wondering if. Given all the privacy problems Zuckerberg and Facebook have been caught up in over the past two years -- starting with the Cambridge Analytica scandal that saw misuse of over 70 million US Facebook users' data for election marketing purposes -- the biggest concern remains how Facebook will handle data privacy and user content (imagine fake news in VR) when the metaverse goes social.

Zuckerberg has no answers there: "Those concerns will continue to exist and I think there will be new concerns," he says without elaborating. "There's still a lot of questions. I don't think these are things that ever get fully answered -- the threats evolve and you need to work on them. But I would hope that by the time these ecosystems are mature, our approaches to those issues will also be quite mature."

Every science fiction vision of the VR/AR future I've loved has imagined plenty of dystopian ways that increased data tracking on headsets, combined with their increased power to create emotional and possibly unreal narratives, could accelerate our biggest fears on the internet. After listening to Zuckerberg, I see no comfort or solutions coming today from Facebook on that front. There's reason to worry, says Fatemeh Khatibloo, a principal analyst at Forrester. "There are very good reasons to be concerned that a Facebook company is on the leading edge of development here."  

What will the future bring?

All the breadcrumbs are already out there for Facebook's eventual AR headset, or glasses. But Zuckerberg seems in no rush to get there. He offered no answers when I asked how things will proceed after so many Facebook executive departures at Oculus over the last few years, choosing instead to reiterate that "VR is going to be a central part of this. I said before that VR is going to be bigger than people realize."

He sees the already-working VR hardware as a way to workshop the future. "Certainly we're doing more long-term research now focused on AR as well. But I think the commitment to VR is certainly still strong, and that's the technology path that we're going to use to get there."

In the future, I see hand tracking on the Oculus Quest, and a budget VR-only HoloLens from Microsoft. And maybe, like Apple working within mobile to figure out AR, Facebook will use VR in the same way. It's probably the right move for a new world that's still weird and unproven. "They're putting together the pieces, and they're all helping because this is such a new arena," says Tuong Nguyen, senior principal analyst at Gartner, about Oculus' efforts to date. "I don't think anybody has the answer yet but they're all discovering a piece of the answer."  

The Oculus Quest is an impressive VR headset that will keep getting better. But with new technology on the horizon, there's a lot more that still needs to happen. The Oculus Quest isn't forever, it's just the hardware for right now.

"If AR were possible to deliver tomorrow, then it's not like we're the company that you would expect to go do that," Zuckerberg says. But he believes Facebook's work in VR and optics will pave the way when the moment is right.

CNET reporter Queenie Wong contributed to this report.