We won't be wearing our magic Tony Stark AR smartglasses this year, or the year after, or maybe not even the year after that. Although Facebook is already working on smartglasses with Luxottica, those won't be world-sensing mixed reality devices yet. But Facebook's Project Aria is ready to start mapping the real world with a head-worn sensor array being deployed to 100 or so testers in Seattle and the San Francisco Bay Area starting this month. The company announced the news Wednesday at its virtual Facebook Connect conference.
Facebook's current immersive product is the Oculus Quest 2, an advanced stand-alone VR headset that can scan the real world a bit. But Facebook's AR ambitions are going a lot further than that: The company's currently working on mapping a spatial scan of the real world, called LiveMaps, that will be the backbone for a future pair of AR glasses. Facebook's VP of AR and VR, Andrew Bosworth, spoke with CNET about the news and where the company's headed with VR, AR and how the company plans to address privacy concerns and universal Facebook logins for VR going forward.
Smartglass sensors, without the smartglass part
Project Aria is most definitely not an actual pair of smartglasses. It's a "sensor array" for those future glasses. Facebook's post on Aria says it will have "the full sensor suite used in VR headsets for spatial awareness," adding, "they also compute location from GPS, take high-res pictures, and capture multichannel audio and eye images." A set of clearly marked testers wearing lanyards will be wearing these research devices out and about, a group of 100 or so Facebook employees and contractors in Seattle and the San Francisco Bay Area "trained in both where and when to use the device, and where and when not to."
The devices will "capture video and audio from the wearer's point of view, as well as eye movement and location data," according to Facebook, but that information will be held on-device and not be seen by Facebook researchers before scrubbing the data of faces and sensitive information like car license plates.
Facebook plans to refine its LiveMaps data, which will be a 3D map array for glasses to navigate through, using the device. But it also plans to train an AI assistant for the glasses, partially through smart spatial audio technology. The eye-movement element also suggests eye tracking -- not present on Facebook's VR headsets -- will be a major part of the smartglass tech when it does arrive.
The research projects brings to mind other Bay Area tech experiments in public spaces, from Google Glass to self-driving cars. The intersection of the real world and the data-collecting challenges smartglasses pose sounds like an extremely tricky problem for Facebook to figure out -- and for people to trust. The mixing of world-scanned map data with location-aware hardware is also what Google is pursuing, as well as Apple, Microsoft, Magic Leap and Niantic.
But for Facebook, this real-world testing outside of the lab is unexplored territory, and will bring a lot of questions. Bosworth says of Facebook's testers, "They have a responsibility still to make sure that it's off where it would not be respectful, in prayer rooms, in bathrooms, the obvious places that you'd expect," to make sure the headsets aren't recording in sensitive areas. But Bosworth sees the person-view info as being important for understanding AR glasses' AI and design. "We have datasets from satellites, you've got datasets from cars, it's just not quite the same as from human height on the sidewalk with trees and things over your head."
The black glasses, with cameras, look very much like everyday glasses from the photo Facebook shared, but a white light will be on when recording.
Bosworth emphasizes that the data on the glasses isn't viewable by other researchers right away. "The data that comes with the glasses is quarantined for three days, and then it's scrubbed so all faces are blurred, license plates are blurred, before anyone can get access to it," he says of Project Aria's collected photo data.
The specific info is also, according to Bosworth, part of getting Facebook to be less reliant on data collection in AR, stripping that away while leaning on the built-up 3D LiveMaps to reduce glasses processing. "We need that egocentric dataset so we can start to understand, 'Hey, how do we actually minimize data collection on real AR glasses?'"
Privacy questions galore
Facebook is promising to take care with data on the glasses, and testers are reportedly being instructed on how to use these devices responsibly. But the glasses will be tested lots of places: in Facebook's offices, in testers' homes and in public. In privately owned public places they're supposed to get consent before recording. The data being recorded is encrypted and then Facebook uses a "secure ingestion system to upload data from the devices to a separate, designated storage space, accessible only to researchers with approved access."
Public-collected data will "not be used to inform the ads people see across Facebook's apps," Facebook's Project Aria privacy guidelines say. But that also suggests that, eventually, information on AR headsets could be used to target ads. That's the weird line Facebook is exploring, and will have to deal with, in an AR smartglass future. And it sounds like this experiment hasn't yet figured out the answers.
Full AR glasses: This decade, but not next year or the year after
Facebook CEO Mark Zuckerberg told CNET last year that AR glasses may be more of a decade-long challenge than a couple-of-years challenge, and Bosworth mirrors that opinion. "It's one of those things where I'm not trying to be coy, there's just a tremendous amount of uncertainty," Bosworth says. "I go on a decade timeframe. I feel really good about us having a really good full-featured AR experience as an industry."
While smartglasses are coming in the next year or so, Bosworth doesn't see true AR glasses coming next year for Facebook... or for anyone else. "The next one or two years, I think I'd be pretty surprised to see [full AR glasses] in the industry. So we're definitely dealing with years -- hopefully not decades."
He sees a lot of unsolved issues for future smartglasses. "It's all a challenge. Let's imagine we could get ahead of all the negative externalities that are possible ... You've still got tremendous technological challenges in terms of the wireless performance into your pocket or into a bag: the thermal dissipation of the computer you have to have either on the headset or uploaded in the headset; the battery life; all fit inside of a thing that's small enough to be lightweight to wear for a long time to be socially acceptable; that has a contrast ratio bright enough to work in the sun or in a dark place; that has the clarity that you can read text on it at various focal lengths; that has a field of view that you don't feel as if you're constantly going through a tunnel, or seeing the stark edges of things.
"This is a playground of challenging problems," Bosworth says. "They're all really good, hard, important problems to go solve. I'm confident that they will be solved. I have a sense of how transformative this will be to society over the long term. But today, we're just dealing with the basics really: When can you gather data and when can you not gather data? What's socially acceptable for you to wear, what's socially acceptable to other people to have you wear? It's a very deep set of challenges that range from technological to social to political."
VR and Facebook and identity: 'We have Batman, we need Bruce Wayne'
Meanwhile, Facebook has the successful Oculus Quest. Although specific sales numbers still aren't being shared, Bosworth does say that 90% of Quest users last year were brand-new to Oculus. The lower-cost Quest 2 points to VR and passthrough-camera mixed reality being building blocks for socially immersive experiences and experiments, at least until smartglasses are ready.
A new Facebook account requirement for Oculus VR points to a real-world identity becoming a bigger part of Facebook's virtual worlds picture, which runs counter to how many people think of the freedoms of virtual spaces and identities.
"When I go to VR, sometimes I want to be Batman, I don't want to not be able to be Bruce Wayne. I'd like to be able to be Bruce Wayne, if that's what I need at that moment," Bosworth says. "But sometimes I want to be Batman! All this change is about empowering the Bruce Wayne use cases, not about altering the Batman use cases. The Batman use cases are there, they're good, they're protected, they're gonna be in good shape." To Bosworth, "the thing that's missing right now in VR is [for people to] also to be able to be themselves."
That future, in Facebook's VR and AR, is going to be very much about being in Facebook's world. "I want everyone to get the heads up to know that Facebook is the one making that headset," Bosworth says. "Facebook is the one who's processing that data. I don't want them to be surprised by that at all."