How Facebook plans to blend the virtual and real with your wrist
Facebook sees AR glasses, a neural wristband and haptic gloves in its future, but the company's sci-fi ambitions also raise privacy questions.
- Nearly 20 years writing about tech, and over a decade reviewing wearable tech, VR, and AR products and apps
- 2022 Eddie award for consumer analysis
Facebook envisions a future where you can type without having to open your laptop or smartphone.
Facebook , a social network that has already changed the way people communicate, has set its sights on another lofty goal: seamlessly blending digital life with the physical world.
A big part of that future will involve wrist-based technology that can sense your neural signals and track your intentions, even without your moving. It sounds like a band that can give you Star Wars-like Force powers.
To explain how this could work, Facebook's research science manager Tanya Jonker describes the process of cooking a zucchini herb omelette with a friend. Currently, people might have to interrupt a conversation to set a timer on their smartphone. In the future, through augmented reality, people will be able to complete this task by simply moving their finger.
"One of the most exciting aspects of [AR] technology is really this opportunity to help people stay more engaged in the real world even while they're interacting with their digital devices," says Jonker, who's part of Facebook Reality Labs, which focuses on the social network's virtual and augmented reality efforts.
Facebook has been working on plans for an advanced pair of AR glasses for years, but it's also leaning on a layer of predictive artificial intelligence that looks like it's aiming to be a total life assistant, raising bigger questions about how much Facebook could aim to run your life through its glasses.
For groups like the Electronic Frontier Foundation, AR glasses also come with privacy concerns, especially for a company known to cross lots of boundaries.
"If smartglasses become as common as smartphones , we risk losing even more of the privacy of crowds," EFF said in an October blog post. "Far more thorough records of our sensitive public actions, including going to a political rally or protest, or even going to a church or a doctor's office, can go down on your permanent record."
A wristband that senses your fingers
In 2019, Facebook acquired neural input tech company CTRL-Labs, which developed an armband to sense neural signals. Facebook plans for a band built on that tech is wrist-based and can sense individual motor neurons, according to the company. It can convert finger motions and even small squeezes and shifts into actions that animate a virtual hand. It looks like what hand tracking already does inthe Oculus Quest VR headset, but in this case it can work without moving your fingers at all.
That's where things could get really strange and fascinating. Facebook's shown aspects of its neural input tech before, and demos of how finger-typing on a desk could eventually adapt into a custom keyboard that learns and improves, almost like how phone-based touch and swipe keyboards do. But the tech could work even without moving any fingers at all. Or, without having fingers.
Tech like this does exist elsewhere: the Mudra Band, a neural input band being made for the Apple Watch this year, similarly senses finger movements and small gestures. Other companies may also have other solutions, but Facebook's work looks like some of the most advanced seen so far.
The company's ambitions in assistive technology are clear, after showing a demo of how the input bands could be used with virtual finger motions even when a hand is missing fingers. Another demo where someone makes a computer dinosaur jump by pressing a spacebar is then shown being done by the wristband's motor neuron measurements, with the hand seemingly not moving.
In a recent podcast conversation with The Information, Facebook CEO Mark Zuckerberg discussed the idea of having extra virtual hands. Facebook Reality Labs' team also invokes this idea, which is brain-boggling. Even more so, there's the question of how these phantom or extra limbs or digits would work with AR glasses.
In the immediate future, Facebook is more focused on specific actions that the band can recognize, the main one being a small finger-press called Intelligent Click. The small actions are meant to accompany heads-up displays that could proactively suggest information (buying something, directions or cooking directions while cooking) that you could click to accept or refuse. The band also looks like it recognizes individual finger movements, thumb movements, squeezes and drawing on your hand like a trackpad. It looks like a potentially more precise way to control AR headsets: Microsoft's HoloLens 2 has no controller at all, and hand gestures can sometimes be difficult to pull off.
Someday, Facebook's research science director Sean Keller says, you'll enter a cafe donning AR glasses and a wristband. Then a digital assistant will ask if you want to order an Americano. An interface will appear that you'll see floating in front of you, like a notification on your phone or the voice on your smart speaker. But to tell it what you want, you won't speak or touch a screen. You'll tap your fingers together in the air, and a neural-input wristband will recognize your subtle smart click.
Wrist squeezes, haptic emoji
Facebook has been working on a way to control a machine by clicking your fingers.
Facebook's been exploring haptic, vibrational feedback wearables for four years, according to Nicholas Colonnese, manager for research science at Facebook Reality Labs. At least a dozen prototypes in the Reality Labs' research division hint at what feedback will be worked into a future band.
One, an inflatable wrist-cuff called the Bellowband, shown off during a video chat presentation, has eight blood pressure cuff-like inflatable zones that squeeze the wrist. A more developed Tasbi (Tactile and Squeeze Bracelet Interface) band also vibrates (with "six vibrotactile actuators") and squeezes. Facebook's demo videos show people wearing one band on each wrist, playing virtual games like tennis and archery, where the bands can give haptic feedback that feels realistic enough to make the floating objects feel tactile. Facebook says the tech works convincingly, even without wearing anything on the fingertips, via a cognitive phenomenon called sensory substitution (similar to how prosthetic devices can be operated).
An example of thumb navigation while wearing the prototype neural wristband.
Facebook is exploring "haptic emoji" for these bands, which would send custom vibrations or squeezes, Colonnese says. It sounds like Apple's early exploration of sending "digital touches" via the Apple Watch. Another use could be subtle pings to navigate while walking, which is also used on some wearables.
Facebook's current Oculus Touch controllers have only basic vibrating feedback, but these haptics look far more advanced. Recent advanced haptics devices, from Apple's subtle Apple Watch pings to Sony's force-feedback adaptive trigger-equipped DualSense PS5 controller, show what Facebook could be aiming for.
Facebook's vision for haptic-feedback virtual archery sounds like it could work as well in VR as AR.
Could this come to VR before AR, and what about smartwatches and smart homes?
The wristband and haptics seem like the sort of evolution Facebook could adopt for VR beyond the Oculus Quest . Facebook acknowledges that VR is part of how the company's been prototyping its neural interfaces but told CNET that, while nothing is planned imminently, there could be possibilities similar to the Quest's hand tracking.
"As you saw with hand tracking, as the technology gets good enough for a consumer to use and is easy to learn, we get to start thinking about how it can fit into a consumer product," a Facebook spokesperson said.
Could this also work on a future Facebook watch in combination with ambient smart home tech and not require a headset at all? Maybe, but not right away.
"This technology could take a lot of forms and cover a lot of use cases, but right now we're currently focused on how the tech we are building can work together (AR glasses + wristband)," a Facebook spokesperson told CNET via email. "Wristband wearables may offer a path to ultra-low-friction, always-available input for AR glasses, but they're not a complete solution on their own -- just as the mouse is one piece of the graphical user interface. They need to be assisted with intent prediction and user modeling that adapts to you and your particular context in real time."
How Facebook's new AR inputs will handle data privacy is still a mystery.
Collecting more intimate data
The future that Facebook envisions where people are able to type, order a coffee or look up a recipe without picking up their smartphone or opening a computer also involves collecting a trove of personal data from people and their surroundings.
Privacy advocates have raised concerns that the technology could be used by governments or law enforcement agencies for constant surveillance. The social network has also been looking at whether to add facial recognition to its smart glasses, and the company's own employees have brought up potential harms such as the abuse of the technology by stalkers.
Kurt Opsahl, deputy executive director and general counsel of EFF, said there's an extra layer of privacy concerns given how much personal data Facebook already collects about its users to serve targeted ads.
"One of the concerns that has come up time and time again over the years is the possibility that Facebook would use that data for a new purpose that [users] didn't agree to when they originally gave the information to Facebook," he said.
The social network has had a poor track record of safeguarding user privacy and has faced more scrutiny from lawmakers and regulators since the Cambridge Analytica data scandal in 2018. Cambridge Analytica, a UK consulting firm hired by former President Donald Trump's 2016 election campaign, harvested the data from up to 87 million Facebook users without their consent as part of an effort to influence the behavior of American voters.
Facebook hasn't provided a lot of details about how it plans to use the data gathered from AR glasses in the future, including whether it will be used to serve people even more targeted advertising. It's unclear if Facebook plans to link data gathered from its smart glasses to a person's Facebook profile.
"If aspects of this research are applied to consumer devices in the future, we will be up front about any plans related to ads," a Facebook spokesperson said in a statement.
Facebook executives have acknowledged that the neural data they're collecting as part ofthe research is highly personal. Chief Technology Officer Mike Schroepfer, though, noted the information collected to decipher a person's intent to move their hand is "nowhere near" collecting information about people's thoughts.
The company also said it's sharing its research so people can weigh in with their own concerns about the technology.
"The reality is that we can't anticipate or solve all the ethical issues associated with this technology on our own," Keller said.