X

Facebook's next step to AR glasses is smart audio

Michael Abrash and Facebook Reality Labs are aiming for audio that can filter out noise and place virtual audio in the real world.

Scott Stein Editor at Large
I started with CNET reviewing laptops in 2009. Now I explore wearable tech, VR/AR, tablets, gaming and future/emerging trends in our changing world. Other obsessions include magic, immersive theater, puzzles, board games, cooking, improv and the New York Jets. My background includes an MFA in theater which I apply to thinking about immersive experiences of the future.
Expertise VR and AR, gaming, metaverse technologies, wearable tech, tablets Credentials
  • Nearly 20 years writing about tech, and over a decade reviewing wearable tech, VR, and AR products and apps
Scott Stein
4 min read
frl-research-audio-iem-new

Facebook Reality Labs' plans for building immersive audio will involve glasses, but could also use earbuds, like this in-ear monitor prototype.

Facebook

Facebook's future plans for AR smartglasses are still in the works, but one part of the equation that could soon see the light of day is smart audio. Spatial audio and augmented audio reality are two arenas in which Facebook is actively interested, and the company's newest Facebook Reality Labs research update details how the company plans to make devices that can tune out the real world and make virtual objects sound like they're right next to you, even if they're not really there.

The two separate technologies, which Facebook is calling Audio Presence and Enhanced Hearing, are both parts of an audio puzzle the company's trying to crack. Meanwhile, the larger augmented reality field remains an unresolved landscape, with players including Apple, Google, Microsoft, Magic Leap and others aiming to drive the next decade of innovation. 

frl-research-audio-demo-1

Facebook Reality Lab's AR audio research is using prototype glasses frames with directional microphones for focusing attention.

Facebook

Facebook originally planned to host visitors for demos ahead of the upcoming Facebook Connect conference later this month. Instead, due to social-distancing concerns because of the coronavirus, reporters were briefed virtually in a chat with Facebook Reality Labs research head Michael Abrash and an audio research team.

Smart audio AR isn't new: Bose had been experimenting with it for years before its AR audio division was apparently shuttered earlier this year. Apple's AirPods Pro have active noise canceling and passthrough, and will be adding 3D spatial audio support this fall. Spatial audio is already a key feature in realistic VR, but Facebook's next-wave plans focus more on conversations, attention and fidelity of audio, including 3D placement of invisible speakers in places that seem like they're in the same room. "The most remarkable thing to me is that everything [about Facebook's new audio tech] is absolutely feasible within a reasonable period of time," Abrash said.

According to Abrash, it's possible the audio tech will appear on VR headsets like the Oculus Quest eventually, or even other Facebook devices. 

The experiments Facebook has been currently running using spatial audio involve using an anechoic chamber and a robot arm with a speaker array to surround a person and calculate their specific spatial audio geometry for their ears, which takes 30 minutes, so it's hardly portable yet. The next step would be to shrink that process down to taking a photograph of an ear to get that measurement (which Facebook calls head-related transfer function, or HRTF). Smartglasses are still a considerable ways away, and the setups that Facebook has been using to test the audio once the measurements are taken are more like glasses with audio playback and focusing microphones, with no visual element yet.

Facebook Reality Lab Research's audio group is also looking to make this audio tech assistive, listening and enhancing conversations and improving clarity of speech while filtering background noise, or other people speaking.

Some of Facebook's concepts of 3D audio and audio-sensing superpowers remind me of the work Doppler Labs tried years ago, on its Here One earbuds with various ambient noise filters. Facebook's research uses beamforming to focus audio, also using spatially aware prototypes worn by testers that have directional microphones. The company has made a custom in-ear monitor to create this audio, with aims to eventually build this tech into future products.

frl-research-audio-demo-2

The spatial audio gear right now is pretty experimental at the moment.

Facebook

I've been wondering for years whether audio augmented reality could be a doorway to making always-on AR glasses work. Audio can be ambient, and people already wear earbuds all the time and have learned to make it work in their lives. Always-on holographic video is another story. On 2D phones, heads-up AR experiments like Google Maps have shown the limits of heads-up displays in the real world, while audio directions can be much less intrusive. Maybe audio could work as a signal cue to trigger visuals afterwards. Facebook hasn't made that process clear yet, but a blog post the company published Thursday promises that this future audio could work to navigate AR-optimized Live Maps that Facebook is currently assembling of the spatially mapped world.

Abrash promises that the company's advances in spatial audio have produced realistic 3D audio that seems like it's coming from the real world and not from headphones, but the current research involves complicated ear scans and a specific, spatially mapped room. The company is working on developing algorithms that could potentially work with a simple camera ear scan at home, making the tech more accessible. 

Would this maybe mean Facebook earbuds? AR apps on mobile? Future directions for the Oculus Quest? Maybe yes to all of these. It's a sign that the bigger, thornier questions of attention and focus in immersive AR wearables still need to be thought through. But it also suggests that Facebook Reality Labs could eventually be working on some sort of headphone technology.