How VR helps you hear what it sounds like in somebody else's head
Creators of virtual-reality experiences are using sound not just to put you in a different environment, but to put you in a different mental and emotional place.
What does the world sound like from inside your head? Or to a kid on the mean streets of New York? Or the cop confronting him? Those are the kinds of questions being asked by virtual-reality creators keen to use sound to take you places both familiar and untraveled.
At this year's Sundance Film Festival, where film fans and filmmakers eagerly tried out new VR headsets and experiences, many for the first time, creators and experts discussed the various tricks and techniques they can use to convince you of an illusion of reality. Crucial among those is the use of sound to help you locate yourself inside the virtual world, tell a story and impact your emotions.
VR is a relatively new medium, with creators still developing the tools and processes. "The process is not defined, the workflow is not defined," composer and sound designer Dražen Bošnjak said of VR production. "Unless you're willing to adopt existing strategy from the gaming industry you really have to reinvent how it's done." Sometimes, he added, strategies for sound in games simply don't work well for VR.
Sound designer Tim Gedemer showed how he reinvented familiar processes to create the sound mix for hard-hitting VR experience "Perspectives: Chapter 2 - The Misdemeanour", which debuted at Sundance.
This is a 360-degree frame of "Perspectives 2", flattened so it can be seen when you're not wearing a VR headset. Each noise is "placed" around the frame so it comes from a consistent direction even when you turn your head.
Gedemer uses audio-editing software Pro Tools and divides each frame of the video into slices, so he can mark where a sound will come from in each shot. Those sounds, coming from different directions, are important to establish the geography of the virtual world, allowing you to orient yourself in a scene as it unfolds. That means the sound mix is crucial, and Gedemer makes the most of the multichannel Dolby Atmos sound system to precisely place the sounds you hear around you.
The sound of storytelling
For filmmakers, your ears are just as important as your eyes when it comes to immersing you in the story. "Sound design is key," said Diana Williams, who creates interactive Star Wars experiences at Lucasfilm's ILMxLAB. "We can use it to drag your eye around to where we want you to look."
But this direction isn't a precise science. "Sometimes it works and sometimes it simply doesn't," Bošnjak said. "If [the viewer] is engaged with the action that's right in front of them, trying to take their attention is much harder than if you have somebody just walking around observing the environment. I think that's comparable to real life. If you're flying a fighter jet and you hear a sound behind you, if there's enough action in front of you that you're directly engaged in, that's the priority for your brain."
Subjective sound
"Perspectives 2" also demonstrates a more nuanced use of sound to put you in not only the physical place of the characters, but in the same mental place. "When we do sound that is super-accurate to reality," Gedemer explained, "it's not evoking the emotion or communicating the idea we want to communicate...A lot of these projects go wrong because you spend so much energy trying to make it sound like you're actually there and that can be a mistake."
The film, which details a confrontation between police officers and teenagers, switches viewpoints between the characters. When you are following the scene through the eyes and ears of one of the police officers involved in the fracas, the creators punch up traffic noise, people yelling and other jarring sounds to make you feel as if you are in unfamiliar and potentially threatening territory.
Conversely, when you are in the shoes of one of the kids involved, sound is used to make the environment a friendlier place, replacing some of the loud traffic noise with birds chirping and children playing in a nearby playground.
To sell the first-person effect, you also hear breathing and muttering from each character as you inhabit him or her. "We used Dolby Atmos technology to put that sound of your voice in your head, close to your ears," Gedemer explained, "and EQ-ed it in a way that sounded more resonant, as your voice does to yourself."
Nuanced sound is clearly crucial to the success of a virtual-reality experience. As Morris May, co-creator of "Perspectives 2", puts it, nuanced and clever use of sound "puts you in a location like nothing else can. It's the missing link for feeling like you're there."