For virtual-reality movies, old methods don't fit new medium

I'm standing on the bow of what looks to be a sunken pirate ship. A blue whale swims near me. Our eyes lock as he glides by and, as he turns to drift away, I see the ridges on his belly.

I am awestruck by his beauty, and it takes a moment for me to recall a crucial element of biology: Whales have tails. Sure enough, his sweeps along the top of the ship. I duck, and then watch the whale swim away.

My underwater encounter took place in a hotel room in Manhattan's trendy Meatpacking District. Instead of a scuba suit, I donned the HTC Vive headset that lets wearers move through another world, courtesy of two beacons in the room that track their position.

Enlarge Image

Welcome to virtual reality, or VR, which many believe marks the next leap in entertainment because it can make viewers feel they're part of the action. But as exciting as that seems, the new medium comes with new challenges. Artists will have to rethink how they write, shoot and edit for a virtual-reality film.

Virtual reality is "very daunting as a writer," said Craig Gilbert, co-founder of TotalCinema 360, a VR video production company. "But I think it can also be very, very liberating."

Still, it's a promising enough field that heavy hitters such as Samsung, Nokia, handset maker HTC, gaming company Valve and Facebook's Oculus have poured significant resources into it.

Pen to paper

The writing of a film is one of the most crucial parts of moviemaking. But while screenwriting has never been easy, the addition of virtual reality now forces writers to account for everything in a scene -- all revolving around the viewers' perspective.

Picture a conversation between a couple at a restaurant. Now, in addition to writing both characters' lines, you also have to choreograph the actions of all the other diners in the room because, in virtual reality, you can't control where the viewer is looking.

"As hard as you try, [viewers] might be staring at the ceiling, they might just want to see what's behind them," Gilbert said. It means having to account for "what's going on in the peripherals, what's going on behind them, what's going on above them, below them -- these are things that you never really thought about as a writer, and now you can and you should."

For Cliff Plumer, president of Jaunt Studios, writing for virtual reality is akin to crafting a play. He says the writer must ask herself, "How else can I take advantage of a full, 360-degree environment in which I can have activity happening all around me, all around the camera, as well as sound?"

Lights, camera, action

Virtual reality's ability to immerse people in the middle of another world also means filmmakers lose the ability to control what a viewer sees.

Anthony Batt, co-founder of WEVR, the company that created the underwater scene I experienced in that Manhattan hotel, found out about the complexities of different perspectives the hard way. "There's people that actually miss the whale," he said.

As a result, filmmakers will have to try different methods to draw viewers' attention to a particular element of a scene. Jessica Brillhart, a filmmaker who works in VR at Google's Creative Lab, uses the environment to point to the action -- angling chairs in a certain direction, for example, or having light pour in from a particular place. And she wants to be sure people who do miss the main action don't completely lose out. "So I might put something a little bit more interesting but maybe irrelevant behind them."

In the "Blu: Encounter" demo, the whale is part of a timed sequence, appearing and then swimming off on cue. For the next version of the demo, WEVR will add story-triggers that let the viewer's gaze dictate when the content appears. That means the action will happen when the viewer looks in a particular direction -- not on a timed schedule. "You're wearing a computer system," Batt said. "We know where you're gazing; so we can cue things."

Then there's the issue of hiding your crew. Lights, wires, tripods and even crew members are all visible to the camera's all-seeing eye. "It's like playing a really fun game of hide-and-seek," Brillhart said, "just figuring out where you can stash yourself."

In one of Jaunt's videos, viewers watch Paul McCartney perform "Live and Let Die" in San Francisco's Candlestick Park football stadium. The tripod, which you would normally see when you look down, is hidden by the company logo. Jaunt's new camera, the Neo, will eliminate having to hide the tripod and rigging. "So the way we're able to assemble those images coming off those cameras, you wouldn't see the tripod or what it's mounted on; you'd see the ground," Plumer said.

Sound and vision

The McCartney VR video highlights another consideration: audio design. In the clip, the audio changes based on where viewers focus. Look at the stage or at McCartney, and the audio comes in clearer. Look at the audience, and you hear more of the crowd. This effect, known as spatial audio, is a big feature in virtual reality.

In her own work, Brillhart has captured four channels of sound to create such spatial audio. This lets her position the sound using the same kind of surround-sound concept at play in a home-theater system.

"When you turn your head left or right, you can sense something not only visually, but audibly in a space," she said. "So if you hear talking behind you, you can turn and see the thing emitting that sound."

This is critical for tying together what viewers specifically see and hear in a scene. If you're looking at two people having a conversation, you hear that conversation clearly. But if you turn your head toward someone else talking nearby on a cell phone, that person's voice becomes the new audible focal point.

There have always been multiple points of audio recording that you can link together, Gilbert said. "Now we have multiple mixes for single shots, so that you can hear very specific things."

Sound engineers can now use audio to direct your attention to the action. "Sound is half the experience of VR, because that can trigger people where to look and just enhance the whole storytelling process," Plumer said.

Putting it together

Once you've written your script, planned your shoot and captured your footage, you have to assemble everything into a full 360-degree video. That's the editor's job, who has to "stitch" together footage from multiple cameras and line up the edges of each shot. "Currently one would say that's a bit of a pain point for creating the art experiences." Brillhart says.

She's not alone in that opinion. "Stitching takes up a vast majority of our postproduction time," Gilbert said.

But newer cameras are aiming to solve the problem.

Jump, an editing program from Google that works with an open-source camera rig of the same name, uses Google's computing power to seamlessly stitch footage automatically, Brillhart said.

The Bublcam, from Bubl Technology, automatically takes care of stitching as well. In a process called Smart Stitch, Bubl's cloud service can also recognize if there's an issue with lens alignment -- due to minute movements of the lenses or the camera momentarily dropping -- and can fix the discrepancy.

The pace of edits presents another issue. Traditional movies can cut rapidly from one scene to another. But in virtual reality, you can't switch scenes quickly without disorienting your viewer. "You can do cuts, you can [have] transitions," Plumer said. "But depending on the piece of content, you've got to be very careful how you do that."

With new power comes new responsibility

When speaking to creators and innovators in the VR community, one thing becomes abundantly clear: They want to be responsible when it comes to making good content. Responsibility, in this sense, takes on many forms.

To Sean Ramsay, CEO of Bubl Technology, there's a responsibility to enable people to create things that they want to see in virtual reality, quickly. "We don't want people to have to wait to get content that they can experience," he said.

For Batt, it means being responsible with the trust the user places in the filmmaker. "When you're actually putting on a VR headset with headphones and eyeglasses, the creator has to say, 'Look, we're either going to scare you and we need to tell you upfront, or we're just going to give you a very beautiful experience,'" he said. "You, as the consumer, have that trust."

As virtual reality is on the precipice of becoming mainstream, the main responsibility they all share: not making content that will get viewers to take off the goggles forever.