At Sundance's New Frontier, VR and AR go epic, weird and theatrical

Virtual live improv theater? Karaoke with digital puppets? Synesthetic VR food fests? The tech-driven show lets artists test their limits.

Joan E. Solsman Former Senior Reporter
Joan E. Solsman was CNET's senior media reporter, covering the intersection of entertainment and technology. She's reported from locations spanning from Disneyland to Serbian refugee camps, and she previously wrote for Dow Jones Newswires and The Wall Street Journal. She bikes to get almost everywhere and has been doored only once.
Expertise Streaming video, film, television and music; virtual, augmented and mixed reality; deep fakes and synthetic media; content moderation and misinformation online Credentials
  • Three Folio Eddie award wins: 2018 science & technology writing (Cartoon bunnies are hacking your brain), 2021 analysis (Deepfakes' election threat isn't what you'd think) and 2022 culture article (Apple's CODA Takes You Into an Inner World of Sign)
Joan E. Solsman
8 min read

Mechanical Souls at New Frontier Central developed artificial intelligence that coordinates with VR to stage something like an immersive-play mystery game.

Digital Rise/Serendipity Films/DVgroup

This year's New Frontier program at Sundance is getting back to its roots. The film festival's high-tech art showcase is going to feel a bit like a party again -- only bigger and more mind-warping.

"We're seeing work being done on a larger scale," Shari Frilot, New Frontier's chief curator, said in an interview Tuesday. "This year, we're ... seeing more kinds of stories being made possible because of innovations" in technology creators are making to realize their visions, Frilot said.

Sundance's New Frontier, designed to foster new forms of storytelling using technology, is the year's first major stop for interactive storytellers to show off their latest work. The program underscores the growing influence of tech like virtual and augmented reality as new mediums for artists at an event once solely devoted to traditional films. This time the program's scale is getting a boost as New Frontier moves to a new, expansive space, allowing for outsize projects to stretch imaginations.

The VR doesn't end with New Frontier. Big names in virtual reality swarmed to Park City to show off their latest projects on the sidelines of the festival, too. In invite-only demos, Oculus debuted sneak peeks of The Under Presents, a social VR narrative game that uses live actors and the company's coming Oculus Quest headset. It's as if the indie video game Journey were adapted into an immersive play like Sleep No More but set in your wildest, most head-scratching dream. And Disney has been showing off a VR short called Cycles at a storefront off Main Street. 


Digital puppet demos will sprinkle New Frontier at Sundance this year. 

Courtesy of the Sundance Institute

At New Frontier itself, festivalgoers this year can try a multiroom mind-warping VR fine-dining installation. Or karaoke with a digital puppet modeled on a monstrosity from a Goya painting. There's an immersive theater piece that sounds like the characters from Crazy Rich Asians got caught up in an Agatha Christie mystery mashed together with Black Mirror.

The Royal Shakespeare Company is even building something like a miniature amphitheater here for a Magic Leap augmented reality play. 

New Frontier kicks off Friday afternoon as the Sundance Film Festival 2019 opens its 10-day run in Park City, Utah. It spans two locations: A large-scale venue called New Frontier Central hosts most of the exhibition, while a smaller space nearby, in the basement of the Ray theater, features the rest of the projects along with a 40-seat VR cinema and all the New Frontier panels.

Oh, and New Frontier Central is where they put the bar.

"We're trying to get back to what we started with and lost over the years: getting back the social intersection element of New Frontier," Frilot said of the social space. "[It's] about getting lots of different kinds of people in the same room talking about work that is really surprising them."

Epic scale

One of the dominating installations at New Frontier Central will be an interactive Immersive Stage in the back corner, an area where the floor and two walls become projection screens for three projects by experimental theater-art production company 3-Legged Dog. (Three years ago, the group helped stage a punk-tinged, projection-heavy opera about human trafficking that won the Pulitzer Prize for music.)


Two Black Lights and One Red is a play within the interactive universe of Esperpento. (Yes, he's swigging a bottle of wine.)

Courtesy of the Sundance Institute

That's where karaoke with digital puppets comes in. Esperpentois a Goya-inspired animated world, which includes a performance of a play with one character represented by a digital puppet and the others played by real people. In one of the "lounges" inside the Esperpento world, artists will be teaching people how to puppeteer alongside somebody singing. Performances of the play within Esperpento, called Two Black Lights and One Red, will take place four times over the course of the fest.

"I've seen things like that in very large studios," Frilot said. "It blew me away when I saw this indie development."

Throughout the festival, the Immersive Stage will alternate between Esperpento and two other works. Dirtscraper is a virtual reality "game" with an aesthetic like that of a bleak dystopian Donkey Kong, and analmosh is an audio/visual performance that links glitchy electronic music to projected imagery that's generated live based on the audio.

The Dial, at the New Frontier Central, also uses projections, as well as augmented reality, to stage a mystery. A woman smashes her car through the stone wall of her wealthy family's compound, action that all takes place around a projection-mapped sculpture of a house inside a glowing translucent cube. Three participants use augmented reality on their phones to interact with the projection and unravel the mystery. 


The Dial

Dougal Henken

Magic leaps

The Royal Shakespeare Company's tiny amphitheater, back at New Frontier Central, is where a pseudo-hologram of Shakespearean actor Robert Gilbert will perform the Seven Ages of Man speech from As You Like It. (High-school English cheat sheet: That's the one with "all the world's a stage" in it.) Gilbert performs as a mixed-reality avatar, appearing in front of participants via a Magic Leap headset.

Watch this: What it's like to experience Magic Leap One

It's one of three Magic Leap projects at New Frontier this year. The secretive augmented-reality device developed by a $1 billion start-up began emerging slowly in the real world only in the last five months.

Mica could be a glimpse at the future of AI assistants.

Magic Leap

Magic Leap itself is presenting Mica, its human-type artificial intelligence technology. Pitched with a first-person invitation from Mica herself -- "Join me at the beginning of my existence, as we pull from the past to create the future," whatever that means -- the tech will interact with you in a "gestural exchange" that's aimed at getting festivalgoers to contemplate her place in our world, says Magic Leap.

And A Jester's Tale integrates robotic avant garde pop star Poppy into the mixed-reality telling of a "psychologically taxing children's fable." By virtue of Poppy's participation, A Jester's Tale, which is at the Ray, has already begun to generate buzz on Twitter. 

Hard and soft

Artists at New Frontier are "out of necessity inventing custom hardware and custom software to be able to tell these stories," Frilot said.

One such customized work is Sweet Dreams, at the Ray, which leads you through multiple rooms with bouncy walls and, using artificial intelligence and virtual reality, aims to give you trippy impressions from things you eat and drink. Synesthesia is a rare neurological phenomenon that mixes up the human senses, causing some people to experience sensations like hearing colors, smelling sounds or seeing words or numbers as colors and shapes. With VR and AI, Sweet Dreams aims to re-create the condition, making it feel like you're eating the sun or sipping the room around you.

Mechanical Souls at New Frontier Central also developed artificial intelligence that coordinates with VR to stage something like an immersive-play mystery game. The premise of the experience is "a sumptuous wedding" for the "heir of the rich Zang family," a ceremony with hired androids on hand to help. When something's gone wrong, six audience members play Mechlife employees to figure out what's happened. The story unfolds in virtual reality with live actors, and each participant may follow a different path of action, depending on where he or she looks. 


Mechanical Souls incorporates live actors with VR.

Digital Rise

The artists behind Interlooped developed new hardware and software with the ambition of creating what Frilot describes as a rabbit hole of reality, where you can't tell what's real and what's not. It includes live volumetric capture -- a technology that essentially scans a whole person to replicate them as a pseudo-hologram in real time. Interlooped, however, applies this kind of live capture to two people at once, something that hasn't been done often in performances.

Being there

Interlooped, Mechanical Souls and all the Magic Leap experiences are examples of a trend Frilot noticed while curating this year's New Frontier. Creators, she said, are putting physical bodies inside the frame of their experiences.

"That's fascinating to me ... particularly because for so long there was that criticism of VR that it takes you out of the real world and into something else," said Frilot.

Another is Embody, at the Ray, described by Frilot as a high-tech yoga experiment. Drawing from dance and the martial art called aikido, Embody is social, movement-focused VR where you see other bodies next to you entering and leaving the experience. Runnin' also incorporates dance into VR, in this instance holding a virtual interactive dance party with comic hip-hop artist Reggie Watts.

VR Box

The festival also includes three VR cinema programs, ticketed events where groups of up to 40 experience the same sequence of VR shorts together in a room called the Box at the Ray.


Traveling While Black uses Ben's Chili Bowl, a DC restaurant that was been a sanctuary for black travelers for decades, as the setting for much of the documentary.  

Felix Paul Studios

The second program focuses on two documentary-style experiences, Traveling While Black and Marshall from Detroit. Both were made in collaboration with Felix & Paul, an immersive storytelling studio that's been a staple of film festivals like Sundance for years. Marshall from Detroit is a 360-video interview with rapper Eminem during a drive around Detroit on a winter's night to discuss his creative process and his hometown.

Traveling While Black used the Green Book -- the travel/survival guide used from the 1930s through the '60s by black motorists to identify places across the US where they could drive, sleep and eat safely -- as a jumping-off point for intimate conversations about hazards to black people's safety in public spaces today. Traveling While Black will also be on view at the Ray in a replica of Ben's Chili Bowl, a Washington DC restaurant that was one of the Green Book's safe spaces and the site of many of the VR experience's interviews. 

The first and third programs at the Box span four far-flung projects each. 

Program 1 includes Ghost Fleet VR, which depicts the Thai slave trade in the fishing industry, while RocketMan360 takes place in the cockpit of a spaceship. Last Whispers: An Immersive Oratorio explores languages that are endangered or extinct, and The Tide: Episodes 1 & 2 invents a tale about man-eating sea creatures.

Program 3 incorporates Live Stream from YUKI <3 about a live-stream influencer who has a secret she hides from her fans. Kaiju Confidential flips the script of a Godzilla movie to explore the point of view of towering animated monsters. Ashe '68 uses re-creations, period archival footage and sand animation to convey the internal pressures felt by black tennis legend Arthur Ashe. And 4 Feet: Blind Date follows a blue-haired teen girl on a blind date.

New Frontier's sprawling showcase includes yet more experiences -- (antiquated) Augmented Reality, The Art of Be(i)ng, AquarelaEmergence, Gloomy Eyes, GrisailleReach, Taking the Horse to Eat Jalebis and Walden -- that CNET doesn't have a bead on yet. We'll be updating with more details as the festival opens.

"It's a very different show from years past," Frilot said. "So people can get ready for it." 

This article was originally published on Jan. 24. 
Updated on Jan. 25 and Jan. 27: With more details about projects in Park City. 

Culture: Your hub for everything from film and television to music, comics, toys and sports.

Movie Magic: The secrets behind the scenes of your favorite films and filmmakers.