At Sundance's New Frontier, VR and AR go epic, weird and theatrical
Virtual live improv theater? Karaoke with digital puppets? Synesthetic VR food fests? The tech-driven show lets artists test their limits.
Joan E. SolsmanFormer Senior Reporter
Joan E. Solsman was CNET's senior media reporter, covering the intersection of entertainment and technology. She's reported from locations spanning from Disneyland to Serbian refugee camps, and she previously wrote for Dow Jones Newswires and The Wall Street Journal. She bikes to get almost everywhere and has been doored only once.
ExpertiseStreaming video, film, television and music; virtual, augmented and mixed reality; deep fakes and synthetic media; content moderation and misinformation onlineCredentials
Three Folio Eddie award wins: 2018 science & technology writing (Cartoon bunnies are hacking your brain), 2021 analysis (Deepfakes' election threat isn't what you'd think) and 2022 culture article (Apple's CODA Takes You Into an Inner World of Sign)
This year's New Frontier program at Sundance is getting back to its roots. The film festival's high-tech art showcase is going to feel a bit like a party again -- only bigger and more mind-warping.
"We're seeing work being done on a larger scale," Shari Frilot, New Frontier's chief curator, said in an interview Tuesday. "This year, we're ... seeing more kinds of stories being made possible because of innovations" in technology creators are making to realize their visions, Frilot said.
Sundance's New Frontier, designed to foster new forms of storytelling using technology, is the year's first major stop for interactive storytellers to show off their latest work. The program underscores the growing influence of tech like virtual and
as new mediums for artists at an event once solely devoted to traditional films. This time the program's scale is getting a boost as New Frontier moves to a new, expansive space, allowing for outsize projects to stretch imaginations.
The VR doesn't end with New Frontier. Big names in virtual reality swarmed to Park City to show off their latest projects on the sidelines of the festival, too. In invite-only demos, Oculus debuted sneak peeks of The Under Presents, a social VR narrative game that uses live actors and the company's coming Oculus Quest headset. It's as if the indie video game Journey were adapted into an immersive play like Sleep No More but set in your wildest, most head-scratching dream. And Disney has been showing off a VR short called Cycles at a storefront off Main Street.
New Frontier kicks off Friday afternoon as the Sundance Film Festival 2019 opens its 10-day run in Park City, Utah. It spans two locations: A large-scale venue called New Frontier Central hosts most of the exhibition, while a smaller space nearby, in the basement of the Ray theater, features the rest of the projects along with a 40-seat VR cinema and all the New Frontier panels.
Oh, and New Frontier Central is where they put the bar.
"We're trying to get back to what we started with and lost over the years: getting back the social intersection element of New Frontier," Frilot said of the social space. "[It's] about getting lots of different kinds of people in the same room talking about work that is really surprising them."
One of the dominating installations at New Frontier Central will be an interactive Immersive Stage in the back corner, an area where the floor and two walls become projection screens for three projects by experimental theater-art production company 3-Legged Dog. (Three years ago, the group helped stage a punk-tinged, projection-heavy opera about human trafficking that won the Pulitzer Prize for music.)
That's where karaoke with digital puppets comes in. Esperpentois a Goya-inspired animated world, which includes a performance of a play with one character represented by a digital puppet and the others played by real people. In one of the "lounges" inside the Esperpento world, artists will be teaching people how to puppeteer alongside somebody singing. Performances of the play within Esperpento, called Two Black Lights and One Red, will take place four times over the course of the fest.
"I've seen things like that in very large studios," Frilot said. "It blew me away when I saw this indie development."
Throughout the festival, the Immersive Stage will alternate between Esperpento and two other works. Dirtscraper is a
"game" with an aesthetic like that of a bleak dystopian Donkey Kong, and analmosh is an audio/visual performance that links glitchy electronic music to projected imagery that's generated live based on the audio.
The Dial, at the New Frontier Central, also uses projections, as well as augmented reality, to stage a mystery. A woman smashes her car through the stone wall of her wealthy family's compound, action that all takes place around a projection-mapped sculpture of a house inside a glowing translucent cube. Three participants use augmented reality on their
to interact with the projection and unravel the mystery.
The Royal Shakespeare Company's tiny amphitheater, back at New Frontier Central, is where a pseudo-hologram of Shakespearean actor Robert Gilbert will perform the Seven Ages of Man speech from As You Like It. (High-school English cheat sheet: That's the one with "all the world's a stage" in it.) Gilbert performs as a mixed-reality avatar, appearing in front of participants via a
Watch this: What it's like to experience Magic Leap One
Magic Leap itself is presenting Mica, its human-type artificial intelligence technology. Pitched with a first-person invitation from Mica herself -- "Join me at the beginning of my existence, as we pull from the past to create the future," whatever that means -- the tech will interact with you in a "gestural exchange" that's aimed at getting festivalgoers to contemplate her place in our world, says Magic Leap.
And A Jester's Tale integrates robotic avant garde pop star Poppy into the mixed-reality telling of a "psychologically taxing children's fable." By virtue of Poppy's participation, A Jester's Tale, which is at the Ray, has already begun to generate buzz on Twitter.
Hard and soft
Artists at New Frontier are "out of necessity inventing custom hardware and custom software to be able to tell these stories," Frilot said.
One such customized work is Sweet Dreams, at the Ray, which leads you through multiple rooms with bouncy walls and, using artificial intelligence and virtual reality, aims to give you trippy impressions from things you eat and drink. Synesthesia is a rare neurological phenomenon that mixes up the human senses, causing some people to experience sensations like hearing colors, smelling sounds or seeing words or numbers as colors and shapes. With VR and AI, Sweet Dreams aims to re-create the condition, making it feel like you're eating the sun or sipping the room around you.
Mechanical Souls at New Frontier Central also developed artificial intelligence that coordinates with VR to stage something like an immersive-play mystery game. The premise of the experience is "a sumptuous wedding" for the "heir of the rich Zang family," a ceremony with hired androids on hand to help. When something's gone wrong, six audience members play Mechlife employees to figure out what's happened. The story unfolds in virtual reality with live actors, and each participant may follow a different path of action, depending on where he or she looks.
The artists behind Interlooped developed new hardware and software with the ambition of creating what Frilot describes as a rabbit hole of reality, where you can't tell what's real and what's not. It includes live volumetric capture -- a technology that essentially scans a whole person to replicate them as a pseudo-hologram in real time. Interlooped, however, applies this kind of live capture to two people at once, something that hasn't been done often in performances.
Interlooped, Mechanical Souls and all the Magic Leap experiences are examples of a trend Frilot noticed while curating this year's New Frontier. Creators, she said, are putting physical bodies inside the frame of their experiences.
"That's fascinating to me ... particularly because for so long there was that criticism of VR that it takes you out of the real world and into something else," said Frilot.
Another is Embody, at the Ray, described by Frilot as a high-tech yoga experiment. Drawing from dance and the martial art called aikido, Embody is social, movement-focused VR where you see other bodies next to you entering and leaving the experience. Runnin' also incorporates dance into VR, in this instance holding a virtual interactive dance party with comic hip-hop artist Reggie Watts.
The festival also includes three VR cinema programs, ticketed events where groups of up to 40 experience the same sequence of VR shorts together in a room called the Box at the Ray.
The second program focuses on two documentary-style experiences, Traveling While Black and Marshall from Detroit. Both were made in collaboration with Felix & Paul, an immersive storytelling studio that's been a staple of film festivals like Sundance for years. Marshall from Detroit is a 360-video interview with rapper Eminem during a drive around Detroit on a winter's night to discuss his creative process and his hometown.
Traveling While Black used the Green Book -- the travel/survival guide used from the 1930s through the '60s by black motorists to identify places across the US where they could drive, sleep and eat safely -- as a jumping-off point for intimate conversations about hazards to black people's safety in public spaces today. Traveling While Black will also be on view at the Ray in a replica of Ben's Chili Bowl, a Washington DC restaurant that was one of the Green Book's safe spaces and the site of many of the VR experience's interviews.
The first and third programs at the Box span four far-flung projects each.
Program 3 incorporates Live Stream from YUKI <3 about a live-stream influencer who has a secret she hides from her fans. Kaiju Confidential flips the script of a Godzilla movie to explore the point of view of towering animated monsters. Ashe '68 uses re-creations, period archival footage and sand animation to convey the internal pressures felt by black tennis legend Arthur Ashe. And 4 Feet: Blind Date follows a blue-haired teen girl on a blind date.