Your body is driving a new wave of AR, VR experiences

Inside the softly glowing room, I pace around a table with an iPhone to reveal an augmented-reality mystery. Digital characters on my screen drop breadcrumbs for me to follow back to the scene of a murder. As I trace their trail of clues, my movements make everything around me change -- the lights, the scenes, the sounds I hear in my ears. 

The use of movement and location to direct the flow of the story is one of the key elements of The Dial, the brainchild of Peter Flaherty. While he wanted to incorporate augmented reality, which overlays digital media on top of the real world, and projection mapping, which shrink-wraps video onto physical objects, the most important technology for telling his story is one we're already constantly using: our own bodies. 

"The more your body's engaged in any interactive or immersive form, the more meaningful it is," Flaherty said in an interview at the Sundance Film Festival on the night before the fest opened late last month. "Those choose-your-own-adventures -- where you choose A or B, pathway one or two -- for me is never that exciting. But if you're actually kinesthetically moving your body, you're engaged."

At the festival's New Frontier showcase of tech-heavy immersive projects, pieces like The Dial are using participants' bodies as the primary mechanism for how they work. Instead of bathing you in passive immersion or forcing fiddly game controllers into your hands, creators are nesting unfamiliar tech with an instrument -- your body -- that's as familiar as the back of your hand. Literally. 

Their hope is to make projects utilizing augmented and virtual reality -- once hot tech trends still struggling to find their way to consumers -- more engrossing. VR and AR could use all the help they can get to actually fulfill their promise of offering us radically different and immersive experiences. The trend could, at the same time, restore some of the connection we've lost to our physical selves that tech has cleaved away. 

"It's a critical moment culturally with the advent of spatial computing and the advent of also machine learning, for us ... to put the body back at the center of the potential relationship of technology and humans," said Melissa Painter, another creator whose project, Embody, is featured at Sundance. "Flat out, people's physical relationship to their own body has somehow been stripped out of the equation."

My technology, myself

Inside the glowing room-sized cube of The Dial, one viewer -- dubbed the navigator -- and two other people walk around a table with iPhones that reveal augmented-reality characters acting around a projection-mapped miniature house. The AR and projections reveal the characters and the sets, but the story's scenes are unlocked depending on where the navigator decides to move. 

It's a little like walking in front of an automatic sliding door at a supermarket, except instead of opening the entrance to a store, you awaken a new scene to play out. 

Your movements also sync with changing colors of light in this glowing cube. When you stand to the south of the miniature house, the lights turn pale blue to match the winter's night setting of all the scenes that take place there. If you stand to the west of the house, the lights turn orange with long shadows to suggest the autumn afternoon. 

All these high-tech orchestrations are designed to happen naturally as you follow whatever path through the story you want to take. Slowly, you unravel an interactive mystery centered on a woman who smashes through the stone wall outside her family home, exposing the underbelly of a formerly wealthy family. 

Flaherty hoped that the end result would be an engrossing story that has an actual arc. Making movements central to the story would allow the interactive methodology starts to "disappear," he said.

"They're not focused on trying to break the tech, they're not focused on trying to push the buttons," Flaherty said of his hypothesis for the project. 

Painter, on the other hand, purposely wanted to avoid telling a story with Embody, an experience that feels like going on a fantasy yoga retreat. (Note that even somebody like me, who is terrible at yoga, wanted to do it again.) Her design was to create something that felt like a product of the future rather than storytelling of the future. 

Share: