360° and VR Video: New Technologies, New Possibilities
2016 is the year of 360° video and its cousin, Virtual Reality. How can filmmakers, documentarians, and journalists use this new medium in the service of storytelling?
You can’t miss the buzz around 360° video (often defined as a live-action and mostly passive experience) and virtual reality (VR, typically computer-generated and often interactive). Many of the pieces are forgettable, so it's tempting to dismiss 360° and VR video as a flash in the pan. But when it’s good, it’s very good, and its strengths—immediacy, intimacy, the ability to transport us anywhere—are obvious.
If you haven’t seen this this new kind of video yet, be warned that viewing on a desktop computer is underwhelming. You can watch it on a phone or device and take advantage of the built-in accelerometer, but without a headset, that’s only incrementally better than the desktop’s mouse-driven experience. For the most immersive, intimate experience, you need a quality headset, and it's even better with stereo images.
FORM SERVING FUNCTION: EXAMPLES
Graham Roberts, senior graphics editor at the New York Times, saw his first VR demo in 2014. He began thinking about how to apply 360° video to journalism. The Times is now all-in on VR, shooting 360 video on the U.S. presidential campaign trail and at the November 2015 Paris vigils, as well as working with VR company Vrse and director Chris Milk on a variety of VR projects. “Journalism has always in some way or another been about transporting you to a place, for a time, to understand it better,” says Roberts, “and I think VR is this amazing new tool to strip away some of that distance.” Check out the Times’ videos.
That ability to place the viewer directly into the middle of situations or cultures they wouldn’t ordinarily have access to, and the immediacy and intimacy of the experience, is the power of the medium, and of VR’s potential as a tool of cultural empathy. Planned Parenthood worked with director Nonny de la Pena to produce Across the Line, a 360° video that follows a patient walking into a clinic, past a gauntlet of vocal and intimidating anti-abortion protestors. It’s not an experience many will have, but the 360° format is a way to put the viewer in the shoes of the patient. “It’s hard to imagine until you are the one experiencing it,” says Planned Parenthood Press officer Catherine Lozada. “We wanted to help people understand what the experience is like. Telling is one thing, experiencing it [in 360°] is completely different. You become a patient.”
In Collisions, director Lynette Wallworth and producer Nicole Newnham tell the story of Nyarri Nyarri Morgan—a Mardu aboriginal—and his experience witnessing the 1950 atomic tests in Australia. Wallworth has worked in both interactive installations and documentary film, and she found VR to be a natural fit for her as well as a perfect form to tell Nyarri’s story. Both the Western Australian landscapes—which show beautifully in vast 360° panoramas—as well as the personal and cultural nature of the piece made VR the right tool for the job. Watch it and you’ll find yourself alone, in the middle of barren western Australia, being sung to by an aboriginal man just a few feet away. The impact is amazingly intimate—you do feel like you are there.
But Wallworth is wary of filmmakers who parachute into cultures and capture and distribute content without context. The ease and convenience with which you can “experience” a distant culture from the comfort of your living room is fraught with ethical complications' it's an even more invasive version of parachute journalism and cultural appropriation. Wallworth sees it as such an invasive technology that unusual sensitivity and care is required, and she feels the viewer should have “an understanding that you are invited to be there—even virtually.” She’d done three previous projects with the Mardu people and believes that the relationship and trust developed as a result were keys to the success of the most recent project. In the case of Collisions, the Mardu had sacred places and things they didn’t want on camera, which is hard when there are cameras pointed everywhere. “This camera captures everything,” says Wallworth, “and everything had to be checked again when it was finished to make sure that nothing sacred or secret was visible to the camera.”
Collisions is a hybrid and experimental piece, incorporating both live-action footage and CG elements to tell Nyarri’s story. While essentially documentary in nature, the filmmakers used both re-creation and multimedia, and the effect here is poetic and visually stunning. The form serves the function perfectly.
CAUTIONARY TECH TALES
But 360° video isn’t good for everything, and the buzziness of the medium, along with the relatively low-cost barrier to entry of the technology, has resulted in a slew of lifeless, boring 360° videos.
“If there is a singular point of action and no reason to see anything else, it’s probably not a good VR shot,” says the Times’ Roberts. “If you need a lot of movement and a lot of cuts, it isn’t a good VR shot. But if you want to give people the feeling that they’re there, and from a journalistic perspective give them back control of the crop, in a sense, let them see the whole scene, that’s good VR.”
The technology is currently low-fidelity and/or kludgey, or proprietary, unwieldy, and expensive. An entry-level GoPro rig typically has six cameras in an array, paired with and controlled by a wireless remote and iPad apps, and it’s not easy to monitor while you’re shooting. Low-end rigs are prone to error, frustration, and poor audio. This isn’t a showstopper for narrative live action, where action can be choreographed and repeated, but for documentary and journalism work, where the ability to be spontaneous and capture quickly unfolding moments is vital, it can be disastrous.
Camera placement is critical to an effective 360° video shot, and the distance of the camera to the subject is also important. Subjects crossing the camera planes less than three or four feet away can get chopped in half by parallax, which can be difficult or impossible to fix when you’re stitching the footage together in post. And because of the throw of the wide-angle lenses, if action happens more than 10 to 15 feet away, that action may be lost. Optimally, subjects stay within four to 12 feet of the cameras.
Because there’s no “behind the camera” in 360°, directors have very little input into the action or movement of their subjects, which could lead to the temptation to manipulate or choreograph the action a little. That’s taboo in journalism and verite style documentary. On the other hand, since you’re picking up 360 degrees, you don’t really need to point the camera. As long as the subject is given basic information on how to relate to the camera and how far away to stay, no directing is needed, at least in theory.
The cameras themselves can create something of an observer effect, as people are distracted by the weird-looking camera rigs. Directors and camera crew either have to hide once the camera rolls, or become part of the movie. (The Collisions crew often had to drive a good distance across the vast Australian plains to get out of the shot. Other times, they hid behind rocks, cars, or houses.)
These factors add up to make 360° doc and journalism work difficult, but not impossible. The most common solution is to place the camera thoughtfully and walk away. The New York Times Paris vigil piece placed the viewer between the shrine and the crowd, giving viewers a great perspective in all directions. “One of our filmmakers in Paris, Ben Solomon, said it really well,” says Roberts. “He said, ‘Traditional filmmaking is like hunting, and VR is like setting a trap.’ With VR you put it there and then get away, and part of the challenge is getting out of the shot.” Roberts sees an advantage to the smaller rigs for newsgathering. “One of the great surprises is that after the initial glance or so, people ignored the camera. Some of these [expensive new] cameras people are coming out with look amazing, but I imagine bringing it to an intimate news scene; everyone would be like …‘Wow.’”
Post-production can be intensive, expensive, and far more time-consuming than production. Between stitching together the multi-camera footage; patching and compositing out parallax artifacts and tripods and other camera support; color correction; adding graphics or VFX; and audio mix, it’s not atypical to spend a day shooting and a month on post. And 360° video really shines on a quality headset with stereo images, but stereo production requires twice the cameras and makes it at least twice as complicated in post.
Maybe even more than in traditional filmmaking, audio and sound design is a key and under-appreciated component of this new video format. Just as the phone or VR device accelerometer controls the crop and movement of the visual scene, it also dictates the audio mix and pan, and a good sound design can greatly improve the experience, bringing it from ordinary to immersive. The Times has experimented with binaural, positional, and tetrahedral audio and will have ambisonic capabilities (more organic-feeling surround sound) in a near future version of the publisher’s app.
“Audio will take on an incredible new role in VR as being a key driver of how we direct your eye gaze,” says Roberts. “I can’t think of many better ways to direct where people look,” agrees Collisions’ Wallworth. Her team did the audio mix at Skywalker Sound, and it paid off. “Nicole and I were just so happy and thrilled when that Atmos sound was layered in, because it gave another whole dimension. It ramped the sense of presence enormously.”
NOT FILMMAKING AS WE KNOW IT
In spite of all the obstacles, VR and 360° remain powerful new tools in the service of documentaries and journalism. How VR and 360° will evolve in the narrative and live-action world is less clear. There are some obvious strengths: Imagine using 360° video in a horror film. You hear someone breathing down your neck, prompting you to turn and discover an unspeakable terror just behind you.
But traditional narrative filmmaking—with its tension-driven pace, rigid structural language, and beginning-middle-end storyline—is a different animal from VR, where the strengths are experiential, immersive, potentially interactive, and paced for exploration and self-discovery. Consider a car chase sequence. In a traditional live-action piece, the director might ratchet tension by telling the viewer exactly where to look: maybe an establishing wide shot to set the scene, a shot of the driver tensely gripping the wheel, a close-up of the gas gauge hovering near empty, a quick medium shot of the passenger, bleeding from a gut shot, a whip pan POV to the chase cars. All of this happens in a few seconds and wrings maximum dramatic tension from the moment. A 360° experience of the same scene would likely take much longer, to allow the viewer to look around and discover the important elements at their own pace, and with the added risk that they’ll miss key information. While 360° filmmakers can try and influence the POV with sound, eyelines, or light and other visual cues, they can’t be sure the viewer will be take in everything, or be in the right place at the right time. Dramatic tension, as we currently know it, ceases. It’s a very different kind of experience, and filmmakers will need to approach it very differently.
Directing actors in VR is different as well. In the journalism and documentary worlds, you essentially put the camera on the floor and walk away. But for narrative pieces, the action needs to be choreographed. “It’s much more like directing theatre or performance than film,” says Wallworth. “You block things out, you can’t create close-ups, you’ve got people moving about in a field of view, all of them visible.” “Directors have to let go a lot more, and let events unfold,” says Osman Zeki, an interactive developer at Turbulent, who partnered with the National Film Board of Canada on the VR story The Unknown Photographer.
The reality is, VR and 360 is not filmmaking as we know it, and talking about it as an extension of film or attempting to wedge Hollywood cinematic rules into VR experiences only exasperates traditionalists and enrages 360° filmmakers. Which isn’t to say that traditional cinematic elements won’t make their way into the VR world—they will.
But a lot of the primary learning has and will come from the gaming world, where designers and developers have been dealing with the challenges around engagement, interactivity, open-ended narratives, and choosing your own adventure for many years now. Storytelling in 360 degrees is really a new form, with only a few crude rules that are already being discarded and broken, and the vocabulary is still unformed. “In very early conversations, everyone we spoke to said you can’t do filmic transitions in this medium,” says Wallworth. “And you see whole bodies of work where the transitions from scene to scene are fading in and out of black. It became a really great challenge with the new technology to find a way to apply filmic transitions—which are beautiful and intriguing—into virtual reality space.”
VR creators will need to learn through doing, and pushing the technology and the form. “There is a different design language with VR that you have to learn,” says Zeki. “You can read articles—we did all the research—but until you’re in the trenches, you’re not going to be able to understand what you need to tell your story.”
Because of the physical qualities of VR headsets, and also tactile feedback devices and controllers, the user experience will drive the form, maybe more so than most other mediums. Best practices around length, motion, transitions, camera position, and so on will all be dictated by what the user will tolerate. “Remember, this is the user’s head,” says Roberts. “You don’t want to toss it around—they’ll feel sick.” The pace is generally much slower, and you have to be careful about how much information is delivered through voice over, and when. Wallworth found that “talking at the beginning of scenes isn’t remembered as well as at the end. You’re overwhelmed by everything you want to look at and not listening to what people are saying.”
Perspective is important, too: Is the viewer a participant? Do the subjects address the viewer (the camera)? Or are viewers witnesses, anonymously observing? And because VR is—for now, at least—primarily a single-person experience, running time matters. “Definitely think about length,” says Louis-Richard Tremblay of the National Filmboard of Canada. “VR is a lonely experience. How long will you accept being lonely in this passive, sometimes aggressive world?”
BREAKING THE RULES IS HALF THE FUN
While the few rules are still being written, 360° artists are also already using the flaws and idiosyncrasies of the technology to try new things. Creative people will find ways to misuse the platform to push it forward. For example, while most of Collisions was stitched and composited flawlessly, the filmmakers intentionally left 360° technology artifacts scattered through the piece like Easter eggs. A camera stand shadow in one scene, out-of-sync footage compressing time in another, a tripod intentionally left in yet another—all were ways for the filmmaker to allude to the presence of the camera and its impact on the subject, while commenting on the medium and storytelling itself. As Wallworth says, “Our interest was not just in the story but in the technology, and in a contemplation of that technology in relation to this story.”
In another case, Bjork’s Stonemilker 360° music video uses intentionally un-synced cameras and different takes to “double expose” multiple Bjorks dancing around a barren, volcanic landscape. Like any medium, breaking the rules is half the fun, and we can expect 360° filmmakers to continue to abuse the technology for their art.
Roberts compares it to the early days of photography. “How often does a completely new medium come along? And that’s how I see this. 3D films weren’t a new medium; they were an enhancement. This [VR and 360° video] lets us as creatives invent what storytelling means in this new space. And I don’t think we’ve figured it out yet.”
“I feel the technology leads at the moment,” says Tremblay. But as they always do, creative people will catch up and find ways to push the medium to new and currently unimagined places. VR will move from purely experiential novelty, to one of our primary vehicles for storytelling.
“Sadly, a lot of things need to be invented in order to let you to tell your story,” says Osman Zeki. “But I think it’s going to shift at some point. Tools are going to get better, and you’re going to be able to tell your story the way you want to. And that’s going to be magical.”