VR Progress

Learning, Testing the Boundaries of New Content Production

“It's not fair. It's too late. It wasn't supposed to happen this way. It *shouldn't* have happened...” – Madeleine, “Vertigo,” Alfred J. Hitchcock Productions, 1958 — *“It’s not fair. It’s too late. It wasn’t supposed to happen this way. It *shouldn’t* have happened…” –* **Madeleine, *“Vertigo,”* Alfred J. Hitchcock Productions, 1958**

There are many ways to describe VR film shooting, production.

For me, it’s like open ocean scuba diving.

Get down 40-60 feet and you’re immersed in a world of nothingness.

All around you, your world goes on forever. You have no point of reference. It’s a damn good thing you’re tethered to the boat with a line.

Yeah, guys and gals who do VR day-in and day-out know the feeling because there are no guidelines, no proven techniques. They’re on the bleeding edge of one of the techniques visual storytellers use to express themselves but VR is way different because the filmmaker will turn over much of the storytelling to the viewer with VR.

And there are people like Steven Spielberg who say VR is dangerous because in his words, “…it gives the viewer a lot of latitude not to take direction from the storytellers but make their own choices of where to look.”

Others are less kind.

Getting Serious – You know your technology has arrived when it gets its own Gartner Hype Cycle. VR got its own a few years ago and has been steadily climbing the charts since. The movie/entertainment industry has taken it very seriously; pouring time, money, effort into bringing it to consumers. — **Getting Serious** – You know your technology has arrived when it gets its own Gartner Hype Cycle. VR got its own a few years ago and has been steadily climbing the charts since. The movie/entertainment industry has taken it very seriously; pouring time, money, effort into bringing it to consumers.

But there will always be an audience for other film techniques – 4K HDR (high dynamic range) stories, 360 video and streams that put you in the center of the action and probably more to come.

Still VR has its place in the visual/audio storytelling arsenal; and while it’s in its infancy, a few filmmakers are doing it very well. Over the next five years, it will get better with its own unique set of tools, guidelines, mind-numbing failures and mind-boggling successes.

To a lesser degree, 360 video is quite similar which is probably why so many people mask their offerings by calling them 360 VR. Except for some differences, the pros we talked with see a future for both forms.

The Script

One of the first issues many filmmakers raise is that you can’t script the VR story because once viewers enter the “scene,” they can go wherever they want, do what they want, explore/experience what they want.

Nick Bicanic, of RVLVR Labs, blew that excuse out of the water when he said, “Cinematic VR is NOT an Easter egg hunt.”

He points out we live life every day in 360 degrees. We can see approximately 170 degrees but can hear/sense 360 with our ears and other visual cues.

Cuban Celebration – Nick isn’t much for grabbing selfies but in documenting the first Rolling Stones concert in Cuba he wanted folks to know he was in the thick of it, along with an audience of close to a million music-loving fans. — **Cuban Celebration** – Nick isn’t much for grabbing selfies but in documenting the first Rolling Stones concert in Cuba he wanted folks to know he was in the thick of it, along with an audience of close to a million music-loving fans.

“People focus on an area of interest (a conversation, a ball, a door, a car, a strawberry/apple juice smoothie, a kiss, or a group of friends),” he explained. “Our attention shifts with each particular scene but we don’t run around like a puppy trying to take it all in before it disappears.”

That’s how you script a 360 video/VR film.

Having shot documentary and scripted narrative 360 and VR projects, he thinks most of the work to date comes up short because the filmmaker doesn’t include the viewer and his/her UX (user experience) at the outset.

Lewis Smithingham, of “30ninjas,” feels his virtual storytelling gets better and better with every project.

“We’re still in the awe phase,” he said. “The people who are really good and serious about this stuff are making major strides in developing the character, plot, drama, beat and viewer guides to make it entertaining and compelling.”

Marcelo Lewing, of DigMediaPros, added, “Don’t bore me. Don’t make me work too hard. Make me feel stuff.”

In other words, it’s all about the story!

Nick, Lewis and Marcelo insist the basic tools are available and are constantly being improved. It’s just up to the filmmaker to shake off what he/she has been doing and work in a new, more open way.

Capture

Want to start shooting? Camera choice depends on whether you’re a hip YouTuber or filmmaker.

The YouTubers can choose from a wide array of smartphone 360 video add-ons and cute 360 devices.

Filmmakers have tougher choices because the benefits of the 360-degree experience is compelling but there are a lot of problems to be solved because the medium is so new.

Jaunt, Nokia Ozo and recently introduced Sphericam are just three of the VR cameras that filmmakers are using to tell their stories.

Pro Tools – The market for 360 degree and VR video is only just beginning but already, there are hundreds of consumer 360 cameras as well as a growing number of units designed for filmmakers. At left is the Nokia Ozo which has already had a major price reduction ($15K), and on the right, is the Sphericam which offers excellent video capture as well as time-saving internal stitching. — **Pro Tools** – The market for 360 degree and VR video is only just beginning but already, there are hundreds of consumer 360 cameras as well as a growing number of units designed for filmmakers. At left is the Nokia Ozo which has already had a major price reduction ($15K), and on the right, is the Sphericam which offers excellent video capture as well as time-saving internal stitching.

Most are fixed focal lenses so they are placed and grab the activity around them, which means everything is spatial and you have to be aware of how the viewer’s attention is moving.

Nick pushes the camera systems with movement, proximity to action and a range of positions for coverage so he has all of the content he needs during production and post.

Lewis captures hours of content, racking up TBs of data so he has all of the story elements the viewer might want or need to experience.

“I easily rack up 20TB of content a day,” Lewis noted, “It isn’t a quantity race, it’s about capturing the moment or series of moments so you have everything you need when you begin to unfold the story for the viewer. In production, you don’t bring a knife to a gunfight. You want all of the ammunition possible.”

Most VR projects require a minimum of crew and equipment, forgoing things like technocranes, dollies and sound booms. Still, the crew and equipment have to be hidden and removed during post. That’s why most folks shoot multiple passes and composite them during production.

VR shooters agree there’s no right answer, just which approach they prefer.

Production, Post

They also agree that the biggest problem in 360, VR filmwork is stitching (merging) all of the footage together from a 360-degree camera rig with multiple lenses facing outward.

Capturing 10 times more data, it’s time-consuming to smooth the edges between one view and another.

Addressing this issue, Sphericam incorporated automatic stitching software in their gear which could be a huge time saver.

Folks who are less than enthusiastic about VR note the projects don’t have conventional frames (film segments/scenes) because the filmmaker doesn’t know where the viewer is going to look. Any cut might confuse him/her.

But Marcelo noted that the filmmakers on DigMediaPros say that the frame is a relative window of experience from the viewer’s field of vision, so the key is approaching production from the user’s perspective.

He cited Google principal VR filmmaker Jessica Brillhart’s explanation that every path exists simultaneously like ripples in a pond.

Ripple Effect – Just as you drop a pebble in the water and the ripples extend in all horizontal directions, 360 degree surrounds you while immersive VR envelops you — **Ripple Effect** – Just as you drop a pebble in the water and the ripples extend in all horizontal directions, 360 degree surrounds you while immersive VR envelops you

While traditional cutting is about doing temporal shifts and emotional juxtapositioning; immersive VR and 360 are about giving the viewer control of the experience, which is a work-in-progress.

Nick has shown that fast match-cutting in VR is clearly possible without using cuts/dissolves or wide shots/close-ups to allow the viewer to move between scenes.

Regarding the state of production tools, we asked Lewis, who has done tentpole 360, VR projects like Conan 360; Marcelo, who regularly does conference/convention 360 streams for DigMedia Pro and Steven Antturi, head of post production at IM360/DigitalDomain, and part of the Rio OBS (Olympic Broadcast Services) team.

They all agreed that the tools are still in their early stages but are improving rapidly.

Multi-Tasking – 360 and VR production isn’t a straightforward process like “conventional” film production; so it requires fast, high-capacity storage and connectivity like Thunderbolt 2 that enables multiple specialists to work on the project simultaneously. — **Multi-Tasking** – 360 and VR production isn’t a straightforward process like “conventional” film production; so it requires fast, high-capacity storage and connectivity like Thunderbolt 2 that enables multiple specialists to work on the project simultaneously.

Obviously, VR workflow breaks a lot of the rules filmmakers are experienced with because transitions are initiated by the viewer, which requires authoring interactivity in the VR streams.

This means massive amounts of blindingly fast storage are needed for simultaneous file access by a number of people using high-performance workstations. Artists, editors and others collaborate to stitch footage from multiple cameras, hide seams, stabilize shots, grade color and carry out other work that demands low latency production storage.

And it’s available!

Convention in the Round – Marcelo has taken his 360-degree camera to a number of conferences and conventions to give visitors to DigMediaPros.com’s site a clear view of being at the event, even if they are thousands of miles away. Sometimes you have to wonder if the combination of a great 360 video stream and online reports might save you time and money. — **Convention in the Round –** Marcelo has taken his 360-degree camera to a number of conferences and conventions to give visitors to DigMediaPros.com’s site a clear view of being at the event, even if they are thousands of miles away. Sometimes you have to wonder if the combination of a great 360 video stream and online reports might save you time and money.

Still, there are production tools Marcelo would like to see which require standards and adoption by the major participants – YouTube, Vimeo, etc. They include:

Keyframes – defining the start/end points of a segment so viewers can turn on/off controls
Keyframe (segment) camera orientation so filmmakers can “re-orient” the camera for the viewer
Ability to add hot spots so the viewer can link to different viewing areas or websites if they want
Ability to stop, start camera movement to smoothly guide (or at least encourage) the viewer through the story
Ability to lock down an axis – tilt, pan, zoom – to facilitate the story
Provide spatial audio and audio queues to deliver a total immersive experience to the viewer

Total Agreement

While the four filmmakers often disagree on priorities, the one thing they agree on with Woody Woodhall, of AlliedPost, is that really good stereo 360 audio is the most important component in telling the immersive story.

Sounds Like – People have always appreciated the work that Allied Post Audio and other audio post production specialists do to enhance video content. But with 360/VR content, it will be even more critical in providing the viewer with audio clues to enjoy the best of the content. — **Sounds Like** – People have always appreciated the work that Allied Post Audio and other audio post production specialists do to enhance video content. But with 360/VR content, it will be even more critical in providing the viewer with audio clues to enjoy the best of the content.

“Some have said audio is 70 percent of a good/great 360/VR project,” he said, “I’ll stick with the more traditional 50 percent of the story. Regardless of the percentage, film storytellers need to guide the viewer with object-based sound, movement, staging and dialogue.”

Realistic 3D positional audio that occurs in the real world goes beyond surround sound and adjusts dynamically based on your head position and the location of the sound source.

HRTF (head-related transfer functions) attempt to reproduce how people hear sounds; but work still needs to be done to blend this with 3D audio to deliver positional audio that is natural to the “viewer.”

When it arrives, it will be a boon to the 360/VR filmmaker and the content consumer!

Enter the Viewer

Intel recently cut the cord for the HMD (head-mounted display) user and it’s great to envision a time when the personal headset won’t be required

Nick summarized the challenge for broad acceptance saying, “VR headsets are a speed bump in how quickly and widely 360 and VR are adopted. People won’t carry headsets and/or things to use with their smartphone so they can enjoy an immersive video like they do when they stop and stream 4K content.”

Short of a brain implant, that won’t happen in the foreseeable future, but it will become less bulky and more stylish.

Clear View – For the foreseeable future, HMDs will be the standard way to enable 360 video and VR content to immerse you into a different virtual world. But don’t worry; they will soon be better than the bricks currently available. — **Clear View** – For the foreseeable future, HMDs will be the standard way to enable 360 video and VR content to immerse you into a different virtual world. But don’t worry; they will soon be better than the bricks currently available.

And the variety, as well as the enjoyment quotients, will rise significantly.

The challenge is the compute power required to properly support 360 video and VR and the fact that I probably don’t want your immersive experience interfering with my immersive experience.

There are still a lot of hardware/software enhancements the content industry needs – advanced rendering, sensor fusion and true 3D audio – to deliver content people really want to watch.

We’re also in the learning phase of mastering which shooting/production rules can be, should be broken.

But Still

We’ve come a long way since Ivan Sutherland and his student Bob Sproull created the first VR/AR HMD. That early work is finally yielding some fantastic content.

Immersive VR is going to be mainstream filmmaking but Lewis knows I’m never going to be a VR shooter like him.