VR Filmwork

The Immersive VR Filmaking Experience Begins to Emerge … Slowly

“I need to have something in the foreground to give it some scale.” **Hooper, “Jaws,” Zanuck/Brown Productions, 1975**

Want to see a storage person’s heart skip a beat … or two? Flutter even?

Just mention VR and BAM!, they get real excited.

Creating good (if not great) VR is getting slightly easier, but it still poses serious challenges for the professionals working in this emerging medium.

We’ve seen some of the stuff Andrew Shulkind and Andrew Cochrane have been up to (scripted and unscripted), and we know it’s good.

We were all fired up to hear Shulkind at CineGear and then later presenting with Cochrane at the Creative Storage Conference to get their insight into the progress that has been made with the technology and when we’ll be able to see VR as “regular fare.”

Before this tag team took the stage at Creative Storage, Tom Coughlin gave his annual overview on the state of storage in the M&E field.

With 4K content consuming about 6,880GB/h and 8K gulping roughly 91,729GB/h, we’re starting to talk some serious storage capacity.

360º and 180º immersive video and AR/VR must be even more data-hungry.

**It’s Great** – Most of the people who discount VR film viewing either haven’t experienced it or watched some really bad VR immersion (it is out there). Once you experience good/great VR storytelling, you’re ready for more.

After all, darn near everyone is ready to immerse themselves in great content.

Taking a conservative approach to forecasting VR storage needs, Coughlin projected that stereo 360º video would require about 115 GB/s of data or roughly 414 TB/h due to its 16,000 x 16,000-pixel resolution, 10+ bits per channel and a frame rate of up to 120fps.

Both Andrews thought these requirements were “on the modest side;” and they should know, they’re busy doing projects around the globe.

For almost two decades, Shulkind has been at the leading edge of where technology meets entertainment. He’s become the “go-to” expert on the future of entertainment; bridging immersive content, advertising, cinematic quality, gaming and mixed reality.

He is a rainmaker for both Hollywood and tech giants making Hollywood-style steps.

He’s done projects all over the globe and even works with foreign governments to help foster local production economies but seems most proud of his advisory work for SpaceCraft VR, a NASA-sponsored interactive infrastructure for space exploration.

**Beyond Gravity** – Shulkind’s work with the Spacecraft VR project, sponsored by NASA, is a worldwide collaborative effort that excites you whether you’re young or old, male or female. It is addictive to see how people come together to expand our understanding of the universe.

Cochrane’s pedigree is just as impressive, including running digital and interactive projects for over six years at Guillermo del Toro’s Mirada, and now managing projects as a consultant and independent creative. Recently, he directed five CG-animated 360º films for the Vegas: Alter Your Reality experience and has created immersive content for USA Today, Disney, Intel and Visa, with multiple projects currently in production.

Before the session, we sat down with Cochrane and Shulkind to get an update on the progress that has been made with VR over the past few years.

**Sound Growth** – Everything is getting better about VR film work, including the experience of the people at the forefront of immersive content. They’re learning how to help viewers tell their story. The only shortfall has been the wild projections of early analysts. It will take time.

Shulkind said the immersive content business is the latest example of how storytellers have always used the best tools available to engage and thrill the biggest audiences in the most immersive way possible.

From stories painted around a cave, to amphitheaters and coliseums, to 19th century panoramic paintings and haunted houses, to Cinemascope and Cinerama, to surround sound and gaming chairs, entertainment has always striven to be engaging and immersive.

Cochrane agreed that the application and purpose for placing a person directly in the movie or experience is only the beginning, and the future of this medium is when users can interact with the content.

“We’ve only just begun to tap the potential of immersive media. From 360º and 180º video to realtime experiences, there’s a growing library of interesting, innovative content that people really want to experience,” he added.

**It’s Alive** – OK, “The Strain” was spooky enough on FX but Cochrane helped people get up close and personal with the evil strain of vampirism that ravaged New York with VR segments. If you aren’t scared when you take off the HMD … you’re one of them.

At the beginning of their presentation, Cochrane told the audience that VR is a massively exciting new medium that has ignited the imaginations of technicians, creatives and businesses.

“We have only scratched the surface as to what can be done with and how people can experience and interact with virtual environments,” he noted. “But what gets lost in the excitement is that it isn’t going to be an instant success. We still have a long way to go and obstacles to overcome.”

Both filmmakers noted that the challenge is being able to efficiently and effectively handle all of the data required to create this content.

**Poor Math** – VR film production done right means that 1+1 doesn’t equal 2, it equals 4 – content for everywhere you look, might look, couldn’t possibly look. As resolution quality increases, so does the data.

Shulkind explained that 1080p per eye can be readily handled within traditional production pipelines. But when you scale up to 3D 4K VR, the requirements grow exponentially. “At the moment, the weakest link is not the capture but the display devices,” he stated. “Outside of a couple of early devices that still have very low adoption (like StarVR 5K and Pimax 8K), every headset is still below the 4K display threshold.

“Even if you wanted to deliver in 4K, you couldn’t see it,” he noted. “The current displays have such low pixel density that when viewed through the cheap plastic magnifiers in the headsets, the image is dramatically softer than what we shot. Newer displays will improve upon this quickly; but in the meantime, you have to take these factors into consideration when selecting your pipeline and how you spend money.”

**Robust** – Filmmakers who understand it well know that the content is more than just HD, 4K or even 8K. VR adds a new dimension of workflow and workload for the production/post teams.

“4K is not twice 2K, it is four times 2K!” quipped Cochrane. “If we want to match the abilities of our eyes, we’ll have to reach 64k resolution, at 72 frames per second or higher with at least 11bits per channel, per eye; so, we’ve got quite a way left to go.

“The production, distribution and viewing issues all have to be addressed with a project, and it isn’t simple,” he added.

**More Glass, More Storage** – VR cameras and rigs are being improved much more rapidly than the headgear people wear to experience the content. Rigs like the Headgear VR unit at the right also use a lot of SD cards to capture 360 degree and immersive content.

Shulkind stated that VR cameras and rigs are still in their evolutionary stage with new and better solutions shown every year.

The one thing they all have in common is that they capture a lot of data – usually on SD cards – which needs to quickly and reliably copied to the computer and larger, more easily managed storage solutions.

Shulkind noted that rendering of the detailed images requires robust CPUs and GPUs because a detailed 4K image can require more than 5.6ms per frame.

“If rendering requires 700ms of every second, the system has to move more 2.988GB in 300ms or 9.9GB/s,” he explained.

**Heavy Lifting** – High-performance CPUs and GPUs are needed to process immersive RAW data during the stitching and rendering processes. And storage has to be fast enough to keep the pipeline full.

“A pivotal point for VR filmmakers and producers, is our ability to move RAW and final data quickly and efficiently which means high performance GPUs and ultra-high-speed storage solutions,” Shulkind continued. “We always have someone on set with multiple laptops churning through rushes and focusing on what is being done so we can more quickly and easily complete the fine stitching after the edit.”

He noted that the fastest consumer SSD available today tops out at 3.5GB/s. As a result, VR filmmakers have either striped several SSDs to handle 4K VR or do what he and several friends have done, moved to the OWC ThunderBlade. The unit can transfer up to 2800MB/s and two units provide speeds of 3800MB/s to handle the high bandwidth content stream requirements.

Technology such as volumetric and light field capture, photogrammetry and spatialized audio will continue to generate extremely large, complex master files that will dwarf what is being done today in terms of both size and required functions

Virtual is Real – Shulkind emphasized to the audience that VR proponents tend to be overly optimistic on when the film experience will be mature; and naysayers tend to point to how growth isn’t keeping pace with industry projections. The truth is somewhere in between, and both progress and acceptance are taking place.

While VR has its optimists and pessimists, both Andrews feel significant progress has been made in the past few years. The new cameras, software, computing assets and storage are paving the way for exciting opportunities for people to immerse themselves in a wide range of content – documentary and entertainment.

“We’re working on new experiences and examining new ways the audience is responding,” Shulkind said. “At the end of the day, when we’ve done our best work, it’s the reaction/interaction of the viewer that keeps us going.”

“We’ve moved into an exciting new space,” Cochrane added. “We’re not just visual storytellers anymore, we are really enabling and encouraging people to be story-livers. VR is quickly becoming dynamic, personal, social, and even transformational.

“The technology, software, talent and user behaviors are still in the early stages,” he continued. “We haven’t seen the extent of what VR can deliver. We don’t want to oversell it, and we don’t want people to sell it short; it’s still in its infancy.”

More than just filmmakers; Shulkind, Cochrane and other active VR professionals are integrating the polish of immersive interactivity and data optimization, setting guidelines, focusing on standards and using AI to turn skeptics and non-believers into VR fans.

For the Creative Storage audience, they had one final message … the volume of data and content they are creating and working with is growing rapidly as they create more interesting, more exciting and more immersive video experiences.

Or, as Brody said, “You’re gonna need a bigger boat.”