The Immersive VR Filmaking Experience Begins to Emerge … Slowly
Want to see a storage person’s heart skip a beat … or two? Flutter even?
Just mention VR and BAM!, they get real excited.
Creating good (if not great) VR is getting slightly easier, but it still poses serious challenges for the professionals working in this emerging medium.
We’ve seen some of the stuff Andrew Shulkind and Andrew Cochrane have been up to (scripted and unscripted), and we know it’s good.
We were all fired up to hear Shulkind at CineGear and then later presenting with Cochrane at the Creative Storage Conference to get their insight into the progress that has been made with the technology and when we’ll be able to see VR as “regular fare.”
Before this tag team took the stage at Creative Storage, Tom Coughlin gave his annual overview on the state of storage in the M&E field.
With 4K content consuming about 6,880GB/h and 8K gulping roughly 91,729GB/h, we’re starting to talk some serious storage capacity.
360º and 180º immersive video and AR/VR must be even more data-hungry.
After all, darn near everyone is ready to immerse themselves in great content.
Taking a conservative approach to forecasting VR storage needs, Coughlin projected that stereo 360º video would require about 115 GB/s of data or roughly 414 TB/h due to its 16,000 x 16,000-pixel resolution, 10+ bits per channel and a frame rate of up to 120fps.
Both Andrews thought these requirements were “on the modest side;” and they should know, they’re busy doing projects around the globe.
For almost two decades, Shulkind has been at the leading edge of where technology meets entertainment. He’s become the “go-to” expert on the future of entertainment; bridging immersive content, advertising, cinematic quality, gaming and mixed reality.
He is a rainmaker for both Hollywood and tech giants making Hollywood-style steps.
He’s done projects all over the globe and even works with foreign governments to help foster local production economies but seems most proud of his advisory work for SpaceCraft VR, a NASA-sponsored interactive infrastructure for space exploration.
Cochrane’s pedigree is just as impressive, including running digital and interactive projects for over six years at Guillermo del Toro’s Mirada, and now managing projects as a consultant and independent creative. Recently, he directed five CG-animated 360º films for the Vegas: Alter Your Reality experience and has created immersive content for USA Today, Disney, Intel and Visa, with multiple projects currently in production.
Before the session, we sat down with Cochrane and Shulkind to get an update on the progress that has been made with VR over the past few years.
Shulkind said the immersive content business is the latest example of how storytellers have always used the best tools available to engage and thrill the biggest audiences in the most immersive way possible.
From stories painted around a cave, to amphitheaters and coliseums, to 19th century panoramic paintings and haunted houses, to Cinemascope and Cinerama, to surround sound and gaming chairs, entertainment has always striven to be engaging and immersive.
Cochrane agreed that the application and purpose for placing a person directly in the movie or experience is only the beginning, and the future of this medium is when users can interact with the content.
“We’ve only just begun to tap the potential of immersive media. From 360º and 180º video to realtime experiences, there’s a growing library of interesting, innovative content that people really want to experience,” he added.
At the beginning of their presentation, Cochrane told the audience that VR is a massively exciting new medium that has ignited the imaginations of technicians, creatives and businesses.
“We have only scratched the surface as to what can be done with and how people can experience and interact with virtual environments,” he noted. “But what gets lost in the excitement is that it isn’t going to be an instant success. We still have a long way to go and obstacles to overcome.”
Both filmmakers noted that the challenge is being able to efficiently and effectively handle all of the data required to create this content.
Shulkind explained that 1080p per eye can be readily handled within traditional production pipelines. But when you scale up to 3D 4K VR, the requirements grow exponentially. “At the moment, the weakest link is not the capture but the display devices,” he stated. “Outside of a couple of early devices that still have very low adoption (like StarVR 5K and Pimax 8K), every headset is still below the 4K display threshold.
“Even if you wanted to deliver in 4K, you couldn’t see it,” he noted. “The current displays have such low pixel density that when viewed through the cheap plastic magnifiers in the headsets, the image is dramatically softer than what we shot. Newer displays will improve upon this quickly; but in the meantime, you have to take these factors into consideration when selecting your pipeline and how you spend money.”
“4K is not twice 2K, it is four times 2K!” quipped Cochrane. “If we want to match the abilities of our eyes, we’ll have to reach 64k resolution, at 72 frames per second or higher with at least 11bits per channel, per eye; so, we’ve got quite a way left to go.
“The production, distribution and viewing issues all have to be addressed with a project, and it isn’t simple,” he added.
Shulkind stated that VR cameras and rigs are still in their evolutionary stage with new and better solutions shown every year.
The one thing they all have in common is that they capture a lot of data – usually on SD cards – which needs to quickly and reliably copied to the computer and larger, more easily managed storage solutions.
Shulkind noted that rendering of the detailed images requires robust CPUs and GPUs because a detailed 4K image can require more than 5.6ms per frame.
“If rendering requires 700ms of every second, the system has to move more 2.988GB in 300ms or 9.9GB/s,” he explained.
“A pivotal point for VR filmmakers and producers, is our ability to move RAW and final data quickly and efficiently which means high performance GPUs and ultra-high-speed storage solutions,” Shulkind continued. “We always have someone on set with multiple laptops churning through rushes and focusing on what is being done so we can more quickly and easily complete the fine stitching after the edit.”
He noted that the fastest consumer SSD available today tops out at 3.5GB/s. As a result, VR filmmakers have either striped several SSDs to handle 4K VR or do what he and several friends have done, moved to the OWC ThunderBlade. The unit can transfer up to 2800MB/s and two units provide speeds of 3800MB/s to handle the high bandwidth content stream requirements.
Technology such as volumetric and light field capture, photogrammetry and spatialized audio will continue to generate extremely large, complex master files that will dwarf what is being done today in terms of both size and required functions
While VR has its optimists and pessimists, both Andrews feel significant progress has been made in the past few years. The new cameras, software, computing assets and storage are paving the way for exciting opportunities for people to immerse themselves in a wide range of content – documentary and entertainment.
“We’re working on new experiences and examining new ways the audience is responding,” Shulkind said. “At the end of the day, when we’ve done our best work, it’s the reaction/interaction of the viewer that keeps us going.”
“We’ve moved into an exciting new space,” Cochrane added. “We’re not just visual storytellers anymore, we are really enabling and encouraging people to be story-livers. VR is quickly becoming dynamic, personal, social, and even transformational.
“The technology, software, talent and user behaviors are still in the early stages,” he continued. “We haven’t seen the extent of what VR can deliver. We don’t want to oversell it, and we don’t want people to sell it short; it’s still in its infancy.”
More than just filmmakers; Shulkind, Cochrane and other active VR professionals are integrating the polish of immersive interactivity and data optimization, setting guidelines, focusing on standards and using AI to turn skeptics and non-believers into VR fans.
For the Creative Storage audience, they had one final message … the volume of data and content they are creating and working with is growing rapidly as they create more interesting, more exciting and more immersive video experiences.
Or, as Brody said, “You’re gonna need a bigger boat.”
# # #