From a certain technical perspective, there has been little change in how we consume media for more than 100 years. Traditional media consumption, whether by 2D or 3D video, is largely passive. It does not give you an option to move around or interact with the scene. Here is how volumetric video is becoming a key content for VR.
We have had 130 years of filmmaking — yet on the surface of it — the art and science of making films and animations have undergone tremendous evolution.
The rapidly growing virtual reality (VR) and augmented reality (AR) scene are set to revolutionize how we consume media.
With the help of mixed reality, we no longer are just passive spectators. We can virtually walk through scenes and interact with the content in a realistic and immersive fashion.
Many of the current virtual reality experiences still offer limited degrees of freedom.
For example, a traditional 360 video rendered in virtual reality is still viewable only from a single perspective. The camera man’s position takes you through the experience. It does not give you the freedom to fully explore the environment by yourself.
How to bring more immersive and realistic VR content
The solution to creating greater immersion in VR is volumetric video. Volumetric capture technology enables images of real people and objects to be simultaneously captured with many cameras at the same time.
The capturing generates hyper-realistic 3D models that move in a natural and organic way. It allows us to view objects at any moment and angle, with six degrees of freedom.
The setup for volumetric capture requires a bulky array of cameras and lighting that can depth-sense 360-degree capture.
The setup also needs a processing unit that stitches all the images together. The rendering can take many hours even with the highest-end graphics cards.
However, the painful setting-up process and processing is rewarded by the ultra-high-quality 3D models. These high-quality models are becoming the core of virtual reality.
The use of volumetric video capture technology has exploded with the rise of virtual reality.
The need for greater immersion and realistic characters in VR has seen many content producers increasingly relying on the volumetric format.
The popularity of the volumetric video has seen many startups emerge that now provide volumetric video capture services. All of them have different approaches and setups. The industry is growing at such a pace that it is expected to be worth $2.78 billion by 2023 according to market research.
Volumetric Capture
Despite its growing popularity — volumetric capture is still experiencing certain bottlenecks in its deployment. Cost and portability are the main problems limiting adoption. However, we can already see that hardware and software improvements are coming that are likely to lower the costs and complexity while increasing the ease of use.
As VR slowly seeps into the mainstream, the technology will become the raison d’être.
VR and AR content producers will find it indispensable in meeting the exacting demands for immersion and realism.
How Volumetric Video Capture Works
Volumetric video capture works on the same principle as photogrammetry only that it records objects in motion.
The resolutions of the cameras used in volumetric capture vary but they capture enormous amounts of data that can take an eternity to process.
Multiple cameras capture the scene from different viewpoints in specially designed volumetric capture studios. It generates a true 3D representation of the space. There are different volumetric video capture systems in place for example Intel Capture or Microsoft Mixed Reality Studios.
The studio is wrapped in a green screen and a massive camera rig. It captures multiple viewpoints of the person standing or performing in the middle.
Once the movements of the person have been recorded you can clean out all of the unwanted stuff.
A powerful computer software is interpolating and triangulating the overlapping visual information. The computer software generates a so-called point cloud. The point cloud consists of various points in space that contain both geometrical and depth information.
The data generated by the computer software is still largely a “noisy” array or representation of the original.
Therefore, the data has to be filtered and polished by connecting the fine points and generating more familiar 3D mesh. The data will then have the appropriate textures from the volumetric capture applied to it.
The point cloud filtering process isn’t fully automated in some volumetric capture setups and that can involve manual cleaning and correction.
Pixels that contains depth and color are the final product.
The content you have made might be for VR but actually it is not limited to it. You can reuse your creation on any platform including AR or on the Web.
The virtual assets created from volumetric capture simplify virtual content creation. People can use these to create new stories or shots without relying on a crew or being physically present at the set.
Challenges with Volumetric Video Capture
While the opportunities for volumetric videos are limitless thanks to the growth in the adoption of virtual reality, the technology still poses a practical change-problem.
The change has mainly to do with data. The cameras in the systems capture vast amounts of data. As a result, volumetric content creators have to grapple with some near debilitating constraints.
Often these constraints can limit the potential and scope of use of volumetric video technology on different devices.
We hope that this challenge will be addressed through hardware and software improvements and innovations — soon.
Content creators working with volumetric capture technology are also likely to face workflow challenges.
Creators have to work with lots of post-production software to stitch things together and simplify the workflow.
Most of these stitch-together-options are proprietary software.
Apart from the software tools challenge, the workflow of the volumetric video will also face a “people and talent” challenge. You have to put a collaborating team together with diverse and specialized skill sets to see the project through.
New Approaches by New Players
Several startups are investing in volumetric video and coming with newer and simpler approaches.
EF EVE dot com offers volumetric capture setup through the use of multiple Microsoft Azure Kinect and other popular 3D cameras.
EF EVE is currently the only software on the market that can automatically calibrate up to 10 sensors.
The EF EVE system allows you to record, edit, export, and publish volumetric videos via its Volcapp and Creator software. The product has created a linear workflow that enables creators to accomplish all the tasks under one roof.
These types of volumetric video software companies are bringing more affordable and portable setups that are now democratizing volumetric video capture technology. With the efforts of EF EVE and other software companies — it will make this type of volumetric video production accessible to hundreds of new companies and institutions — and may to you, too!
Image Credit: David Attenborough, “Hold the World.”
The post Volumetric Video is Becoming a Key Content for VR appeared first on ReadWrite.
