We have been watching video in two dimensions for too long! Most 3D content people experience today is computer generated, such as VR games. Diverse use cases can benefit from real-world 3D video, such as music performances, training, family memories, and the metaverse. Why isn’t there more real-world 3D video? Because it has been difficult and expensive to create, stream, and playback real-world, camera-captured 3D video content.
I am an IEEE Fellow and inventor of over 200 patents. Basel has a PhD in electrical engineering with deep experience in VR/AR/3D video. While at Intel (as Chief Media Architect and Intel Fellow), I led an MPEG standards workgroup on 360/VR video. I found that 360/VR video’s limitation to 3 Degrees of Freedom (DoF) caused discomfort or even nausea in some viewers, because we experience the real world in 6 DoF (controlling both position + orientation), not in 3DoF (just orientation). I initiated an activity in MPEG to develop the MPEG Immersive Video (MIV) standard, which provides 6DoF. I became the lead editor of the MIV standard, and Basel was the lead editor of the test model.
While at Intel, we developed a MIV 3D video player for Intel GPUs and observed the greater engagement that 3D video provides to viewers. However there was no content available for the new MIV standard, and creation of 3D video content was a very difficult and expensive process. We realized that if 3D video were to become widely used, the creation and distribution processes needed to be simplified. We founded Vimmerse with a mission to greatly expand access to 3D video.
Businesses can build their own services using our APIs to upload captured content and prepare 3D video on our platform. Our platform is capture agnostic, meaning it can work with any video device suitable for 3D capture, such as iPhones or Microsoft Azure Kinect depth sensors. More than 60% of iPhone 12 and 13 models sold (Pro and Pro Max) have LiDAR depth sensors, which can be used for capturing 3D video content.
The Vimmerse platform prepares 3D content from the uploaded capture files. Our approach is built on top of industry standard video streaming protocols and codecs, so existing video streaming servers and hardware video decoders can be utilized. The content preparation platform creates two types of output bitstreams from the uploaded captures: bullet video and 3D video. Bullet video (named after the Matrix movie’s bullet effect) is a 2D video representation of the 3D video, following a predetermined navigation path selected by the content creator. 3D video gives viewers the ability to control navigation with 6 Degrees of Freedom (6DoF), where they can pan around or step into the scene. Bullet video may be streamed (HLS) or downloaded (MP4) for playback on any device. 3D video playback may be streamed (HLS) to the Vimmerse 3D video player.
Services may use the Vimmerse 3D video player app, or developers can use our player SDK inside their own apps. Viewers have the ability to control navigation using any viewer input method: device motion, mouse/keyboard, touch controls, head/gesture tracking. The player SDK renders views for the selected 6DoF position and orientation.
We haven’t published pricing yet, but our plan is to charge for our content preparation APIs based on usage (e.g. minutes of video processed and streamed) and player SDKs based on number of units.
The Vimmerse website https://vimmerse.net provides a no code way to test out our platform or view featured content. We invite the community to upload their own test content. Instructions for preparing content are available at https://blog.vimmerse.net/freeport-platform-usage-instructio.... Sign up for an account to upload content, or use the guest account (login: guest, password: Guest#123). The Vimmerse 3D player for Android is available in the Google Play Store at https://play.google.com/store/apps/details?id=net.vimmerse.p....
Please share your thoughts and experiences with 3D video, and your ideas for use cases that would benefit the most from 3D video. Are there any features we should add, or capture devices that you would like to have supported? Looking forward to getting your feedback.
by Andrey
Azimov