How artists and performers can blend virtual production and motion capture.
By PETE CARVILL, NEXR Technologies
I once wrote that every band eventually becomes their own tribute act, a touring collection of their own greatest hits. Even though the members are still there, they begin over time to lose their relevance, to move back from the cutting edge of music. Their audience changes, too, even if those attending their concerts are the same people. Those who come are no longer young, or the young, even if their names have not changed.
As the older acts eventually leave the stage, the traditional setup of a concert — lead singer, guitarist, drummer, bass, backing singers — also shifts. It would have been, in the early days of rock and roll, anathema to consider lights and pyrotechnics as part of the experience.
And, yet, today it would be strange to go to a concert and NOT have some kind of video production integrated into the performance. Even the most-staid of performers have added flair and fireworks and video to their shows, knowing that modern audiences demand more bang for their buck.
And, yet, there is a technology that has not yet reached mainstream adoption. From our studios in Berlin, we have been working for the last YEAR on assembling a product that will move live music to the next level.
The coronavirus pandemic has lit a fire under this technology by killing off live music for eighteen months, maybe longer. And until we know whether it is if audiences return or when, it may be that musicians decide to offer performances and concerts virtually that can be broadcast over the internet.
All of this has been combined into what we call the NeXR Show, which combines motion capture, computer graphics, and live performance. The crux of the product is to offer fully-immersive and real-time custom-made virtual stages and digital assets for artists and performers.
When we set out to create the NeXR Show, we wanted to put together a product that could by used by any artist or performer. The events of the last year have given us the opportunity to experiment and fine tune it.
The NeXR Show is a set of digital tools to help artists create better, more-immersive experiences for their audiences. While we can never replicate a real-world show, a virtual production still offers audiences the chance to watch an artist, buy merchandise, and hang out with friends. In putting this together, we wanted to maximise the experience of a virtual show. This technology allows that to happen.
The process begins with the team at OnPoint working with the client to decide upon, and design, the stage for the performance and the avatar for the client. We have a number of custom-made offers, including futuristic science fiction stages, industrial club-like sets, and real-world based classical stages.
Once the design for the stage has been set, we then work on building an avatar (digitised representation of the performer). These avatars, animated by using motion capture, can take any form such as a fantastical creature, a robot, or a rendering of the performer themselves.
Once the previsualization work has been done, it is time to record at OnPoint Studios. Located in the north of Berlin, the studio has thirty-six Optitrack Prime 13 cameras, a full green screen setup, and can handle previsualization in Motive, Motionbuilder, Unreal Engine, and Unity. It also has a live remote mocap broadcasting setup and can handle up to five performers at the same time. And with a green screen setup, there is no limit of the number of people that can be streamed.
After the performance is recorded using green screen and data capture technology, everything is integrated into the custom-made environment before broadcast. This, itself, can happen on any platform, including, YouTube and Twitch. It can also be streamed live over a custom website.
The innovation does not stop there. One feature we have implemented into our product line is that of the ‘reactive stage’ – which can get audio and visual feedback through chat and emojis in real time through our innovative ‘virtual audience’ system. As the performance is underway, the ‘reactive stage’ has visual effects that can be triggered in real-time by the fans. These may include firework displays and lighting effects within the generated environment.
There are myriad benefits for performers:
Firstly, this technology allows events to be streamed simultaneously to across different platforms to all four corners of the globe. This means that an artist putting on a show at the Berlin studio can be performing to audience members in Singapore, Sao Paolo, and San Francisco. This, in turn, would reduce the demand for tours, given that one show could be done for an unlimited number of places.
Secondly, the artist-friendly production environment at NeXR Technologies means that performers can put together a show that aligns with their own unique vision and performance, while elevating that show through customized special effects. And, lastly, there are new, multiple possibilities for monetization through creating gamified event features.
There are great possibilities for putting concerts on using a product such as the NeXR Show. For more information, why not come and visit our site and see what we can do for you?
