NVIDIA Omniverse Replicator And AI Avatars Aim To Help Us Pioneer The Metaverse
Long before Facebook rebranded itself as Meta, and even before its CEO introduced the idea of the "metaverse" to the masses, it's been an attractive concept to creators. After all, one perfect use-case for the "virtual conference" concept is as a collaborative creative space. Indeed, NVIDIA already markets exactly such a product, known as "Omniverse" and has been working towards it for years.
NVIDIA's Omniverse isn't exactly like the metaverse that Zuck talks about, though. Rather than being structured like a video game with a player character that you control, Omniverse is more akin to professional collaboration and screen-sharing, but without the screen. It's like cooperatively editing content, except the content is a 3D space chock-full of AI and machine-learning elements, described to the virtual environment by Pixar's Universal Scene Descriptor (USD) format.
This morning at GTC, NVIDIA announced two new technologies for the Omniverse: Avatar and Replicator. By combining its Riva speech recognition SDK, the Megatron 530B language model, the Merlin recommender framework, and NVIDIA's Metropolis computer vision framework, the Omniverse Avatar toolkit allows for the accelerated creation of seemingly-intelligent virtual assistants that can be interacted with through the Omniverse by touch or other input, or by using natural-language voice input.
Next up was the DRIVE Concierge AI platform. If you've ever wished you could have Alexa or Cortana in your car, well, that's the idea. Finally, Huang demonstrated Project Maxine, an avatar intended to be used alongside video conferencing applications. Maxine cleans up background noise in the remote caller's chat, and then translates her English speech into German, French, and Spanish with the same voice and intonation.
All of the demonstrations come complete with dynamically-animated 3D avatars, powered by NVIDIA Video2Face and Audio2Face, AI-driven facial animation and rendering packages. It's not hard to imagine how avatars like this, with a little more development, could quickly become a part of everyday life.
The other bit of kit, Omniverse Replicator, is more straightforward than Avatar. Replicator is what NVIDIA calls "a powerful synthetic data-generation engine." Essentially, it is a tool for using Omniverse-based "digital twins" of real-world spaces and objects to generate physically-simulated data that can be used to train AIs about the real world.
One of the biggest deciding factors in the quality of an AI is the size of its training set, and having the ability to train robots in a physically-simulated virtual space that very closely matches reality will allow researchers and engineers to make high-quality AIs with consistent performance much more quickly. Actually, NVIDIA says it is already using this technique for its autonomous vehicle neural networks, and that it "has saved months of development time."
Replicator currently comes in two forms, one for NVIDIA DRIVE Sim, intended for training autonomous vehicles, and the other for Isaac Sim, intended for training "manipulation robots". As NVIDIA notes, besides allowing AI to learn about the environment in a simulated space, the virtual nature of the "digital twin" allows AIs to test and learn about "rare or dangerous" conditions that can't be tested in reality, like driving partially out of your lane.
NVIDIA hasn't given any timeline for broader availability of Avatar, but the company says that Omniverse Replicator should be available to developers "to build domain-specific data-generation engines" next year.
