Welcome to the age of neurographics.

Nvidia is aligning itself with the metaverse and positioning its Omniverse as the foundation for building a metaverse. It has introduced new AI tools and greatly expanded the partners and participants in its Omniverse, and is offering various open source software tools for the Omniverse, and they are calling it all neurographics.

Neurographics will be the essential pillar of the emerging metaverse—the next evolution of the internet according to Jensen Huang. Two decades ago, the internet was about hyperlinked web pages he observed; a decade ago Web 20 and cloud services. Now web 30 is here and the metaverse is a network of virtual interconnected worlds. The metaverse will extend 2D web pages into 3D spaces and worlds. Hyperlinking will evolve into hyper jumping between 3D worlds, forecasted Huang.

Metaverse apps are already here; fashion designers and furniture catalogs are offering virtual 3D products you can try. Telcos are creating digital twins of their radio networks for tower location optimization. Manufacturers are creating digital twins of warehouses and factories and Nvidia is building a digital twin of the earth to predict the client decades into the future. The metaverse will grow organically as the internet did continuously and simultaneously across all industries but exponentially because of computing compounding network effects. The metaverse is a computing platform that requires a new programming model, a new computing architecture, and new standards. HTML is the standard 2D language of the web. USD is likely the best language for the metaverse. Omniverse is a USD platform, a tool kit for building metaverses. Robotics is one of the most exciting aspects of the Omniverse, said Huang. Robotics is the next wave of AI and Omniverse is essential to design, train, and operate robots. One of the most widely used robots will be a digital human or avatar. Avatars will populate virtual worlds to help create and build things, make recommendations, provide service and support.

Sanja Fidler, Nvidia’s VP of AI Research, said Nvidia presented 16 papers at Siggraph on AI neurographics and the mathematical foundation on graphics from 56 researchers across 20 Univ and two were honored as best papers award.

Nvidia released new research and tools to apply power of neurographics toward the creation and animation of virtual worlds and are introducing the Kaolin Wisp research-oriented library and framework for neural fields. Nvidia hopes it will enable and inspire fast-paced progress in informational neurographics.

Rev Lebaredian, VP Omniverse & Simulation technology, said the metaverse is a 3D spatial overlay of the web linking the digital world to our physical world.

In this new iteration of the internet, said Lebaredian, websites will become interconnected 3D spaces akin to the world we live in every day. Many of the virtual worlds will be representative of the real world. XR devices will act as portals to the virtual world while AIs will portal out to our world. No one can predict how large the metaverse will be but today we can lay the foundation he added. The 3D internet or metaverse will require a standard way to describe all the things in the 3D worlds and Nvidia thinks USD is that standard scene description. USD is far more than a static file format; it is a 3D composition engine with APIs for composing, editing, queering, rendering, collaborating, and simulating virtual worlds. The Omniverse framework is highly extensible allowing for custom schemes for workloads, said Lebaredian.

HTML 1.0, introduced in 1993, only describes simplistic web pages. HTML 5 describes rich dynamic media applications. USD is evolving along a similar path from its origins in M&E as a static description of virtual worlds to a system for dynamic, procedural, and real time worlds. Most recently, Nvidia contributed a material definition language (MDL) schema to represent physically accurate materials and parameters and upgraded USC from python 2 to python 3 bindings and with Apple and Pixar support for rigid body synthesis. Nvidia sees USD becoming a compliment to existing 3D standards such as glTF. Nvidia is developing an open-source USD file format plugin to glTF. With it, developers can leverage layering, compositions, and non-destructive editing capabilities of used with their existing glTF asset library. Next, Nvidia wants to make USD performance for real time large scale virtual worlds and industrial digital twins. Nvidia has developed a just-in-time pixel rendering for optimizer of digital twin simulation achieving, says Nvidia, a 10-times speed up in load time and a 10-times speed in rendering.

Steven Parker, Vice President, Professional Graphics, spoke about some of the developments Nvidia made in MDL and associates apps for Omniverse.

MDL is 10 years old and Nvidia made it open source at Siggraph 2018. Now they are evolving it so developers can pick and choose what elements of the materials they want to include, eliminating some to improve performance.

The new program introduces the distillation of MDL materials to a fixed target model and bakes material sub-expressions to a texture which greatly speeds up rendering.

OpenVDB is an Academy Award-winning, industry standard library for sparse dynamic volumes. It is used throughout the visual effects industry for simulation and rendering of water, fire, smoke, clouds, and a host of other effects that rely on sparse volume data. The library includes a hierarchical, dynamic data structure and a suite of tools for the efficient storage and manipulation of sparse volumetric data discretized on three-dimensional grids.

Nvidia announced its NeuralVDB that introduces neural representations of values and the underlining tree structure, which drastically reduces the memory requirement allowing interaction with extremely large and accurate complex volumetric datasets in real time. It also includes NanoVDB Nvidia introduced in 2021, offering up to 60× compression to speed up training by 2× by using the weights of the previous frame to be used by the subsequent frames and enables temporal coherency (smooth encoding) reducing the need for post-production effects like motion blur and has extended its application to other fields like robotics, heath care, and climatology.

And Nvidia introduced Omnilive workflows that introduces non-destructive live workflows at increased speed and performance to users interconnecting and collaborating between different applications.

Shared virtual worlds will enable the next wave of AI and will profoundly impact today’s industries. Omniverse, says Parker, is where all Nvidia’s technology comes together and enables industry to realize the metaverse opportunity.

Nvidia claims they have a 150,000-user base of Omniverse.

One of the features of the metaverse will be avatars, and Omniverse has that covered too. For avatars to be believable, and not fall into the uncanny valley, they have to have facial movement that matches their speaking, expressions, and emotions.

Nvidia’s latest AI-assisted software is Audio2face that can create facial animation directly from audio. A developer can use a recording or animate live. They just have to record a voice audio track, input into the app, and see their 3D face come alive. Facial animations can even be generated live using a microphone.

Huang said in his wrap up remarks that there will be billions of avatars and announced Nvidia Ace that runs in the cloud.

Nvidia’s Omniverse avatar cloud engine (ACE) is a suite of cloud-native AI models and services to simplify building and customizing lifelike virtual assistants and digital humans.

ACE is built on Nvidia’s unified compute framework, which provides access to software tools and APIs for realistic, interactive avatars. These include Nvidia Riva for developing speech AI applications, Metropolis for computer vision and intelligent video analytics, Merlin for high-performing recommender systems, NeMo Megatron for large language models with natural language understanding, and of course Omniverse for AI-enabled animation.

All in all, it was a very impressive (as usual) presentation that ended with a fly through of Nvidia’s building, which was really impressive.