NVIDIA showcased a wave of cutting-edge artificial intelligence research that will allow developers and artists to bring their ideas to life, whether they're still or moving images, 2D or 3D, hyperrealistic or fantastical.
About XNUMX NVIDIA Research papers advancing generative AI and neural graphs — including collaborations with more than a dozen universities in the United States, Europe, and Israel — will be showcased at SIGGRAPH 2023, the premier computer graphics conference, to be held August 6-10 in Los Angeles.
NVIDIA RESEARCH research publications can be accessed here.
Articles include generative AI models that turn text into custom images; reverse rendering tools that turn still images into 3D objects; neural physics models that use AI to simulate complex 3D elements with stunning realism; and neural rendering models that unlock new capabilities to generate AI-powered real-time visual detail. Here are some examples.
Rethinking the text-to-image model
Generative AI models that turn text into images are powerful tools for creating concept art or storyboards for movies, video games, and 3D virtual worlds. Text-to-image AI tools can turn a prompt like "kids toys" into near-endless visuals a creator can use as inspiration — generating images of stuffed animals, blocks, or puzzles .
However, artists may have a particular subject in mind. A creative director for a toy brand, for example, might plan an ad campaign around a new teddy bear and want to visualize the toy in different situations, like a tea party with a teddy bear. To allow for this level of specificity, researchers at Tel Aviv University and NVIDIA have two SIGGRAPH papers that allow users to provide sample images that the model learns quickly.
An article describes a technique that requires a single sample image to personalize its output, speeding up the personalization process from minutes to about 11 seconds on a single NVIDIA A100 Tensor Core GPU, more than 60 times faster than previous personalization approaches.
A second article presents a very compact model called drip, which takes a handful of concept images to allow users to combine multiple custom elements – like a specific teddy bear and teapot – in one AI-generated visual:
Bringing 2D to life
Once a creator comes up with “concept art” for a virtual world, the next step is to render it and fill it with 3D objects and characters. NVIDIA Research invents artificial intelligence techniques to speed up this time-consuming process by automatically transforming 2D images and videos into 3D representations that creators can import into graphics applications for further editing.
A third paper created with researchers at the University of California, San Diego unveils technology capable of generating and render a photorealistic 3D model of the head and shoulders based on a single 2D portrait – a major breakthrough that makes 3D avatar creation and 3D video conferencing accessible with AI. The method executes in real time on a consumer desktop computer and can generate photorealistic or stylized 3D telepresence using only conventional webcams or smartphone cameras.
A fourth project, resulting from a collaboration with Stanford University, brings realistic movement to 3D characters. Researchers have created an artificial intelligence system capable of learning a range of tennis techniques from 2D video recordings of real tennis matches and to apply this movement to 3D characters. Simulated tennis players can hit the ball with precision to target positions on a virtual court and even play extended rallies with other characters.
Beyond the tennis test, this SIGGRAPH article addresses the difficult challenge of producing 3D characters capable of performing various skills with realistic movements – without the use of expensive motion capture data.
The difficult physics of hair, simplified by AI
Once a 3D character is generated, artists can overlay realistic details such as hair – a complex and computationally expensive challenge for animators.
Humans have an average of 100.000 hairs on their heads (I'm obviously lowering that average), each reacting dynamically to an individual's movement and the environment around them. Traditionally, creators use physical formulas or dedicated plugins to calculate hair movement, simplifying movements based on available resources. That's why virtual characters in a big-budget movie sport much more detailed hair than real-time video game avatars.
A fifth article presents a method capable of simulate tens of thousands of hairs in high resolution and in real time using neural physics, an AI technique that teaches a neural network to predict how an object would move in the real world.
Neural rendering for cinematic quality in real time
Once an environment is filled with animated 3D objects and characters, real-time rendering simulates the physics of light reflected through the virtual scene. Recent research from NVIDIA shows how AI models for textures, materials, and volumes can deliver real-time, cinematic-quality photorealistic visuals for video games.
In a sixth SIGGRAPH article, NVIDIA will present neural texture compression that delivers up to 16x more texture detail without using additional GPU memory. Neural Texture Compression can dramatically increase the realism of 3D scenes, as seen in the image below, which shows how Neural Compressed Textures (right) capture sharper detail than previous formats, where text remains blurry (In the center).
What you have just read are just a few examples that will be presented at SIGGRAPH. All of NVIDIA's research publications are available at this address: https://research.nvidia.com/publications
It includes studies on the simulation of fluids, the physics of tissues, or surface effects. SIGGRAPH 2023 will certainly be under the sign of AI.