Nvidia's GauGAN -- one of the company's technology showcases at the graphics-research-focused Siggraph show this week -- is one of the coolest uses for AI in graphics that I've seen in a while. Essentially, it lets you paint with smart fills and brushes that are based on real-world images.
It starts with an image that it deconstructs into a segmentation map -- a breakdown of the image by object types such as sky, grass, mountains and clouds -- or you can start from scratch and scribble in the various regions yourself. Then it uses the GAN-based content to fill in each area with the appropriate type of artificially-generated images. You can also apply generated styles based off real paintings.
It has the potential to be a huge timesaver for all sorts of designers.
GauGAN, which debuted in March and has been publicly available on Nvidia's web site for about a month, is just the latest in a series of
projects to showcase the company's Generalized Adversarial Network-focused AI research, which include its StyleGAN-based deepfake generator and older face-aware fill-in.
Other tech the company is showing off at Siggraph includes an AR headset that incorporates foveated rendering, a technique which prioritizes rendering quality for the parts of a scene you're looking at to save processing power. It's usually used for VR, which is less processing-intensive than AR, because it doesn't have to worry about overlaying on the real world.
And because the Apollo 11 anniversary is such a hot topic that people are creating butter sculptures of the crew, Nvidia's highlighting its Omniverse platform to put attendees virtually on the moon using its AI pose estimation, a combination of motion capture, AI and its RTX ray-tracing technology.