The current wave of AI may be traced back to a 2012 academic competition that evaluated how effectively algorithms could distinguish items in images.
Researchers discovered that sending hundreds of photos into an algorithm partly inspired by how neurons in the brain respond to information resulted in a substantial gain in accuracy that year. The breakthrough generated a wave of academic and commercial activity that is reshaping several businesses and sectors.
Now in the realms of both computer graphics and AI, a new approach that includes training the same sort of AI algorithm to transform 2D photos into a rich 3D perspective of a scene is generating buzz. The method has the potential to revolutionize video games, virtual reality, robotics, and self-driving vehicles. It may even help robots observe and reason about the world in a more intelligent—or at least humanlike—way, according to some researchers.
“It’s quite hot, and there’s a lot of buzz,” says Ken Goldberg, a robotics at the University of California, Berkeley, who is utilizing the technique to help AI-enhanced robots understand unknown forms. According to Goldberg, the technique has “hundreds of applications” in a variety of industries, including entertainment and architecture.
The new process, termed “neural rendering,” involves utilising a neural network to acquire and synthesise 3D data from a few 2D photos. It developed from the fusion of concepts circulating in computer graphics and AI, but attention peaked in April 2020 when researchers at UC Berkeley and Google demonstrated that a neural network could photorealistically capture a scene in 3D merely by examining multiple 2D photos of it.
This technique takes advantage of the way light travels through the air and uses it to compute the density and colour of points in three-dimensional space. This allows 2D photos to be converted into a lifelike 3D representation that can be viewed from any angle. Its core is a neural network similar to the one used in the 2012 image-recognition programme, which analyses pixels in a 2D picture. The new techniques turn 2D pixels into voxels, which are 3D equivalents. Videos of the technique, dubbed Neural Radiance Fields (NeRF) by the researchers, astounded the scientific world.
According to Dellaert, the method is a game-changer for everyone working in computer graphics. Normally, creating a complex, realistic 3D scene involves hours of tedious hand labour. The new approach allows amazing scenarios to be created in minutes from conventional images. It also introduces a brand-new method for creating and manipulating synthetic settings. “It’s pivotal and vital,” he adds, which is a lot to say for a work that’s just two years old.
The pace and range of ideas that have evolved since then, according to Dellaert, has been amazing. Others have utilised the concept to make moving selfies (or “nerfies”), which allow you to pan around a person’s head based on a few stills; 3D avatars from a single headshot; and a technique to automatically relight situations differently.
The project has received a surprising amount of interest in the industry. The blooming of research and development, according to Ben Mildenhall, one of the researchers behind NeRF who is now at Google, is “a steady tidal wave.”
Nvidia, a company that develops computer processors for both AI and video games, has released studies that employ NeRF to build 3D pictures from photo collections, create more realistic textures in animation, and point to video game advancements. Facebook (now Meta) has created a NeRF-like method that might be used to flesh out scenes in Mark Zuckerberg’s much-hyped Metaverse. The latest study is “fascinating,” according to Yann LeCun, chief AI scientist at Meta and a pioneer of the method that shook things up in 2012.
NeRF might be particularly beneficial for machines that work in the actual world. Goldberg, one of the world’s foremost specialists in robotic grasping, and colleagues utilised NeRF to teach robots to understand transparent objects, which are generally difficult to understand due to the way they reflect light, by allowing them to deduce the form of an item from a video clip.
Self-driving car manufacturers are also exploring applications for the concept. Andrej Karpathy, Tesla’s head of AI, claimed the firm was using the technology to build 3D sceneries to train its self-driving algorithms to detect and react to more on-road events during a presentation in August.
The concepts underpinning NeRF might be crucial for AI as a whole. This is due to the fact that comprehending the physical features of the real world is essential for making sense of it.
“These computer graphics approaches are having a significant influence on AI,” says Josh Tenenbaum, an MIT professor who researches the computational foundations underlie human learning and reasoning.