20 June 2018 | Salt Lake City, USA

Realistic Avatars for the Virtual Zoo

On the basis of a few photos alone, a new technique creates realistic avatars of animals that look and move like real animals

Filmmakers and developers of computer games will have a new way of animating animals in the future. A team led by researchers at the Max Planck Institute for Intelligent Systems in Tübingen, Germany, has developed a technique that uses photographs alone to create lifelike 3D models of almost all quadrupeds. These avatars can be animated to realistically imitate the movements of animals. But the simple method of bringing animals to life on the computer is not only interesting for the entertainment industry. Many people have lost a beloved family pet. Now this technology can bring them back to “life” as a virtual 3D avatar. It could also benefit biologists in species protection and help to make children in particular aware of the importance of biodiversity.

“Populating the virtual world with animals used to be a lot of manual work“, says Michael J. Black, director of the Perceiving Systems department at the MPI for Intelligent Systems. Today, animators have to laboriously design 3D animals using graphics software. While there are technologies to create realistic 3D avatars of humans, no such technology is available to automatically capture the shape of wild animals. “You can’t bring a tiger into the lab to be scanned and you can’t take expensive scanners out into nature,” says Black.

The key is to start with a rough 3D shape

The team, whose technique creates avatars of animals including their fur or skin with reasonable effort, includes scientists from the CNR Institute for Applied Mathematics and Information Technology (IMATI) in Milan and the University of California in Berkeley as well as Max Planck researchers. "Ultimately, 3D animal capture is about reconstructing the 3D structure from 2D images of moving objects captured by various unknown and moving cameras," explains Silvia Zuffi, the lead author on the project from IMATI.

In their new method, the researchers only need a few photos of an animal in different views to calculate realistic models that can then be animated naturally. Unlike other approaches, the images from which the avatars are generatedcan also come from different cameras. The computer program delivers such convincing results because it doesn't start from scratch when it creates a model. “The key is to start with a rough 3D shape model that can explain the shapes of many animals and then refine this based on the observed images,” says Michael J. Black.

Helping endangered animals in the wild

In their current work the researchers demonstrate that their method works for as different animals as a tiger, a bear, a horse or a rhino. "Our algorithm generates realistic models of animals for which no previous 3D models exist," says Silvia Zuffi. For example, the researchers were able to reconstruct the 3D shape of the extinct Tasmanian tiger from low-quality film footage shot in the 1930’s. The researchers now want to make the method fully automated so that it can be run on thousands of images of animals in the wild.

Fully automated 3D animal shape capture would not only facilitate the work of game developers and animators but also supports animal conservation. Given photographs of animals captured by camera traps, the method could determine whether their shape, i.e. their weight, has changed. Species conservationists would thus receive quick and reliable statements as to whether endangered animals are doing well.

PH / MJB

The work appears in the annual IEEE Conference on Computer Vision and Pattern Recognition (CVPR) held in Salt Lake City. CVPR is highly competitive and is regarded as the world´s top conference in the field.

The work appears as:
Silvia Zuffi, Angjoo Kanazawa, Michael J. Black, “Lions and Tigers and Bears: Capturing Non-Rigid, 3D, Articulated Shape from Images,” Proc. CVPR, 2018.

For more about the research see: smalr

The 3D models are freely available for download from: sketchfab

More Information