When we sit down to solve a puzzle, there is always one thing we take for granted: the picture on the box. Without that benchmark, we’d be tearing our hair out, trying and failing to reconstruct a bunch of various parts.
This is exactly what happened in the ancient Roman city of Pompeii, where more than 10,000 fragmented pieces of 2,000-year-old frescoes have been lying around for decades, waiting for someone to solve the puzzle. Today, a team of scientists led by the Italian Institute of Technology based in Venice may have found a solution: train a robot to do it.
Called “Reconstructing the Past: Artificial Intelligence and Robotics Meet Cultural Heritage” (RePAIR), the project is funded by a grant of 3.5 million euros (just under $ 4 million) from a European Commission which supports high-risk projects aimed at »radically new future technologies. “The project will be developed in two phases: first, an algorithm will digitally reconstruct the puzzle, then a pair of robotic hands will piece together the puzzle (i.e. the frescoes). This is the first time that the AI will be used as an archaeological tool on such a large scale, and the first time robotic hands will be loaded with so many pieces. If the project works, scientists hope to deploy the technology to other cultural heritage sites across the world. world, like the historic churches in Italy, or even the ancient city of Palmyra in war-torn Syria.
A key part of the project will be teaching the algorithm how to study like an archaeologist and think like a puzzle master. The puzzle-solving AI was developed in collaboration with a team from Ben-Gurion University in the Negev, Israel, and it functions as an infinitely more complex version of the popular “Find the Pair” memory game. The computer software compares all the fragments in pairs and assesses their degree of similarity based on the shape of the pieces, how they fit together and the correspondence of the illustrations on the fragments. Normally, this process can be done manually (using a computer), but the team is now teaching the algorithm to compare parts on its own.
Thanks to a team of archaeologists from the University of Lausanne, who have tried to solve the puzzle in the past, they already have about ten reconstructed clusters (about 10 pieces each) which they feed into the algorithm. If the computer can reconstruct these items, it will know that the system can be deployed on a larger scale.
The robot will be deployed in Pompeii next summer, but for now, scientists are working on several projects in parallel. While one team builds the algorithm, another 3D scans a large sample of fragments so that they can be placed in the database (once the robot is fully operational, it will scan them on its own). And another team is working on the physical infrastructure and the robotic hands that will eventually pick up the pieces and rebuild the frescoes. “One day, you will take all the pieces, put them in a room, lock the door, come back after a few days, and you will find the fresco completely reassembled”, explains Marcello Pelillo, professor of computer science and artificial intelligence. at the University of Venice. (Although he admits things probably won’t turn out so well.)
The fragments come from two separate rooms (including the ceilings) of a building called the House of Painters at Work (named because the artists were painting when Vesuvius erupted in AD 79). the whole range of small fragments to palm-sized coins, most of them damaged, and many of them missing. Scientists are working with archaeologists and art historians, who will reduce the pieces to a first thousand that they believe belong to the same group, or at least on the same wall.
Within a year or two, Pelillo says they should have a preliminary image of the frescoes, but a full, final image will take longer. When the robotic platform is finished, the plan is for the robot to do everything at the same time: “The robot will do the sweep itself, then after doing the sweep it will solve the puzzle, then after solving the puzzle, he will reassemble it. , Pelillo said. (The installation will look like a bridge, with two robotic arms hanging from a metal frame.)
At the end of the project, the frescoes will be on display in the Pompeii Archaeological Park for visitors to discover, but they will also inform future research. Ariana Traviglia, director of the IIT Center for Cultural Heritage and Technology, says we can learn a lot from the reconstructed frescoes and the patterns in them. “The frescoes weren’t like wallpaper, they weren’t all the same,” she says. “In each of them, the artist put something new according to the owner’s taste.” Already, Traviglia can say that the family who lived there were rich: “the stucco all around the ceiling is of very high quality.
If the project is successful, technology can save restaurateurs valuable time that can be spent on restoration. And although this project currently focuses on relatively flat fragments, Traviglia wants to take it one step further and then try more complex shapes, like ancient statues, wrecks and amphorae.
Eventually, the team could put all of this into practice in other cultural heritage sites, such as the many ancient churches that collapsed in earthquakes in central Italy four years ago. “All over the world, we have so many pieces of broken frescoes and broken objects from our past, and they are so small that we don’t have the time or the staff to put them together,” she says. “It’s really going to change the things we can do.”