The country is facing the biggest maintenance in history. Hundreds of bridges, viaducts and waterworks are scheduled for renewal in the coming decades. This maintenance is imminent to keep the society running and safe. 

“It's a massive operation that the Netherlands hasn't been able to keep up with for years. There simply aren't enough people to inspect all those bridges and plan maintenance, so we're lagging behind,” says the STRUCTURE project leader Egor Bondarev. He wants to develop autonomous drones with multi-modal sensors that use AI to independently inspect bridges and other infrastructure. A welcome solution to the time-consuming process Rijkswaterstaat currently faces.

One week

Manually inspecting bridges is not only time-consuming and slow, but also expensive, as labor is costly in the Netherlands. There's a significant shortage of engineers, something that can't be solved overnight. “Currently, one maintenance team can inspect one bridge per week. With thousands of projects, we'll never be able to inspect all the inventory in time,” Bondarev explains. He is also the director of the AI for Multi-modal Sensing (or AIMS) lab. 

Researchers in this department develop AI models for systems equipped with sensors of various modalities. This includes a combination of RGB, thermal, depth, LiDAR, acoustic, sonar, and radar sensor data. When these multimodal sensors are combined into a sensor suite, they often offer capabilities comparable to the human “five sense system” for complete perception of what’s around you.

Bridges have had to be temporarily closed before because inspectors suddenly noticed dangerous defects. For example, the Houtribbrug at the beginning of this year and the Merwedebrug in 2016. A thorough investigation has shown that it's a miracle the latter didn't collapse at the time. Dangerous situations could become more common if we fall further behind maintenance schedule in the coming years.

Senses

The inspection drones will use a unique approach during their inspection of the concrete and steel structures. They will combine different sensors that operate a bit like senses to find defects at the surface and subsurface levels: infrared, X-ray, radar, RGB (color sensor), and acoustic data. “For example, no one has ever used X-rays to examine bridges before,” Bondarev beams.

The sensors will be mounted on the Avular drones. They come together digitally in a sensor suite, a kind of platform on which they can collaborate. After the inspection routine, all types of data are ‘merged’ to create a single picture that clarifies the condition of the bridge.

No data

Within STRUCTURE, Bondrev's AIMS lab will focus primarily on AI analysis and merging the results of the various measurement types. The AIMS team, including the post-doc Erkut Akdag and two PhD students, will research and develop AI models that can detect defects both on and in the structures. Examples of defects are cracks, the detachment of the top layer of the concrete, deformations, fatigue of the concrete, et cetera.

The training of the AI will be quite challenging because there's essentially no good data for it to learn from. The domain transfer, self-created data and zero-shot learning techniques may come to the rescue. Zero-shot learning (ZSL) is a learning method in which an AI model is trained to recognize and categorize objects or concepts without having ever seen examples of such before. The researchers will, for example, generate an X-ray model which can highlight any irregularities in the test images for the detection of possible defects. 

Acoustics

The acoustic component also requires data, but Bondarev already has a plan for that. “If you stand under a bridge, you hear the sound of cars. The armored concrete sections conduct the sound of the cars driving over it. If the concrete is healthy, it sounds different than if it has defects. We can record these different sounds to train the AI ​​on this as well, so it can eventually ‘hear’ cracks in the bridge.” 

The largest gain, but maybe also the biggest challenge, is the merging of all this different data into one verdict about the state of the bridge. X-ray, radar, LiDAR, imaging and acoustic data types are very different by nature, therefore, merging them together to bring the five-sense human-like system is difficult, but can bring very high detection accuracy. 

Digital Twin

The results of the AI ​​analysis are stored in DTaaS: Digital Twin as a Service. This is a framework that generates and stores 3D digital copies of bridges, along with numerous images of cracks and other problems localized in the foundations, allowing for testing without directly impacting actual bridges.

Apart from the virtual stress testing, DTaaS provides great visualization capabilities for servicing. A service officer can walk atop of the bridge with his tablet-PC, on which the 3D model of the bridge is rendered together with all the defects located precisely under his current position on the bottom side of the section.

Predicting

Besides saving time, Bondarev believes the new system will also offer a significant second advantage. The current inspection method is limited in its approach: it's done with cameras, and the inspector inspects the exterior of the structure. “Currently, they don't look inside the concrete, so you miss a lot of important information. Drones will soon be able to do that.”

With the interior inspection, they will soon be able to predict when maintenance will be needed. This so-called predictive maintenance will help to plan maintenance more efficiently, but also to prevent damage such as that to the Merwedebrug. “The AI ​​will continue to learn and build a database with knowledge of how concrete and steel structures develop over the long term. With this historical knowledge, the AI ​​system can calculate and make predictions.”

The new working method will allow for significant gains in infrastructure inspections. “Instead of a week, a drone can do it in one day,” Bondarev explains. “And once developed, it will be easy to equip many drones with this technology. This will really speed up this huge infrastructure operation.”