Coordinated disruptions are haunting networks around the world. Here’s how researchers solved the mystery.

For years, mysterious electrical anomalies have baffled grid managers around the world. Every 60 seconds, the frequency on the grid decreased just a little bit.

Even stranger: the drops occurred at exactly the same time on the power grids of all countries and continents.

It was a mystery that Dirk Witthaut, a researcher at Forschungszentrum Jülich in Germany, could not let go.

Germany’s largest transmission system operator, Tennet Germany, brought the conundrum to Witthaut, and it took years of investigative and modeling work to determine the cause. From Europe to Australia, from the United States to Malaysia, the same pattern appears in the data.

Such frequency drops are not uncommon, but they are troubling. And the regularity with which they occur can make the phenomenon even more worrying.

Balance the network

The proper functioning of the network relies on maintaining the frequency within a certain range. If the oscillations become too large or if grid operators do not know the cause and react inappropriately, they can lead to power outages. Even if subsidence does not cause power outages, it can damage power plant equipment if left unchecked, particularly in hydroelectric turbines.

Hydroelectric power plants have always been one of the main drivers of so-called ultra-low frequency oscillations (UFLO). Each turbine vibrates, and if these vibrations occur with a certain regularity, called a resonant frequency, they can amplify, sometimes to dangerous levels. Think of a glass of wine broken in front of the powerful trumpets of an opera singer: that’s resonance.

So Witthaut and his team began investigating hydropower, the most obvious potential culprit. They used a computer model of a control system for a hydroelectric plant on the Scandinavian grid to test this hypothesis.

Although they managed to find conditions that could generate a resonance peaking once per minute, they could not explain why it persisted for so long. Generally, resonances in mechanical systems come and go, as does the accompanying signal on the electrical network. But this drop in frequency continued to occur minute after minute. Every 60 seconds, without fail.

The team then realized that anything that worked with such a clockwork mechanism required… a clock. How else could they explain this strange timing?

Looking for clues

To address this hypothesis, researchers started close to home by installing more than 80 smart energy quality meters around the Juliers campus. The research site is equipped with medium and low voltage distribution lines, transformers, solar panels and battery energy storage systems that serve as an uninterruptible power supply (UPS). It also has a high-performance computer system as well as the usual desktop computers powered by much smaller UPS devices. The power quality meters sampled a series of attributes, including frequency, voltage, current and power, every three seconds.

Before long, the research team had their answer. Measurements recorded on the transformer connected to the battery energy storage system peaked every minute, just like on larger power grids.

It turns out that many stationary batteries, from data center UPSs to utility-scale batteries, check their state of charge every minute. Since these devices are connected to the Internet, their clocks are in almost perfect sync. Each time we check its load, the disturbance is imperceptible on the wide network. But when they all do it together, this rift becomes apparent.

To find evidence that UPSs were primarily responsible for the network, Witthaut and his team turned to data provided by network operators. They found their answer in Ireland.

There, data centers have grown to consume much of the country’s electricity. By 2022, data centers accounted for 18% of Ireland’s grid load, and each facility is served by a fleet of UPSs. Indeed, between 2016 and 2021, the minute-by-minute frequency drop on the Irish network quadrupled, from 3 milli-Hertz (mHz) in 2016 to 12 MHz in 2021. More recently, some drops have exceeded 20 MHz.

Ireland’s 50 Hz network is designed to withstand oscillations of plus or minus 200 MHz, although the network operator prefers to limit oscillations to 100 MHz. In this range, 20 MHz isn’t much, but data centers aren’t the only loads on the network. If other loads also cause frequency sag, the added bump of each UPS checking its state of charge could push the grid toward blackout conditions.

The solution

The problem is entirely avoidable. Rather than checking the state of charge exactly to the minute, Witthaus and his team recommend that grid-scale inverter and battery manufacturers add some randomization to the schedule.

Even if UPSs are the cause of the problem today, there is a risk that the phenomenon will persist. The researchers also tested IoT devices. In devices as diverse as a smart oven and Raspberry Pi clusters, they were able to detect periodic power fluctuations. The patch is also a simple software fix, but many IoT devices are not patched. A recent study found that less than 3% of IoT devices are running the latest firmware.

Regulators may soon have to step in to demand changes, but they can only look after their corner of the network. Given the global nature of the problem, this will ultimately require global coordination. The mystery may be solved, but the problem could persist for years.