This is one of three root causes of the outage that affected businesses, hospitals, governments and other organizations around the world.
Microsoft has revealed more details about the outage that took services including Outlook, Teams and Copilot 365 offline, affecting businesses, hospitals, law enforcement and other organizations across the world. Charles Lamanna, the company’s executive vice president for co-pilot and AI agents, explained on X that there were three root causes for the outage.
He said Microsoft typically maintains two network gateway systems that connect separate systems and allow data to pass between them. One of these gateways was not working for Microsoft Teams and was overwhelming the primary gateway. It also turns out that almost 14% of this gateway’s capacity was out of service due to an unrelated issue that occurred the day before. These two factors led this gateway to encounter a latent bug present in Microsoft systems since July.
Together, the higher load that overwhelmed the gateway, the reduced server capacity, and the latent bug led to increased traffic until Microsoft’s servers gave way and stopped working. All this happened on a Thursday afternoon, during rush hour in the United States.
The outage lasted almost a full day. Microsoft worked on fixing the problem overnight, but the bug was several months old at that point and the company wasn’t able to simply roll back to quickly undo the changes it caused. Additionally, its servers were overwhelmed with pending traffic after restarting and became unresponsive.
“We are deeply sorry for the impact on all of our customers and recognize the impact this is having on their normal business and operations,” Lamanna wrote in his message. These details come from Microsoft’s preliminary post-incident report, and the company said it will release more details in the final report to be released later this week.
Gn headline
