Microsoft Blames Exchange Online Outage on a July Gateway Defect
Software / news
Microsoft Blames Exchange Online Outage on a July Gateway Defect
Its Nov. 15 preliminary report says load tests ran on hardware with fewer cores than the servers where the bug bit.

Microsoft traced a roughly 24-hour Exchange Online outage on Nov. 12 and 13 to a defect in a Windows change it shipped to its Microsoft 365 network gateway servers in July, according to its preliminary post-incident report dated Nov. 15 and The Register's account of it.
The report says the change passed load testing. Microsoft said: "Our load testing tended to focus on weaker hardware with fewer cores." It added: "The contention issue is worsened on machines with more cores and newer hardware." A final report is due Nov. 20.
Two events pushed the gateways over
The defect is lock contention in an internal Windows process, where tasks queue behind one lock until CPU and memory run out. It sat in production for about four months without visible effect.
Two things changed on Nov. 12. Teams traffic failed over to the primary gateway that also serves Exchange Online, starting at 01:07 UTC, adding load. Nearly 14 percent of that gateway's server capacity was also out of service from an unrelated issue the day before. Microsoft said the service "had sufficient capacity headroom," but the combination was enough to trigger the bug. An independent write-up by Enea Krähenbühl on his personal blog, who says he is unaffiliated with Microsoft and worked from the same report, adds that load climbed through the afternoon and servers began locking up in numbers around the 16:00 UTC US peak.

Timeline from the preliminary report
| Time (UTC) | Event |
|---|---|
| Nov. 12, 01:07 | Teams failover to the Exchange-supporting gateway begins |
| Nov. 12, 16:03 | Connectivity anomaly monitor flags the problem |
| Nov. 12, 16:33 | First support case arrives |
| Nov. 13, 00:30 | Connection negotiation identified as the failure point |
| Nov. 13, 07:20 | Update reverting the defective change begins rolling out |
| Nov. 13, 16:10 | Services confirmed healthy |
The times come from The Register's reading of the report. Krähenbühl puts the start of the outage at 16:20 UTC and the total at 23 hours 50 minutes, so the exact minute depends on which log line is counted. At the low point, more than 80 percent of gateway servers were unresponsive.
Why a rollback did not work
For an admin watching the Microsoft 365 service health page, the slow part was recovery. Microsoft paused deployments and configuration changes, added capacity and rerouted traffic, but servers kept failing. The Register reports that automated recovery was not built for a fleet-wide failure of this kind.
A plain rollback was also unavailable, because the defect was months old and there was nothing recent to revert to. Engineers instead shipped a fix returning the affected code path to a previously known good Windows state and applied it to server groups in stages. Krähenbühl counts about 15 hours from first alert to a working fix and about nine more to roll it out.
Teams, OneDrive and SharePoint could be moved off the faulty gateways. Exchange Online could not, "for scale reasons," in Microsoft's words as quoted by Krähenbühl. That is why a Teams failover ended up taking down email.
Related coverage in this section includes the Claude Code 2.1.313 Cursor SSH issue and the Lightpanda benchmark.
What the report leaves open
Krähenbühl names the incident IDs as EX1550543 and MO1550821; The Register did not print an ID, so treat them as single-source. He also flags the biggest open question: the regional isolation meant to contain a failure did not, and Microsoft's preliminary explanation is only that it happened "due to a separate issue."
Microsoft has not said whether it will change how it picks hardware for load tests. The final post-incident report on Nov. 20 should say whether it does.
Sources
More in Software
- 01Zigzag, a 35 MB Mac Browser, Courts Arc Users 35 Days Before ShutdownThe WebKit-based browser imports Arc spaces and pinned tabs, but its README lists no sync, no Boosts and a partial Chrome extension API.
- 02Nimony 0.6.2 Sets a Year-End Goal for Nim 3 With 44 Days LeftThe compiler that will become Nim 3 is MIT-licensed and builds natively without LLVM, but its FAQ says it cannot yet compile Nim 2 code unchanged.
- 03Alloy CSS Runs Menus and Tabs Without JavaScript, but Dialogs Need Safari 26.2The 24 KB MIT stylesheet leans on popover and invoker commands, so its no-script promise has browser floors that differ by component.
- 04Windows 11 Telemetry Flags Which Coding Agents You Run, Researcher FindsXusheng Li traced a service-experiment feature in build 26200.9974 that matches executable names such as claude.exe and queues the result as optional diagnostic data.