GitHub found the broken component about three hours into the August 17 outage. They were another four hours and thirty-nine minutes from done.
The status timeline has it at 16:36 UTC: “identified the problematic component and have taken corrective actions.” Then 17:30, Git Operations degraded again. 19:13, they partially disabled authentication token retries. Resolved 21:15. Seven hours forty-seven minutes, and most of it came after they knew what was wrong.
Nobody deployed anything. From the writeup, about this one and the one before it: “Neither outage was caused by a code or configuration change. Both incidents were capacity failures at their core.” Load balancers in Central US saturated on a new traffic peak, an Istio sidecar hit its concurrency limit without scaling, and four HAProxy nodes went with it.
Then the retries piled on. The Copilot token service normally sits around 7-9K RPS. During the incident it took 70-100K. That’s not user traffic. That’s retry logic doing what it was told.
The growth in that post is the context. 1.4 billion commits in April, 2.9 billion in August. Azure went from 12% of platform load in May to 58%. Move that fast and something in the stack is going to be the piece that didn’t scale.
Coming back up is a different engineering problem than not going down, and it usually gets less attention.
Prevention gets the budget: headroom, redundancy, review gates. The recovery path gets a quarterly drill where everything cooperates. GitHub’s remediation list is mostly the second category. Retry budgets, variable timeouts, isolating critical systems, removing shared dependencies.
Measure how long it takes you to come back. That number matters as much as the one on your status page.