Background

On August 17, 2026, GitHub — the world's largest code-hosting platform — suffered an outage that lasted 7 hours and 47 minutes. The incident began at 13:28 UTC and was not fully resolved until 21:15 UTC. During that window, Issues, Pull Requests, API, Actions and Copilot all surfaced widespread errors: Web and API error rates climbed to around 20%, while raw-content and archive downloads hit roughly 50%.

The outage dragged down every team that depends on GitHub for collaboration and CI/CD, and the Copilot Token Service was hit hardest. The RPS feeding the service that authenticates and meters Copilot usage jumped from a normal 7–9K to 70–100K — a near-tenfold amplification.

Three Layers, Three Failures

The official post-incident report frames the event as a classic three-layer cascade.

The first layer sat in the data-center network. The trigger was network saturation on a group of load balancers in the Central US region. The actual cause was an Istio sidecar hitting its concurrency limit while GitHub's autoscaling policy was watching only host-service metrics — not sidecar capacity. One pod buckled, the failure cascaded to four HAProxy nodes exhausting their flow limits, and the auth path collapsed.

The second layer was "optimistic retry" stacking. An internal gateway used optimistic retries meant for graceful degradation; in this case it slammed load balancers that were already on their last legs. Recovery only began once GitHub paused those four HAProxy nodes at the same time.

The third layer was an implicit client-side retry in VS Code. According to GitHub, a delayed response to one internal endpoint in the Northern Virginia region triggered a latent retry bug in Visual Studio Code that amplified the delay by roughly 10x in request volume — converting an ordinary endpoint hiccup into a request storm targeting Copilot Token Service.

Containment on this layer required finer-grained mitigation: temporarily dialing gateway retries down via a PR, blocking retry-triggering responses at the load-balancer with 403s, then ramping traffic back up per-site so clients could slowly break their retry loops.

Why This Hurts More in the AI Era

Several features of this incident deserve separate attention from AI engineers.

First, developer reliance on Copilot-grade coding assistants has reached the point where an authentication-path outage directly disrupts daily development. Copilot is no longer a "nice to have" — it is part of many teams' GitOps workflows.

Second, incident response now has to manage two fundamentally different time constants simultaneously: infrastructure-layer container and network faults (visible in seconds, recoverable in minutes) and client-layer retry storms (which can unfold over hours). AI-assistant clients typically enable aggressive auto-retry by default in pursuit of "invisible to the user" experience, but during incidents that pattern can hammer the backend into the ground.

Third, autoscaling policies that only watch host CPU and memory while ignoring service-mesh sidecar concurrency and throughput are sitting on a live bomb in today's meshed architectures. Upgrade the control plane or change the traffic profile, and these policies suddenly cannot see the real bottleneck.

The Remediation List and What It Signals

GitHub's report enumerates concrete follow-ups:

  • Fix autoscaling policies to account for service-mesh sidecar concurrency and capacity.
  • Audit Istio request, concurrency, and scaling limits across affected services.
  • Review retry ceilings and backoff behavior across gateways and clients.
  • Address the VS Code retry behavior that amplified Copilot Token Service traffic.
  • Improve load-balancer capacity monitoring and regional failover safeguards.

These lessons apply equally to teams operating large-scale AI inference platforms and coding-assistant services domestically. Once a model is in production, what determines availability is rarely its benchmark score — it is the boring machinery of traffic scheduling, rate limiting, client retry and mesh configuration. A seemingly ordinary HAProxy saturation, in a high-QPS auth path like Copilot's, is easily amplified tenfold by client retries and measured in hours of recovery.

By clearly attributing responsibility across three layers — data center, gateway, VS Code client — in this eight-hour incident, GitHub is signaling that Copilot-dependent developers should now treat their AI tooling with the same operational seriousness as a managed database: disable client auto-retry, keep a backup IDE ready, and tighten CI retry policies. The deeper AI tools sink into our workflows, the closer our availability expectations should hew to that of mission-critical infrastructure.

Full incident report available at the GitHub Status Incident page (githubstatus.com).