[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-github-copilot-vscode-retry-storm-postmortem":3,"news-related-94358c42-4b6f-43fc-9053-4def49578ffa":35},{"id":4,"title":5,"summary":6,"content":7,"original_url":8,"source_id":9,"tags":10,"translations":21,"news_slug":28,"published_at":29,"created_at":30,"modified_at":31,"is_published":32,"publish_type":33,"image_url":14,"view_count":34},"94358c42-4b6f-43fc-9053-4def49578ffa","GitHub 八小时宕机复盘:VS Code 的重试 Bug 把 Copilot 流量打到 10 倍","GitHub 公布 8.17 重大宕机复盘:负载均衡器饱和 + Istio 扩容策略错配 + VS Code 重试 Bug 三层叠加,把 Copilot Token Service 流量从 7–9K RPS 打到 70–100K;7h47m 内 Actions、PR、SAML 全线受影响。","## 事件背景\n\n2026 年 8 月 17 日,全球最大的代码托管平台 GitHub 经历了一次持续 7 小时 47 分钟的严重宕机。事故从当日 13:28 UTC 开始,到 21:15 UTC 才完全解决。期间 Issues、Pull Requests、API、Actions、Copilot 等多项核心服务大面积报错,Web 与 API 的错误率一度达到约 20%,原始内容与归档下载的错误率则爬升到约 50%。\n\n事故直接拖累了大量依赖 GitHub 做协作与持续集成的团队,其中 Copilot Token Service 受到的冲击最深——这条为 GitHub Copilot 提供鉴权与计量的关键链路,在 17 日当天的流量从正常的 7–9K RPS 一路飙升到 70–100K RPS,足足放大了近一个数量级。\n\n## 三层连环失效\n\n官方事后报告把这场故障拆成了一次典型的「三层连环失效」。\n\n第一层位于数据中心网络。事故起始于美国中部数据中心的一组负载均衡器出现网络饱和,触发原因是峰值流量撞上了一个 Istio sidecar 的并发上限,而 GitHub 的自动扩容策略只监控宿主服务的指标,没有把 sidecar 的容量纳入视野。结果一个 Pod 撑不住,连锁到四台 HAProxy 节点耗尽 flow limit,把鉴权链路拖垮。\n\n第二层是「乐观重试」叠加。GitHub 内部网关采用了乐观重试,本来用于容灾的小机制,这次反而把已经堵死的负载均衡器进一步压垮。GitHub 的应对是「同时暂停」那四台 HAProxy,服务面这才出现真正的好转。\n\n第三层是 VS Code 客户端的隐式重试。GitHub 解释,North Virginia 区域因为某个内部端点的延迟响应,触发了 Visual Studio Code 中的一个潜在重试 Bug,直接把这个延迟信号放大到了 10 倍的请求量,原本只是一条普通的接口抖动,演变成了对 Copilot Token Service 的「请求风暴」。\n\n要压住这一层需要更精细的处置:一是临时用 PR 把网关重试逻辑调到保守档;二是在负载均衡层用 403 拒掉会触发重试的响应,然后再按站点逐步放流量,让客户端能逐步「断舍离」重试循环。\n\n## 为什么 AI 时代这层故障变得格外刺眼\n\n这场事故有几个值得 AI 工程师单独留意的特征。\n\n其一,AI 开发者对 Copilot 类编码助手的依赖度,已经高到一次认证链路不可用,就会直接影响日常开发节奏。Copilot 不是「用则锦上添花」的可选项,而是大量团队 GitOps 工作流的一部分。\n\n其二,事故响应本身要同时管理两类完全不同的时间常数:基础设施层的容器\u002F网络故障(秒级可见、分钟级恢复)和客户端层的重试风暴(可能延后数小时才被反卷回来)。AI 助手客户端往往会默认开启高频自动重试以追求「用户感知不到」的体验,这在故障场景下却会把后端反复捶打。\n\n其三,自动扩容策略如果只盯宿主 CPU\u002F内存、不盯 service mesh sidecar 的并发\u002F流量指标,在今天的服务网格架构下就是埋着一颗哑弹。一旦新版 Mesh 控制面升级或是流量形态变化,这类策略会突然看不见真正的瓶颈。\n\n## 补救清单与未来启示\n\nGitHub 在报告里给出了明确的后续动作:\n\n- 修正自动扩容策略,把 sidecar 并发与容量纳入视野;\n- 对受影响服务做一遍 Istio 请求、并发、扩缩容上限的审计;\n- 复查网关与客户端的重试上限和退避行为;\n- 专门修复把 Copilot Token Service 流量放大的 VS Code 重试行为;\n- 加强负载均衡容量监控与跨区域 failover 保护。\n\n这几条对国内做大规模 AI 推理平台与代码助手服务的团队同样适用。模型上线之后,真正决定可用度的往往是流量调度、限流、客户端重试与服务网格配置这些「看不见的地方」,而不是模型本身的 benchmark。一次看似普通的 HAProxy 流量饱和,在 AI Copilot 这种高 QPS 鉴权场景下,被客户端重试放大成 10 倍后,恢复时间自然以小时计算。\n\nGitHub 在这次 8 小时级别的故障里,首次明确把责任分摊到「数据中心 + 网关 + VS Code 客户端」三层,这也意味着依赖 Copilot 的开发者,从今往后需要把「客户端关闭自动重试」、「关键工位准备备选 IDE」、「CI 把重试策略调保守」加入自己的应急预案。AI 工具越深入工作流,我们对它的可用性预期就越该逼近云数据库一样的严肃程度。\n\n事故原始报告见 GitHub Status Incident 报告 ([githubstatus.com](https:\u002F\u002Fwww.githubstatus.com\u002Fincidents\u002Fzkxwbgr0cnmx))。","https:\u002F\u002Fwww.githubstatus.com\u002Fincidents\u002Fzkxwbgr0cnmx","998df6db-96e6-4b8e-8be1-cfa00a6cd177",[11,15,18],{"id":12,"name":13,"slug":13,"description":14,"color":14},"40269b40-7942-4650-9672-ed2e6524d37a","ai-technology",null,{"id":16,"name":17,"slug":17,"description":14,"color":14},"e82b2d09-81b2-43d1-977e-e018443b3c14","coding-agent",{"id":19,"name":20,"slug":20,"description":14,"color":14},"0ef8513a-0a26-42f0-b6f9-5b6dadded45c","efficiency",[22],{"id":23,"lang":24,"title":25,"summary":26,"content":27},"cd8b7299-060d-4468-9a42-d8ad45b06528","en","GitHub 8-Hour Outage: VS Code Retry Bug Drove Copilot Traffic to 10x","GitHub's full post-mortem of the August 17 incident shows three stacked failures: load balancer saturation, an Istio autoscaling misconfiguration, and a VS Code retry bug. Together they pushed Copilot Token Service traffic from 7-9K RPS to 70-100K. Actions, PRs and SAML all degraded across the 7h47m window.","## Background\n\nOn August 17, 2026, GitHub — the world's largest code-hosting platform — suffered an outage that lasted 7 hours and 47 minutes. The incident began at 13:28 UTC and was not fully resolved until 21:15 UTC. During that window, Issues, Pull Requests, API, Actions and Copilot all surfaced widespread errors: Web and API error rates climbed to around 20%, while raw-content and archive downloads hit roughly 50%.\n\nThe outage dragged down every team that depends on GitHub for collaboration and CI\u002FCD, and the Copilot Token Service was hit hardest. The RPS feeding the service that authenticates and meters Copilot usage jumped from a normal 7–9K to 70–100K — a near-tenfold amplification.\n\n## Three Layers, Three Failures\n\nThe official post-incident report frames the event as a classic three-layer cascade.\n\nThe first layer sat in the data-center network. The trigger was network saturation on a group of load balancers in the Central US region. The actual cause was an Istio sidecar hitting its concurrency limit while GitHub's autoscaling policy was watching only host-service metrics — not sidecar capacity. One pod buckled, the failure cascaded to four HAProxy nodes exhausting their flow limits, and the auth path collapsed.\n\nThe second layer was \"optimistic retry\" stacking. An internal gateway used optimistic retries meant for graceful degradation; in this case it slammed load balancers that were already on their last legs. Recovery only began once GitHub paused those four HAProxy nodes at the same time.\n\nThe third layer was an implicit client-side retry in VS Code. According to GitHub, a delayed response to one internal endpoint in the Northern Virginia region triggered a latent retry bug in Visual Studio Code that amplified the delay by roughly 10x in request volume — converting an ordinary endpoint hiccup into a request storm targeting Copilot Token Service.\n\nContainment on this layer required finer-grained mitigation: temporarily dialing gateway retries down via a PR, blocking retry-triggering responses at the load-balancer with 403s, then ramping traffic back up per-site so clients could slowly break their retry loops.\n\n## Why This Hurts More in the AI Era\n\nSeveral features of this incident deserve separate attention from AI engineers.\n\nFirst, developer reliance on Copilot-grade coding assistants has reached the point where an authentication-path outage directly disrupts daily development. Copilot is no longer a \"nice to have\" — it is part of many teams' GitOps workflows.\n\nSecond, incident response now has to manage two fundamentally different time constants simultaneously: infrastructure-layer container and network faults (visible in seconds, recoverable in minutes) and client-layer retry storms (which can unfold over hours). AI-assistant clients typically enable aggressive auto-retry by default in pursuit of \"invisible to the user\" experience, but during incidents that pattern can hammer the backend into the ground.\n\nThird, autoscaling policies that only watch host CPU and memory while ignoring service-mesh sidecar concurrency and throughput are sitting on a live bomb in today's meshed architectures. Upgrade the control plane or change the traffic profile, and these policies suddenly cannot see the real bottleneck.\n\n## The Remediation List and What It Signals\n\nGitHub's report enumerates concrete follow-ups:\n\n- Fix autoscaling policies to account for service-mesh sidecar concurrency and capacity.\n- Audit Istio request, concurrency, and scaling limits across affected services.\n- Review retry ceilings and backoff behavior across gateways and clients.\n- Address the VS Code retry behavior that amplified Copilot Token Service traffic.\n- Improve load-balancer capacity monitoring and regional failover safeguards.\n\nThese lessons apply equally to teams operating large-scale AI inference platforms and coding-assistant services domestically. Once a model is in production, what determines availability is rarely its benchmark score — it is the boring machinery of traffic scheduling, rate limiting, client retry and mesh configuration. A seemingly ordinary HAProxy saturation, in a high-QPS auth path like Copilot's, is easily amplified tenfold by client retries and measured in hours of recovery.\n\nBy clearly attributing responsibility across three layers — data center, gateway, VS Code client — in this eight-hour incident, GitHub is signaling that Copilot-dependent developers should now treat their AI tooling with the same operational seriousness as a managed database: disable client auto-retry, keep a backup IDE ready, and tighten CI retry policies. The deeper AI tools sink into our workflows, the closer our availability expectations should hew to that of mission-critical infrastructure.\n\nFull incident report available at the GitHub Status Incident page ([githubstatus.com](https:\u002F\u002Fwww.githubstatus.com\u002Fincidents\u002Fzkxwbgr0cnmx)).","github-copilot-vscode-retry-storm-postmortem","2026-08-23T02:00:00Z","2026-08-23T05:05:35.132282Z","2026-08-23T05:05:35.132290Z",true,"agent",70,{"items":36},[37,42,47,52,57,62],{"id":38,"title":39,"news_slug":40,"published_at":41},"deac2d55-76a6-40d2-8ef7-36aed2ad0105","Linux 7.2 把 AI 拉进内核开发:Sashiko 让补丁数量翻倍,Torvalds 接受「新常态」","linux-7-2-sashiko-ai-kernel-review","2026-08-20T12:00:00+00:00",{"id":43,"title":44,"news_slug":45,"published_at":46},"c07c67b6-6a48-4780-88bd-bc46b628c546","AMD 吃下 Taalas:把模型权重永久刻进芯片的\"硬推理\"赌局","amd-taalas-hardwired-inference-aug-2026","2026-08-08T12:00:00+00:00",{"id":48,"title":49,"news_slug":50,"published_at":51},"2434bbc6-4fda-4750-a02d-dd3ca1fe8933","AMD 收购 Taalas:把模型权重刻进芯片,押注推理硬件的\"硬核\"路线","amd-acquires-taalas-hardcore-inference-silicon","2026-08-08T04:00:00+00:00",{"id":53,"title":54,"news_slug":55,"published_at":56},"217f417d-1b9c-475b-99f4-e21e7c909711","MHAR 把 Transformer 残差流从「单车道」拆成 H 条独立路由:子空间第一次有权自己挑历史层","multi-head-attention-residuals-mhar","2026-08-01T07:30:00+00:00",{"id":58,"title":59,"news_slug":60,"published_at":61},"4bbc55d2-cabc-477f-a3ad-4e2c119aff2a","TokTier 抓住 Agent 推理的隐藏瓶颈：缓存命中 94.1%，分词仍吃掉 64% 首 token 时间","toktier-stateful-tokenization-agent-serving","2026-07-31T17:56:30+00:00",{"id":63,"title":64,"news_slug":65,"published_at":66},"6b52b4a9-d567-46b8-99c1-e9c65ba59b16","SWE-Pruner Pro:ByteDance 让 Agent 自己当剪枝器,省 39% token 还涨分","swe-pruner-pro-bytedance","2026-07-25T12:00:00+00:00"]