[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-kaplan-scaling-law-bug":3,"news-related-d132e369-59e6-40a6-9d20-0c5d94535026":33},{"id":4,"title":5,"summary":6,"content":6,"original_url":7,"source_id":8,"tags":9,"translations":20,"news_slug":26,"published_at":27,"created_at":28,"modified_at":29,"is_published":30,"publish_type":31,"image_url":13,"view_count":32},"d132e369-59e6-40a6-9d20-0c5d94535026","Kaplan scaling law 原论文有 bug:原作者 ICML 2026 公开承认,行业被'大而不精'拐了两年","2026 年 7 月 ICML 韩国现场,DeepMind 研究员、Kaplan et al. (2020) 原始 scaling law 论文合著者 Sander Dieleman 在 X 上公开承认:\n\n> \"The original scaling laws were wrong due to a bug, which probably led to a lot of wasted compute on oversized undertrained models.\"\n\n这句话在 LLM 圈炸了锅 —— Kaplan 2020 那篇论文,是整个\"越大越好\"路线的理论基石。\n\n**Bug 出在哪**\n\nTim Pearce 与 Jinyeop Song 在 TMLR 2024 论文《Reconciling Kaplan and Chinchilla Scaling Laws》(arXiv:2406.12907) 中点破:最优规模公式 Kaplan 给的是 N_optimal ∝ C^0.73,而 Chinchilla (Hoffmann et al. 2022) 修正为 N_optimal ∝ C^0.50。差距源于 Kaplan **数参数时只算 non-embedding 部分**,且分析仅在小规模数据完成。两重偏差叠加,系统性推高最优模型规模。用 Chinchilla 协议 + Kaplan 的\"non-embedding\"口径重跑,正好复现 0.73 —— 证明不是算法错,而是分母选错了。\n\n**两年学费**\n\n按 Kaplan 配置,2020-2022 整个行业训练了一批\"大而不精\"的模型:GPT-3 (175B 训 300B tokens,ratio 1.7:1)、Gopher (280B)、PaLM (540B) 全数严重训不足。175B 模型按 Chinchilla 应配 ~3.5T tokens,实际只吃到 1\u002F12。修正后行业策略翻面 —— \"造小训多\":Llama 3 (8B \u002F 15T tokens,tokens\u002Fparam 比例近 1900:1)、DeepSeek V3 (MoE 671B \u002F 14.8T tokens) 都是这条线的产物。\n\n**今天意味着什么**\n\n第一,bug 不是 framework 错。Power law + (N, D, C) 这套数学框架今天仍是 foundation model planning 的默认工具,只是常数被修了。第二,2026 真正的新轴 —— test-time compute (o1\u002Fo3 类推理)、agent deployment scaling (ByteDance EdgeBench 那种\"3 个月学习速度翻倍\") —— Kaplan 那套根本不覆盖。承认 bug 是给\"预训练 scaling 触顶\"加注脚,**不等于 scaling law 整体失效**。Karpathy 1 月在 nanochat miniseries 用 total-params 重跑,复现干净的 0.5\u002F0.5 配比,亲口说\"我有点庆幸能用个简单常数\"。\n\n定调:这次承认不是改写历史,而是**给历史盖上了干净公章**。今天再谈 scaling law,讨论的不再是\"模型多大\",而是\"test-time \u002F agent \u002F post-training 这几条 Kaplan 没碰过的新轴\"该怎么 budget。","https:\u002F\u002Farxiv.org\u002Fabs\u002F2406.12907","7437aeb9-930c-4866-a2e9-48003c1a792b",[10,14,17],{"id":11,"name":12,"slug":12,"description":13,"color":13},"7ac06d8e-b074-4147-abfc-ffaa4c6b8744","ai-efficiency",null,{"id":15,"name":16,"slug":16,"description":13,"color":13},"01598627-1ea6-4b27-a5d8-874971571a71","llm",{"id":18,"name":19,"slug":19,"description":13,"color":13},"4f214978-cac1-4f39-aa4b-f92a0d0934b7","transformer",[21],{"id":22,"lang":23,"title":24,"summary":25,"content":13},"76d7604e-2be3-4ab2-8570-97d1873916c1","en","The Kaplan scaling-law paper had a bug, author admits at ICML","At ICML 2026 in South Korea, DeepMind researcher and co-author of the original Kaplan et al. (2020) scaling-law paper, Sander Dieleman, publicly admitted on X: > \"The original scaling laws were wrong due to a bug, which probably led to a lot of wasted compute on oversized undertrained models.\" This statement exploded in the LLM circle — Kaplan's 2020 paper is the theoretical cornerstone of the entire \"bigger is better\" route. **Where the bug was**: Tim Pearce and Jinyeop Song, in their TMLR 2024 paper \"Reconciling Kaplan and Chinchilla Scaling Laws\" (arXiv:2406.12907), point out: the optimal-size formula Kaplan gave is N_optimal ∝ C^0.73, while Chinchilla (Hoffmann et al. 2022) corrects it to N_optimal ∝ C^0.50. The difference stems from Kaplan **counting parameters only as the non-embedding part**, and the analysis is done only on small-scale data. The two biases stack, systematically pushing up the optimal model size. Using the Chinchilla protocol + Kaplan's \"non-embedding\" convention to re-run, it exactly reproduces 0.73 — proving the algorithm isn't wrong, but the denominator was selected wrong. **Two years of tuition**: Following Kaplan's configuration, from 2020-2022 the entire industry trained a batch of \"big but undertrained\" models: GPT-3 (175B trained on 300B tokens, ratio 1.7:1), Gopher (280B), PaLM (540B) all severely undertrained. The 175B model according to Chinchilla should have ~3.5T tokens, but actually only ate 1\u002F12. After correction, the industry strategy flipped — \"make small, train many\": Llama 3 (8B \u002F 15T tokens, tokens\u002Fparam ratio near 1900:1), DeepSeek V3 (MoE 671B \u002F 14.8T tokens) are products of this line. **What it means today**: First, the bug isn't a framework error. The mathematical framework of power law + (N, D, C) is still the default tool for foundation-model planning today, only the constants have been fixed. Second, the truly new axes of 2026 — test-time compute (o1\u002Fo3-style reasoning), agent deployment scaling (ByteDance EdgeBench's \"3-month learning speed doubling\") — Kaplan's set simply doesn't cover. Acknowledging the bug adds a footnote to \"pretraining scaling has topped out\", **not equivalent to scaling-law as a whole failing**. Karpathy, in his January nanochat miniseries, re-ran with total-params, reproducing clean 0.5\u002F0.5 ratios, saying in person \"I'm kind of glad to be able to use simple constants\". Calibration: this admission isn't rewriting history, but **putting a clean official seal on history**. When we talk about scaling law today, the discussion is no longer \"how big should the model be\", but \"how should test-time \u002F agent \u002F post-training, these new axes Kaplan never touched, be budgeted\".","kaplan-scaling-law-bug","2026-07-05T15:00:00Z","2026-07-05T15:08:17.432357Z","2026-08-19T02:08:40.142862Z",true,"agent",121,{"items":34},[35,40,45,50,55,60],{"id":36,"title":37,"news_slug":38,"published_at":39},"e3f049f5-2f0d-48d2-8e88-246ef006fa16","LoopMTP 给循环 Transformer 装上前瞻路标：固定参数下让每一轮都做不同的事","loopmtp-latent-multi-token-loop-guidance","2026-08-04T13:13:09+00:00",{"id":41,"title":42,"news_slug":43,"published_at":44},"c48681ff-ddbb-402c-ade8-23b584a06aea","更强教师反而教不动学生：Lightning OPD 2.0 剥掉蒸馏中的“文风噪声”","lightning-opd-2-cross-teacher-style-bias","2026-07-30T16:17:15+00:00",{"id":46,"title":47,"news_slug":48,"published_at":49},"a151db0c-d832-4df2-ac03-2d4e58b26e99","Kimi K3 跑通 MiniTriton:Moonshot 让 LLM 第一次从零编译出自己的 GPU 编译器","kimi-k3-minitriton-gpu-compiler","2026-07-26T14:00:00+00:00",{"id":51,"title":52,"news_slug":53,"published_at":54},"582462cb-d952-4136-818b-3edd6fc9ad0c","LongStraw 把百万 token 拉进 RL 后训练:8 张 H20 跑通 2.1M token 端到端","longstraw-2-1m-token-rl","2026-07-16T13:00:00+00:00",{"id":56,"title":57,"news_slug":58,"published_at":59},"7312935f-7432-4dbd-9cff-c665fa7b4765","FlashMorph:ByteDance Seed 把混合注意力的\"层选择\"做成预算约束优化","flashmorph-hybrid-attention-layer","2026-07-05T02:01:00+00:00",{"id":61,"title":62,"news_slug":63,"published_at":64},"b47914fe-d539-440f-ab02-3b089120051f","MultiHashFormer 用多重哈希签名改写 LLM 词表嵌入：3B 模型跑赢标准 Transformer","multihashformer-hash-embedding-3b","2026-06-29T04:20:00+00:00"]