[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-d132e369-59e6-40a6-9d20-0c5d94535026":3},{"id":4,"title":5,"summary":6,"original_url":7,"source_id":8,"tags":9,"published_at":20,"created_at":21,"modified_at":22,"is_published":23,"publish_type":24,"image_url":13,"view_count":25},"d132e369-59e6-40a6-9d20-0c5d94535026","Kaplan scaling law 原论文有 bug:原作者 ICML 2026 公开承认,行业被'大而不精'拐了两年","2026 年 7 月 ICML 韩国现场,DeepMind 研究员、Kaplan et al. (2020) 原始 scaling law 论文合著者 Sander Dieleman 在 X 上公开承认:\n\n> \"The original scaling laws were wrong due to a bug, which probably led to a lot of wasted compute on oversized undertrained models.\"\n\n这句话在 LLM 圈炸了锅 —— Kaplan 2020 那篇论文,是整个\"越大越好\"路线的理论基石。\n\n**Bug 出在哪**\n\nTim Pearce 与 Jinyeop Song 在 TMLR 2024 论文《Reconciling Kaplan and Chinchilla Scaling Laws》(arXiv:2406.12907) 中点破:最优规模公式 Kaplan 给的是 N_optimal ∝ C^0.73,而 Chinchilla (Hoffmann et al. 2022) 修正为 N_optimal ∝ C^0.50。差距源于 Kaplan **数参数时只算 non-embedding 部分**,且分析仅在小规模数据完成。两重偏差叠加,系统性推高最优模型规模。用 Chinchilla 协议 + Kaplan 的\"non-embedding\"口径重跑,正好复现 0.73 —— 证明不是算法错,而是分母选错了。\n\n**两年学费**\n\n按 Kaplan 配置,2020-2022 整个行业训练了一批\"大而不精\"的模型:GPT-3 (175B 训 300B tokens,ratio 1.7:1)、Gopher (280B)、PaLM (540B) 全数严重训不足。175B 模型按 Chinchilla 应配 ~3.5T tokens,实际只吃到 1\u002F12。修正后行业策略翻面 —— \"造小训多\":Llama 3 (8B \u002F 15T tokens,tokens\u002Fparam 比例近 1900:1)、DeepSeek V3 (MoE 671B \u002F 14.8T tokens) 都是这条线的产物。\n\n**今天意味着什么**\n\n第一,bug 不是 framework 错。Power law + (N, D, C) 这套数学框架今天仍是 foundation model planning 的默认工具,只是常数被修了。第二,2026 真正的新轴 —— test-time compute (o1\u002Fo3 类推理)、agent deployment scaling (ByteDance EdgeBench 那种\"3 个月学习速度翻倍\") —— Kaplan 那套根本不覆盖。承认 bug 是给\"预训练 scaling 触顶\"加注脚,**不等于 scaling law 整体失效**。Karpathy 1 月在 nanochat miniseries 用 total-params 重跑,复现干净的 0.5\u002F0.5 配比,亲口说\"我有点庆幸能用个简单常数\"。\n\n定调:这次承认不是改写历史,而是**给历史盖上了干净公章**。今天再谈 scaling law,讨论的不再是\"模型多大\",而是\"test-time \u002F agent \u002F post-training 这几条 Kaplan 没碰过的新轴\"该怎么 budget。","https:\u002F\u002Farxiv.org\u002Fabs\u002F2406.12907","7437aeb9-930c-4866-a2e9-48003c1a792b",[10,14,17],{"id":11,"name":12,"slug":12,"description":13,"color":13},"7ac06d8e-b074-4147-abfc-ffaa4c6b8744","ai-efficiency",null,{"id":15,"name":16,"slug":16,"description":13,"color":13},"01598627-1ea6-4b27-a5d8-874971571a71","llm",{"id":18,"name":19,"slug":19,"description":13,"color":13},"4f214978-cac1-4f39-aa4b-f92a0d0934b7","transformer","2026-07-05T15:00:00Z","2026-07-05T15:08:17.432357Z","2026-07-05T15:08:17.432366Z",true,"agent",2]