[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-dont-drop-dropout-layer-sparsity":3,"topics-all":38,"news-related-0c29e1ad-914a-4b79-a153-445c087acb03":39},{"id":4,"title":5,"summary":6,"content":7,"original_url":8,"source_id":9,"tags":10,"translations":24,"news_slug":31,"published_at":32,"created_at":33,"modified_at":34,"is_published":35,"publish_type":36,"image_url":14,"view_count":37},"0c29e1ad-914a-4b79-a153-445c087acb03","被 LLM 抛弃的 dropout 翻身:Cerebras 称调好可省 25% 训练 FLOPs","层dropout(随机深度)已被逐出大模型预训练配方。Cerebras团队在ICML 2026论文中用2400+次实验(271M–8.2B参数、160B token)验证:配置得当时同等算力loss更低,最多省25%训练FLOPs,还为early exit与自投机解码铺路,推理最高提速1.5倍。","在 2017 年的 Transformer 论文里,dropout 还是标配正则化;到了大模型时代它几乎绝迹——GPT-3、OPT 还保留着它,PaLM 只在微调阶段用,LLaMA 之后的模型干脆不再写进配方。理由听上去充分:单轮扫过数万亿 token 的预训练几乎没有经典意义上的过拟合,还有研究报道 dropout 在这个 regime 里掉点。Cerebras Systems 团队新挂在 arXiv 的论文「Don't Drop Dropout」(ICML 2026 发表后的扩展版,[arXiv:2609.05275](https:\u002F\u002Farxiv.org\u002Fabs\u002F2609.05275))给出相反结论:被抛弃的可能不是 dropout 本身,而是没调对的用法。\n\n## 2400+ 次实验,重新给层 dropout 定位\n\n研究聚焦层 dropout(随机深度):训练时以一定概率整块跳过 transformer 层。与神经元级稀疏不同,跳过整层是结构化稀疏,省下的 FLOPs 几乎与丢弃率线性对应。团队在 Cerebras CS-3 系统上跑了 2400+ 次预训练实验,模型从 271M 到 8.2B 参数、数据最高 160B token,按 20 token\u002F参数的 compute-optimal 预算执行,架构沿用 Celerity 系列(ALiBi 位置编码、squared ReLU、Llama3 词表)。\n\n核心结论是一组组合拳:dropout 率沿层深递增(浅层少丢、深层多丢)、随训练进程递减(开局高噪声探索、后期收敛),再配上按丢弃率重调的优化器超参。这样配下来,同等训练 FLOPs 下 loss 反而更低;固定训练步数时,最多省 25% 训练 FLOPs 而 loss 持平或更好。论文称,在 503M 和 906M 模型上省 5% FLOPs 的配置击败了 dense 基线,并称这是首次有实验证明「更少训练算力也能打过全量基线」。采样粒度上,按序列独立采样优于按 batch 整体采样——这是该团队自称首次系统对比的维度。\n\n## 推理端的「免费」弹性\n\n层 dropout 训出的模型自带深度弹性:zero-shot 做 early exit(提前若干层出口)或跳过中间层,loss 退化平缓,而 dense 模型几乎一跳就崩。这意味着一个大模型可以按延迟预算动态缩到小模型水平,不用重训。叠加自投机解码等后训练手段,推理最高提速 1.5 倍、精度损失可忽略。层分布上存在明确取舍:交替分布(ALD,隔层丢)对跳层最鲁棒,递增分布(ILD)基线精度和 early exit 表现更好,论文建议规模化后选 ILD。\n\n## 所以呢\n\n这篇论文把「dropout 在大模型没用」重新定性为「配置问题」:它不再只是防过拟合的正则项,而是训练省算力加推理可弹性的结构化稀疏开关。对算力预算敏感的预训练团队,25% FLOPs 是笔不小的账。但要记住边界:实验上限 8.2B 参数,更大的前沿尺度未验证;结论基于 Celerity 式 decoder-only 架构与 20 token\u002F参数预算,数字为团队在其自有系统上的自报结果,尚待独立复现。下一次看到某家预训练成本突然降一截,配方表里的 dropout 也许就是答案。","https:\u002F\u002Farxiv.org\u002Fabs\u002F2609.05275","7437aeb9-930c-4866-a2e9-48003c1a792b",[11,15,18,21],{"id":12,"name":13,"slug":13,"description":14,"color":14},"0ef8513a-0a26-42f0-b6f9-5b6dadded45c","efficiency",null,{"id":16,"name":17,"slug":17,"description":14,"color":14},"0a93ec8e-ea39-4693-81de-563ca8c173f7","inference",{"id":19,"name":20,"slug":20,"description":14,"color":14},"01598627-1ea6-4b27-a5d8-874971571a71","llm",{"id":22,"name":23,"slug":23,"description":14,"color":14},"b9bd9039-fcdb-41a8-b85b-fc1587def2b9","open-source",[25],{"id":26,"lang":27,"title":28,"summary":29,"content":30},"d38dce6a-1e86-4e2d-9de7-812c74264c4a","en","Don't Drop Dropout: Cerebras Study Cuts LLM Training FLOPs 25%","Cerebras shows tuned layer dropout cuts LLM training FLOPs up to 25% and speeds inference 1.5x, reviving a technique large models abandoned.","Dropout was standard regularization in the 2017 Transformer paper, yet it has all but vanished from large language model pretraining. GPT-3 and OPT kept it; PaLM used it only during finetuning; LLaMA-style models no longer document it at all. The reasoning sounds solid: single-epoch pretraining over trillions of tokens leaves little room for classical overfitting, and some studies report that dropout degrades accuracy in this regime. A new paper from the Cerebras Systems team, \"Don't Drop Dropout\" (an extended version of their ICML 2026 paper, [arXiv:2609.05275](https:\u002F\u002Farxiv.org\u002Fabs\u002F2609.05275)), argues the opposite: what got abandoned may not be dropout itself, but misconfigured dropout.\n\n## 2400+ Experiments Reframe Layer Dropout\n\nThe study focuses on layer dropout, also known as stochastic depth: during training, entire transformer blocks are skipped with some probability. Unlike neuron-level sparsity, skipping whole layers is structured sparsity, and the FLOPs saved scale almost linearly with the dropout rate. The team ran more than 2400 pretraining experiments on Cerebras CS-3 systems, spanning models from 271M to 8.2B parameters and datasets up to 160B tokens, at a compute-optimal budget of 20 tokens per parameter, using the Celerity-style architecture (ALiBi position embeddings, squared ReLU activations, Llama3 vocabulary).\n\nThe core result is a combined recipe: increase the dropout rate across depth (drop less in shallow layers, more in deep layers), decrease it over training time (high initial noise for exploration, then convergence), and re-tune optimizer hyperparameters for each dropout rate. With this configuration, layer dropout reaches lower loss at the same training FLOPs; for a fixed number of steps, it saves up to 25% of training FLOPs while matching or beating the validation loss. The paper reports that at 5% FLOPs savings on 503M and 906M models, the dropout configuration beat the dense baseline, which the authors describe as the first demonstration that fewer training FLOPs can outperform a full-compute baseline. On sampling granularity, per-sequence dropout yields lower loss than per-batch dropout, a comparison the authors say they are the first to systematically study.\n\n## \"Free\" Elasticity at Inference Time\n\nModels pretrained with layer dropout gain depth elasticity for free: zero-shot early exit or intermediate layer skipping degrades loss gracefully, while dense models spike almost immediately. One large model can thus shrink toward smaller-model performance levels to fit latency budgets, without retraining. Combined with post-training tricks like self-speculative decoding, inference speeds up by up to 1.5x with negligible accuracy loss. The layer distribution carries a clear trade-off: the alternating distribution (ALD, dropping every other layer) is most robust to layer skipping, while the increasing distribution (ILD) gives better base accuracy and early-exit behavior; the paper recommends ILD as models scale.\n\n## So What\n\nThis work reframes \"dropout is useless at scale\" as a configuration problem. Layer dropout is no longer just an anti-overfitting regularizer; it is a structured-sparsity switch that saves training compute and buys inference elasticity. For teams with tight pretraining budgets, 25% of FLOPs is a serious line item. But keep the boundaries in mind: experiments top out at 8.2B parameters, so frontier-scale behavior is unverified; conclusions rest on the Celerity-style decoder-only architecture and a 20 tokens-per-parameter budget; and the numbers are self-reported on the team's own hardware, awaiting independent replication. The next time a lab's pretraining bill drops unexpectedly, check whether dropout quietly returned to the recipe.","dont-drop-dropout-layer-sparsity","2026-09-07T21:06:35Z","2026-09-07T21:06:49.344273Z","2026-09-07T21:06:49.344284Z",true,"agent",33,[],{"items":40},[41,46,51,56,61,66],{"id":42,"title":43,"news_slug":44,"published_at":45},"199cd4ef-f092-45a5-8635-91778dd2bce2","编译即训练：一句规约炼出 83.6% 准确率的本地神经函数，教师模型只用一次","compile-by-training-neural-functions","2026-09-04T23:08:03+00:00",{"id":47,"title":48,"news_slug":49,"published_at":50},"7a6d28b6-65da-4a29-96b1-dedb9894de97","随机驱逐追平最强打分器:Salesforce 重写 KV Cache 压缩常识","random-attention-kv-cache-eviction","2026-09-04T19:08:26+00:00",{"id":52,"title":53,"news_slug":54,"published_at":55},"44a035c8-b8a3-48e5-af4f-c76323dac7b5","RWKV7-G1j 13.3B 开源:不用注意力,每 token 推理成本是常数","rwkv7-g1j-13b-attention-free","2026-09-03T13:14:19+00:00",{"id":57,"title":58,"news_slug":59,"published_at":60},"ff3b7c9f-e338-4af9-a562-4fda8b1bc929","vLLM 0.28 发布:584 项提交押注 Kimi K3,每卡省 17GB 显存","vllm-0-28-kimi-k3-optimization","2026-08-29T19:05:00+00:00",{"id":62,"title":63,"news_slug":64,"published_at":65},"cac485ab-a429-4ebc-88c6-a1f924f978ff","AWS 开源 KeysAndValues:微调时就让模型学会“遗忘”,单张 A100 撑住 128K","aws-keysvalues-sparse-attention-finetuning","2026-08-26T05:20:00+00:00",{"id":67,"title":68,"news_slug":69,"published_at":70},"c94bdf86-5de9-49fe-8c98-0f5c47611bfe","SGLang v0.5.18 发布:大模型冷启动提速 2.38 倍,710 个 PR 都改了什么","sglang-v0-5-18-cold-start-2-38x","2026-08-24T23:15:00+00:00"]