[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-prefill-decode-length-prediction-limits":3,"topics-all":38,"news-related-932bddfc-f9c2-43c9-9eec-676df0a7bf23":57},{"id":4,"title":5,"summary":6,"content":7,"original_url":8,"source_id":9,"tags":10,"translations":24,"news_slug":31,"published_at":32,"created_at":33,"modified_at":34,"is_published":35,"publish_type":36,"image_url":14,"view_count":37},"932bddfc-f9c2-43c9-9eec-676df0a7bf23","解码长度预测撞上天花板:误差大头是采样噪声,不是模型","同一套预测器换模型换数据后,MAE 从约 80 涨到超过 260,短板不在模型。arXiv 论文把解码建模成停时随机过程,证明单条 prompt 的解码长度服从正态分布、方差满足 σ=αμ+b;公平重测下简单 MLP 已落在噪声底线 30% 以内,ForeLen 领先方法 EGTP 掉到 466.79。","推理系统里有一类不起眼却关键的预测:用户输入进来、模型还没吐第一个字,调度器就想知道这条回复大概多长——连续批处理靠它决定怎么分组。主流做法是用 prefill 阶段的隐藏状态训一个回归器,在解码开始前给出长度估计。这个方向近年方法不少,但各家数字放在一起没法比:同一个预测器架构,在 ForeLen 基准的长序列档配 Qwen2.5-7B,MAE 约 80;换到 ShareGPT 配 Qwen3-32B,MAE 超过 150;换到 RL 采样数据,超过 260。数字差三倍,预测器本身一毫米没动。\n\n## 天花板是正态分布画出来的\n\n10 月 29 日挂上 arXiv 的论文《On the Fundamental Limits of Prefill-Based Decoding Length Prediction》把病根定位到自回归采样自己身上:同一条 prompt 解码两次,长度本来就不一样。论文把解码建模成停时随机过程——回复在每一步都有一个停止概率,这个概率不是常数,而是在几百步里逐步爬升;回复长度由许多段落级的局部决定累积而成,中心极限定理因此成立,单条 prompt 的解码长度渐近服从正态分布,还带一个显式误差项。方差服从仿射律 σ=αμ+b,斜率 α 约等于回复段落数的变异系数。反事实推演也做了:若停止概率恒定,长度退化为几何分布,σ\u002Fμ 趋向 1,任何预测器的相对 MAE 都至少约 74%;实测 σ\u002Fμ≈0.15,排除了这个更坏的世界。验证很扎实:300 条 ShareGPT prompt,每条用 Qwen3-32B(temperature 0.6、top-k 20、top-p 0.95)独立解码 200 次,其中两条完全确定(200 次都返回同一个 8-token 答案);其余 298 条上,高斯公式 σ√(2\u002Fπ) 算出的每-prompt MAE 与实测差距在 1.9% 以内,概率质量中部差距不超过 2 个百分点,残余偏离(中位偏度 +0.27、超额峰度 +0.35)正是采样噪声修正项预言的方向。\n\n## 哪些模型「话长」更飘\n\nα、b 谁大谁小,论文给出五模型表(均为 ShareGPT、300 prompt×200 解码):Qwen3-4B-Instruct 最稳,α=0.1068;关掉 thinking 的 Qwen3-4B 掉到 0.2029;Llama3-8B 为 0.2042;Qwen3-32B 为 0.1486。最有意思的是 DeepSeek-R1-distilled-thinking:斜率不高(0.1156),截距却高达 147.6——作者把它读作思维链固有的探索随机性,哪怕最终答案很短,推理路径的长度也在大幅波动。方差不同,天花板就不同:Llama3-8B 平均 σ 只有 112,Qwen3-30B-A3B 则到 261。\n\n## 公平重测之后\n\n把模型(Qwen3-32B)、数据(约 1.8 万条 ShareGPT 训练、18000–18300 共 300 条测试)、采样参数全部钉死,八种预测器同台:最简单的 MLP-Concat(拼接全部层最后 token 的隐藏向量,约 10 万维喂给 3 层 MLP)拿到 MAE 169.31,加 LayerNorm 后 168.13,是八种里最好的;三个 MoE 变种落在 170.70–177.91;Soft-Label、FiLM 在 180 上下。而 EGTP 熵引导池化——它在 ForeLen 上报过相对基线最高 29.16% 的 MAE 降幅——在这里掉到 466.79,连最朴素的 MLP-Mean(181.21)都远远不如。论文的解释:当 token 级熵分布与长度信号不对齐,熵加权聚合反而注入噪声。理论底线同时算清了:300 条测试 prompt 的平均 σ 为 162.3(中位 133.9,10% 超过 300),对应高斯噪声地板 129.5;最好的预测器 168.1,超出地板 38.6 个 token,而这部分富余至多相当于系统误差的 30%。Acc@100 的理论天花板 57.9%,实测最高 48.9%,差 8.9 个点;想要 90% 的 Acc@100,得先把 σ 压到 61 以下。跨域同样贴着这条线走:LMSYS-200K 训练、ShareGPT 测试,MAE 182.40;LeetCode 训练、HumanEval 测试,MAE 29.28、Acc@100 96.28%——域对齐了就轻松,不对齐就贴地板。\n\n## 所以呢:先看 σ,再谈预测器\n\n论文结论很直接:prefill 侧长度预测是噪声受限问题,系统误差压过模型误差,复杂架构的边际收益趋零甚至为负。想提升,路只有两条:控制生成本身的方差(更低温度、约束解码、prompt 设计),或在解码开始后做迭代式重估——每 50–100 token 更新一次估计,理论上每 token 都更新时剩余长度退化为确定性倒计时、MAE 界趋零,代价是每次更新多一次编码器前向。评测层面,作者建议固定模型、数据与采样参数,报告每-prompt σ,并把相对误差当货币。论文也自曝短板:特征与标签由 PD 分离管线的不同 worker 产出,批大小与数值行为未严格受控;训练标签是单次解码、自带噪声。代码以 MIT 协议开源在 [GitHub](https:\u002F\u002Fgithub.com\u002FShwStone\u002FDecodingPredict),论文见 [arXiv:2610.40541](https:\u002F\u002Farxiv.org\u002Fabs\u002F2610.40541)。下次看到解码长度预测的新榜,先问一句:这局的 σ 是多少?","https:\u002F\u002Farxiv.org\u002Fabs\u002F2610.40541","7437aeb9-930c-4866-a2e9-48003c1a792b",[11,15,18,21],{"id":12,"name":13,"slug":13,"description":14,"color":14},"120fa59a-ff6f-4537-9bf5-f818df636a0e","benchmark",null,{"id":16,"name":17,"slug":17,"description":14,"color":14},"0ef8513a-0a26-42f0-b6f9-5b6dadded45c","efficiency",{"id":19,"name":20,"slug":20,"description":14,"color":14},"0a93ec8e-ea39-4693-81de-563ca8c173f7","inference",{"id":22,"name":23,"slug":23,"description":14,"color":14},"01598627-1ea6-4b27-a5d8-874971571a71","llm",[25],{"id":26,"lang":27,"title":28,"summary":29,"content":30},"bc008b7f-fdc7-4032-9d98-6925df7191dc","en","Decode-length prediction hits a sampling-noise floor","Decode length is per-prompt normal; a plain MLP sits within 30% of the 129.5 MAE noise floor, while EGTP degrades to 466.79 under a fixed setup.","Inference stacks quietly depend on a small prediction: before the model emits its first token, a scheduler wants to know how long the reply will be, because continuous batching uses that estimate to group requests. The dominant recipe trains a regressor on prefill-phase hidden states and outputs a length before decoding starts. The method list has grown; the numbers have not converged. The paper \"On the Fundamental Limits of Prefill-Based Decoding Length Prediction\" (arXiv:2610.40541, posted October 29) opens with an uncomfortable observation: the same predictor architecture reports MAE around 80 on ForeLen's long-sequence split with Qwen2.5-7B, degrades past 150 on ShareGPT with Qwen3-32B, and past 260 on RL-sampled data — a three-fold swing with the predictor untouched.\n\n## The ceiling is drawn by a normal distribution\n\nThe authors locate the root cause inside autoregressive sampling itself: decode the same prompt twice and the lengths differ. They model decoding as a stopped stochastic process. Each step carries a stopping probability that is not constant but ramps up over hundreds of steps; because response length accumulates from many segment-level local decisions, a central limit theorem kicks in, making the per-prompt decode length asymptotically normal with an explicit error term. The variance obeys an affine law σ=αμ+b, whose slope α approximately equals the coefficient of variation of the number of response segments. A counterfactual sharpens the point: if the stopping probability were constant, length would be geometric, σ\u002Fμ would tend to 1, and every predictor would face an irreducible relative MAE of about 74%; the measured σ\u002Fμ≈0.15 rules that darker world out. The validation is thorough: 300 ShareGPT prompts, each decoded 200 independent times with Qwen3-32B (temperature 0.6, top-k 20, top-p 0.95); two of the 300 are fully deterministic (all 200 decodes return the same 8-token answer). Across the remaining 298, the Gaussian value σ√(2\u002Fπ) reproduces the per-prompt MAE to within 1.9%, the central mass to within two percentage points, and the residual departures (median skewness +0.27, excess kurtosis +0.35) point the way the sampling-noise correction predicts.\n\n## Which models drift\n\nTable 1 fits (α, b) per model on ShareGPT at 300 prompts × 200 decodes. Qwen3-4B-Instruct is the most stable (α=0.1068); Qwen3-4B with thinking disabled drops to 0.2029; Llama3-8B sits at 0.2042; Qwen3-32B at 0.1486. The most telling row is DeepSeek-R1-distilled-thinking: a modest slope (0.1156) paired with an unusually large intercept (147.6), which the authors read as the exploratory randomness intrinsic to chain-of-thought generation — even when the final answer is short, the reasoning path length swings widely. Different variance means different ceilings: Llama3-8B averages σ of 112, while Qwen3-30B-A3B reaches 261.\n\n## After a fair retest\n\nWith the model (Qwen3-32B), the data (roughly 18,000 ShareGPT prompts for training, IDs 18000–18300 as the 300-prompt test set), and the sampling parameters all pinned, eight predictors share one stage. The simplest, MLP-Concat (concatenating the last-token hidden vector of all layers, roughly 100k dimensions, into a 3-layer MLP), scores MAE 169.31; adding LayerNorm improves it to 168.13, the best of the eight. Three MoE variants land between 170.70 and 177.91; Soft-Label and FiLM hover around 180. EGTP-style entropy-guided pooling — the method that reported up to 29.16% MAE reduction over baselines on ForeLen — collapses to 466.79 here, far behind even the plain MLP-Mean at 181.21. The authors' explanation: when the token-level entropy distribution does not align with the length signal, entropy-weighted aggregation injects noise. The theoretical floor is settled in the same table: the 300 test prompts average σ of 162.3 (median 133.9; 10% of prompts exceed 300), which puts the Gaussian noise floor at 129.5; the best predictor at 168.1 exceeds it by 38.6 tokens, and the paper proves that surplus is at most 30% of the system error. For tolerance metrics, the Acc@100 ceiling is 57.9% while the measured best is 48.9% — 8.9 points short; reaching a 90% Acc@100 would require pushing σ below 61. Cross-domain results hug the same line: training on LMSYS-200K and testing on ShareGPT yields MAE 182.40; training on LeetCode and testing on HumanEval yields 29.28 with Acc@100 at 96.28% — align the domain and the task gets easy, misalign it and you sit on the floor.\n\n## So what: read σ before buying a predictor\n\nThe conclusion is blunt: prefill-side length prediction is a noise-limited problem where system error dominates model error, and architectural elaboration buys marginal or negative returns. Improvement has two honest paths — control the variance of generation itself (lower temperature, constrained decoding, prompt design), or re-estimate iteratively once decoding has begun (updates every 50–100 tokens; in the every-token limit the remaining length collapses to a deterministic countdown and the MAE bound tends to zero, at the price of an encoder forward pass per update). For benchmarks, the authors recommend fixing model, data, and sampling, reporting per-prompt σ, and treating relative error as the currency. The paper also discloses its own weaknesses: features and labels came from different workers of a PD-disaggregated pipeline with batching and numerical behavior not strictly controlled, and training labels are single decodes that carry noise of their own. Code is open-sourced under MIT on [GitHub](https:\u002F\u002Fgithub.com\u002FShwStone\u002FDecodingPredict); the paper is at [arXiv:2610.40541](https:\u002F\u002Farxiv.org\u002Fabs\u002F2610.40541). Next time a decode-length predictor posts a new low on a leaderboard, ask first: what is σ in that setup?","prefill-decode-length-prediction-limits","2026-11-01T23:09:55Z","2026-11-01T23:09:59.131952Z","2026-11-01T23:09:59.131960Z",true,"agent",116,[39,48],{"slug":40,"tag_slug":40,"title_zh":41,"title_en":42,"intro_zh":43,"intro_en":44,"id":45,"is_active":35,"created_at":46,"modified_at":47},"ai-for-science","AI for Science 2026：从 UniPert 到 GPT-Rosalind 的硬核进化","AI for Science 2026: from UniPert to GPT-Rosalind","生命科学、化学材料、物理世界模型——AI 正在从\"语言工具\"变成\"实验伙伴\"。本专题收录 AI 在三大科学方向的关键节点：UniPert 统一基因与化学扰动空间、GPT-Rosalind 端到端生命科学推理、达摩院 AI 智能体 28 小时找到 4 种超导新材料、Anthropic Claude Science 把工作台做成标准品。","From language tool to lab partner — AI is reshaping life sciences, chemistry\u002Fmaterials, and physical world models. This topic covers the key milestones: UniPert unifying genetic-chemical perturbation spaces, GPT-Rosalind's end-to-end life-sciences reasoning, DAMO's AI agent discovering 4 superconducting materials in 28 hours, and Anthropic's Claude Science workbench going mainstream.","988a4300-5fab-41c4-b5d8-63711a2dc757","2026-09-10T01:34:15.296649Z","2026-09-10T01:34:15.296663Z",{"slug":49,"tag_slug":49,"title_zh":50,"title_en":51,"intro_zh":52,"intro_en":53,"id":54,"is_active":35,"created_at":55,"modified_at":56},"h3-series","MiniMax H3 系列：从开源权重到 35 倍吞吐","MiniMax H3 Series: from open weights to 35x throughput","MiniMax H3 自 2026 年 8 月开源以来节奏密集：官方把生成、参考与编辑收回一个模型；ComfyUI 当天压进 RTX 3060；摩尔线程 3 小时完成国产 GPU 适配；fal 后训练版把吞吐拉到 35 倍；FastH3 蒸馏再砍推理成本。本专题持续追踪 H3 的发布—开源—蒸馏—部署全链路。","Since MiniMax open-sourced H3 in August 2026 the pace has been relentless: one unified omni-modal model, same-day ComfyUI support down to an RTX 3060, a 3-hour Day-0 port to Moore Threads GPUs, fal's post-trained H3 Max at 35x throughput, and FastH3 distillation cutting inference cost further. This topic tracks the full H3 chain — release, open weights, distillation, deployment.","83ef0daa-3c31-4cb3-86ed-e5ee58654d5f","2026-09-08T07:33:19.942193Z","2026-09-08T07:33:19.942209Z",{"items":58},[59,64,69,74,79,84],{"id":60,"title":61,"news_slug":62,"published_at":63},"a550dbc5-240b-4747-b625-9aa874aad444","工具拉满的AI停在0.48倍:会提问的做到1.13倍","kernelseek-question-driven-cuda-kernel-agent","2026-11-03T09:05:00+00:00",{"id":65,"title":66,"news_slug":67,"published_at":68},"c2e41947-c219-4fbe-83f2-5dd9e4501e73","信号处理审计:推理链置信度评分其实量的是长度","token-confidence-chain-length-audit","2026-10-28T11:12:21+00:00",{"id":70,"title":71,"news_slug":72,"published_at":73},"98bf8c61-e559-4f96-9c3a-e7049b493f6c","小模型自信≠正确:4.2万次实测拆解LLM路由","confidence-based-llm-routing-multilingual","2026-10-24T19:30:00+00:00",{"id":75,"title":76,"news_slug":77,"published_at":78},"13ed0c94-9bea-440c-93c4-771c38ba9834","sink 消失后，KV 驱逐改看 value 几何","valuediff-value-geometric-kv-eviction","2026-09-22T17:05:00+00:00",{"id":80,"title":81,"news_slug":82,"published_at":83},"7cacddc6-fa02-4de9-84a2-c3320e225571","因果归因剪枝 CAP：让 LLM 推理能力不再随稀疏化而流失","cap-causal-attribution-pruning-arc-61pct","2026-06-20T22:14:08.915874+00:00",{"id":85,"title":86,"news_slug":87,"published_at":88},"9a1e1c85-60eb-47c6-92b5-bace1746e217","大模型竞争进入下半场：从「比参数」到「比部署」——2026年5月技术格局观察","llm-2nd-half-deploy-vs-params-may-2026","2026-05-25T05:15:00+00:00"]