[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-token-confidence-chain-length-audit":3,"topics-all":38,"news-related-c2e41947-c219-4fbe-83f2-5dd9e4501e73":57},{"id":4,"title":5,"summary":6,"content":7,"original_url":8,"source_id":9,"tags":10,"translations":24,"news_slug":31,"published_at":32,"created_at":33,"modified_at":34,"is_published":35,"publish_type":36,"image_url":14,"view_count":37},"c2e41947-c219-4fbe-83f2-5dd9e4501e73","信号处理审计:推理链置信度评分其实量的是长度","论文用信号处理重做 token confidence 打分,发现 DeepConf 式链筛选评分主要反映链长而非正确性:在 DeepSeek-R1-Distill-Qwen-1.5B 的 8000 条推理链上,按长度排序即匹敌,剔除长度后对正确性无信息量,早停也不如直接少采样几条链。","DeepConf 是 Meta AI 与 UCSD 团队 2025 年 8 月提出的并行思考方法,原论文《Deep Think with Confidence》口径抢眼:AIME 2025 上最高 99.9% 准确率,相比完整并行思考最多省下 84.7% 的生成 token,无需额外训练或调超参,可接进 vLLM。([DeepConf 原论文](https:\u002F\u002Farxiv.org\u002Fabs\u002F2508.15260))\n\n核心机制是一个叫 token confidence 的量:模型每一步输出词表上的概率分布,取前 20 个候选概率的平均负对数得到该 token 的置信度;再用 2048 token 的滑动窗口取平均,取整条链上这个平滑序列的最小值作为该链的「最低组置信度」。低分链在投票前被丢掉,分数跌破阈值就提前停掉。\n\n## 把置信度当成信号来量\n\n10 月 27 日提交的一篇审计论文([arXiv:2610.36709](https:\u002F\u002Farxiv.org\u002Fabs\u002F2610.36709),作者 Pragna Mallappa,Keshav Memorial Institute of Technology)换了个视角:token 置信度既然是逐 token 记下的序列,那就把它当成音频或无线电意义上的离散信号,再用信号处理的标准工具去体检。([审计论文全文](https:\u002F\u002Farxiv.org\u002Fhtml\u002F2610.36709v1))\n\n转换很直接。滑动窗口平均就是一个箱式滤波器,属于低通滤波器:保留慢变化、压制快波动;取最小值则是一个检测器。DeepConf 由此被拆成「滤波 + 检测」两个可单独检验的部件。论文还指出,2048 这个窗口长度并非从对信号的测量中得来,只是滤波器空间里的一个点。\n\n## 长度效应有多强\n\n实验规模不小:DeepSeek-R1-Distill-Qwen-1.5B,一个 15 亿参数的蒸馏推理模型;MATH500 前 200 题加 GSM8K 前 300 题,每题采 16 条链,温度 0.6、top-p 0.95、最多 8192 个生成 token,vLLM 跑在单张 RTX 4090 上,两次运行共生成 2200 万 token、耗时 84 分钟,共 8000 条链。\n\n关键对比是 AUROC(组内比较,衡量一个分数把好链排在坏链前面的能力):DeepConf 的 2048 箱式滤波器在 MATH500 上是 0.585,而单纯按链长排序是 0.715,前者被反超 0.130(区间 0.019 到 0.257);GSM8K 上长度排序 0.636,与 DeepConf 的 0.625 持平。\n\n再把长度的影响直接扣掉:论文对每道题的分数和链长对数各减去组内均值,拟合一个全局斜率,再算斜率解释不了的残差 AUROC。最低组置信度在 MATH500 上掉到 0.486(区间 0.372 到 0.607),GSM8K 上是 0.520(区间 0.463 到 0.575),两个基准都落在随机水平。\n\n## 机制:谱是 1\u002Ff 的,取最小值必然偏向长链\n\n为什么长度会混进来?论文测了置信度信号的功率谱。正确链与错误链的谱斜率在 MATH500 上是 -0.80 和 -0.76,GSM8K 上是 -0.70 和 -0.68,都接近 1\u002Ff 过程的 -1,意味着方差主要堆在慢漂移里。低通滤波器几乎削不掉这种方差,于是链越长,平滑后取最小值越容易碰到低点,下限随长度下走。\n\n论文用「同一条链内部的前缀」把链间差异消掉:前缀从 2048 增长到 6144 token 时,MATH500 上最低组置信度下降 0.74(0.65 到 0.84),GSM8K 下降 0.40(0.33 到 0.48);把同一条链的 token 打乱(值不变、谱变平),降幅缩到 0.15;只把傅里叶相位随机化(谱完全保留),降幅是 1.10 和 0.75。对照白噪声:平均 w 个样本会让标准差缩到约 1\u002F√w,w=2048 时约 45 倍,最小值几乎不动;1\u002Ff 信号不同,慢漂移能穿过滤波。\n\n## 谱里还剩一点信号,但带不动投票\n\n有一条残余信号是长度解释不了的:错误链的谱更平,快波动占比更高——两者斜率之差在 MATH500 上是 0.041(区间 0.018 到 0.064),GSM8K 上是 0.020(0.007 到 0.032)。约 0.06 cycles\u002Ftoken(周期约 18 个 token)的频带在正确链上功率更高,AUROC 0.64(0.53 到 0.74)和 0.57(0.52 到 0.62);周期两三 token 的最快频带则在错误链上更强,AUROC 0.38(0.27 到 0.49)和 0.39(0.34 到 0.44)。论文说这可能意味着正确链的节奏更稳,但也明说未经进一步验证。\n\n论文还测了「反思词」的响应:模型写出 Wait 时置信度骤降约 1.3 个标准差(MATH500)、1.0(GSM8K),约 50 个 token 后恢复;错误链的反思词更多,每千 token 6.7 对 5.8、7.7 对 6.9。\n\n问题是残余信号太弱。高频能量检测器(不需要训练)在 MATH500 上保住 0.554、GSM8K 上 0.549;学出来的滤波器组在 GSM8K 上 0.601、MATH500 上只有 0.509。把高频分和长度分等权相加,AUROC 没有一个基准能超过单用长度分。\n\n## 早停也没赢过「少采几条链」\n\n论文把提前停止改写成变化检测:用几条热身链定出置信度的基线均值和标准差,再用经典的 CUSUM 累积和检测器盯着标准化后的置信度,跌破阈值就停。结论在 MATH500 上很清楚——花一半的钱,直接少采样几条链拿到 90.0%,检测器这边只有 88.4%;花七成钱时,少采样 90.8%,CUSUM 89.4%,箱式滤波器 88.7%。GSM8K 上从 40% 到 70% 成本区间内,所有方法差距不超过 0.6 个百分点。固定 token 上限明显更差,它砍掉的正是那些本该投出正确答案的长链。\n\n## 这篇论文没说什么\n\n得说清楚边界。作者用的是 15 亿参数的蒸馏模型和两个数学基准,MATH500 上好链坏链混杂的题只有 31 道,区间因此偏宽。DeepConf 原文的收益来自大得多的模型和更难的竞赛题,那里链更长、长度与正确率的关系也可能不同。论文的原话是:它并不显示置信度筛选在那种设定下失效,只显示在自己的数据上这个分数没有超出长度的贡献。它也只用一个温度、一个 k 值。\n\n## 所以呢\n\n这篇论文的价值不在于推翻谁,在于它把一条被广泛复用的工程惯例拆开验了一遍:一个看起来在测「模型有多确信」的分数,可能只是在测「这条链有多长」。论文的建议很具体——以后做置信度链选择,应把链长作为基线一并报告,并给出剔除长度后的 AUROC,否则长度效应会冒充成置信度效应。\n\n对做推理服务的人,这也意味着一个更朴素的降本思路值得先试:与其部署一套复杂的置信度调度,不如先少采样几条链,看钱省在哪儿。","https:\u002F\u002Farxiv.org\u002Fabs\u002F2610.36709","7437aeb9-930c-4866-a2e9-48003c1a792b",[11,15,18,21],{"id":12,"name":13,"slug":13,"description":14,"color":14},"120fa59a-ff6f-4537-9bf5-f818df636a0e","benchmark",null,{"id":16,"name":17,"slug":17,"description":14,"color":14},"0ef8513a-0a26-42f0-b6f9-5b6dadded45c","efficiency",{"id":19,"name":20,"slug":20,"description":14,"color":14},"0a93ec8e-ea39-4693-81de-563ca8c173f7","inference",{"id":22,"name":23,"slug":23,"description":14,"color":14},"01598627-1ea6-4b27-a5d8-874971571a71","llm",[25],{"id":26,"lang":27,"title":28,"summary":29,"content":30},"16fb32ad-1bc8-4a4e-a36c-ee24a963de2e","en","Signal-Processing Audit: Token Confidence Is Mostly Chain Length","A signal-processing audit finds token confidence in reasoning chains mostly measures chain length: on 8000 chains, ranking by length alone matches it.","DeepConf is a parallel-thinking method proposed in August 2025 by a team from Meta AI and UCSD. The original paper made a striking set of claims: up to 99.9% accuracy on AIME 2025, up to 84.7% fewer generated tokens compared with full parallel thinking, no extra training and no hyperparameter tuning, and drop-in compatibility with serving frameworks such as vLLM. ([original paper](https:\u002F\u002Farxiv.org\u002Fabs\u002F2508.15260))\n\nThe mechanism rests on a quantity called token confidence. At every step the model produces a probability distribution over its vocabulary; taking the top 20 candidates and averaging their negative log probabilities yields a per-token confidence value. Those values are averaged over a sliding window of 2048 tokens, and the minimum of that smoothed sequence across the whole chain becomes the chain’s \"lowest group confidence\". Low-scoring chains get dropped before the vote; chains whose score falls below a threshold get stopped early. It is a clean idea: use the model’s own internal uncertainty as its own quality inspector.\n\n## Treating confidence as a signal\n\nAn audit paper submitted on 27 October (arXiv:2610.36709, by Pragna Mallappa of Keshav Memorial Institute of Technology) takes a different route to the same computation. Token confidences are recorded token by token, so they form a discrete-time signal in exactly the sense an audio or radio waveform does. That means the standard tools of signal processing can be pointed at it. ([full text](https:\u002F\u002Farxiv.org\u002Fhtml\u002F2610.36709v1))\n\nThe translation is straightforward. A sliding-window average is a boxcar filter with all-equal weights, hence a low-pass filter: it keeps slow changes and suppresses fast ones. Taking the minimum is a detector. DeepConf’s whole method decomposes into \"filtering, then detection\", two parts that can be tested separately. The paper also notes that the window length of 2048 was never derived from a measurement of the confidence signal; it is one point in a large space of filters.\n\n## How strong the length effect is\n\nThe experimental setup is substantial: DeepSeek-R1-Distill-Qwen-1.5B, a distilled reasoning model with 1.5 billion parameters; the first 200 problems of MATH500 plus the first 300 of GSM8K; 16 chains sampled per problem at temperature 0.6 with top-p 0.95 and a cap of 8192 generated tokens; vLLM on a single RTX 4090. The two runs generated 22.0 million tokens in 84 minutes of generation, 8000 chains in total.\n\nThe decisive comparison is AUROC, measured within a problem, which asks how often a score ranks a correct chain above an incorrect one. DeepConf’s 2048-token boxcar reaches 0.585 on MATH500, while simply ranking chains from shortest to longest reaches 0.715 — the length baseline beats the confidence score by 0.130, with a 95% interval of 0.019 to 0.257. On GSM8K the length baseline scores 0.636, statistically tied with DeepConf’s 0.625.\n\nNow subtract length directly. Within each problem the paper subtracts the mean from both the score and the log of chain length, fits a single global slope across all problems, and computes the AUROC of what that slope fails to explain. DeepConf’s lowest group confidence falls to 0.486 on MATH500 (interval 0.372 to 0.607) and 0.520 on GSM8K (interval 0.463 to 0.575) — chance on both benchmarks. The plain mean confidence score behaves the same way.\n\n## The mechanism: a 1\u002Ff spectrum and a minimum operation\n\nWhy does length leak in? The paper measures the power spectrum of the confidence signal. The spectral slope is -0.80 for correct chains and -0.76 for incorrect chains on MATH500, and -0.70 and -0.68 on GSM8K. Both are close to the -1 of a 1\u002Ff process, meaning most of the variance sits in slow drifts. A low-pass filter removes almost none of that variance, so the longer the chain, the more chances the smoothed signal has to wander down — and the minimum over it keeps sliding lower.\n\nThe paper then removes all chain-to-chain differences by looking inside a single chain. As the prefix grows from 2048 to 6144 tokens, the lowest group confidence falls by 0.74 on MATH500 (0.65 to 0.84) and by 0.40 on GSM8K (0.33 to 0.48). Shuffling the tokens of the same chain — same values, flat spectrum — shrinks the fall to 0.15. Randomizing only the Fourier phases, which preserves the spectrum exactly, gives 1.10 and 0.75. The spectrum alone predicts a fall at least as large as the observed one. For contrast, averaging w samples of white noise shrinks the standard deviation by roughly the square root of w, about 45 times for w=2048, so the minimum barely moves; a 1\u002Ff signal behaves differently because its slow drift survives the filter.\n\n## A residual signal, but too weak to help\n\nSome signal survives that length cannot explain: incorrect chains have flatter spectra, meaning relatively more of their variance sits in fast fluctuations. The slope difference is 0.041 on MATH500 (interval 0.018 to 0.064) and 0.020 on GSM8K (0.007 to 0.032). Resolved into bands, the band around 0.06 cycles per token — a period of about 18 tokens, roughly a short sentence or one step of an equation — carries more power in correct chains, with AUROC 0.64 (0.53 to 0.74) and 0.57 (0.52 to 0.62). The fastest band, with periods of two to three tokens, goes the other way, with AUROC 0.38 (0.27 to 0.49) and 0.39 (0.34 to 0.44). One reading is that correct chains work through their reasoning at a steadier rhythm, though the paper is explicit that this reading has not been tested.\n\nThe paper also measures the response to reflection words. When a reasoning model writes something like \"Wait\" or \"Alternatively\", confidence drops sharply — to about 1.3 standard deviations below the chain mean on MATH500 and 1.0 on GSM8K — and recovers over roughly 50 tokens. Incorrect chains contain more reflection words per thousand tokens: 6.7 against 5.8 on MATH500 (a difference of 0.9) and 7.7 against 6.9 on GSM8K (a difference of 0.8).\n\nThe problem is that this residual is too weak. A high-frequency energy detector requiring no training retains an AUROC of 0.554 on MATH500 and 0.549 on GSM8K; a learned filter bank reaches 0.601 on GSM8K but only 0.509 on MATH500. Adding the high-frequency score to the length score with equal weight after standardization does not push the AUROC above length alone on either benchmark.\n\n## Early stopping does not beat sampling fewer chains\n\nThe paper recasts early stopping as change detection. A few warmup chains fix the baseline mean and standard deviation of token confidence, then a classical CUSUM detector watches the standardized signal and stops the chain when it crosses a threshold. On MATH500 the conclusion is blunt: at half the full cost, simply sampling fewer chains reaches 90.0% against 88.4% for the detector; at 70% of the cost, sampling fewer chains reaches 90.8%, CUSUM 89.4%, and the 2048-token boxcar 88.7%. On GSM8K all methods lie within 0.6 points of each other at every cost between 40% and 70%, which is roughly two problems out of 300. A fixed token cap is clearly worse on both benchmarks, because it removes long chains that would have voted correctly.\n\n## What the paper does not claim\n\nThe boundaries matter. The study uses a single 1.5-billion-parameter distilled model and two math benchmarks, and only 31 MATH500 problems are mixed enough to offer both correct and incorrect answered chains, which leaves wide intervals. DeepConf’s original gains are reported for much larger models on harder competition problems, where chains are longer and the relation between length and correctness may differ. The paper’s own wording is that its results do not show confidence filtering fails in that setting; they show that on this data the score adds nothing beyond length. It also uses a single sampling temperature and a single value of k, and the early-stopping results are simulated on already-finished chains.\n\n## So what\n\nThe value here is not in overturning somebody’s method. It is in taking a widely reused engineering habit apart and checking it: a score that appears to measure how sure the model is may simply be measuring how long the chain is. The recommendation is concrete — work on confidence-based chain selection should report chain length as a baseline and report AUROC with length removed, since otherwise a length effect can easily pass for a confidence effect.\n\nFor anyone running inference in production, there is a plainer takeaway worth trying first: before deploying a complicated confidence scheduler, sample fewer chains and find out where the money was actually going.","token-confidence-chain-length-audit","2026-10-28T11:12:21Z","2026-10-28T11:12:23.677532Z","2026-10-28T11:12:23.677540Z",true,"agent",180,[39,48],{"slug":40,"tag_slug":40,"title_zh":41,"title_en":42,"intro_zh":43,"intro_en":44,"id":45,"is_active":35,"created_at":46,"modified_at":47},"ai-for-science","AI for Science 2026：从 UniPert 到 GPT-Rosalind 的硬核进化","AI for Science 2026: from UniPert to GPT-Rosalind","生命科学、化学材料、物理世界模型——AI 正在从\"语言工具\"变成\"实验伙伴\"。本专题收录 AI 在三大科学方向的关键节点：UniPert 统一基因与化学扰动空间、GPT-Rosalind 端到端生命科学推理、达摩院 AI 智能体 28 小时找到 4 种超导新材料、Anthropic Claude Science 把工作台做成标准品。","From language tool to lab partner — AI is reshaping life sciences, chemistry\u002Fmaterials, and physical world models. This topic covers the key milestones: UniPert unifying genetic-chemical perturbation spaces, GPT-Rosalind's end-to-end life-sciences reasoning, DAMO's AI agent discovering 4 superconducting materials in 28 hours, and Anthropic's Claude Science workbench going mainstream.","988a4300-5fab-41c4-b5d8-63711a2dc757","2026-09-10T01:34:15.296649Z","2026-09-10T01:34:15.296663Z",{"slug":49,"tag_slug":49,"title_zh":50,"title_en":51,"intro_zh":52,"intro_en":53,"id":54,"is_active":35,"created_at":55,"modified_at":56},"h3-series","MiniMax H3 系列：从开源权重到 35 倍吞吐","MiniMax H3 Series: from open weights to 35x throughput","MiniMax H3 自 2026 年 8 月开源以来节奏密集：官方把生成、参考与编辑收回一个模型；ComfyUI 当天压进 RTX 3060；摩尔线程 3 小时完成国产 GPU 适配；fal 后训练版把吞吐拉到 35 倍；FastH3 蒸馏再砍推理成本。本专题持续追踪 H3 的发布—开源—蒸馏—部署全链路。","Since MiniMax open-sourced H3 in August 2026 the pace has been relentless: one unified omni-modal model, same-day ComfyUI support down to an RTX 3060, a 3-hour Day-0 port to Moore Threads GPUs, fal's post-trained H3 Max at 35x throughput, and FastH3 distillation cutting inference cost further. This topic tracks the full H3 chain — release, open weights, distillation, deployment.","83ef0daa-3c31-4cb3-86ed-e5ee58654d5f","2026-09-08T07:33:19.942193Z","2026-09-08T07:33:19.942209Z",{"items":58},[59,64,69,74,79,84],{"id":60,"title":61,"news_slug":62,"published_at":63},"a550dbc5-240b-4747-b625-9aa874aad444","工具拉满的AI停在0.48倍:会提问的做到1.13倍","kernelseek-question-driven-cuda-kernel-agent","2026-11-03T09:05:00+00:00",{"id":65,"title":66,"news_slug":67,"published_at":68},"932bddfc-f9c2-43c9-9eec-676df0a7bf23","解码长度预测撞上天花板:误差大头是采样噪声,不是模型","prefill-decode-length-prediction-limits","2026-11-01T23:09:55+00:00",{"id":70,"title":71,"news_slug":72,"published_at":73},"98bf8c61-e559-4f96-9c3a-e7049b493f6c","小模型自信≠正确:4.2万次实测拆解LLM路由","confidence-based-llm-routing-multilingual","2026-10-24T19:30:00+00:00",{"id":75,"title":76,"news_slug":77,"published_at":78},"13ed0c94-9bea-440c-93c4-771c38ba9834","sink 消失后，KV 驱逐改看 value 几何","valuediff-value-geometric-kv-eviction","2026-09-22T17:05:00+00:00",{"id":80,"title":81,"news_slug":82,"published_at":83},"7cacddc6-fa02-4de9-84a2-c3320e225571","因果归因剪枝 CAP：让 LLM 推理能力不再随稀疏化而流失","cap-causal-attribution-pruning-arc-61pct","2026-06-20T22:14:08.915874+00:00",{"id":85,"title":86,"news_slug":87,"published_at":88},"9a1e1c85-60eb-47c6-92b5-bace1746e217","大模型竞争进入下半场：从「比参数」到「比部署」——2026年5月技术格局观察","llm-2nd-half-deploy-vs-params-may-2026","2026-05-25T05:15:00+00:00"]