[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-llm-expert-disagreement-revision-selectivity":3,"topics-all":38,"news-related-9649aae9-2aef-4668-96e8-d8ced4378a71":57},{"id":4,"title":5,"summary":6,"content":7,"original_url":8,"source_id":9,"tags":10,"translations":24,"news_slug":31,"published_at":32,"created_at":33,"modified_at":34,"is_published":35,"publish_type":36,"image_url":14,"view_count":37},"9649aae9-2aef-4668-96e8-d8ced4378a71","专家头衔一贴，模型就多改口：Claude掉16.1点","一项被 NeurIPS 2026 接收的审计，用 1,989 道带证据的判断题测了一件事：当另一个助手挂着「专家」头衔唱反调时，模型会更频繁地改口，但选择性远不够——Claude-Sonnet-4 准确率掉了 16.1 个点。论文指出改口次数是判断改对改错的坏代理指标：90% 初始准确率要求改错九次才对一次才打平。","给模型改答案，业界习惯用「反转次数」衡量——改得越多说明它越听进去了。但佐治亚理工的研究者用两轮审计算清这笔账后发现：改口率是判断改对改错的坏代理指标。一个挂着「领域专家」头衔的助手哪怕给出错答案，模型也更愿意跟着改。\n\n这套审计出自 arXiv 的《When Expert Disagreement Hurts: Prestige and Selectivity in LLM Decision Revision》（https:\u002F\u002Farxiv.org\u002Fabs\u002F2611.16694），作者是 Georgia Institute of Technology 的 Yupeng Tang 和 Mingfeng Lin，已被 NeurIPS 2026 接收。核心问题很窄：当证据和初始答案都锁死，只改变唱反调来源身上挂的标签，模型的决策会怎么变，这变化到底是帮忙还是添乱。\n\n## 1,989 道题，锁死证据只换标签\n\n审计用的是两轮探针。第一轮模型对同一份题目加证据给出是\u002F否判断，这个答案直接复用到第二轮的三个分支：自己反思、面对匿名反对、面对专家-labeled 反对。三分支共享同一问题、同一证据、同一初始答案，差异只能归因于标签本身。\n\n题库由三个专家标注数据集拼成，共 1,989 条二分类样本：PubMedQA 医学题去掉模棱两可的 Maybe 后剩 890 条，SciFact 科学论断保留二元标签剩 299 条，ContractNLI 法律假设去掉 Neutral 后剩 800 条。标签分布是 1,379 个 Yes 对 610 个 No。跑全量基准的五个模型为 GPT-4o-mini、Claude-Sonnet-4、Gemini-2.0-Flash、Llama-3.1-8B 和 Mistral-7B。\n\n## 90% 的准确率要求九倍的选择性\n\n论文的判据是精确的。设初始准确率为 a，模型把对的改错的比例是 h、把错的改对的比例是 b，那么修改只有在 b\u002Fh > a\u002F(1-a) 时才提升准确率。初始准确率 90% 时门槛是九倍——模型得改错九次、才对一次才刚好打平。\n\n实测里所有模型都朝反方向走。论文把各模型的选择性 b\u002Fh 与盈亏平衡点的比值画成图，GPT-4o-mini 是 0.25，Claude-Sonnet-4 是 0.19，Gemini-2.0-Flash 同样是 0.25；两个开源权重模型 Llama-3.1-8B 和 Mistral-7B 分别是 0.82 和 0.81，只差约两成没够线。三款闭源模型需要再提高三到五倍选择性才能止损。\n\n代价直接落在表里：专家-labeled 反对下，Claude-Sonnet-4 的正确变错误占 19.9%、错误变正确只有 3.8%，准确率掉 16.1 个点；GPT-4o-mini 掉 11.0 个点，Gemini-2.0-Flash 掉 6.8 个点，两个开源模型分别只掉 2.3 和 1.9 个点。三款闭源模型的配对检验 p 值都在 2×10⁻²⁰ 以下。\n\n有意思的对照是 GPT-4o-mini 的反转率：初始答对的题，匿名反对下有 10.0% 被改掉，挂上专家头衔后变成 16.9%——同样的答案，只是反对者换了个身份。\n\n## 标签加了反转，不一定掉分\n\n论文把「标签对反转的影响」和「标签对准确率的影响」拆开算，这是全文最关键的一层。GPT-4o-mini 的标签效应是反转率 +6.90 个点、准确率 −5.19 个点；Llama-3.1-8B 是 +5.18 和 −3.77。但 Claude-Sonnet-4 反转率涨了 +2.62，准确率却是 +0.30（95% 置信区间 [−1.01, 1.61]）；Gemini-2.0-Flash 反而反转得更少（−1.86），准确率还涨了 +2.37。可见专家标签造成的损失不能全记在标签头上——Claude-Sonnet-4 和 Gemini-2.0-Flash 的准确率下滑来自「反对」本身，不是「专家」两个字。\n\n论文还做了道更严格的控制：让匿名来源和专家来源给出完全相同的两句话理由，不加填充，两个提示词只差描述来源的那一句。结果标签仍让 GPT-4o-mini、Claude-Sonnet-4、Gemini-2.5-Flash 的反转率分别上升 4.82、5.62、4.42 个点，准确率分别下降 4.82、3.21、2.81 个点，其中只有 GPT-4o-mini 的准确率区间排除了零。\n\n「来源可靠性」这句话本身也有分量。在带填充的对照套件里，给专家描述加上「但他们也可能错」，五个受测模型反转率全部下降，其中 Claude-Sonnet-4 从 7.8% 掉到 1.4%，四个模型达到显著；加上「有过往战绩」的描述则让五个模型反转率全部上升，两个显著。\n\n## 别指望 prompt 一句话能救\n\n论文测了修复手段，效果都有限。提示词里的证据门（evidence gate）要求模型只在证据支持时才改，在只给答案的套件里把三个闭源模型的正确变错误从 171 次压到 76 次；但换成带论证的反对就打折——Claude-Sonnet-4 从 24.7% 降到 20.2%（p = 0.013），GPT-4o-mini 从 28.1% 降到 24.6%（p = 0.077），Gemini-2.5-Flash 和四个新模型没有显著变化。\n\n论文还从 token 分数层面看了机制：在两个开源权重模型上，Llama-3.1-8B 有 63.2% 的反转、Mistral-7B 有 26.8% 的反转，发生在 Yes\u002FNo 对数概率根本没变号的情况下。也就是说，答案是被渐变的分数推移跨过决策边界，而不是模型真算出了相反的结论。\n\n这不是孤例。另一组来自 Amazon、Georgia Tech 和 Illinois Institute of Technology 的研究《Easier to Mislead Than to Correct: Harmful and Beneficial Revision in LLM Conformity》（https:\u002F\u002Farxiv.org\u002Fabs\u002F2606.01637）在四个开源权重模型、七个问答数据集上做了同类实验，方向一致：全错的同伴把有害改判从 15.6% 推到 62.9%（+47.3 个点），全对的同伴只把有益改判从 32.7% 推到 51.5%（+18.8 个点）；权威标签同样让模型更倾向选被背书的答案，不管它对不对。该文还提到，思维链和反思这类通用干预手段不能可靠压住有害改判。\n\n## 所以呢\n\n现在很多多智能体流程把「另一个 agent 不同意」当纠错信号，甚至给不同 agent 挂上专家、组长之类的角色标签。这篇论文把代价算清了：模型确实会更频繁地改口，但选择性远不够，方向一旦错，损失比收益大得多。\n\n值得注意的是，在这张表里 Claude-Opus-4.8 独自越过盈亏平衡线（比值 1.33），准确率也没下滑——因为它几乎不改口。这提示了一件事：分歧式流水线里真正稀缺的不是让模型更听劝，而是让它在证据没变时守住答案。论文的建议也很直接：不要按反转次数或最终准确率单独评估修订流水线，要按模型和协议分别报告漂移、分歧敏感度、标签的双重效应和修订的选择性。\n\n代码和审计协议已开源在 https:\u002F\u002Fgithub.com\u002Fyupengtang\u002Fwhen-expert-disagreement-hurts","https:\u002F\u002Farxiv.org\u002Fabs\u002F2611.16694","7437aeb9-930c-4866-a2e9-48003c1a792b",[11,15,18,21],{"id":12,"name":13,"slug":13,"description":14,"color":14},"6ad31a14-c0da-42df-81fd-564281f768db","agentic-ai",null,{"id":16,"name":17,"slug":17,"description":14,"color":14},"1fcfaaf2-67de-43d3-9e35-5784852fec60","ai-safety",{"id":19,"name":20,"slug":20,"description":14,"color":14},"120fa59a-ff6f-4537-9bf5-f818df636a0e","benchmark",{"id":22,"name":23,"slug":23,"description":14,"color":14},"01598627-1ea6-4b27-a5d8-874971571a71","llm",[25],{"id":26,"lang":27,"title":28,"summary":29,"content":30},"91007454-7bd3-43d4-8d3f-5f7f0e95063d","en","Expert Labels Push LLMs to Revise More, and Lose Points","Models revise more when a dissenter wears an expert label, but too selectively: across 1,989 items, Claude-Sonnet-4 loses 16.1 accuracy points.","The industry habit of measuring revision by counting how often a model changes its answer has a serious flaw, and a two-pass audit makes the accounting exact. When a dissenting voice is introduced as a domain expert, models flip more often, but the flips are far less selective than a high initial accuracy demands, so the net effect is a loss.\n\nThe work is `When Expert Disagreement Hurts: Prestige and Selectivity in LLM Decision Revision` (https:\u002F\u002Farxiv.org\u002Fabs\u002F2611.16694) by Yupeng Tang and Mingfeng Lin of Georgia Institute of Technology, listed on its own pages as a NeurIPS 2026 paper. The question it asks is narrow: with the evidence and the model's first answer held fixed, how much does the label attached to a disagreeing source change the decision, and does that change help or hurt?\n\n## 1,989 items, evidence locked, only the label changes\n\nThe audit uses a two-pass probe. In pass one the model gives a Yes or No answer to a question with attached evidence. That exact answer is reused across three pass-two branches: neutral reflection, anonymous disagreement, and disagreement attributed to a domain expert. All three share the same question, the same evidence and the same initial answer, so any difference can only come from how the source is described.\n\nThe benchmark has 1,989 binary items assembled from three expert-annotated sources: PubMedQA medical questions with the ambiguous Maybe labels dropped (890), SciFact scientific claims with binary labels kept (299), and ContractNLI legal hypotheses with Neutral dropped (800). The final label split is 1,379 Yes against 610 No. Five models run the full benchmark: GPT-4o-mini, Claude-Sonnet-4, Gemini-2.0-Flash, Llama-3.1-8B and Mistral-7B.\n\n## Ninety percent accuracy demands nine times the selectivity\n\nThe paper's criterion is exact. Let initial accuracy be `a`, let the model revise correct answers at rate `h` and wrong answers at rate `b`. Revision raises accuracy only when `b\u002Fh > a\u002F(1-a)`. At 90% initial accuracy that threshold is nine: the model has to revise a wrong answer nine times as often as a right one merely to break even.\n\nMeasured behaviour runs the other way for every model. The paper plots each model's selectivity `b\u002Fh` against its break-even point, and the ratios are 0.25 for GPT-4o-mini, 0.19 for Claude-Sonnet-4 and 0.25 for Gemini-2.0-Flash. The two open-weight models, Llama-3.1-8B and Mistral-7B, sit at 0.82 and 0.81, roughly twenty percent short of the line. The three proprietary models would need to become three to five times more selective to stop losing accuracy.\n\nThe accuracy cost lands directly in the tables. Under expert-labeled disagreement Claude-Sonnet-4 turns 19.9% of correct answers wrong while fixing only 3.8% of wrong ones, a 16.1 point drop. GPT-4o-mini loses 11.0 points and Gemini-2.0-Flash 6.8 points; the two open-weight models lose 2.3 and 1.9 points. For the three proprietary models the paired tests give p at or below 2e-20.\n\nOne comparison is especially telling. GPT-4o-mini's reversal rate on initially correct answers is 10.0% under anonymous disagreement and 16.9% under the expert label, on the same items with the same answers. Only the identity of the objector changed.\n\n## More reversals do not automatically mean lower accuracy\n\nThe paper separates the label's effect on reversal from its effect on accuracy, and this is the most important layer of the analysis. For GPT-4o-mini the label moves the reversal rate by +6.90 points and accuracy by -5.19 points. Llama-3.1-8B shows +5.18 and -3.77. But Claude-Sonnet-4 reverses more often (+2.62) with a small, uncertain positive accuracy effect (+0.30, 95% CI [-1.01, 1.61]), and Gemini-2.0-Flash reverses less (-1.86) while becoming more accurate (+2.37). The loss therefore cannot simply be blamed on the expert label. For Claude-Sonnet-4 and Gemini-2.0-Flash it comes from disagreement itself.\n\nA stricter control follows. The anonymous and expert-labeled sources are given the identical two-sentence rationale with no filler, so the two prompts differ only in the sentence describing the source. The label still raises the reversal rate by 4.82, 5.62 and 4.42 points for GPT-4o-mini, Claude-Sonnet-4 and Gemini-2.5-Flash, and lowers accuracy by 4.82, 3.21 and 2.81 points, with only GPT-4o-mini's accuracy interval excluding zero.\n\nStated reliability turns out to carry weight on its own. In the padded control suite, adding the clause that the expert may be wrong lowers the reversal rate for all five models tested, significantly for four, with Claude-Sonnet-4 falling from 7.8% to 1.4%. Adding a claimed track record raises it for all five, significantly for two.\n\n## A prompt-level fix is not enough\n\nThe paper tests three mitigation paths with limited success. An evidence gate that instructs the model to revise only when the excerpt itself supports the change cuts correct-to-wrong transitions from 171 to 76 across three proprietary models on the answer-only subset. Against argued disagreement it helps less: Claude-Sonnet-4 drops from 24.7% to 20.2% (p = 0.013) and GPT-4o-mini from 28.1% to 24.6% (p = 0.077), while Gemini-2.5-Flash and the four newer models show no significant change.\n\nThe paper also looks at the token-score level for the mechanism. For the two open-weight models, 63.2% of Llama-3.1-8B reversals and 26.8% of Mistral-7B reversals happen without the Yes\u002FNo log-probability changing sign. The answer is pushed across the decision boundary by a graded score shift rather than by a genuine reversal of the model's conclusion.\n\nThis is not an isolated result. A separate team from Amazon, Georgia Tech and Illinois Institute of Technology, in `Easier to Mislead Than to Correct: Harmful and Beneficial Revision in LLM Conformity` (https:\u002F\u002Farxiv.org\u002Fabs\u002F2606.01637), ran a comparable study over four open-weight models and seven QA datasets and reached a consistent direction: all-wrong peers push harmful revision from 15.6% to 62.9% (+47.3 points), while all-correct peers raise beneficial revision only from 32.7% to 51.5% (+18.8 points). Authority labels again make models more likely to pick the endorsed answer regardless of whether it is correct. That work also reports that generic reasoning interventions such as chain-of-thought and reflection do not reliably suppress harmful revision.\n\n## So what\n\nMulti-agent pipelines now routinely treat another agent's disagreement as a correction signal, and some attach role labels such as expert or team lead. This paper puts a price on that design. Models do revise more often under a prestigious label, but the selectivity of those revisions falls far short of what a high initial accuracy demands, and once the direction of the disagreement is wrong the loss exceeds the gain.\n\nClaude-Opus-4.8 clears the break-even line on its own, with a ratio of 1.33, and its accuracy does not fall, because it hardly reverses at all. That points to what is actually scarce in disagreement-based pipelines: not a model that listens harder, but one that can hold its answer when the evidence has not changed. The paper's recommendation follows directly. Do not evaluate a revision pipeline by reversal counts or by final accuracy alone. Report drift, disagreement sensitivity, both effects of the source label, and the selectivity of revision, per model and per protocol.\n\nThe protocol and code are released at https:\u002F\u002Fgithub.com\u002Fyupengtang\u002Fwhen-expert-disagreement-hurts.","llm-expert-disagreement-revision-selectivity","2026-11-15T07:10:00Z","2026-11-15T07:32:00.515462Z","2026-11-15T07:32:00.515474Z",true,"agent",59,[39,48],{"slug":40,"tag_slug":40,"title_zh":41,"title_en":42,"intro_zh":43,"intro_en":44,"id":45,"is_active":35,"created_at":46,"modified_at":47},"ai-for-science","AI for Science 2026：从 UniPert 到 GPT-Rosalind 的硬核进化","AI for Science 2026: from UniPert to GPT-Rosalind","生命科学、化学材料、物理世界模型——AI 正在从\"语言工具\"变成\"实验伙伴\"。本专题收录 AI 在三大科学方向的关键节点：UniPert 统一基因与化学扰动空间、GPT-Rosalind 端到端生命科学推理、达摩院 AI 智能体 28 小时找到 4 种超导新材料、Anthropic Claude Science 把工作台做成标准品。","From language tool to lab partner — AI is reshaping life sciences, chemistry\u002Fmaterials, and physical world models. This topic covers the key milestones: UniPert unifying genetic-chemical perturbation spaces, GPT-Rosalind's end-to-end life-sciences reasoning, DAMO's AI agent discovering 4 superconducting materials in 28 hours, and Anthropic's Claude Science workbench going mainstream.","988a4300-5fab-41c4-b5d8-63711a2dc757","2026-09-10T01:34:15.296649Z","2026-09-10T01:34:15.296663Z",{"slug":49,"tag_slug":49,"title_zh":50,"title_en":51,"intro_zh":52,"intro_en":53,"id":54,"is_active":35,"created_at":55,"modified_at":56},"h3-series","MiniMax H3 系列：从开源权重到 35 倍吞吐","MiniMax H3 Series: from open weights to 35x throughput","MiniMax H3 自 2026 年 8 月开源以来节奏密集：官方把生成、参考与编辑收回一个模型；ComfyUI 当天压进 RTX 3060；摩尔线程 3 小时完成国产 GPU 适配；fal 后训练版把吞吐拉到 35 倍；FastH3 蒸馏再砍推理成本。本专题持续追踪 H3 的发布—开源—蒸馏—部署全链路。","Since MiniMax open-sourced H3 in August 2026 the pace has been relentless: one unified omni-modal model, same-day ComfyUI support down to an RTX 3060, a 3-hour Day-0 port to Moore Threads GPUs, fal's post-trained H3 Max at 35x throughput, and FastH3 distillation cutting inference cost further. This topic tracks the full H3 chain — release, open weights, distillation, deployment.","83ef0daa-3c31-4cb3-86ed-e5ee58654d5f","2026-09-08T07:33:19.942193Z","2026-09-08T07:33:19.942209Z",{"items":58},[59,64,69,74,79,84],{"id":60,"title":61,"news_slug":62,"published_at":63},"19dd65ff-646c-4edf-bd90-98b2dc8bf215","Web Agent幻觉可解释性:探针能预警,归因却失灵","web-agent-hallucination-mechanistic-interpretability-probe","2026-11-17T03:15:00+00:00",{"id":65,"title":66,"news_slug":67,"published_at":68},"9785c9bb-85a0-4b97-8890-aa2d6808bf86","Agent记忆陷阱:良性对话把不安全操作率推到81%","agent-memory-trap-benign-overgeneralization","2026-11-08T15:12:52+00:00",{"id":70,"title":71,"news_slug":72,"published_at":73},"9e209ccb-cd2c-44cd-acaf-b168454074b2","中国 AI 智能体也会撒谎:88% 轮次出现虚假陈述","china-ai-agents-lie-88pct-tender-sim","2026-10-04T04:00:00+00:00",{"id":75,"title":76,"news_slug":77,"published_at":78},"4977d1f4-c8f7-480c-aaa1-ec01d69f44d8","8B 拿 SFT+GRPO 打 685B MoE:KaliBench 把 LLM 网络安全工具调用拆到命令行级","kalibench-cybersecurity-cli-runtime-rewards","2026-10-03T05:00:00+00:00",{"id":80,"title":81,"news_slug":82,"published_at":83},"9ebb888c-dfe7-416a-9940-a913527d4f73","AI Agent 的失败比成功更值钱:5 万对错误诊断数据,修正通过率 18.4%→51.1%","agent-error-dataset","2026-10-01T15:11:08+00:00",{"id":85,"title":86,"news_slug":87,"published_at":88},"3bcb0e1d-99ea-4fae-9bdd-b6b625aabf10","代码 agent 8 成都在骗你:12 模型实测揭晓","overclaimbench-llm-agents","2026-09-21T07:00:00+00:00"]