[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-multimodal-eval-answer-readout-protocol-score-drift":3,"topics-all":38,"news-related-cde1231e-8cfc-428e-9880-2d9a803c2133":67},{"id":4,"title":5,"summary":6,"content":7,"original_url":8,"source_id":9,"tags":10,"translations":24,"news_slug":31,"published_at":32,"created_at":33,"modified_at":34,"is_published":35,"publish_type":36,"image_url":14,"view_count":37},"cde1231e-8cfc-428e-9880-2d9a803c2133","同一模型换个读法掉20分:多模态评测的答案读出协议","一份 arXiv 预印本量了多模态评测里最容易被忽略的环节:答案读出协议。500 道 POPE 对抗题上,同一模型只换读答案的接口,SmolVLM-256M 从 71.8% 掉到 51.6%,差 20.2 个点;TinyLLaVA-1.1B 差 16.4 点,Qwen2.5-VL-3B 只差 1.8 点。","多模态模型的评测榜单，最近被一份 arXiv 预印本撕开了一道口子。\n\n论文叫 RAMP，2026 年 11 月 16 日挂出（https:\u002F\u002Farxiv.org\u002Fabs\u002F2611.20653），作者来自中国科学技术大学、中南大学和燕山大学。它做的事很朴素：把多模态评测里一个长期没人较真的环节单独拎出来——**答案读出协议**（answer readout），也就是评测程序到底从模型输出里怎么把答案抠出来。\n\n作者的结论是：同一张图、同一道题、同一个模型权重，仅仅换一个「怎么读答案」的接口，榜单分数能差出 20 个百分点。\n\n## 差距到底出在哪\n\n论文在 POPE 对抗性数据集的 500 道题上，用了 SmolVLM-256M、TinyLLaVA-1.1B、Qwen2.5-VL-3B 三个骨干，比较四种读出方式：直接读 yes\u002Fno 的原始 token（bare）、把大小写等表面变体都算进去取最高分（surface）、以及给模型看 A. yes \u002F B. no 这种带选项的提示再读字母（typed），外加把选项顺序颠倒的反向版本。\n\n结果是：SmolVLM-256M 从 bare 的 71.8% 掉到 typed 的 51.6%，差 20.2 个百分点，95% 置信区间落在 [-27.6, -13.2]。TinyLLaVA-1.1B 从 82.4% 掉到 66.0%，差 16.4 点。Qwen2.5-VL-3B 从 85.2% 到 87.0%，只差 1.8 点。同一族更大一点的 SmolVLM-500M 在自己那套 CPU 配置下，从 82.2% 掉到 60.0%，差 22.2 点。\n\n规律很直白：**小模型在读出协议面前极其脆弱，参数多一点的就稳得多**。这个反差比「换个提示会掉分」更值得玩味——它说明很多小模型的分数本来就不该被当真。而且换 surface 读法，SmolVLM-256M 反而从 71.8% 涨到 75.0%，TinyLLaVA 从 82.4% 掉到 78.4%，方向都不一致。\n\n还有一组候选集实验：往 yes\u002Fno 之外再塞 2 到 50 个诱饵选项，三个较大骨干的准确率基本不动，SmolVLM-256M 则卡在 64.0% 左右，只有一个 50 选项的种子掉到 63.8%。论文据此写下一条构造层面的单调性关系——候选集越大，准确率不会更好。\n\n## 换协议，答案全变了\n\n更要命的是，作者发现准确率这个汇总数字会骗人。\n\n（下面这组数字取自论文的归档实验，和上一节新跑的一批分开统计。）在归档的 SmolVLM-256M 那组 POPE 实验里，从 bare 切到 typed，有 221 道题从对变错、121 道题从错变对，500 道题里 342 道的对错归属整个翻转，净差只有 -20.0 个点。TinyLLaVA 则是 119 道对转错、38 道错转对，157 道题变了，净差 -16.2 点。Qwen 只变了 33 道题。\n\n更极端的例子在 ChartQA 的 104 道原生是非题上：SmolVLM-500M 有 40 道从错转对、39 道从对转错，79 道题改了答案，占这个子集的 76.0%，而最终准确率只动了 +1.0 点。\n\n也就是说，榜单上那个「差不多」的分数，可能是两份完全不同的答卷互相抵消出来的。把 A\u002FB 选项顺序对调，SmolVLM-256M 有 401 道题的语义预测直接翻转；Qwen 那边 487 道共同观测题里只有 7 道变。\n\n## 换协议，连置信度一起塌\n\n作者还顺手量了校准。SmolVLM-256M 的 15 分箱正确性 ECE 从 0.221 涨到 0.409，SmolVLM-500M 从 0.093 涨到 0.238；TinyLLaVA 从 0.061 涨到 0.084。只有 Qwen 是反方向的，从 0.065 降到 0.050。AUROC 也跟着掉：SmolVLM-256M 从 0.7011 降到 0.6084，Qwen 从 0.7927 降到 0.7762。\n\n提示词措辞那一层同样敏感。SmolVLM-256M 在一种问法下相对原提示涨 13.2 个点，把答案 token 首字母大写则掉 10.2 点；TinyLLaVA 首字母大写的变体直接掉 17.2 点。Qwen 那一列的波动区间始终在 -0.4 到 +2.0 之间。选项放到问题前面还是后面也另有一组 64 题的小实验，TinyLLaVA 在这里掉了 23.44 个点。\n\n## 同一份代码，换个机器结果就不一样\n\n作者做了一件同行很少做的事：把原始 logit、候选 token ID、语义映射、精度、图像策略全部留档，然后从保存的数据里独立重放。他们从 JSONL 里重放了 16,140 个首 token 决策，逐条跟处理后的预测完全一致；另一条路从 57,073 行的读出表重算，也复现了全部 217 个导出的准确率格。\n\n重放过程还暴露了一件让人不太舒服的事：同一个 SmolVLM-256M 的 POPE 500 题实验，跑在 CPU\u002Ffp32 上是 -14.4 个点，跑在 MPS\u002Ffp16 上是 -20.2 个点。语义预测在那 64 道对照题上一个都没变，但分数边界的差异最大能到 0.136694。**结论的量级本身，也取决于跑在什么设备上。**\n\n## 这不是新问题，但没人这么量过\n\n「读出方式会改变结论」这个机制本身不新鲜。2024 年的 Order Matters 一篇论文（https:\u002F\u002Farxiv.org\u002Fabs\u002F2410.16983）就发现多模态大模型对输入顺序极度敏感，顺序一变，表现能在接近最好和随机猜之间跳，并提出位置不变准确率（PIA）来纠正；2026 年 6 月的 Facet-Probe（https:\u002F\u002Farxiv.org\u002Fabs\u002F2606.26079）审计了 18 个前沿和开源多模态模型，跨选项、证据块、文档排序、图像集、图文混合五个维度做扰动，发现没有一个模型是顺序不变的，各维度翻转率在 24% 到 50% 之间，论文里表现最好的模型仍有 13.4% 的试验在翻转。更早的 State of What Art?（https:\u002F\u002Farxiv.org\u002Fabs\u002F2401.00595）在 650 万次实例、20 个 LLM、39 个任务的规模上证明了单提示评测的脆弱性，那篇已被 TACL 接收。\n\nRAMP 的增量在于把它拆成了可复算的协议层，并且留了完整的审计记录。\n\n## 所以呢\n\n多模态榜单上的数字，有一部分根本不归模型管。同一场评测里，「问 yes 还是问 A」这种工程细节，能吃掉小模型二十分的成绩。\n\n如果你是发模型的，别只报裸准确率，把读出协议和置信度校准一起报。如果你是看榜单的，先问一句：这个分数是按哪种接口抠出来的？\n\n模型在被评测之前，得先被读懂。读法不对，分数就是误会。\n","https:\u002F\u002Farxiv.org\u002Fabs\u002F2611.20653","7437aeb9-930c-4866-a2e9-48003c1a792b",[11,15,18,21],{"id":12,"name":13,"slug":13,"description":14,"color":14},"40269b40-7942-4650-9672-ed2e6524d37a","ai-technology",null,{"id":16,"name":17,"slug":17,"description":14,"color":14},"120fa59a-ff6f-4537-9bf5-f818df636a0e","benchmark",{"id":19,"name":20,"slug":20,"description":14,"color":14},"01598627-1ea6-4b27-a5d8-874971571a71","llm",{"id":22,"name":23,"slug":23,"description":14,"color":14},"499f4b56-819d-49a3-9609-33e775143b86","multimodal",[25],{"id":26,"lang":27,"title":28,"summary":29,"content":30},"317c8022-e969-49b8-99bd-2dc31561ab8f","en","Same Model, Different Readout: 20 Points of Score Drift","An arXiv preprint measures how answers get read out. On 500 POPE questions, SmolVLM-256M drops 20 points but Qwen2.5-VL-3B barely moves.","Multimodal leaderboards just got a crack in them, and the crack comes from an arXiv preprint.\n\nThe paper is called RAMP. It went up on 16 November 2026 (https:\u002F\u002Farxiv.org\u002Fabs\u002F2611.20653), authored by researchers from the University of Science and Technology of China, Central South University and Yanshan University. What it does is deliberately unglamorous: it isolates one step of multimodal evaluation that almost nobody scrutinizes — the **answer readout protocol**, meaning how the evaluation program actually pulls an answer out of the model's output.\n\nThe headline finding is blunt. Same image, same question, same model weights, and simply changing the interface used to read the answer shifts leaderboard scores by up to 20 percentage points.\n\n## Where the gap comes from\n\nOn 500 adversarial POPE questions, the paper runs three backbones — SmolVLM-256M, TinyLLaVA-1.1B and Qwen2.5-VL-3B — and compares four readout families: bare (score the raw yes\u002Fno tokens directly), surface-max (fold in capitalization and other single-token spellings and take the highest), typed (show the model 'A. yes' and 'B. no' and score the option letters), plus a reversed variant that flips the option order.\n\nSmolVLM-256M falls from 71.8% under bare scoring to 51.6% under typed, a 20.2 point drop with a 95% paired interval of [-27.6, -13.2]. TinyLLaVA-1.1B goes from 82.4% to 66.0%, a 16.4 point gap. Qwen2.5-VL-3B moves from 85.2% to 87.0%, only 1.8 points. The larger SmolVLM-500M, on its own CPU configuration, drops from 82.2% to 60.0% — 22.2 points.\n\nThe pattern is plain: **small models are extremely fragile under readout protocols, while the ones with more parameters hold steady**. That contrast is more interesting than the familiar 'prompt wording changes the score' story, because it suggests many small-model scores were never trustworthy in the first place. Swapping to the surface readout also does not push everything one way — SmolVLM-256M actually climbs from 71.8% to 75.0% while TinyLLaVA drops from 82.4% to 78.4%.\n\n## Swap the protocol, change every answer\n\nMore unsettling is what the authors found about the aggregate number itself.\n\nIn the archived SmolVLM-256M POPE run (counted separately from the newly executed batch above), switching from bare to typed moves 221 items from correct to incorrect and 121 from incorrect to correct — 342 of 500 items flip correctness entirely, while the net effect is only -20.0 points. TinyLLaVA records 119 correct-to-wrong and 38 wrong-to-correct transitions, 157 changed items, net -16.2 points. Qwen changes just 33 items.\n\nThe starkest case sits on a 104-question native yes\u002Fno subset of ChartQA: SmolVLM-500M has 40 wrong-to-correct and 39 correct-to-wrong transitions. Seventy-nine answers change, 76.0% of the subset, and the final accuracy moves by only +1.0 point.\n\nIn other words, a leaderboard score that looks stable can be two entirely different answer sheets cancelling each other out. Reverse the A\u002FB option order and 401 of SmolVLM-256M's semantic predictions flip; for Qwen, only 7 of 487 jointly observed reversed items change.\n\n## Swap the protocol, and confidence collapses too\n\nThe paper also measures calibration. SmolVLM-256M's 15-bin correctness ECE rises from 0.221 to 0.409, and SmolVLM-500M's from 0.093 to 0.238; TinyLLaVA goes from 0.061 to 0.084. Qwen moves the other way, from 0.065 down to 0.050. AUROC tracks along: SmolVLM-256M drops from 0.7011 to 0.6084, and Qwen from 0.7927 to 0.7762.\n\nPrompt wording is just as sensitive. One formulation lifts SmolVLM-256M by 13.2 points relative to the original prompt, while capitalizing the answer token costs 10.2 points; TinyLLaVA loses 17.2 points on the capitalized variant. Qwen's column stays inside a -0.4 to +2.0 band. A separate 64-question experiment on whether the options sit before or after the question produces a -23.44 point result for TinyLLaVA.\n\n## Same code, different machine, different number\n\nThe authors did something few of their peers do: they archived raw logits, candidate token IDs, semantic mappings, precision and image policy, then independently replayed the evaluation from that stored evidence. They reconstruct 16,140 first-token decisions from the JSONL and match every processed prediction exactly; a second pass over the 57,073-row readout table reproduces all 217 exported accuracy cells.\n\nThe replay also surfaced an uncomfortable fact. Two SmolVLM-256M POPE runs on 500 questions produced typed-minus-bare changes of -14.4 points on CPU\u002Ffp32 and -20.2 points on MPS\u002Ffp16. The semantic predictions were identical on all 64 control questions, yet individual score margins differed by as much as 0.136694. **The magnitude of the conclusion depends on what hardware you ran it on.**\n\n## Not a new mechanism, but a new level of measurement\n\nThat readout style changes conclusions is not itself new. The 2024 paper Order Matters (https:\u002F\u002Farxiv.org\u002Fabs\u002F2410.16983) showed that multimodal large language models are acutely sensitive to input order, that reordering can swing performance between near-best results and random guessing, and proposed Position-Invariant Accuracy to correct for it. A June 2026 audit, Facet-Probe (https:\u002F\u002Farxiv.org\u002Fabs\u002F2606.26079), perturbed 18 frontier and open-weight multimodal models across five facets — option, evidence-chunk, document-rank, image-set and mixed-modality ordering — and found none of them order-invariant, with per-facet flip rates spanning 24% to 50% and the best model in that audit still flipping on 13.4% of trials. Earlier still, State of What Art? (https:\u002F\u002Farxiv.org\u002Fabs\u002F2401.00595) demonstrated the brittleness of single-prompt evaluation across 6.5M instances, 20 LLMs and 39 tasks, and was accepted at TACL.\n\nWhat RAMP adds is decomposing the effect into a recomputable protocol layer, with a complete audit trail behind it.\n\n## So what\n\nA chunk of any multimodal leaderboard number does not belong to the model. Within a single evaluation, an engineering detail like 'ask yes or ask A' can eat twenty points off a small model's score.\n\nIf you are shipping a model, stop reporting bare accuracy — report the readout protocol and the calibration alongside it. If you are reading a leaderboard, ask one question first: which interface was this score extracted through?\n\nA model has to be read correctly before it can be evaluated. Read it wrong, and the score is just a misunderstanding.\n","multimodal-eval-answer-readout-protocol-score-drift","2026-11-17T07:20:00Z","2026-11-17T07:19:27.371294Z","2026-11-17T07:19:27.371309Z",true,"agent",129,[39,49,58],{"slug":40,"tag_slug":41,"title_zh":42,"title_en":43,"intro_zh":44,"intro_en":45,"id":46,"is_active":35,"created_at":47,"modified_at":48},"image-generation","text-to-image","图像生成：从文生图到逐框编辑的模型竞赛","Image Generation: from text-to-image to frame-level editing","文生图已是红海，战场正转向编辑与可控性：FLUX.2\u002F3 用开源和量化把生图搬进消费级显卡，Qwen-Image、Seedream 等国产模型紧咬不放，GPT Image 2 则把推理能力带进像素。本专题追踪图像生成与编辑模型的发布、架构演进、加速量化与商业化数据，看「好图」如何变成「可用的图」。","Text-to-image is a crowded race, and the frontier has moved to editing and control: FLUX.2\u002F3 push open weights and quantization onto consumer GPUs, Qwen-Image and Seedream close in from China, and GPT Image 2 brings reasoning into pixels. This topic tracks model releases, architecture shifts, acceleration work, and the business numbers behind \"pretty\" becoming \"usable\".","52de7c59-7864-4569-8ae7-3d656d26a1a1","2026-11-20T07:20:23.534506Z","2026-11-20T07:20:23.534527Z",{"slug":50,"tag_slug":50,"title_zh":51,"title_en":52,"intro_zh":53,"intro_en":54,"id":55,"is_active":35,"created_at":56,"modified_at":57},"ai-for-science","AI for Science 2026：从 UniPert 到 GPT-Rosalind 的硬核进化","AI for Science 2026: from UniPert to GPT-Rosalind","生命科学、化学材料、物理世界模型——AI 正在从\"语言工具\"变成\"实验伙伴\"。本专题收录 AI 在三大科学方向的关键节点：UniPert 统一基因与化学扰动空间、GPT-Rosalind 端到端生命科学推理、达摩院 AI 智能体 28 小时找到 4 种超导新材料、Anthropic Claude Science 把工作台做成标准品。","From language tool to lab partner — AI is reshaping life sciences, chemistry\u002Fmaterials, and physical world models. This topic covers the key milestones: UniPert unifying genetic-chemical perturbation spaces, GPT-Rosalind's end-to-end life-sciences reasoning, DAMO's AI agent discovering 4 superconducting materials in 28 hours, and Anthropic's Claude Science workbench going mainstream.","988a4300-5fab-41c4-b5d8-63711a2dc757","2026-09-10T01:34:15.296649Z","2026-09-10T01:34:15.296663Z",{"slug":59,"tag_slug":59,"title_zh":60,"title_en":61,"intro_zh":62,"intro_en":63,"id":64,"is_active":35,"created_at":65,"modified_at":66},"h3-series","MiniMax H3 系列：从开源权重到 35 倍吞吐","MiniMax H3 Series: from open weights to 35x throughput","MiniMax H3 自 2026 年 8 月开源以来节奏密集：官方把生成、参考与编辑收回一个模型；ComfyUI 当天压进 RTX 3060；摩尔线程 3 小时完成国产 GPU 适配；fal 后训练版把吞吐拉到 35 倍；FastH3 蒸馏再砍推理成本。本专题持续追踪 H3 的发布—开源—蒸馏—部署全链路。","Since MiniMax open-sourced H3 in August 2026 the pace has been relentless: one unified omni-modal model, same-day ComfyUI support down to an RTX 3060, a 3-hour Day-0 port to Moore Threads GPUs, fal's post-trained H3 Max at 35x throughput, and FastH3 distillation cutting inference cost further. This topic tracks the full H3 chain — release, open weights, distillation, deployment.","83ef0daa-3c31-4cb3-86ed-e5ee58654d5f","2026-09-08T07:33:19.942193Z","2026-09-08T07:33:19.942209Z",{"items":68},[69,74,79,84,89,94],{"id":70,"title":71,"news_slug":72,"published_at":73},"0b895f86-1c24-4c60-9f1f-c2fd0da42b2c","PO降幻觉:提前闭嘴能复现58%到86%的收益","po-hallucination-gain-reproduced-by-early-stopping","2026-11-19T01:30:00+00:00",{"id":75,"title":76,"news_slug":77,"published_at":78},"1f38ebab-de2b-4b33-9566-2eaf380eef42","8B LLM 当 AutoML 搜索器:跨三模态零样本","ncode-llm-automl-searcher","2026-11-10T23:30:00+00:00",{"id":80,"title":81,"news_slug":82,"published_at":83},"51e65509-14a8-442b-bc70-9864569dc57f","8个开源VLM数数实测:硬上限大约5个物体","vlm-counting-horizon-geoscope","2026-11-10T09:00:00+00:00",{"id":85,"title":86,"news_slug":87,"published_at":88},"e4ac848d-d24a-465f-bece-37e090da86de","全双工语音模型长对话必崩:KAIST 揪出初始化锚点被挤走","full-duplex-speech-collapse-fd-lhb-kv-cache","2026-10-27T03:20:00+00:00",{"id":90,"title":91,"news_slug":92,"published_at":93},"42f5ea83-365c-4a4e-a318-17d7c4c11c15","论文实测:自洽性证书在AIME上24%的题上会误触发","self-consistency-certificates-misfire-correlated-reasoning","2026-11-21T09:30:00+00:00",{"id":95,"title":96,"news_slug":97,"published_at":98},"53dd30bb-ba19-45b6-a2de-5011709608b6","论文实测:一段文本认出39%审稿人,靠的是曝光量","frontier-models-name-authors-by-exposure","2026-11-21T01:40:00+00:00"]