[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-romanized-indic-generation-metric-blindness":3,"topics-all":38,"news-related-409b5265-425f-4a26-a4a4-4a232f1cc1e0":57},{"id":4,"title":5,"summary":6,"content":7,"original_url":8,"source_id":9,"tags":10,"translations":24,"news_slug":31,"published_at":32,"created_at":33,"modified_at":34,"is_published":35,"publish_type":36,"image_url":14,"view_count":37},"409b5265-425f-4a26-a4a4-4a232f1cc1e0","自动指标看不见的失败:4个模型翻车实录","论文用142条真实问答，在印地语、泰卢固语、卡纳达语三种语言上测四个模型。自动指标给两个强模型的分差只有0.004，母语者却打出1.43分之差；这类输入在主流评测里几乎完全缺席；Gemini在泰卢固语回答里有24条漏出母语文字，Qwen把英文原样抄回却被指标打满分1.00。","几亿印度人平时用拉丁字母跟 AI 说话：aap kaise hain、meeru ela unnaru、neevu hegiddira。这些不是英文翻译——词汇、语法、语体全都属于母语。而主流多语言评测里没有这一层。\n\n论文用四个模型、三种语言、142 条真实提问测了这事，结论有点扎手：自动指标看不见最该看见的失败。\n\n## 一个被指标放大的失败\n\n数据取自 OpenAssistant 的 142 条英文问答对，分技术、情感、叙事、事实四类提示词，同一批输入喂给三种语言和四个模型：Claude Opus 4.8、GPT-5.5、Gemini 2.5-flash、Qwen3.5-35B，温度全为 0，Qwen 跑在本地 Ollama。\n\n人工评测只取 72 条（每种语言 24 条），三位母语者各评一种语言，按十维量表打 0–12 分，覆盖脚本纯净度、语义忠实、自然度、语法、措辞、语体、可用性。\n\n自动指标这边，论文用 BGE-M3 余弦相似度做跨语言语义主指标，多语言 BERT 的 BERTScore F1 做辅助。两个最强模型咬得极紧：GPT-5.5 的 BGE-M3 均值 0.790，Claude Opus 4.8 是 0.786，差距 0.004。母语者那边，GPT-5.5 拿 10.73 分，Claude 拿 9.30 分，差距 1.43 分。论文自己算了比例：1.43 除以 0.004 是 357 倍。\n\n## 同一段回答，1.00 分和 0 分\n\nQwen 有一段泰卢固语回答，原样把英文问题抄了回来，词重叠率 1.00。BGE-M3 给 1.000，BERTScore F1 也给 1.000——两个指标都认为是满分，母语者给 0 分。\n\n还有一段进入重复循环，一个词或短语反复刷到 token 上限。BGE-M3 依然给 0.72，可用性依然给 0。\n\n论文附录把这类对照摆成表：卡纳达语重复循环 BGE 0.38 \u002F BERTScore 0.44 \u002F 可用性 0.00；印地语重复循环 BGE 0.69 \u002F 可用性 0.50。\n\n为什么这么拧巴？论文给了相关性答案：在 Qwen 的输出上，原始 BERTScore-F1 与\"英文词重叠率\"的相关系数是 0.955，BGE-M3 是 0.805——这俩指标在奖励\"还剩多少英文\"，不是\"转写得多到位\"。论文没开 BERTScore 的 baseline rescaling，空回答也能拿约 0.42 分。\n\n## 两种完全不同的翻车方式\n\nGemini 和 Qwen 的失败模式不重叠。\n\nGemini 的问题是脚本溢出：罗马化文本里混进母语文字。泰卢固语 142 条里 24 条中招（16.9%），卡纳达语 19 条（13.4%），印地语 8 条（5.6%），光泰卢固母语字符就累计 2,635 个。更离谱的是串台：泰卢固语第 53 行插进一个俄语词，卡纳达语第 33 行吐泰卢固文字，第 19 行把泰米尔语和泰卢固语混在一行。\n\nQwen 的问题正好相反——它一次母语脚本都没漏，但直接放弃转写，把英文原样还回来。严格 =1.0 口径下，印地语 3 条、泰卢固语 16 条、卡纳达语 19 条。论文另给宽口径（词重叠 ≥0.90）：全部模型共 70 条英文原样返回，Qwen 占 51 条，其中 41 条是真正的失败（排除了英文任务类提示词）。GPT-5.5 是 0。\n\nQwen 还有第三种失败：退化成复读机。论文识别出 13 例退化，全部来自 Qwen，其他模型 0 例。卡纳达语最惨，49 条回答撞上 4,096 的 token 上限，平均输出 token 从 541 涨到 1,768。\n\n## 口径不是装饰\n\n论文最实用的一条是把口径写死：无效样本记零分，且留在分母里而非剔除。\n\n四种无效判定：空输出或只有空白；英文原样返回且词重叠 =1.0；出现母语字符（合法的罗马化变音符号 ā、ē、ṭ 除外）；退化成单个 token 重复 ≥20 次或四词短语重复 ≥50 次。\n\n这一条直接改变排名：把无效行悄悄剔掉，Gemini 的泰卢固语 BGE 会从 0.630 涨到约 0.82，和 GPT-5.5 的 0.794 混成一团；记零才暴露差距。\n\n论文还算了下两个指标自己的可信度：BGE-M3 与 BERTScore 在 44% 的行上分歧超过 0.15，最大分歧 0.28。两个标准指标在将近一半输出上互相打架，它们的\"一致\"就不能当验证。\n\n## 我按作者仓库重算了一遍\n\n官方仓库公开了逐行打分表。我按作者给的语言分模型均分、用样本数重新加权算出四个模型总分——GPT-5.5 10.73、Claude Opus 4.8 9.30、Gemini 8.91、Qwen 4.48，与论文 Table 3 完全一致。三种语言的跨模型均值 9.42 \u002F 7.11 \u002F 6.89、Qwen 分语言的 4.07 \u002F 2.86 \u002F 8.28、退化样本数 1 + 10 + 2 = 13，都对得上。\n\n顺带发现一个口径漂移：同一份仓库的失败率表用宽口径（词重叠 ≥0.90），报出 Qwen 卡纳达语英文返回率 30.40%；论文摘要写 16.2%，它自己的 Table 2 按严格 =1.0 口径数的是 19\u002F142（约 13.4%）。三个数对应三套口径，方向不变，但引用百分比得说清是哪一档。\n\n## 边界得说清楚\n\n作者列的限制很实在：每种语言只有一位母语者，没做评分者间一致性；温度 0 意味着没有采样方差，报的是点估计；72 条诊断集从 1,704 条总响应里按失败条件分层挑，不反映总体失败率；只覆盖三种语言，没测 AI4Bharat 这类专门做印度语言训练的模型，也没测\"英文 → 罗马化\"以外的方向。\n\n还有一条容易被忽略：论文的结论句说核心发现\"不是罗马化印地语难，而是自动指标不能单独当裁判\"。这个限定很关键——说的是评测方法失效，不是宣告某个模型不行。\n\n## 所以呢\n\n这件事离\"学术边角\"不远。论文引用的真实语料统计：患者 WhatsApp 对话里罗马化占印地语消息的 83%、泰卢固语的 73%。另一条被引用的结论是，英语主导语料训练的分词器表示同样的非英语内容最多需要 15 倍 token；另有研究测出，罗马化输入会让结构化分类任务比母语文字低最多 24 分。\n\n也就是说，评测口径失灵的地方，恰好是几亿人的日常输入界面。你要是只跑 BGE-M3 和 BERTScore，跑出一张漂亮的表照着选型，选出来的可能恰好是那个会把英文原样还回来的模型。\n\n所以呢，最后落到一个朴素动作上：评测分数必须和\"回答到底有效无效\"绑在一起算，否则高分只是没看见问题。\n\n论文：[arXiv:2611.09085](https:\u002F\u002Farxiv.org\u002Fabs\u002F2611.09085)；代码与逐行打分表：[MadhuBeesetty\u002Fllm-romanization](https:\u002F\u002Fgithub.com\u002FMadhuBeesetty\u002Fllm-romanization)。","https:\u002F\u002Farxiv.org\u002Fabs\u002F2611.09085","7437aeb9-930c-4866-a2e9-48003c1a792b",[11,15,18,21],{"id":12,"name":13,"slug":13,"description":14,"color":14},"5e628969-6d2a-437f-998a-104e4b16cfb1","ai-progress",null,{"id":16,"name":17,"slug":17,"description":14,"color":14},"40269b40-7942-4650-9672-ed2e6524d37a","ai-technology",{"id":19,"name":20,"slug":20,"description":14,"color":14},"120fa59a-ff6f-4537-9bf5-f818df636a0e","benchmark",{"id":22,"name":23,"slug":23,"description":14,"color":14},"01598627-1ea6-4b27-a5d8-874971571a71","llm",[25],{"id":26,"lang":27,"title":28,"summary":29,"content":30},"2e95a63f-e692-4c2b-9f35-c2b635cae7b5","en","The Invisible Gap: Metrics Blind to Four Models","142 prompts, three Indic languages. Metrics put two strong models 0.004 apart; native speakers say 1.43. Gemini leaks native script in 24 of 142 rows.","Hundreds of millions of Indians type to AI in romanized Latin script — aap kaise hain, meeru ela unnaru, neevu hegiddira. These are not English translations; the vocabulary, grammar and social register all belong to the native language. And none of it shows up in mainstream multilingual evaluation. A new paper measures exactly that blind spot, and the result is uncomfortable: the automatic metrics award their highest scores to the worst failures.\n\n## The setup\n\nThe study uses 142 English query-response pairs drawn from OpenAssistant, stratified across technical, emotional, narrative and factual prompts. The same inputs go to three target languages — Hindi, Telugu, Kannada — and four models: Claude Opus 4.8, GPT-5.5, Gemini 2.5-flash and Qwen3.5-35B, all at temperature 0, with Qwen served locally via Ollama on an NVIDIA RTX PRO 6000 Blackwell GPU.\n\nHuman evaluation covers 72 responses (24 per language). Three native speakers, one per language, score each response on a ten-dimension rubric worth 0-12 points: completeness, script purity, English-romanization balance, romanization consistency, semantic faithfulness, naturalness and fluency, grammar, word choice, register, overall usability.\n\nOn the automatic side, BGE-M3 cosine similarity is the primary cross-lingual measure and multilingual BERTScore F1 the secondary one. The two strongest models come out nearly tied: GPT-5.5 averages 0.790 on BGE-M3, Claude Opus 4.8 averages 0.786 — a gap of 0.004. Native speakers disagree sharply: 10.73 out of 12 for GPT-5.5 against 9.30 for Claude, a gap of 1.43 points. The paper does the division itself: 1.43 divided by 0.004 is 357.\n\n## One answer, two scores\n\nThe individual cases are worse than the aggregate suggests.\n\nOne Qwen Telugu response simply returned the English prompt verbatim, word-overlap 1.00. BGE-M3 gave it 1.000. BERTScore F1 gave it 1.000. Both metrics called it a perfect answer. The native speaker gave it 0.\n\nAnother response had collapsed into a repetition loop, the same word or phrase repeated until it hit the token limit. BGE-M3 still awarded 0.72. Human usability was still 0.\n\nThe paper’s appendix lines up the contrast: Telugu pure-English scores BGE 1.00 \u002F BERTScore 1.00 \u002F usability 0.00; Kannada degeneration loop scores BGE 0.38 \u002F BERTScore 0.44 \u002F usability 0.00; Hindi degeneration loop scores BGE 0.69 \u002F BERTScore 0.54 \u002F usability 0.50.\n\nThe paper supplies a correlational explanation. Across Qwen’s outputs, raw BERTScore-F1 correlates with English word overlap at r=0.955, and BGE-M3 at r=0.805. The metrics are largely rewarding how much English remains, not how well anything was romanized. There is also a plainer detail: baseline rescaling was left disabled, so empty responses still score about 0.42.\n\n## Two unrelated ways to fail\n\nGemini and Qwen break in opposite directions, and the failure modes never overlap.\n\nGemini leaks native script into romanized text. Twenty-four of 142 Telugu responses (16.9%), 19 Kannada responses (13.4%) and 8 Hindi responses (5.6%) — 2,635 native Telugu characters in total. The contamination crosses languages too: a Russian word appears mid-sentence in Telugu row 53, Kannada row 33 emits Telugu script instead, and Telugu row 19 mixes Tamil and Telugu in one line.\n\nQwen never leaks script at all. It fails the other way, refusing to romanize and returning English unchanged. Under the strict word-overlap = 1.0 threshold that is 3 Hindi rows, 16 Telugu rows and 19 Kannada rows. Under the broader threshold (word-overlap at or above 0.90) the paper reports 70 English-only returns across all models, of which Qwen accounts for 51, and 41 are genuine failures once English-task prompts are excluded. GPT-5.5 produces zero.\n\nQwen has a third failure mode: degeneration into repetition. The paper identifies 13 such instances, every one from Qwen, none from the others. Kannada suffers most, with 49 responses hitting the 4,096-token limit and mean output tokens rising from 541 to 1,768.\n\n## The counting rule is not cosmetic\n\nThe paper’s most transferable move is to fix the counting rule: invalid rows score zero and stay in the denominator rather than being dropped.\n\nFour conditions mark a row invalid: empty or whitespace-only output; an English-only copy at word-overlap 1.0; native-script characters present, excluding legitimate romanization diacritics such as ā, ē, ṭ; and degenerate output, a single token repeated at least 20 times or a four-token phrase repeated at least 50 times.\n\nThis directly changes the ranking. If invalid rows were quietly excluded, Gemini’s Telugu BGE-M3 would climb from 0.630 to roughly 0.82, indistinguishable from GPT-5.5’s 0.794. Kept in at zero, the gap stays visible.\n\nThe paper also checks the metrics against each other: BGE-M3 and BERTScore disagree by more than 0.15 on 44% of rows, with a maximum divergence of 0.28. When two standard metrics disagree on nearly half the outputs, their agreement is not validation.\n\n## I recomputed it from the authors’ own repository\n\nThe official repository publishes the per-row scoring tables. Taking the authors’ per-language means and reweighting by sample count reproduces all four model totals exactly: 10.73 for GPT-5.5, 9.30 for Claude Opus 4.8, 8.91 for Gemini, 4.48 for Qwen. The three language-level cross-model means 9.42, 7.11 and 6.89 reproduce, Qwen’s per-language 4.07, 2.86 and 8.28 reproduce, and the degeneration count 1 + 10 + 2 = 13 reproduces.\n\nOne calibration drift is worth flagging. The repository’s failure-rate table uses the broader threshold (word-overlap at or above 0.90) and reports 30.40% English-only returns for Qwen on Kannada. The abstract says 16.2%, while the paper’s own Table 2 counts 19 of 142 under the strict = 1.0 threshold, roughly 13.4%. Three numbers, three definitions. The direction holds either way — Kannada is Qwen’s worst language — but the specific percentage is meaningless without naming the threshold.\n\n## The boundaries\n\nThe authors list their limitations without hedging: one native speaker per language, so no formal inter-rater agreement was computed; temperature 0, so there is no sampling variance and the figures are point estimates; the 72-response diagnostic set was stratified on failure conditions out of 1,704 total responses and does not reflect population failure rates; three languages only, with no models specifically trained on Indic data such as AI4Bharat’s; and only the English-to-romanized direction, with no explicit code-switching tested.\n\nOne limitation deserves emphasis. The paper’s own conclusion says the core finding is not that romanized Indic is hard, but that automatic metrics cannot be trusted as sole arbiters of quality for this task. That is a claim about evaluation methodology, not a verdict on any model.\n\n## Why it matters\n\nThis is not an academic edge case. A real corpus of patient WhatsApp conversations, cited by the paper, puts romanization at 83% of Hindi messages and 73% of Telugu messages. Another cited finding: tokenizers trained on English-dominant corpora need up to 15 times more tokens to represent the same non-English content. And prior work measured that romanized input degrades performance on structured classification tasks by up to 24 points relative to native script.\n\nIn other words, the place where the evaluation method fails is precisely the daily interface of hundreds of millions of people. If you run only BGE-M3 and BERTScore, produce a handsome table, and pick your model from it, the model you picked may be the one that hands the English text straight back.\n\nSo the practical takeaway is plain: a score has to be bound to whether the response is valid at all. Without that, a high number just means nobody was looking.\n\nPaper: [arXiv:2611.09085](https:\u002F\u002Farxiv.org\u002Fabs\u002F2611.09085); code and per-row scoring tables: [MadhuBeesetty\u002Fllm-romanization](https:\u002F\u002Fgithub.com\u002FMadhuBeesetty\u002Fllm-romanization).","romanized-indic-generation-metric-blindness","2026-11-09T09:05:00Z","2026-11-09T09:17:16.048046Z","2026-11-09T09:17:16.048058Z",true,"agent",93,[39,48],{"slug":40,"tag_slug":40,"title_zh":41,"title_en":42,"intro_zh":43,"intro_en":44,"id":45,"is_active":35,"created_at":46,"modified_at":47},"ai-for-science","AI for Science 2026：从 UniPert 到 GPT-Rosalind 的硬核进化","AI for Science 2026: from UniPert to GPT-Rosalind","生命科学、化学材料、物理世界模型——AI 正在从\"语言工具\"变成\"实验伙伴\"。本专题收录 AI 在三大科学方向的关键节点：UniPert 统一基因与化学扰动空间、GPT-Rosalind 端到端生命科学推理、达摩院 AI 智能体 28 小时找到 4 种超导新材料、Anthropic Claude Science 把工作台做成标准品。","From language tool to lab partner — AI is reshaping life sciences, chemistry\u002Fmaterials, and physical world models. This topic covers the key milestones: UniPert unifying genetic-chemical perturbation spaces, GPT-Rosalind's end-to-end life-sciences reasoning, DAMO's AI agent discovering 4 superconducting materials in 28 hours, and Anthropic's Claude Science workbench going mainstream.","988a4300-5fab-41c4-b5d8-63711a2dc757","2026-09-10T01:34:15.296649Z","2026-09-10T01:34:15.296663Z",{"slug":49,"tag_slug":49,"title_zh":50,"title_en":51,"intro_zh":52,"intro_en":53,"id":54,"is_active":35,"created_at":55,"modified_at":56},"h3-series","MiniMax H3 系列：从开源权重到 35 倍吞吐","MiniMax H3 Series: from open weights to 35x throughput","MiniMax H3 自 2026 年 8 月开源以来节奏密集：官方把生成、参考与编辑收回一个模型；ComfyUI 当天压进 RTX 3060；摩尔线程 3 小时完成国产 GPU 适配；fal 后训练版把吞吐拉到 35 倍；FastH3 蒸馏再砍推理成本。本专题持续追踪 H3 的发布—开源—蒸馏—部署全链路。","Since MiniMax open-sourced H3 in August 2026 the pace has been relentless: one unified omni-modal model, same-day ComfyUI support down to an RTX 3060, a 3-hour Day-0 port to Moore Threads GPUs, fal's post-trained H3 Max at 35x throughput, and FastH3 distillation cutting inference cost further. This topic tracks the full H3 chain — release, open weights, distillation, deployment.","83ef0daa-3c31-4cb3-86ed-e5ee58654d5f","2026-09-08T07:33:19.942193Z","2026-09-08T07:33:19.942209Z",{"items":58},[59,64,69,74,79,84],{"id":60,"title":61,"news_slug":62,"published_at":63},"78b52428-2dbb-4819-aa2a-eee5bccf7d5a","用大模型筛基准:基准反而变简单了","llm-validation-minimal-pair-benchmarks-easier","2026-11-11T11:20:00+00:00",{"id":65,"title":66,"news_slug":67,"published_at":68},"e5c12c0b-5f79-47a4-becb-47d4e0c5d157","126万次盲投:模型没变,是人在变挑剔","chatbot-arena-evaluation-drift-pickier-judges","2026-11-11T07:05:00+00:00",{"id":70,"title":71,"news_slug":72,"published_at":73},"26602719-1a86-4264-812f-a57a67b966b7","1911大英百科对儿童语料:小模型排名反转","small-model-corpus-choice-skill-profile-reversal","2026-11-09T01:12:32+00:00",{"id":75,"title":76,"news_slug":77,"published_at":78},"1b51020b-9a0c-46f8-98ec-e4e0c8c051a6","基准分其实是难度的加权平均:25 个主流基准的体检报告","benchmark-difficulty-measure-coverage-gaps","2026-11-01T01:16:42+00:00",{"id":80,"title":81,"news_slug":82,"published_at":83},"4c0315b0-aae1-4483-ab12-3281f8b729d7","批量打分翻车:LLM 判决 11%–22% 会翻转","llm-judge-batch-grading-instability","2026-10-30T07:28:14+00:00",{"id":85,"title":86,"news_slug":87,"published_at":88},"c780f08d-fa7e-4c4b-8cf5-ee367bde4c06","零 RL 涨 42.9 分几乎不说「等等」:它来自训练数据","reasoning-wait-marker-provenance-rl-training-data","2026-10-25T05:15:00+00:00"]