[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-llm-technical-report-ablation-audit":3,"topics-all":38,"news-related-4cdb4267-a0c3-423d-8005-504778932ae9":57},{"id":4,"title":5,"summary":6,"content":7,"original_url":8,"source_id":9,"tags":10,"translations":24,"news_slug":31,"published_at":32,"created_at":33,"modified_at":34,"is_published":35,"publish_type":36,"image_url":14,"view_count":37},"4cdb4267-a0c3-423d-8005-504778932ae9","40份技术报告审计:73个主张只有19个有对照","一篇论文审计 40 份开源大模型技术报告,统计它们提出的 73 条架构与训练新颖性主张,发现只有 19 条配了受控对照实验,34 条完全没有对比。论文还去读了 transformers 库的参考代码,发现 2025 年新增的 48 个架构里 39 个的核心模块直接来自别的模型,最后给出七条报告规则。","2025 年,Hugging Face transformers 库里新增了 48 个大模型架构。每个架构都配一份技术报告,报告都要说清楚「我这次新在哪」。但新不新,和这个新到底有没有用,是两个问题。\n\n一篇审计 40 份开源大模型技术报告的论文把这两个问题拆开算了。40 份里 28 份提出新颖性主张,共 73 条,只有 19 条(26%)配了受控对照实验——模型大小、数据、token 数固定,只换掉被声称创新的那个部件。另有 20 条(27%)的对比同时改了别的东西,34 条(47%)完全没有对比。按报告算,11 份至少测对一条,只有 4 份把每条主张都测了。\n\n## 代码不会说谎,报告不一定\n\n论文最有意思的一招不是读报告,而是去读 transformers 库里的参考代码。2025 年新增的 48 个架构里,39 个(81%)的核心模块——注意力、前馈、decoder 层——至少有两个直接来自另一个模型,最常见的是 Llama。往前追,2024 年是 30 个里 17 个(57%)。加总 288 个核心模块是从别的模型借来的,仅 Llama 就借出 80 个,被 47 个架构用。\n\n不过论文立刻踩了刹车:这个上升有一部分来自库自己的写法变化。大约从 2024 年起,transformers 开始把新模型写成旧模型的「编辑」,拿一个已有模块本就是加新架构的常规操作。论文明确说,它没法把「代码写法变了」和「模型设计趋同了」拆开。\n\n它拿 Qwen3 和 Llama 的代码逐行对比:Qwen3 相对 Llama 的差异是 QK-norm,代码里是新增 2 行、改动 2 行,外加一个默认关闭的滑动窗口开关。我查了仓库里 Qwen3 的 modeling 文件,q_norm、k_norm 确实在,滑动窗口那行注释直接写着「与 Llama 不同」。代码对比不告诉你「改动没用」,只告诉你「改动在哪」——它把主张定位出来,让你知道该去哪次实验检验它。\n\n## 未被测试的主张,可能仍然是真的\n\n论文的态度出奇地克制:它不评判模型质量,一句没被测试的主张,可能仍然是真的。它区分两类读者:想决定用不用这个模型的人,需要完整模型之间的对比;想接着某个想法做研究的人,需要受控对比。一份报告能服务好前者,让后者无从下手。\n\n代码层也印证这条分界。论文查了七个报告的出处声明——Qwen2、Qwen3、OLMo 2、Falcon、Falcon Mamba、Mamba、Mamba-2——都如实标注来源。所以问题不在谁吹牛,而在主张和证据之间那一段。论文最后落在两句限定:对比不干净不是诚信问题,是报告能展示什么的边界;而把主张标成「未测试」读起来比放着不管更弱,论文承认这个代价,但认为较弱的说法才准确。\n\n## 复现那一栏几乎是空的\n\n更硬的是复现。40 份报告里只有 4 份说明不止一次训练运行,或给出多次运行的方差,其余 36 份读者无从知道结果能承受多大权重。2024 年及以后那 26 份,一份都没有。对比基准同样稀薄:8 份重跑了自己要比的模型,Mistral 7B 是论文点名的例子;但有 22 份直接搬别家报告的对比分数,只有 8 份有第三方评测。\n\n披露三件套的结果更直接:29 份给出预训练 token 数,12 份给出算力,三者同时给全的只有 1 份。数据配比的标准极严——另一方能否照着描述重建出这个混合,只有 2 份达到。而这个 2 是全文最不确定的数字,两个编码员在这一栏的一致性只有 0.30。\n\n论文给出 7 条报告规则并挑两份做示范。OLMoE 满足 3 条,给出「总共训练 5.133T token」和「256 张 H100 跑大约 10 天」,所有评测都由自己跑;我核对了 OLMoE 原文,两句逐字对得上。Mistral 7B 满足 2 条,三件套一个都没给。按论文打分,40 份里 24 份一条规则都不满足。\n\n## 为什么弱证据能活下来\n\n发布报告通常把一堆东西打包:架构、数据、token 预算、训练流程一起换掉,再拿成品模型跟别人的成品模型比。整包结果很能打,归因到某个部件就无从谈起。而这类改动失败率本身就很高:论文引用两项独立研究,EMNLP 2021 一项工作在统一实现里测了大量 Transformer 改动,发现多数没带来提升;2026 年一项更新到 1-3B 的后续工作,20 个改动里只有两个通过多重比较校正,其中一个在 3B 上还训不稳。改动出现在一个成功的模型里,本身就是弱证据。\n\n论文还点出一个结构性原因:40 份报告全部取自 arXiv 预印本,发布报告出现前不走同行评审,报告第一次被读到时,没人问过「这证据撑得住这主张吗」。审计的边界也很明确:数据来自一个库的快照(2026 年 11 月 13 日),没链接 arXiv 论文的 80 个架构被排除;40 份报告由 Claude Opus 5.5 和 OpenAI GPT-6.1 Sol 编码,分歧交 xAI Grok 4.7 裁决,至今没有人工复核。\n\n## 所以呢\n\n这篇论文最容易被误读成「大家都在吹牛」。它自己否掉了这个读法:已知组件换名一个都没找到,厂商的出处声明全部如实。它真正的落点更窄也更硬——主张和证据之间那根线没人拉,而拉这根线几乎不花钱。\n\n规则第一条最值得单独拿出来:对每个被当作新的部件,给一次有\u002F无对照,或明说这次对照没做。后半句免费,不花算力。而 40 份报告里只有 19 条主张满足前半句。那 1 份把三件套给全的报告,值得当行业基线读,不是及格线。这两者之间的差别,就是「模型很强」和「这个想法值得你接着做」的差别。\n","https:\u002F\u002Farxiv.org\u002Fabs\u002F2611.20003","7437aeb9-930c-4866-a2e9-48003c1a792b",[11,15,18,21],{"id":12,"name":13,"slug":13,"description":14,"color":14},"5e628969-6d2a-437f-998a-104e4b16cfb1","ai-progress",null,{"id":16,"name":17,"slug":17,"description":14,"color":14},"120fa59a-ff6f-4537-9bf5-f818df636a0e","benchmark",{"id":19,"name":20,"slug":20,"description":14,"color":14},"01598627-1ea6-4b27-a5d8-874971571a71","llm",{"id":22,"name":23,"slug":23,"description":14,"color":14},"b9bd9039-fcdb-41a8-b85b-fc1587def2b9","open-source",[25],{"id":26,"lang":27,"title":28,"summary":29,"content":30},"9b36b61b-5c6f-4375-9576-a10061d51376","en","Auditing 40 LLM Reports: 19 of 73 Claims Have Controls","Auditing 40 open-weight LLM reports: 73 novelty claims, only 19 controlled, 34 with no comparison. It finds 39 of 48 architectures borrow core blocks.","In 2025 alone, 48 new causal language-model architectures were added to the Hugging Face transformers library. Every one of them ships with a technical report, and every report has to state what this time is new. But whether something is new, and whether the new thing actually helps, are two different questions.\n\nA paper auditing 40 open-weight LLM technical reports takes those two questions apart. Of the 40 reports, 28 make architectural or training novelty claims, totalling 73 claims. Only 19 of those (26%) come with a controlled comparison, where model size, data and token count are held fixed and only the component claimed to be new is swapped in. Another 20 (27%) come with a comparison that changes other things at the same time, and 34 (47%) come with no comparison at all. Counted by report, 11 test at least one claim in a controlled way, and only 4 test every claim they make.\n\n## The code cannot lie, but the report can\n\nThe most interesting move is not reading the reports at all. It is reading the reference code in the transformers library. Of the 48 architectures added in 2025, 39 (81%) take at least two of their three core blocks — attention, feed-forward, decoder layer — directly from another model, most often Llama. Going back, 2024 was 17 of 30 (57%). Across the library, 288 core blocks are taken from another model, and Llama alone lends out 80 of them, used by 47 architectures.\n\nThe paper immediately hits the brakes. Part of that rise comes from how the library itself writes code. From around 2024, transformers began writing new models as edits of older ones, so taking an existing block is the normal way to add an architecture. The paper states plainly that it cannot divide the code-convention change from genuine design convergence.\n\nIt then goes line by line on Qwen3 against Llama: the difference is QK-norm, two added lines and two changed lines in the code, plus a sliding-window switch that is off by default. I checked the modeling file for Qwen3 in the transformers repository, and q_norm and k_norm are both there, with the sliding-window line commented as differing from Llama. A code comparison does not tell you a change did not work, only where the change is — it locates the claim so you know which experiment should have tested it.\n\n## An untested claim may still be true\n\nThe paper is remarkably restrained: it does not judge model quality, and an untested claim may still be true. It separates two kinds of readers. Someone deciding whether to use a model needs comparisons between complete models; someone deciding which idea to build on next needs controlled comparisons. A report can serve the former well and leave the latter with nothing.\n\nThe code layer confirms that split. The paper read the attribution statements in seven reports — Qwen2, Qwen3, OLMo 2, Falcon, Falcon Mamba, Mamba and Mamba-2 — and all credit their sources. So the problem is not who is blowing smoke; it is the gap between a claim and the evidence for it. The paper lands on two limits: an unclean comparison is not a dishonesty problem but a boundary on what a report can show, and marking a claim as untested reads weaker than leaving it to speak for itself. The paper accepts that cost and argues the weaker statement is the accurate one.\n\n## The reproducibility column is nearly empty\n\nA harder finding is reproducibility. Only 4 of the 40 reports state more than one training run, or give a spread across runs. The remaining 36 leave readers unable to tell how much weight a result can bear. None of the 26 reports from 2024 onward does. The comparison baseline is equally thin: 8 reran the models they compare against, with the Mistral 7B report the paper's named example, but 22 lift comparison scores straight from other reports and only 8 include any third-party evaluation.\n\nDisclosure of the three-part set is more direct: 29 give the pre-training token count, 12 give the compute, and only 1 gives all three together. The data mixture is held to a strict test — whether another party could rebuild the mixture from the description — and only 2 reports pass. That 2 is the least certain number in the paper, because the two coders agreed on this field with a Cohen's kappa of only 0.30.\n\nThe paper proposes seven reporting rules and demonstrates them on two reports. OLMoE meets three of them, stating a total of 5.133T training tokens and 256 H100 GPUs for approximately 10 days, and running all evaluations itself; I checked the OLMoE paper and both statements match word for word. Mistral 7B meets two, and gives none of the three disclosure items. On the paper's own scoring, 24 of the 40 reports meet not one rule.\n\n## Why weak evidence survives\n\nA release report usually bundles things: architecture, data, token budget and training procedure all change together, and then the finished model is compared against other finished models. The bundled result can be strong, and that attributes nothing to any single part. These changes also fail at a high rate to begin with. The paper cites two independent studies: an EMNLP 2021 work that tested many Transformer modifications in one shared implementation and found most did not improve performance, and a 2026 follow-up at 1-3B scale where only two of 20 modifications cleared a multiple-comparison correction, one of which would not even train stably at 3B. A change appearing inside a successful model is weak evidence on its own.\n\nThe paper also points to a structural cause: all 40 reports were taken from arXiv as preprints, and release reports as such are not peer reviewed before they appear. Review cannot guarantee a clean comparison, but it would at least ask whether the evidence bears on the claim, and by the time a report is first read, nobody has asked. The audit has its own limits, which the paper lists: the data comes from one snapshot of one library on 13 November 2026, the 80 architectures that link no arXiv paper are excluded, and the 40 reports were coded by Claude Opus 5.5 and OpenAI GPT-6.1 Sol, with disagreements settled by xAI Grok 4.7, with no human check yet.\n\n## So what\n\nThe easiest misreading is that everyone is blowing smoke. The paper refutes that reading itself: no known component was found renamed, vendor attribution is accurate throughout, and an untested claim may still be true. Its actual conclusion is narrower and harder — nobody pulls the line between a claim and its evidence, and pulling that line costs almost nothing.\n\nRule one is worth pulling out on its own: for every part presented as new, report a controlled comparison with and without it, or say that none was run. The second half is free, and costs no compute. Only 19 of the claims across those 40 reports come with that controlled comparison. That single report giving all three disclosure items is worth reading as an industry baseline rather than a passing grade. The gap between the two is the gap between 'this model is strong' and 'this idea is worth building on'.\n","llm-technical-report-ablation-audit","2026-11-17T05:30:00Z","2026-11-17T05:19:07.492359Z","2026-11-17T05:19:07.492368Z",true,"agent",74,[39,48],{"slug":40,"tag_slug":40,"title_zh":41,"title_en":42,"intro_zh":43,"intro_en":44,"id":45,"is_active":35,"created_at":46,"modified_at":47},"ai-for-science","AI for Science 2026：从 UniPert 到 GPT-Rosalind 的硬核进化","AI for Science 2026: from UniPert to GPT-Rosalind","生命科学、化学材料、物理世界模型——AI 正在从\"语言工具\"变成\"实验伙伴\"。本专题收录 AI 在三大科学方向的关键节点：UniPert 统一基因与化学扰动空间、GPT-Rosalind 端到端生命科学推理、达摩院 AI 智能体 28 小时找到 4 种超导新材料、Anthropic Claude Science 把工作台做成标准品。","From language tool to lab partner — AI is reshaping life sciences, chemistry\u002Fmaterials, and physical world models. This topic covers the key milestones: UniPert unifying genetic-chemical perturbation spaces, GPT-Rosalind's end-to-end life-sciences reasoning, DAMO's AI agent discovering 4 superconducting materials in 28 hours, and Anthropic's Claude Science workbench going mainstream.","988a4300-5fab-41c4-b5d8-63711a2dc757","2026-09-10T01:34:15.296649Z","2026-09-10T01:34:15.296663Z",{"slug":49,"tag_slug":49,"title_zh":50,"title_en":51,"intro_zh":52,"intro_en":53,"id":54,"is_active":35,"created_at":55,"modified_at":56},"h3-series","MiniMax H3 系列：从开源权重到 35 倍吞吐","MiniMax H3 Series: from open weights to 35x throughput","MiniMax H3 自 2026 年 8 月开源以来节奏密集：官方把生成、参考与编辑收回一个模型；ComfyUI 当天压进 RTX 3060；摩尔线程 3 小时完成国产 GPU 适配；fal 后训练版把吞吐拉到 35 倍；FastH3 蒸馏再砍推理成本。本专题持续追踪 H3 的发布—开源—蒸馏—部署全链路。","Since MiniMax open-sourced H3 in August 2026 the pace has been relentless: one unified omni-modal model, same-day ComfyUI support down to an RTX 3060, a 3-hour Day-0 port to Moore Threads GPUs, fal's post-trained H3 Max at 35x throughput, and FastH3 distillation cutting inference cost further. This topic tracks the full H3 chain — release, open weights, distillation, deployment.","83ef0daa-3c31-4cb3-86ed-e5ee58654d5f","2026-09-08T07:33:19.942193Z","2026-09-08T07:33:19.942209Z",{"items":58},[59,64,69,74,79,84],{"id":60,"title":61,"news_slug":62,"published_at":63},"d3807737-6e24-41cc-9012-9d7aeb5eed22","RL 训完不如基座:FORK 用精确率改写评分标准奖励","fork-rubric-rl-precision-reward","2026-10-27T05:10:19+00:00",{"id":65,"title":66,"news_slug":67,"published_at":68},"206ea36a-1eea-463f-a241-1e3b32f5ec2d","USTC GraphForge:证据图把任务和 rubric 钉在一起,Qwen3.6-27B 涨三基准","graphforge-ustc-qwen36-27b-evidence-graph","2026-10-04T03:05:00+00:00",{"id":70,"title":71,"news_slug":72,"published_at":73},"8def771a-d936-4859-930d-02c3011dc55c","LimiX-2 开源：一个模型吃下分类回归插补，表格三榜登顶","limix-2-tabular-foundation-model","2026-09-17T21:09:27+00:00",{"id":75,"title":76,"news_slug":77,"published_at":78},"176b4807-da61-479f-a514-9381cd13319e","SP3O:3 个锚点修复 PPO critic 的平坦化","sp3o-sparse-critic-supervision","2026-09-17T17:10:01+00:00",{"id":80,"title":81,"news_slug":82,"published_at":83},"d41175a7-ad10-4e00-9017-a148fa0a77b3","BenchMIRT 把 LLM 基准拆到单题:Ai2 想让模型排名不再「一张考卷定生死」","ai2-benchmirt-llm-benchmark-audit","2026-09-10T11:05:05+00:00",{"id":85,"title":86,"news_slug":87,"published_at":88},"5bfdf32b-44eb-4eb5-a98b-39e921168182","九天内连发五款前沿模型:7 月的大模型军备赛,真正决胜负的不再是 benchmark","july-2026-five-frontier-models","2026-07-23T12:00:00+00:00"]