In 2025 alone, 48 new causal language-model architectures were added to the Hugging Face transformers library. Every one of them ships with a technical report, and every report has to state what this time is new. But whether something is new, and whether the new thing actually helps, are two different questions.
A paper auditing 40 open-weight LLM technical reports takes those two questions apart. Of the 40 reports, 28 make architectural or training novelty claims, totalling 73 claims. Only 19 of those (26%) come with a controlled comparison, where model size, data and token count are held fixed and only the component claimed to be new is swapped in. Another 20 (27%) come with a comparison that changes other things at the same time, and 34 (47%) come with no comparison at all. Counted by report, 11 test at least one claim in a controlled way, and only 4 test every claim they make.
The code cannot lie, but the report can
The most interesting move is not reading the reports at all. It is reading the reference code in the transformers library. Of the 48 architectures added in 2025, 39 (81%) take at least two of their three core blocks — attention, feed-forward, decoder layer — directly from another model, most often Llama. Going back, 2024 was 17 of 30 (57%). Across the library, 288 core blocks are taken from another model, and Llama alone lends out 80 of them, used by 47 architectures.
The paper immediately hits the brakes. Part of that rise comes from how the library itself writes code. From around 2024, transformers began writing new models as edits of older ones, so taking an existing block is the normal way to add an architecture. The paper states plainly that it cannot divide the code-convention change from genuine design convergence.
It then goes line by line on Qwen3 against Llama: the difference is QK-norm, two added lines and two changed lines in the code, plus a sliding-window switch that is off by default. I checked the modeling file for Qwen3 in the transformers repository, and q_norm and k_norm are both there, with the sliding-window line commented as differing from Llama. A code comparison does not tell you a change did not work, only where the change is — it locates the claim so you know which experiment should have tested it.
An untested claim may still be true
The paper is remarkably restrained: it does not judge model quality, and an untested claim may still be true. It separates two kinds of readers. Someone deciding whether to use a model needs comparisons between complete models; someone deciding which idea to build on next needs controlled comparisons. A report can serve the former well and leave the latter with nothing.
The code layer confirms that split. The paper read the attribution statements in seven reports — Qwen2, Qwen3, OLMo 2, Falcon, Falcon Mamba, Mamba and Mamba-2 — and all credit their sources. So the problem is not who is blowing smoke; it is the gap between a claim and the evidence for it. The paper lands on two limits: an unclean comparison is not a dishonesty problem but a boundary on what a report can show, and marking a claim as untested reads weaker than leaving it to speak for itself. The paper accepts that cost and argues the weaker statement is the accurate one.
The reproducibility column is nearly empty
A harder finding is reproducibility. Only 4 of the 40 reports state more than one training run, or give a spread across runs. The remaining 36 leave readers unable to tell how much weight a result can bear. None of the 26 reports from 2024 onward does. The comparison baseline is equally thin: 8 reran the models they compare against, with the Mistral 7B report the paper's named example, but 22 lift comparison scores straight from other reports and only 8 include any third-party evaluation.
Disclosure of the three-part set is more direct: 29 give the pre-training token count, 12 give the compute, and only 1 gives all three together. The data mixture is held to a strict test — whether another party could rebuild the mixture from the description — and only 2 reports pass. That 2 is the least certain number in the paper, because the two coders agreed on this field with a Cohen's kappa of only 0.30.
The paper proposes seven reporting rules and demonstrates them on two reports. OLMoE meets three of them, stating a total of 5.133T training tokens and 256 H100 GPUs for approximately 10 days, and running all evaluations itself; I checked the OLMoE paper and both statements match word for word. Mistral 7B meets two, and gives none of the three disclosure items. On the paper's own scoring, 24 of the 40 reports meet not one rule.
Why weak evidence survives
A release report usually bundles things: architecture, data, token budget and training procedure all change together, and then the finished model is compared against other finished models. The bundled result can be strong, and that attributes nothing to any single part. These changes also fail at a high rate to begin with. The paper cites two independent studies: an EMNLP 2021 work that tested many Transformer modifications in one shared implementation and found most did not improve performance, and a 2026 follow-up at 1-3B scale where only two of 20 modifications cleared a multiple-comparison correction, one of which would not even train stably at 3B. A change appearing inside a successful model is weak evidence on its own.
The paper also points to a structural cause: all 40 reports were taken from arXiv as preprints, and release reports as such are not peer reviewed before they appear. Review cannot guarantee a clean comparison, but it would at least ask whether the evidence bears on the claim, and by the time a report is first read, nobody has asked. The audit has its own limits, which the paper lists: the data comes from one snapshot of one library on 13 November 2026, the 80 architectures that link no arXiv paper are excluded, and the 40 reports were coded by Claude Opus 5.5 and OpenAI GPT-6.1 Sol, with disagreements settled by xAI Grok 4.7, with no human check yet.
So what
The easiest misreading is that everyone is blowing smoke. The paper refutes that reading itself: no known component was found renamed, vendor attribution is accurate throughout, and an untested claim may still be true. Its actual conclusion is narrower and harder — nobody pulls the line between a claim and its evidence, and pulling that line costs almost nothing.
Rule one is worth pulling out on its own: for every part presented as new, report a controlled comparison with and without it, or say that none was run. The second half is free, and costs no compute. Only 19 of the claims across those 40 reports come with that controlled comparison. That single report giving all three disclosure items is worth reading as an industry baseline rather than a passing grade. The gap between the two is the gap between 'this model is strong' and 'this idea is worth building on'.