On August 20, the open-weight model family Ornith-1.5 landed on Hugging Face and quickly climbed to the Hacker News front page with 165+ points and 58+ comments. The official pitch is loud: the 397B flagship scores 86.1 on Terminal-Bench 2.1, narrowly beating Claude Opus 4.8 at 85.0, MIT-licensed, shipped in 397B/35B/9B tiers. But almost at the same time, a set of independent community test numbers poured cold water on that report card.
The Self-Reported Scorecard: Winning by a Hair
According to the official tables compiled by explainx, Ornith-1.5-397B reports 86.1 on Terminal-Bench 2.1 (Claude Opus 4.8 at 85.0) and 88.4 on SWE-bench Verified (versus 87.6); on DeepSWE, however, its 56.0 actually trails Opus at 59.0. The team notes the figures are averaged over five runs, with anti-cheating measures in place — stripping git history from repos before evaluation and disabling network access during solving. In a benchmark scene plagued by reward-hacking scandals, that counts as a plus.
The technical story has genuine novelty too. Ornith-1.5 turns "self-improvement" into a training-time closed loop: the model generates its own tasks, builds its own scaffolds, and runs its own solution rollouts, with all three stages jointly trained via GRPO. The task-generation reward has three components — validity acts as a hard gate, with invalid tasks scoring zero no matter how hard they look; difficulty targets an empirical success rate around 0.2, where most attempts fail but rare successes still teach; and novelty serves as a secondary signal against regenerating near-duplicate busywork. The mechanism aims straight at the ceiling of fixed, human-curated task sets.
The Independent Run Tells a Different Story
The problem sits in the 35B tier. In the official tables, Ornith-1.5-35B reports 74.8 on Terminal-Bench 2.1, leaving Qwen3.6-35B-A3B at 52.5 more than twenty points behind. But as explainx reported, one HN user ran their own comparison of Ornith-1.5-35B against Qwen3.8-27B, and the direction flipped completely: on Terminal-Bench 2.1 it trailed 67.8 to 73.0, on SWE-bench Pro 59.6 to 61.7, on HLE 25.6 to 30.8; the largest gap was DeepSWE, 22.0 versus 42.2, nearly a two-fold difference. Across six comparisons, Ornith only overtook on NL2Repo, with GPQA Diamond tied.
Caveats apply: this was a single person's informal re-run, pitting a 27B dense model against a 35B MoE — not an apples-to-apples architecture match. But DeepSWE specifically stresses long-horizon software engineering, the hardest kind of benchmark to cram for, and a gap this large in that dimension is hard to explain away entirely by test-environment differences.
Self-Reported Is Self-Reported
The community dug up two more details. First, some users found in identity probes that the model "reliably claims to be Claude" — usually a sign of training-data contamination from Claude-generated text rather than copied weights. Second, HN research suggests the 397B is derived from Qwen3.5-397B-A17B via post-training, with Jiwei Li behind the team — both unconfirmed by the official announcement, so take them with a grain of salt.
To be fair, Ornith's self-improvement training method deserves study, and the quantized 9B-Mobile build, claimed to deploy on iPhone and Android, is a practical direction for the edge. But this episode repeats an old lesson: the gap between vendor tables and independent reproductions is structural in the open-model scene, and five-run averages do not change it. Next time you see a headline saying "open model beats Claude," ask first — who ran the benchmark, how many times, and in whose environment.