A 27B-parameter open-weight model has taken the top spot on a machine-learning engineering benchmark, outscoring every closed flagship in its comparison table — GPT-5.6 Sol, Claude Opus 4.8, GPT-5.5 — while holding roughly one hundredth of the parameters of the 2.8T Kimi-K3. The model is AREX-2 from the Beijing Academy of Artificial Intelligence (BAAI): Apache 2.0, weights on Hugging Face, paper 2609.38288 first posted in September, and this week it went viral across overseas developer communities — a daily.dev explainer picked up 538 upvotes and Sam Witteveen published a dedicated video.
The loop was never in the model
The paper opens by naming the industry default: in most agent systems the retry loop lives largely in the scaffold. The harness decides when to retry and what to keep; the model mostly produces one attempt at a time. AREX-2 flips the question: how do you move the loop inside the model? The model runs its own propose–measure–reflect–revise loop over multiple test-time rounds, and it is told its budget of rounds and wall-clock time up front — on the order of hours and hundreds of tool calls per task. The detail matters: knowing the budget changes how the agent works. It establishes a baseline first, spends early rounds measuring, and saves revisions for later, instead of submitting its best guess at once. That is the paper’s definition of self-improvement: turning more rounds on one task into a better solution — test-time scaling within a single task.
Training data that keeps the failures
The data construction is the tasteful part. A teacher model compiles GitHub repositories and online-judge problems into executable environments — a task plus a scoring function. An agent then works each task over many rounds, and trajectories are selected whole: kept when the final score is high and the process meets requirements, with no requirement that any individual round succeed. Failed runs, regressions, and abandoned approaches all stay in the training data, on the explicit logic that keeping them is what teaches the model to keep iterating after a setback. Training itself fine-tunes Qwen3.8-27B on a mixture of these trajectories and the existing AREX deep-research data — the paper says fine-tune, and its method section contains no reinforcement learning; daily.dev’s write-up calls out the same gap between the word “reinforcing” in launch language and the supervised fine-tuning actually used. And note the boundary: this self-improvement is test-time only. Weights do not change between tasks; recursive self-improvement would require feeding trajectories back into training, which the paper itself lists as the next step.
Read the scores together with their caveats
Per the model card: MLE-bench Lite (Any Medal averaged over three seeds) at 81.8 is the highest in its comparison table — GPT-5.6 Sol 72.7, Kimi-K3 (2.8T) 72.7, Naive-N0.5-Flash (309B) 73.7, Claude Opus 4.8 63.6. Frontier-CS (the 188-task Agent Track) at 70.7 is the best among open weights but sits below Sol’s 76.4, Opus 4.8’s 74.5, and GPT-5.5’s 72.1. On deep research, with no new training data added: BrowseComp 84.0, HLE 52.6 (text-only subset), GAIA 92.2, DeepSearchQA 93.8 — the GAIA number beats every closed model in the table (GPT-5.5’s 87.4 is the frontier best), though the 35B XYZ-Aquila-mini still posts 97.1 in the same table, so talk of crushing would be premature. The ablation explains the mechanism best: the base model alone scores 28.8; general skills lift it to 41.2; task-specific skills plus more rounds — still untrained — reach 68.2; swapping in the trained AREX-2 reaches 81.8, a net training contribution of 13.6 points. The trained model with fewer rounds beats the untrained one with more rounds by 7.6 points, and more rounds still buy another 6.0 — on Frontier-CS the curve keeps rising through the fifth hour.
Three takeaways
First, on long-horizon tasks parameter count is not the bottleneck — trajectory data shape is. The 2.8T Kimi-K3 and the 1.6T DeepSeek-V4-Pro (54.5) are both left behind by a 27B model; what differs is whole-improvement-trajectory data, not parameters bought with compute. Second, this is a direct challenge to the external-harness route: this platform covered DeepSeek’s pluggable external Harness in September, and BAAI now welds the loop into the weights — whether the loop lives in the harness or in the model is now a two-sided bet. Third, do not relay test-time self-improvement as RSI: weights are fixed, nothing accumulates across tasks, and the paper itself says the recursive version starts when the model’s own trajectories become its next training data.
Reproduction is unusually open: the VectorSpaceLab/AREX-2 repository (49 stars, 80 commits) ships evaluation runners and dataset definitions for six tracks, with the four deep-research benchmarks sharing one default profile. For anyone building agents, the question is changing — from “is my harness good enough?” to “does my model improve itself?”