In AI slang, 'alchemy' means the craft of training and tuning models. A preprint posted November 17 by authors from Sun Yat-sen University and other institutions (https://arxiv.org/abs/2611.23138) argues that this craft is now something a model can learn. RSI-Prime packs 350 complete fine-tuning experiments into a dataset; after training on it, GLM-5.3 lifts its PostTrainBench overall score from 36.56 to 48.24, a 31.95% relative gain as measured by the paper.
The experiment process becomes data
First, the arena. PostTrainBench, built by an independent team, hands an agent one H100 GPU, a 10-hour budget and a Qwen3-4B-Base target, then lets it find data, train, iterate and submit a model. Closed-source pairs lead this board: per the paper's table, Fable 5.1 + Claude Code sits at 53.68, while GLM-5.3 starts at 36.56.
RSI-Prime turns the experiment process itself into supervision: 350 complete trajectories across seven task categories, decomposed into 90,738 decision records. A trajectory only enters the set if its submitted model clears a bar — relative improvement exceeding half the reference Instruct-Base gap (g>0.5) — and it must pass PostTrainBench's four trajectory judges; failed intermediate attempts stay in context.
Good runs contain bad decisions
The sharpest observation in the paper: a successful experiment does not mean every decision deserves imitation. Among five action categories, evaluation and diagnosis account for 38.43% of records and data preparation 27.09% — the workload's center of gravity is not training execution itself. 92.42% of records are labeled high quality, but implementation-optimization decisions show a 16.89% problem rate, and data-source selection shows 12.80%. Blind imitation learns the pitfalls along with the skill.
Their fix is decision-level loss masking: an assistant message is supervised only when every decision linked to it passes quality screening; otherwise the whole message is masked. The conservative rule keeps 76.25% of supervised messages and 75.48% of supervised tokens — 68,027 of 89,211 messages. Quality labels come from GLM-5.3 itself, three assessments combined by majority voting, plus a 400-record human audit with acceptance rates between 97.50% and 99.50%; the authors state that this audit does not estimate the recall of decision identification.
Less supervision, higher scores
Here is the counterintuitive part. Full-trajectory fine-tuning takes GLM-5.3 from 36.56 to 45.61, and decision-level supervision pushes further to 48.24, improving all six sub-tasks, with AIME 2025 up 12.23 points and HealthBench up 26.63 points. Both natural controls fall behind: masking on tool failures, which keeps 89.15% of supervised tokens, scores only 45.12; step-importance selection adapted from ATLAS, which keeps 76.63%, scores 43.21 — nearly the same supervision volume as decision-level, yet 5.03 points lower.
Behavior improves too: on evaluation trajectories the decision-quality pass rate reaches 96.11%, against 87.25% for full-trajectory training. Per the paper's table, 48.24 already sits above Gemini 4 Argon (47.92), Opus 5.5 (47.84), GPT-6.1 Sol (46.45) and Fable 5 (46.31), 5.44 points behind the leader, Fable 5.1 + Claude Code.
Caveats worth stating
- The training corpus contains HumanEval-related content, which the paper itself flags as train-evaluation overlap on that task;
- The three reference runs of baseline GLM-5.3 scored 25.79, 37.03 and 52.57 on Arena Hard Writing — sizable run-to-run variance, and three-run averaging was applied only to the original agent;
- The quality annotator is the very model being fine-tuned, and the human audit covers 400 records;
- All benchmark numbers are self-reported by the paper, with no independent replication yet.
Not a lone data point
The direction is not one team's bet. RSI-Dojo, released in October by a different group (https://arxiv.org/abs/2611.03698), likewise teaches agents post-training from curated trajectories. An August empirical study by Lim et al. (https://arxiv.org/abs/2608.19072) analyzed 1,338 post-training trajectories and found that only 2.1% of transitions between adjacent training runs ever change strategy — agents repair implementations but rarely revise their plans. Decision-level supervision targets exactly this gap: teaching the model when to change approach, not just how to run a fixed pipeline smoothly.
So what: alchemy experience has become a reusable data asset, and knowing how to train models is now a capability you can fine-tune into a model. The sharper question — what happens when a model trained this way goes on to train the next generation, and the loop starts turning — is left open by the paper. But the road is now paved to that door.