Letting AI agents run post-training is no longer novel, but researchers who actually tried it observed an awkward pattern: hand a language model agent the full training stack, and it almost always converges on supervised fine-tuning (SFT) — spending most of its effort on one thing, revising the training data. A team affiliated with the TsinghuaNLP lab (code released under the Thinking-Space organization) put this problem on the table with AutoSFT, a multi-agent system built specifically to automate SFT data research. The paper is on arXiv (2610.22194, 27 pages, 16 figures).
How the single-agent loop fails
The paper first calls out the limitation of existing systems: they give the agent exactly one kind of control — either selecting rows from a fixed pool or calling a generator — whereas a human data engineer adjusts proportions, prunes, repairs, replaces, and synthesizes data. The authors designed a data action space covering those changes and first let a single agent run the full research loop. It learned exactly one action, adjusting category proportions, while its context filled up with execution output and the search was discarded mid-run.
Three-layer division: execute, analyze, synthesize
AutoSFT's answer is multi-agent division of labor: a meta agent separates execution from analysis, gathers proposals from independent angles, and calls a synthesis subagent for capabilities the data pool cannot cover — with the model, the trainer, and the budget fixed within each comparison. Ablations validate each component in turn: the executor halves context loss, the analysis subagents lift the best IFEval score by 6.5 points, and synthesis adds another 3.4 IFEval and 2.9 IFBench. The same loop transfers to code, where analysis adds +3.7 MBPP+ over a single-agent baseline that does not improve at all. Notably, every result comes from runs that retrain from the base model each round, so each component's effect is individually attributable.
Commentary: action coverage beats single-action strength
The takeaway the authors themselves state in the paper (arXiv:2610.22194): progress in automated SFT data research depends more on covering diverse data actions than on making a single action stronger. That mirrors how human data teams work — recipe tuning, cleaning, and gap-filling each have owners. In the community discussion, the authors position the work as "a first step toward RSI-Service for post-training," an ambition that goes beyond automating the data pipeline. The cold water is equally clear: the GitHub repository (Thinking-Space/AutoSFT) currently holds only a description page with source code marked "coming soon," so third-party replication has no path yet and all numbers above are self-reported in the paper. For post-training teams, the more immediately useful part is the architecture itself — separating execution from analysis and explicitly modeling a data action space are two moves any data team can borrow today.
So the next time you see an "AI improving AI" narrative, ask one question first: how many distinct data operations does its action space cover? AutoSFT's answer is trading single-agent depth for multi-agent coverage — a direction worth tracking.