Reinforcement learning with verifiable rewards (RLVR) is now the standard recipe for post-training language models on reasoning and agentic tasks, but its information bottleneck is rarely tallied: when every rollout in a group succeeds, or every one fails, the group's rewards have zero variance and produce no gradient. The paper offers a real snapshot: in one block of math training, 575 of 2,560 problems were all-fail and 156 all-pass, so 28.6% of the problems contributed no learning signal at all. The student reasoned, acted, and failed in legible ways — the training algorithm only received a 0.

A team from the University of Minnesota, Twin Cities and JPMorgan AI Research decided to reclaim that discarded experience, and the result is AutoScaffold.

How the teacher builds scaffolds

The idea borrows from educational scaffolding — Wood 1976 and Vygotsky 1978's notion of temporary, learner-contingent support that is withdrawn as competence grows. Throughout training, a frozen GPT-5.5 acts as the teacher. Every 10 steps it intervenes, issuing at most 15 queries into the training log: statistics, raw trajectories, whole groups of attempts on the same task, and its own past decisions. After diagnosing where the student is stuck, it writes scaffolds at two levels — class-level strategies shared across a task class, injected with a probability capped at 0.5, and instance-level hints for individual hard tasks; in math the hint is a prefix of the reference solution, with the ratio capped at 50%. Revisions are first validated on held-out tasks and only deployed if they beat the current scaffold. At evaluation, every scaffold is removed; all scores are the student's own.

The mechanism sits in the math of binary rewards: u(p) = 1 - (1-p)^G - p^G. Only groups that mix successes and failures carry learning signal; all-success and all-failure groups are dead. Problems where every rollout fails have a name in the paper: the learning cliff. The scaffold's job is to push all-fail groups into mixed ones — too weak and nothing moves, too strong and everyone coasts to success, which zeroes the signal again; what AutoScaffold does, in effect, is lower a rope over the cliff edge.

The scoreboard across four domains

  • ALFWorld (embodied household tasks, Qwen2.5-1.5B + GiGPO): 93.9% success, 7.2 points above the baseline, ahead of SkillRL (89.9%) and SkillC (90.6%), which use a larger 7B model;
  • Search-based QA (Qwen2.5-3B + GiGPO): 46.1 average exact match, 4.0 above baseline;
  • Math (Qwen3-4B, GRPO setting): 70.1 average, 1.4 above GRPO, on par with SAGE (70.0), the strongest hand-designed scaffolding method;
  • Math, second tier (Nemotron-1.5B, QuestA setting): 62.4, 1.3 above QuestA, using 350 steps versus QuestA's 1,100 — roughly one-third;
  • KernelBench (Qwen3-8B, StitchCUDA setting): Level 1 correctness from 59% to 63%, Level 2 from 50% to 54%, and the share of correct kernels faster than the PyTorch reference from 20% to 25%.

The details are more interesting than the totals

Two ablations sting: removing held-out validation costs 14.1 points on ALFWorld, and bolting on Skill0's published skill bank as a fixed scaffold lands 15.3 points below AutoScaffold and 8.1 below the plain RL baseline — guidance is not a case of more is better; guidance misaligned with the student's current ability is negative equity.

The paper also reproduces a real diagnosis from cycle 10 of math training: the teacher saw that most of the 575 all-fail problems were already at the maximum hint dose, so it pushed the low-dose ones upward — problems previously lifted from ratio 25 to 45 or 50 escaped all-fail about 55-60% of the time when seen again — and its two proposed class-level notes were rejected by the framework, because the eight notes already in force were the cap. The contrast is telling: QuestA's prefix ratio follows a hand-designed schedule that cuts from 50% to 25% at a fixed step, while AutoScaffold's mean ratio peaks at 33.7% and then withdraws problem by problem — problems reduced to a zero prefix grow from 130 to 303, and graduated problems rise from 46 to 90. Hint-free pass@1 on the held-out set climbs from 28.8% to 38.3%, so the student is not learning hint dependence. The bill is modest: teacher API cost runs 2.35 to 5.57 dollars per training run, validation rollouts run on the training cluster's own 4-8 H200 GPUs, and each update takes roughly 15 to 26 minutes in total; all scores are scaffold-free evaluations averaged over 3 seeds.

So what

The paper positions itself as AI-for-AI: no change to the RL algorithm, no change to model architecture, no extra training data — just handing the job of reading experience and writing prescriptions from human researchers to a frozen strong model, with positive gains across all four domains. The implicit claim is more interesting than the numbers: in the RLVR era, the bottleneck is shifting from reward design to the utilization of training experience. The reward tells you right or wrong; a teacher can tell you why. With teacher models renting for a few dollars, who reads the training logs looks like the next job to be automated.