Most foundation-model training pipelines today look roughly the same: pour in web text, top up with domain data, align, then run reinforcement learning. A new paper from Carnegie Mellon University and Johns Hopkins University argues that this staged separation is not actually fundamental. Rewrite reinforcement learning as maximum-likelihood fitting to the model's own generation distribution, and the stages collapse into one weighted likelihood loss handled by a single trainer.

The paper does not stop at formulas. It runs a full comparison at two scales: a 124M-parameter model (20B-token pretraining corpus, 510M-token midtraining corpus) and a 0.76B-parameter model (240M tokens from 24 components, with Math12K supplying the math RL tasks and 11977 prompts surviving length filtering). The conclusion is subtler than "unified is better." The real difference shows up not in the current round's score, but in the bill that is deferred and in the starting point you hand to the next round of training.

The forgetting bill staged training leaves open

Start with the most alarming set of numbers. Before RL even begins, held-out pretraining loss (ClimbMix and Dolmino combined at a 75/25 weighting) is 2.537 when no midtraining happens at all, with a mean general-benchmark score of 42.37. Staged training puts midtraining before RL, at a peak learning rate of 2x10^-4, for one epoch: held-out loss jumps to 12.150 and the benchmark mean drops to 35.00. Four epochs gives 15.104 and 35.61. Lowering the peak learning rate to 5x10^-5 for one epoch still leaves held-out loss at 4.135 and the mean at 38.49.

In the paper's own retention comparison, then, staged midtraining wrecks the model's fit to the pretraining corpus several times over, and drags the benchmark score down with it. Unified training instead mixes the same midtraining data into the last 8147 pretraining updates (38147 updates in total, 256 packed sequences of 2048 tokens each, 20000014336 tokens processed). After that, held-out loss stays between 2.537 and 2.540 and the benchmark mean between 41.85 and 42.42, essentially level with the do-nothing baseline.

The real dividing line shows up in the next training round

If you look only at the math task, the two approaches are close to level. After 12000 steps of math RL, sweeping the midtraining mix from 0% to 5% moves MATH pass@1 within a narrow band of 16.11 to 16.91 while GSM pass@1 slides from 32.68 down to 30.80. In the same table, the base model is at 6.54 and 17.84, and the staged baseline is at 16.98 and 31.35, so staged actually edges slightly ahead. Retention, though, moves monotonically: 0% mixing gives +0.0737 (worse than the base model), 0.1% already gives -0.0154, and 5% gives -0.1474, against +0.0472 for the staged baseline. Even one part in a thousand of mixing leaves less forgetting than either baseline.

The coding bill only comes due in the next round. When math RL ends, the paper's own comparison puts the staged model at 15.51 on HumanEval+ and 25.74 on MBPP+, while the unified model mixed at 5% is at 10.90 and 20.97, so staged looks like it is ahead by nearly five points. Then every model gets the same code RL on top: 20000 Python problems from KodCode-V1, 4000 updates, evaluated over 164 HumanEval+ and 378 MBPP+ problems. All four unified models overtake staged on pass@1 for both benchmarks, and the 5% mixture posts 24.96 against 22.94 on HumanEval+ and 32.69 against 29.81 on MBPP+.

The paper then repeats the comparison under a total-compute accounting that estimates model FLOPs across pretraining, midtraining and RL. The larger the budget, the more clearly unified training leads on accuracy and coverage, and its edge is most stable on problems outside the training distribution. The authors caution, though, that the resulting frontier only characterizes the configurations they tested: it is neither a global optimum nor a scaling law with predictive power.

Three caveats worth stating

First, every number here comes from the authors' own experiments, and there is no third-party replication yet. The paper draws its own boundaries plainly: the compute study does not establish scaling laws with predictive power, how optimization should change with the data schedule and mixture design is not studied in depth, most runs optimize a single RL task at a time, and the authors explicitly acknowledge that they do not fully eliminate the notion of stages, since adapting a foundation model to a new downstream task will inevitably require a separate training stage.

Second, discount the throughput gap. Production logs record roughly 3.7 seconds per update for unified training at 0% mixing versus 28.8 seconds per update for staged RL, but the two systems use 24 and 16 GPUs respectively, the logs exclude launch and restart downtime, and the asynchronous trainer can reuse existing responses, so what is measured is optimization throughput rather than the rate of producing fresh rollouts.

Third, can regularizing toward the base model substitute for mixing data? The paper runs a KL-regularization comparison: raising the penalty does hold back the deterioration in held-out loss, but the strongest penalty visibly cuts mathematical accuracy and coverage. The conclusion is that within the range they tested, no penalty reproduces the combination of math performance, corpus loss, SciQ and MMLU accuracy that unified training achieves.

The engineering entry point is open: the yudasong/unified-training repository ships the trainer, the generator, experiment scripts, and a third-party NeMo-RL integration, with the README documenting three separate environment setups and requiring an NVIDIA driver of 580 or newer. The compute came from ASCR Leadership Computing Challenge project ERCAP0034861 plus the Oak Ridge and Argonne national laboratory facilities.

So what

Judging a training recipe is most easily misled by the scoreboard, since the recipe you are evaluating is the one that scores highest on the thing you happen to be testing today. The readings worth remembering from this paper are the gap between 2.537 and 12.150, and the curve where 15.51 on HumanEval+ becomes 10.90, then becomes 24.96 after 4000 more updates. One is a forgetting cost you cannot see in the moment; the other is a starting point you misread in the moment. The paper's own formulation fits neatly here: the value of a data source depends on when it is introduced and on how it interacts with subsequent learning. Unified training does not abolish stages. It turns "staged" from an architectural decision into a scheduling parameter you can change at will.

Full paper: Training Foundation Models with a Single Algorithm - project page - code repository