Reasoning large language models (RLLMs) work by progressively decomposing complex problems to reach answers, performing well on advanced coding and multi-step planning tasks. But their training process faces a serious efficiency bottleneck: an MIT research team found that in RL training, the rollout phase generating multiple candidate answers occupies up to 85% of execution time, while the actual model weight update — the real training part — takes relatively little. When some high-performance processors are busy generating candidate answers, others can only sit idle, wasting huge amounts of compute.
To address this, a joint team from MIT, NVIDIA, ETH Zurich, the MIT-IBM Watson AI Lab, and UMass Amherst proposed an adaptive training method: a smaller, faster auxiliary model predicts the main reasoning model's output, and the main model verifies those predictions. When some processors are idle, the auxiliary model takes over their compute; when the main model needs to verify, the auxiliary model pauses. This adaptive scheduling ensures no chip in the GPU cluster sits idle, doubling training speed without losing accuracy, while reducing energy consumption and cost.
The broader significance of this work: it exposes a systemic flaw in the current RL training paradigm. As the industry universally pursues larger parameters and more compute, the efficiency of the training process itself is often ignored. For the industry, the takeaway is this: more efficient RL training methods mean future reasoning models can be trained with fewer resources; optimization of the training system itself may be the key lever for the next stage of AI progress.