Looped language models have become a distinct line of attack over the past two years: instead of stacking ever more different layers, you run the same block of layers several times. Depth becomes a setting you can vary rather than something baked into the weights — which opens the door to the appealing idea of adaptive compute, spending two iterations on an easy token and eight on a hard one.

The trouble is deciding when to stop. The standard recipe adds a small halting head: after each iteration it outputs the probability that the token should stop there. Sound good in principle, but two separate gaps sit between "the head predicts a stop" and "compute is actually saved" — and a paper submitted to arXiv on October 23 (by Tsung-Han Wu, Minsuk Heo and Kenny Tan at Apple) walks straight into both of them.

Paper: arXiv:2610.32246

An average is a trap, not a saving

The first gap is the difference between an average and a percentile.

The probabilities a halting head outputs only become a real stopping decision once an exit rule converts them: the token halts as soon as the running total of its stopping probabilities reaches 1−ε (the paper uses a 90% threshold). That rule fires at a percentile of the probability distribution, not at the mean. The paper's illustrative token is the textbook case: 33% of its mass sits on iteration 1, 15% on iteration 2, and 18% on iteration 8, giving an average of 3.7 iterations. But the running total only crosses 90% at iteration 8, so the token dutifully runs all eight.

The authors pretrained looped models at four scales from 140M to 1.3B parameters on two corpora (FineWeb-Edu and Common Corpus). The result: heads trained with the mixture loss — the ACT-style training objective — predict that tokens need fewer than 4 of 8 iterations on average, yet under the 90% threshold those tokens actually run 7.6 to 7.9 iterations. Essentially no compute is saved.

Forcing them to stop earlier does not help either. The paper plots BPB (bits per byte, lower is better) against realized steps for each stopping threshold, next to a "sampled-depth baseline" trained at the same number of iterations: the mixture-loss model can save at most about one iteration before it falls behind the baseline. The compute saved by stopping early does not cover the quality it costs.

The training loss is the culprit

The second finding is blunter: the problem lives in the loss function.

The ACT-style mixture loss forms a single weighted mixture of a token's states across all iterations and scores only the prediction decoded from that mixture. It never scores any individual iteration's state on its own — yet the exit rule hands the readout exactly that: a single state. The verification is direct. Even with no early exit at all, at the cap, swapping the readout from "reads the mixture" to "reads the state at the cap alone" costs the mixture-loss model about 0.2 BPB (a +0.189 readout gap at 1.3B). That single gap is most of the 0.24 BPB difference between the smallest and largest models in the paper.

Switching to PonderNet's per-step loss (scoring every iteration) fixes the readout problem, but the loss alone is not enough. Each iteration's loss is weighted by the head's own probabilities, so training only improves the iterations the head already favours; only the KL term pushes depth downward, and at a weak prior it simply loses — even at ε=0.95, tokens still run more than 6 of 8 iterations.

The real switch is the weight of the prior. Raising it from β=0.01 to 0.1 moves the per-step model at one iteration from about 0.3 BPB worse than the sampled-depth baseline to about 0.06 BPB better, and it is never behind at any depth (the authors call this depth-robustness). The paper's tables show the stronger prior improves the per-step model by over 0.25 BPB (a −0.394 change at 140M) while doing nothing for the mixture-loss model. Loss and prior are both required.

With both in place the model does pull ahead: it spends about three iterations on average and beats the sampled-depth baseline run for the same number by about 0.02 BPB. The paper then decomposes that 0.02: shuffling the iteration counts the exit rule assigned across tokens costs 0.008 BPB, showing roughly two-fifths of the lead comes from allocation — giving more iterations to the tokens that need them. Retraining the baseline to draw shallower depths shrinks the lead to about a third, leaving allocation as the dominant remaining source.

Batching eats most of what is left

Then there is the real engineering setting. A server generates many sequences at once, and their tokens share one batched pass — and that pass only ends once its deepest token has stopped.

The paper decodes 32 sequences together: the per-step model's tokens stop after about three iterations on average, but one batched pass actually runs 4.4 to 4.7 iterations. Timed wall-clock at 370M with 32 sequences decoded together, the per-step model runs about 1.6× as fast as eight iterations; run at a fixed depth equal to its realized steps, the same weights would run about 2.3× as fast. The difference is the cost of waiting for the deepest token.

Compared at the iterations the pass actually runs, the baseline is better, and the remaining lead shrinks to 0.003 BPB at 770M. A released model shows the same pattern: Ouro's 2.6B model, which has a trained halting head, has its tokens skip about a tenth of its iterations, but a pass over 32 tokens skips about one percent.

There are two ways around the wait. The first is to gather still-running tokens into a smaller batch before each iteration, as Mixture-of-Depths routers do; in the authors' implementation this did not help when decoding. The second is decoding one sequence at a time — but a stopped token never computes its keys and values at the deeper iterations, even though later tokens attend to them there. The authors verify that reusing the stopped iteration's keys and values in those slots (as Geiping et al. do, with no measurable time cost) preserves almost all the lead. Even then, the model still does not beat a specialist trained to always run as many iterations as it runs on average.

The paper closes with a feasibility test that runs before any head is trained: the depth curve bounds what a head could win. Aiming at three iterations, the bound is about 0.03 BPB when one sequence is decoded at a time; with 32 sequences decoded together the comparison moves to a deeper point on the curve, and the bound falls to about 0.01.

So what

The value of this paper is not the announcement that adaptive depth does not work. It is that a compute ledger that has been quietly hand-waved has been laid out and actually totalled, along with a reporting standard: looped models with a halting head should report three numbers — the expected halting steps, the realized steps under a stated exit rule, and the iterations a batched pass actually runs — and should state how the deeper keys and values of stopped tokens are filled in.

My read: this line of work is about to be repriced. Before betting on how much a halting head saves, the question worth asking is whether your workload's batching will eat the savings entirely. "Adaptive" sounds elegant at the model layer; the queue at the system layer is the bill that actually gets paid.

Do you think adaptive depth is a fake need, or just badly implemented so far?