There is one component of the Transformer that I think has gone largely untouched for the better part of a decade: the residual connection. Every layer’s output gets added back into the trunk. It is blunt, it is stable, and nearly all subsequent work has circled around it rather than through it — gating it, scaling it, splitting it into multiple lanes. Until recently, somebody finally went after it properly. The ARLoop work released by the NLP group at Ohio State takes a more radical idea and carries it into looped transformers: let the residual stream decide for itself which earlier layers are worth reusing.

The result is somewhat counterintuitive. A 1.2B-parameter looped model trained on just 0.5T tokens is reported to beat Huginn, a 3.5B-parameter model trained with up to 64 recurrences, across every benchmark in the paper’s evaluation table, and, the paper claims, matches or exceeds Qwen3.5-2B, a non-looped model trained on 36T tokens. The training code, the data-processing scripts, and the intermediate checkpoints are all released.

The residual stream as a pipe with no choices

In a standard PreNorm Transformer, the input to layer i is h_i = h_{i-1} + f_i(Norm(h_{i-1})). Unroll that and one fact becomes obvious: every residual update produced by an earlier layer gets added, with a fixed unit coefficient, into every residual state that comes after it. This single residual stream offers no mechanism for selection. A later layer that wants to use an earlier layer’s representation can either take the whole stream or not; it cannot pick and choose.

ARLoop borrows from the attention residuals proposed by the Kimi team in 2026, which treat depth itself as something attention can operate over. Each layer uses its own learned query vector to run a softmax over the available residual sources and decide the aggregation weights on its own. To keep the overhead manageable, the paper uses a block-level form: consecutive sublayers are summed into a block-level residual representation, addition happens inside a block as in PreNorm, and attention operates across blocks.

Under recurrent depth, history turns out to be redundant

The paper contains a counterintuitive finding. On looped models, letting depth attention see only the most recent residual blocks, plus the input embedding, is not merely competitive — at larger recurrence depths it is better. The same sliding-window restriction applied to a standard transformer reduces the gains, so the contrast is doing real work here.

Attention visualizations and pairwise cosine similarity explain why. In the 140M-scale ARLoopfull at R=4, sublayer 47 places 96% of its total attention weight on the three most recent residual blocks plus the embedding block. The culprit is parameter sharing. Recurrence applies the same layers over and over, so representations produced at different recurrence depths end up highly similar. Distant history carries little new information and can even introduce interference.

Moving to a realistic pretraining recipe

In the controlled experiments, at 140M and 370M parameters, trained on 11.2B and 29.6B tokens of FineWeb-Edu at a context length of 2048, ARLoop posts the lowest perplexity at every recurrence depth tested. At 140M with R=3 it reaches 16.38, against 16.91 for PreNormLoop, 16.60 for Parcae without input injection and 16.55 with it, 17.05 for DeepLoop, and 17.19 for Sandwich-Norm. At 370M with R=3 it reaches 12.17 against 12.48 for PreNormLoop. At the 140M scale, the gain over PreNormLoop grows from 0.11 at R=1 to 0.53 at R=3, which suggests the method genuinely exploits the extra computation that recurrent depth buys.

Scale up to a realistic recipe — SmolLM-style, grouped-query attention with SwiGLU, FineWeb-Edu, FineMath and Stack-Edu as data, context length 4096 — and the picture sharpens. The 0.6B model, trained on 180B tokens, is compared against an architecturally identical PreNormLoop-0.6B on the same budget: GSM8K 40.9 against 35.7, MATH 26.3 against 20.9, MBPP 36.2 against 33.1. It wins seven of eight evaluated tasks. The one it loses is ARC-C, 45.5 against 46.3. The 1.2B model, with 24 layers, R=3 and 500B tokens, is more aggressive still: GSM8K 63.1, MATH 37.8, MBPP 52.1, HumanEval 33.1. The paper claims it outperforms Huginn-3.5B on every evaluated task — that model’s 0.8T-token run scores 26.1 on GSM8K and 14.1 on MATH — and claims it matches or exceeds Qwen3.5-2B on the evaluated benchmarks, where Qwen3.5-2B scores 60.6 on GSM8K and 35.8 on MATH.

Three bills worth adding up

The first is throughput. ARLoopfull runs about 15% slower than PreNorm on average; the sliding-window variant cuts that gap to roughly 10%. Memory is essentially unchanged — 66.5 GiB against 66.6 GiB at the 0.6B scale. The paper argues depth attention adds little arithmetic and that the main cost is extra memory bandwidth traffic, which leaves room for kernel optimization later.

The second is data. The paper itself concedes that ARLoop-1.2B trails Ouro-1.4B on most tasks — Ouro scores 78.2 on GSM8K and 70.5 on MATH — and attributes the gap to a substantially larger pretraining data budget and higher-quality data sources. Matching a 36T-token Qwen3.5-2B with 0.5T tokens looks impressive, but it mostly says that a small base model does not mean a small bill. At the same parameter count, a non-looped model still has to pay the data budget.

The third is extrapolation. One of the selling points of looped models is training at R=3 and running more recurrences at inference. ARLoop’s conclusion here is cool to the point of deflationary: performance improves up to the maximum training recurrence and does not improve beyond it. Making residual access smarter does not automatically unlock inference-time recurrence extrapolation. Organizing information within the trained depth range and generalizing recurrent computation past that range turn out to be separate problems.

What full openness actually buys

The codebase ships the data preparation pipeline, model implementation, training recipes and evaluation code. The model card ships the full checkpoint trail for the 1.2B 500B-token run: 23 intermediate checkpoints from step 10,000 through step 230,000, plus the final revision at step 238,417, for 499,996,688,384 training tokens total, under an Apache 2.0 license. Architecturally, 24 physical layers are shared across 3 recurrent passes for an effective depth of 72, with hidden width 2048, MLP intermediate width 5632, 16 attention heads against 4 KV heads, and a context length of 4096.

Plenty of papers have come out of the loop direction over the past two years, but most release weights only, or nothing at all. This one lays the entire training trajectory out, which means someone else can pick up from a mid-run checkpoint instead of reproducing a training run from scratch. The paper’s own accounting: roughly 12K GPU hours for the controlled experiments, about a week on 8 H100s for the 0.6B model, about two weeks on 16 B200s for the 1.2B one.

So what

Before the residual stream was pried open, most attention went to layer counts, hidden widths and activation functions. ARLoop points at something else: a model’s depth is not just its physical layer count, it also includes the topology through which information flows between layers. Looped models reuse the same batch of layers, so redundancy is unavoidable, and in that setting letting the model pick its own history beats making it accept everything indiscriminately.

Still, this reads more like a proof of feasibility than a conclusion. A 1.2B model trained on 0.5T tokens is not enough to settle any judgement about the loop direction. What is worth waiting for is whether the 10% to 15% throughput cost holds steady at larger scale. If it does, the residual-stream rewrite stops being a laboratory trick and becomes something every looped model should copy.