The standard ways to add computation to a language model are roughly three: go deeper, loop, or widen the residual stream itself. A paper from MIT CSAIL and Adobe (arXiv:2611.21608, submitted 16 November 2026, first author Shivam Duggal) asks a different question. Not how much computation each token gets, but how many active computational states the model maintains at the same time.
The answer is called the Lattice of Latents. Alongside the vertical residual stream that runs through network depth, the method attaches a horizontal latent stream to every layer, running recurrently along time. That stream updates through an affine recurrence, persists across tokens, and directly conditions the vertical stream's attention, KV-cache writes, MLP computation, and decoding.
This is where it separates from hybrid Transformer-SSM architectures. In those, the recurrent state mainly serves as internal memory for sequence mixing. The Lattice's horizontal latent does not replace attention; it is a companion state that sits beside the residual stream. The paper stresses that it adds no per-token KV-cache entries, which is a categorically different bet from adding layers or looped reasoning steps.
The numbers: from 15.41 to 14.40
After pretraining on 20B FineWeb-Edu tokens, the paper reports FineWeb-Edu validation perplexity as follows, in its own measured terms:
- Parameter-matched baseline Transformer Base-24L (406.7M): 15.410
- LLLM (945, 256) (406.5M): 14.545
- LLLM (908, 256) with input injection and depth-wise readout stream (405.6M): 14.400
- The same model rolled out sequentially, without Jacobi sweeps: 14.733
So a 406M-scale horizontal-latent model pushes perplexity from 15.41 down to 14.40, and that number matches a 40-layer Transformer with 1.5 times more parameters (Base-40L, 609.1M, also 14.400). Per-token compute barely moves: the baseline is 0.71 GFLOP/token, the Lattice 0.72.
The variant with nonlinear top-to-bottom feedback goes further. The paper approximates that feedback path with three Jacobi sweeps, costing about 2.6 times the FLOPs during training, and reaches perplexity 14.026. At inference the same model can instead be rolled out sequentially at the base FLOPs, and the perplexity becomes 14.733. The trade is plain: spend compute for accuracy, or the other way around.
Composing: 13.61 and 13.76 do not agree
One selling point is that the Lattice is orthogonal to existing residual-stream enhancements and can be stacked with them. The paper runs a comparison at matched scale (400M parameters, 20B tokens). On its own the Lattice takes 15.41 down to 14.47, ahead of hyper-connections at 15.28, manifold-constrained hyper-connections at 14.93, and value residuals at 14.69, though behind input residuals at 14.34.
The interesting part is the stacking. With the Lattice added, hyper-connections fall from 15.28 to 14.21 (a 7.0% drop), and hyper-connections plus input residuals fall from 14.22 to 13.61 (a 4.3% drop). The paper puts that combination at the top of the configurations it tested, calling it 11.7% below the plain Transformer and 3.9% below the pure Transformer composition of input residuals plus value residuals (at 14.16).
One detail is worth recording. The abstract and Section 4.2 both say 13.61, but the Figure 3 caption says 'Lattice + HC + x0 reaches 13.76, the best overall'. A passage in the appendix separately states that learned value residuals do little for the Lattice (13.61 to 13.62). Three numbers for the same combination, and they do not line up. Quoting 13.61 from the body text is the safer bet, but the mismatch itself suggests the paper has not been through a full proofread.
Math reasoning: same perplexity, 10.1 points apart
The bill beyond perplexity comes due on GSM8K. The paper finetunes each 20B-token checkpoint on GSM8K-Aug (385,620 problems) with chain-of-thought. At 400M scale, the baseline Transformer's five-benchmark mean accuracy is 36.02, while the Jacobi-refined LLLM reaches 48.10, a gain of 12.08 points. At 600M scale, the mean rises from 40.24 to 47.74, a gain of 7.50 points. The paper's claim is that at both scales every Lattice variant beats every Transformer baseline by at least 10.1 GSM8K points.
The more telling pair of numbers: Base-40L and LLLM (908, 256) have exactly the same pretraining perplexity, 14.40, and nearly the same zero-shot average (47.36 against 47.38), yet they differ by 10.1 points on GSM8K (33.5 against 43.6). Same starting point, a ten-point gap after finetuning. Pretraining perplexity therefore does not determine finetuned reasoning ability; something between the two is not captured by that metric.
Recurrent backbones: beating baselines with 4x the state
The paper also mounts the Lattice on Gated DeltaNet, Mamba-2, Triadic GDN, and GDN-Mamba-2 hybrids (10B tokens, roughly 405M parameters), with the horizontal latents modulating each backbone's memory read, write, and update. Perplexity improves at every native state size tested, and both Lattice-GDN and Lattice-Triadic-GDN outperform pure baselines that hold 4 times as many native-state entries.
On the nine-task downstream average, the Lattice leads in 29 of 30 pairs. The only regression is the hybrid at 128K state in the five-shot setting, dropping from 45.02 to 44.80.
A harder layer of evidence comes from probing GDN's recurrent memory directly. At matched state sizes, Lattice models hold more concentrated memory (rent falls by 9.6% to 18.3%), and the alignment gap between queries and the leading memory direction widens from +1.89 at 16K to +5.36 at 256K, with alignment reaching 13.3 times chance against 8.0 times for the pure model.
Three caveats
Three things have to be read alongside the results.
First, scale. Every experiment here runs at 400M to 600M parameters and 10B to 30B tokens, with a 2048-token context, a LLaMA-style decoder, and the GPT-NeoX tokenizer. The paper limits its own conclusion to the configurations tested, and does not extrapolate to larger scale.
Second, reproducibility. Each GSM8K finetune runs a single seed (seed 0) per model. The paper explicitly declines to interpret the small differences among Lattice variants, offering only the larger and consistent gap to the Transformer baselines. The coupling used for the recurrent backbones was also selected post hoc, and the paper notes the gains should be read as those of the best coupling rather than of any single preselected one.
Third, code. On the project page, both the Paper and Code links currently read 'coming soon', so there is no implementation to work with yet (project page: https://shivamduggal4.github.io/latent-lattice-language-models/).
So what
What is worth carrying away from this paper is not the 14.40. It is that the paper puts an assumed premise on the table: scaling computation does not have to mean making each token do more work. It can also mean letting the model hold more things at once. A horizontal latent is an axis of computation independent of depth, and it compounds with the residual-stream engineering that already exists.
For teams still stacking depth or adding KV-cache entries, this points at a bill that had rarely been calculated carefully: raising the number of active states each layer maintains, without adding per-token KV-cache entries. Whether these numbers survive at real hundred-billion-parameter scale depends on the code landing, and on other people rerunning the experiment at larger sizes. The paper itself: https://arxiv.org/abs/2611.21608