Transformers are packed with non-linear components — softmax attention, MLP blocks with non-linear activations — so the prevailing assumption is that processing multiple independent text streams requires separate forward passes or architecture changes to avoid destructive interference. A paper submitted to arXiv on Sep 24 (arXiv:2609.29845) offers counterevidence: when the embeddings of two texts are averaged token-wise, a pretrained LLM's output approximates a superposition of the two individual next-token distributions. The authors name this the Superposition Linearity Hypothesis.

The signal survives, near the top

The researchers evaluated pretrained models across five families — Pythia, Qwen, Llama, OLMo, Gemma. After mixing embeddings of two unrelated texts, the single-stream predicted token still lands in the mixed distribution's top-10 in roughly 30-40% of cases, top-50 in 50-60%, and top-100 in 60-65%. With vocabularies of at least 50,000 tokens, this recovery rate sits far above the noise floor.

The "lucky high-frequency token" explanation does not hold: under a random frequency baseline, the cross-stream ground-truth token reaches top-10 in only 2.63% and top-100 in 10.41% of cases, while the superposed forward pass exceeds the baseline by an order of magnitude — the effect comes from the superposition structure itself, not a Zipf artifact.

The longer it trains, the weaker it gets

The most counterintuitive finding: superposition fidelity is strongest at initialization and decays monotonically as pretraining proceeds. This is not a capability the model learns; it is a factory setting of the architecture that the training objective steadily erodes. The authors track hidden-state additivity error checkpoint by checkpoint and find it grows monotonically. Layer-wise linearity shows a U-shaped profile — deep layers (beyond roughly 2L/3) stay near-linear — offering a geometric explanation for why the superposed signal survives to the output logits.

Lightweight fine-tuning pulls linearity back

Fine-tuning on less than 0.025% of the original pretraining data substantially reduces the divergence between the mixed-input distribution and the ideal superposition target. The cost is stated plainly: the same objective measurably hurts single-stream quality — Pythia-2.8B drops from 0.544 to 0.357 on LAMBADA, Qwen2.5-3B from 0.602 to 0.460. Compute is not free either: the Pythia-2.8B run of 150,000 steps took about 114 hours on 2×A100 80GB GPUs.

Decoding two continuations from one forward pass

The geometric-mean obstruction is the core difficulty: under averaged logits, probabilities scale with the geometric mean of the two streams' distributions, so any token highly likely in one stream gets suppressed by the other. Joint Contrastive decoding uses a small auxiliary model for per-stream guidance, lifting Llama-3.2-3B's mixed-pass LAMBADA mean accuracy from 0.182 to 0.430 (small-model single-stream baseline: 0.540) while pressing inter-stream Jaccard overlap down to 0.067. The conclusion states that with full disentanglement, a single forward pass could yield two coherent continuations — theoretically 2x throughput and half the KV-cache footprint per active stream.

So what

The paper puts both the promise and the boundaries of parallel inference on the table at once. On the positive side, superposition is intrinsic to the architecture, recoverable, and partially disentangleable. On the sober side, the 0.430-vs-0.540 LAMBADA gap, the single-stream quality sacrifice, and the short L≤512 contexts all mark this as a proof of concept. Compared with DataMUX, Superposed Decoding, and other schemes that bolt on mux/demux structure, the difference here is that superposition does not need to be engineered in — it was already there. For teams sensitive to inference cost, this is a thread worth tracking: if the disentanglement loss can be narrowed further, 2x throughput is not marketing talk but structural.