RoPE is the industry standard for large language models, employed by major families like Llama and DeepSeek, yet its length extrapolation has long been criticized, spawning patches such as positional interpolation and YaRN. On September 10, Daniel Henrik Nevermann and Claudius Gros of Goethe University Frankfurt posted a paper (arXiv:2609.11913) asking a more basic question: does positional encoding actually help with "distance generalization"? The measured answer is counterintuitive — in most cases, turning it off entirely works best.
Two kinds of generalization
Length generalization studies models trained on short contexts and evaluated on longer ones, so unseen positions and unseen token dependencies fail together, hopelessly confounded. This paper defines distance generalization: context length stays fixed while the gap between source tokens and their recall varies, separating "can't handle unseen positions" from "can't handle unseen dependencies". Two synthetic delay-copy tasks are used: full copy recalls 10 source tokens after a delay, while selective copy recalls only the even-numbered tokens. Training distances are sampled uniformly between 15 and 25; test distances stretch from 1 to 120, far outside the training distribution.
NoPE wins in most settings
The model is a standard 8-layer, 8-head, 512-dimension decoder-only Transformer, run in three configurations: RoPE, ALiBi, and NoPE (no positional encoding). Distance generalization turns out to be highly sensitive to the encoding scheme, with NoPE generalizing best in most settings — consistent with Kazemnejad et al.'s 2023 findings on length generalization. The sting for RoPE: it falls behind even though test sequences never exceed the training context, so RoPE never faces untrained angular regimes. The blame cannot be pinned on extrapolating to unfamiliar angles; the mechanism lies elsewhere.
Two side findings: diversity and transfer
On data diversity, enlarging the set of inter-token distances seen in training lifts absolute performance, but relative returns diminish strongly. On transfer learning, training a main task together with an auxiliary one cuts both ways: transferring from the more complex selective copy to full copy yields positive gains, yet destructive effects appear within full copy's own training range. RoPE transfers weakly overall but has one distinct advantage: almost no destructive interference.
Don't rip out RoPE just yet
Two caveats matter. First, NoPE is sensitive to model scale: as models shrink, NoPE even fails to learn the in-distribution copy mechanism, while explicit encodings remain far more stable. Second, these are results from an 8-layer model on synthetic tasks; the authors state in their discussion that distance generalization ultimately needs to be tested on pretrained models — scale, pretraining data, and multi-task mixing still stand between synthetic tasks and LLMs.
For long-context engineers, the value of this paper is not "drop RoPE now" but a clean unconfounding of variables: many "long context fails" problems may not be positional extrapolation's fault at all, but the model never learning to handle unseen token dependencies. Next time RULER scores stall, ask first: is it the positions that are unseen, or the dependencies?