Reasoning models pay a memory bill for their "thinking." When a model uses test-time scaling to solve hard problems, it generates a long reasoning chain, and mainstream architectures keep that entire trace in memory via full attention — the longer it thinks, the larger the KV cache grows, and compute and memory costs balloon with it. Hard problems are exactly the ones that need long thinking, so the scenarios that need the most capability are also the most expensive ones.

The core insight: intermediate tokens go "stale"

A paper published on arXiv on August 26, "Prefix Sliding for efficient test-time scaling" (2608.26070), starts from a counterintuitive observation: as the model continues reasoning, most intermediate reasoning tokens lose importance. Since those tokens are barely needed later, is retaining them worth the cost? This insight directly challenges a hidden assumption — that every token in a reasoning trace matters equally and must participate in attention for the whole run.

The method: prefix + sliding window

The paper proposes Prefix Sliding: during reasoning, attention covers only two groups of tokens — the prefix (key instructions, available tools, and other metadata) and a sliding window of the last few thousand tokens (the reasoning the model is currently working on) — while intermediate tokens in between are simply discarded. This caps total memory requirements no matter how long the model reasons.

Applied to existing models without any training, the paper reports a 3x speedup while maintaining performance; combined with reinforcement learning training, it enables scaling to reasoning traces beyond a hundred thousand tokens — a regime that is hard to reach under full-attention memory budgets. Ablations show the approach outperforms two intuitive alternatives: summarizing intermediate tokens, and a vanilla sliding window.

The engineering: it lands on vLLM and flash-attn

This is not a paper-only proposal. The GitHub repository (Muennighoff/prefix-sliding, Apache-2.0 license) provides an implementation built on vLLM and flash-attn branches; the README demonstrates enabling a 4096-token sliding window on Qwen3-1.7B for 32K-length generation. Evaluation covers AIME, GPQA, MATH500, HealthBench, and LiveCodeBench (the latter run at 262144 max tokens). Evaluation result files and the RL training dataset (prefixsliding/train_v6_filtered) are hosted on HuggingFace, and the RL side hooks into both prime-rl and trl.

The author lineup is also notable: beyond first author Niklas Muennighoff, the paper lists Percy Liang, Jason Wei, Andrew Y. Ng, Yejin Choi, Luke Zettlemoyer, and Mike Lewis among its 18 authors, spanning 28 pages (9 main). That configuration usually means the community will replicate and build on the method quickly.

So what

The mainstream narrative of test-time scaling is "think longer, score higher," but the cost curve decides whether it can spread. The significance of Prefix Sliding is that it turns the marginal memory cost of thinking long into a constant — a direct win for reasoning API pricing, on-device long reasoning, and long-horizon agents. The more durable takeaway is the insight itself: intermediate steps in a reasoning chain are perishable goods, not assets. Next time you see a model "think for a hundred thousand tokens," ask yourself: how much of that really needed to be remembered?

Paper: https://arxiv.org/abs/2608.26070
Code: https://github.com/Muennighoff/prefix-sliding