Xinyu Zhou et al. in arXiv:2606.11052 tear open a neglected wound in hybrid linear-attention LLMs: Chain-of-Thought supervised fine-tuning, while improving reasoning ability, systematically destroys long-context retrieval.
The paper uses HypeNet and Jet-Nemotron as samples. HypeNet-9B's NIAH-S2@256K plummets from 67.2% to 9.4% — near-amnesia. This phenomenon is named "Attention Amnesia": the CoT supervision signal concentrates gradients on short-range patterns, rewriting the W_Q and W_K projection matrices that handle long-range routing into "nearsightedness."
The fix is unexpectedly simple. QK-Restore is a post-training rollback: just "weld" the W_Q and W_K weights from the pre-SFT checkpoint back in, while keeping the rest of the CoT-tuned parameters. HypeNet-5B's S3@256K goes from 65.4% to 76.4%, with reasoning scores not regressing. The paper also gives a Procrustes variant that uses orthogonal constraints to find a smoother trade-off between "preserving routing" and "adapting to reasoning."
The engineering value is clear: long context and reasoning ability have traditionally been near-zero-sum in the SFT stage, and QK-Restore offers an almost-zero-cost dual-repair path. Compared to retraining an entire suite, surgically fixing two rows of matrices — this restraint is precisely the kind of sobriety that is becoming rarer in today's large-model research.