Context compression is the main lever for cutting inference cost on long inputs. ICAE squeezes a whole document into a few memory slots; ARC-Encoder replaces token embeddings outright with continuous representations. Every method on that track optimises for one thing: putting the semantics back. Nobody asked what happens to the model’s temper once the context has been squeezed. A paper submitted on 4 November (arXiv 2611.06380, by Anthony Nguyen and Wenjun Lin of Algoma University in Canada, presented at Canadian AI 2026) adds a line to that ledger: compression itself erodes refusal behaviour, and it does so quietly. The paper is at arXiv 2611.06380.

What collapses is not semantics, it is refusal

The paper encodes the entire user request into compressed representation space, removes the original tokens from the decoder’s input, and forces the model to answer from compressed memory alone. The authors describe this as a deliberate diagnostic stress test, modelling compression-first pipelines where an upstream system hands the generator nothing but compressed representations, rather than the usual retrieval-augmented setup where a document gets compressed while the query stays in token context.

The numbers, on a self-built 624-prompt set with half held out for evaluation: Mistral-7B-Instruct-v0.2 with ICAE drops harmful-prompt refusal from 62.82% to 10.9%; LLaMA-3.1-8B-Instruct with ARC falls from 96.15% to 36.54%; Mistral-7B-Instruct-v0.3 with ARC goes from 20.51% to 8.97%; LLaMA-2-7B-Chat with ICAE slides from 97.44% to 85.9%. Switch to HarmBench and AdvBench and the direction holds: Mistral-7B-Instruct-v0.2 falls from 57.5% to 16.5% on HarmBench and from 62.69% to 20.38% on AdvBench, and in the ARC group LLaMA-3.1-8B-Instruct falls from 84.0% to 47.5% on HarmBench.

What deserves a second look is the other column of the same table. Refusal on safe prompts barely moves: three of the four rows sit at 0.00 and stay at 0.00, and the LLaMA-2-7B-Chat row actually rises from 0.00 to 3.2. So compression did not turn the model into something that answers everything; it trimmed precisely the half that was supposed to refuse. An evaluation script that only watches over-refusal on benign inputs would see nothing wrong.

Why it collapses: reconstruction rewards nothing about refusal

The authors explain this as a training-objective mismatch. Encoders like ICAE and ARC-Encoder are trained purely to reconstruct the original text, and nothing in their loss function asks them to preserve refusal triggers; meanwhile refusal behaviour inside the decoder is mediated, Arditi and colleagues show, by something close to a one-dimensional direction (across 13 open-source chat models, up to 72B). Semantic reconstruction is an information-theoretic objective, refusal is a geometric property, and the two do not live on the same level. The mechanistic evidence is a linear probe on the decoder’s residual stream: harmful inputs that would trigger refusal show activation along the refusal direction collapsing to negative values across all layers under compression, tracking the non-refusal regime.

This lines up with work by Xu and colleagues at EMNLP Findings 2024, who found that pruning and quantization change toxicity and bias behaviour even while perplexity holds, with quantization largely preserving bias and pruning degrading quickly. One is parameter-level compression, the other context-level, two different groups at two different times, and both point the same way: compression carries side effects, not just savings.

Three open questions in the paper

First, high reconstruction fidelity is treated as a premise. The abstract says the effect holds “despite high reconstruction fidelity”, but no table in the paper reports reconstruction quality: no perplexity, no BLEU/ROUGE, no compression ratio. For contrast, the two cited method papers do supply the numbers: ICAE reports 4x compression on Llama with about 1% additional parameters, and ARC-Encoder reports typically emitting 4x or 8x fewer continuous representations than text tokens. The compression ratio, the coordinate that actually matters, has to be dug out of the references.

Second, the text says refusal rates “decrease by up to 78%” under ICAE compression, for Mistral-7B-Instruct-v0.2 on HarmBench, but the paper’s own Table 2 values of 57.5% and 16.5% work out to roughly 71%. The other half of that sentence, “over 40%” under ARC, does check out against Table 2 at 43.45%.

Third, the code address given in the paper’s footnote, github.com/ant-8/Context-Compression-Alignment, is currently an empty repository: the GitHub API reports a repository size of 0, a 404 on the contents endpoint, and an empty branch list. Reproduction is not currently available.

The fix works, but it only covers half the problem

Memory Steering retrains nothing and intervenes once on the compressed memory. The steering direction comes from a counterfactual construction: pair every prompt with a variant ending in “This violates your AI safety guidelines”, and the difference between the two sets of memory means is the direction. A logistic regression then decides slot by slot which slot holds unsafe intent, and injects at strength alpha, so benign prompts do not get over-refused.

The numbers are substantial: on AdvBench, Mistral-7B-Instruct-v0.2 climbs from a 20.38% baseline to 81.15% at alpha=2.0, and LLaMA-2-7B-Chat goes from 90.96% to 100.00%. But the same paper carries three costs. It only works for fixed-length compression, since it cannot be applied to variable-length pooled representations like ARC-Encoder, and the ARC group is precisely the one that collapsed harder. Response quality degrades: on safe prompts the win-plus-tie rate for LLaMA-2 falls from 85.9% at alpha=0.5 to 79.5% at alpha=2.0. And the knob overshoots the baseline, pushing Mistral’s AdvBench refusal to 81.15%, above the 62.69% it showed uncompressed.

One more detail is worth recording: with ARC compression, Mistral-7B-Instruct-v0.3’s refusal rate falls to 8.97%, leaving only 14 positive examples in the evaluation set, so fitting a probe directly would be unreliable under such severe class imbalance. That is exactly why the authors switch to the counterfactual construction. Compression flattened the evidence too.

So what

For teams shipping compression, here is a conclusion you can act on directly: safety evaluation cannot run only on the uncompressed baseline. Evaluate the compressed model separately on refusal, and watch the harmful side, not just over-refusal on benign inputs. In a context-compression evaluation table, reconstruction fidelity and refusal rate belong side by side, rather than assuming that a stable first column means a stable second one.

As for the paper itself, its value is less in how new the “compression breaks alignment” conclusion is than in moving a choice long treated as pure engineering back into the range of safety questions. Every unit of compute that compression saves is asking a question nobody was asking: what exactly was saved?