Speculative decoding has become the standard remedy for LLM inference acceleration: a lightweight draft model proposes tokens, the target model verifies them in parallel, and the more drafts accepted, the bigger the win. But which drafting route to take has long split the community into two camps — neural drafts (EAGLE3 and kin) stay robust across settings, while context-based copying (PLD, CopySpec) runs faster in repetition-heavy regimes. A paper submitted to arXiv on September 17, "To Copy or Not to Copy: Controlling Speculative Decoding via Intrinsic Model Signals", proposes a third path: stop guessing at the text surface and ask the model itself.
Copying pays, mis-copying taxes
The logic of context copying is plain: if the text you are about to produce already appeared in the context, copy it forward and let the target model verify — nearly free money. In the paper's illustration, a templated "positive number decomposition" output lets a copy draft accept 20 tokens in one shot while a neural draft accepts only 2. The reverse is uglier: when a surface n-gram collides but the model has no real intent to copy, the copy draft eats a 0-token acceptance while the neural draft accepts 8 at the same position — all the drafting and verification compute burned for nothing. The authors measure that in reasoning-heavy tasks, such false-positive triggers account for over 50% of all copy attempts. In other words, more than half of the copying budget is spent in the wrong place.
A copy-intent detector inside the model
At its core, SwitchSD is a lightweight linear probe that reads copy intent directly from the target model's hidden representations. Building training labels takes a clever shortcut: use Claude Sonnet to generate 1,000 prompts with diverse repetition structures, have the target model complete them, and label any position where "at least 5 consecutive tokens verbatim-match earlier context" as a copy instance — yielding hundreds of thousands of token-level labels in one pass. Probes are trained layer by layer, and a single optimal layer is picked by greedy F1 maximization on a held-out split (layer 14 for Llama-3.1-8B), reaching an AUC above 0.99 for copy-intent detection. At decode time each position first asks the probe: copying? If yes, take the copy path — with the extra guard that the current 5-gram must actually occur in the context; if no, hand drafting back to the neural speculator (SPS or EAGLE3).
Three tables: consistently past EAGLE3
Across Llama-3.1-8B-Instruct, Llama-3.3-70B-Instruct, and Qwen3-8B, on Math500, HumanEval, and CNN-DailyMail, SwitchSD beats EAGLE3 across the board. For the 8B model: HumanEval speedup 2.58x vs 2.28x, Math500 2.29x vs 2.05x, CNN/DailyMail 2.01x vs 1.85x. The paper's own framing is up to 15% additional throughput over strongest baselines like EAGLE3 — and since speculative decoding is lossless by construction, outputs remain token-identical to the original model.
So what
The memorable part of this paper is not one more speedup ratio, but the fact that "a model's internal state can serve as a control signal" has been wired into the decode loop, the front line of inference engineering. The authors push the direction further in their conclusion: the same probe framework could predict optimal speculation lengths or route among multiple specialized draft models. A bucket of cold water, though: the paper page lists no code repository yet, and the probe's transfer cost to new models plus end-to-end gains in real serving environments await replication. The idea stands; the engineering has just started.
Reference: arXiv:2609.20186 https://arxiv.org/abs/2609.20186