Speculative decoding has had a default script since it was proposed in 2023: a small model guesses a few tokens, the big model verifies them all in one pass, and a correct guess saves time. The script assumes the small model should be as capable as possible.

Recent work at INLG 2026 turns that script inside out. In Towards Green Text Generation: Memory-Based Speculative Decoding (arXiv:2611.16714, https://arxiv.org/abs/2611.16714), Antal van den Bosch changes the order of that script: once the verifier uses a KV cache, a 1.1B neural draft model does not merely fail to help, it drags throughput down to a fraction of the baseline and raises energy use. The fix is not a stronger draft model but demoting the draft back to a lookup table.

The verifier is Mistral-7B-v0.1 base on a Mac Studio with an Apple M1 Max and 32 GB of RAM. The paper sweeps 7 draft strategies times 3 draft lengths (k=3/5/8) times 300 prompts, which is 6,300 generations per regime, and samples whole-system power at 1 Hz with macOS powermetrics across CPU, GPU and Neural Engine.

Two regimes, two answers

Without a KV cache, the verifier recomputes attention over the whole context on every step, so appending draft tokens is nearly free and the cheapest possible draft wins. A 4-gram hash lookup answers in 0.09 ms, the Olifant variants in 0.16 to 0.25 ms, and TinyLlama-1.1B in 254 ms per token. On EU legal text the domain-matched Olifant-eurlex lifts throughput from 2.29 to 5.55 tps, a 2.43x gain, and cuts energy from 1,326 to 529 J per 100 tokens, a 2.51x reduction.

Speed even improves with longer drafts here: as k grows from 3 to 8, acceptance falls from 38.1% to 19.9% but the mean accepted length rises from 1.13 to 1.53 tokens, so net throughput keeps climbing. Extending the sweep to k of 12, 16 and 24 shows throughput peaking at k=8 and settling near 2.2x beyond it.

With a KV cache, the baseline itself accelerates by 6 to 7 times (plain decoding goes from 2.3-2.8 tps to 16.3-16.9 tps) and energy drops to a tenth, 96 to 106 J per 100 tokens. The rules invert. The neural draft runs k sequential small-model forward passes per step, so verifying what it produced costs more than it saves: its speedup falls from 1.15-1.20x to 0.47-0.29x, and its energy goes from saving 1.2-1.4x to costing 2.5-3.8x, or 243 to 382 J per 100 tokens.

The draft demoted to a lookup table

Olifant is not a neural network. It uses the IGTree algorithm from TiMBL: the training set is compressed into a prefix tree whose levels are ordered by information gain, and prediction walks a single path from the root and returns the majority label stored at the deepest matching node. Nodes whose majority label agrees with their parent's are pruned at training time, shrinking the tree by about 95%, and the tree's depth is bounded by the context window rather than the corpus size.

The corpus is tokenized with Mistral's SentencePiece vocabulary, the same units the verifier uses, so drafts drop straight into verification with no representation mismatch. Four instance bases cover 1.089B tokens of FineWeb-Edu, 536M of PubMed Summarization, 130M of WikiText-103 and 94M of EurLex. Compiling the largest one took about six hours on the evaluation machine and roughly half a megajoule, which the paper says is recouped within roughly a thousand 100-token generations. TinyLlama sits at the opposite end: pretrained on 3 trillion tokens on 16 A100 GPUs over about 90 days, around 10^10 J of GPU board power alone, four to five orders of magnitude more.

Matching the corpus beats matching the method

The paper also rules out a confound: maybe legal text is simply repetitive and therefore easy to guess. It is not. Legal text has a within-text repetition rate of 12.1%, against 21.2% for medical and 29.6% for general text. What runs high is cross-document overlap: 73.6% of distinct legal 4-grams recur in other legal documents, versus 26.7% and 14.9% elsewhere. The gain comes from offline training on the domain corpus, not from repeating itself.

Holding the text fixed at k=3 on legal prompts and changing only the draft model, acceptance spans 7.5% for the PubMed-trained model to 38.1% for the EurLex-trained one, with the in-session 4-gram at 24.7% and the neural draft at 21.1%. The 1.1B neural network does not even beat the 4-gram that needs no training corpus at all.

The boundaries the paper draws

Several caveats belong in the same reading. Every experiment ran on a single M1 Max machine, with a single verifier, greedy decoding and batch size one; the energy figures are whole-system measurements that support relative comparisons on identical hardware, not absolute cross-platform claims. In the cached regime the draft and plain outputs differ in ROUGE-L, 0.955-0.959 against 0.971, which the paper attributes to float16 non-determinism on MPS rather than to the algorithm changing what is produced. A footnote records that power during measurement came from a solar panel and home battery combination requiring almost no grid electricity, and repeated measurements of the plain configuration agree within 2% in both regimes.

One set of numbers should not be mixed together. The abstract pairs 38.1% acceptance with 2.43x throughput and 2.51x energy, but Table 1, Table 3 and Appendix A place 2.43x and 2.51x at k=8, where acceptance is 19.9%, while 38.1% belongs to k=3, which gives 2.06x throughput and 2.13x energy.

So what

The interesting part is not that things got faster. It is that the paper changes how speculative decoding gets judged: past acceptance rate, now the cost per drafted token and whole-system energy. The verifier side keeps improving, with KV caching, batching, quantization and faster kernels all squeezing the budget available to drafting, and the paper expects that squeeze to tighten further. The draft side has only one direction, cheap enough to ignore.

A separate team at ACL 2026 Findings measured something similar in Speculative Verification (https://aclanthology.org/2026.findings-acl.1809.pdf): over 40% of verification effort went to rejected tokens, 48% of speculative steps cost more than decoding directly with the target model, and at large batch sizes the draft model and verification overhead eat gains that had already shrunk. Different teams, different methods, different settings, the same conclusion. Once the verifier is no longer the bottleneck, the decisive question becomes how cheap the draft is.

So the next question may be not how large a draft model should be, but how cheap it should be. 0.2 ms on CPU, no accelerator memory, no backpropagation, and swapping an instance base to swap domains is a table worth looking at before another 1.1B model. Code and models are released at https://github.com/antalvdb/olifant-speculative and https://huggingface.co/antalvdb/olifant-speculative-ibases.