One line item that open-weight language models now routinely carry is easy to overlook: the n-gram embedding table. At every token position the model hashes the adjacent pair or triple of tokens into a large lookup table, pulls a vector out, and adds it directly to the token embedding. The lookup itself adds almost nothing to compute, but the table can be made enormous — DeepSeek-V4.1-Flash allocates 196 billion parameters to its Engram conditional memory, a figure stated plainly in the official model card.

On 22 October 2026, Stephen Chung and Wenyu Du posted a paper on arXiv (arXiv:2610.31646) asking a very practical question: can the memory cost of that table be cut while keeping the performance gain it buys?

The approach: a codebook plus a scale, down to 4 bits

The method is not complicated. Each table gets a codebook, and every embedding component is stored as a b-bit index plus a single 16-bit scale factor; on lookup the index selects a codebook value that is multiplied back by the scale. For an embedding with p=128 components, memory falls from 16p bits to p·b+16 bits — at two bits that is 2,048 bits down to 272, roughly an 87% reduction.

The paper compares three families of codebooks (uniform, normal-quantile, and Lloyd–Max fitted from the trained table) against three ways of choosing the scale (maximum-based, RMS, and error-fitted). The finding is slightly counter-intuitive: how the scale is chosen matters more than which codebook is used. With error-fitted scales, the three codebooks all retain 86.36%–86.52% of the improvement. Uniform codebooks with maximum-based scaling retain only 58.04%. Lloyd–Max with RMS scaling reaches 86.19%, so it becomes the default.

Four bits costs almost nothing on models trained from scratch

The authors trained five models from scratch (nanochat 26M/50M, a GPT-2-family 30M, and Llama-family 31M/322M), each with four tables by default and about 262k embeddings per table, trained on ClimbMix with three random seeds per experiment. The metric is deliberately clean: how much of the validation-loss improvement that the tables add survives compression.

The paper reports that four-bit tables save 74%–75% of table memory while retaining more than 99% of the improvement. Two-bit tables save 86%–87% but retain only 85.9%–91.0%. Bigram and trigram tables turn out to differ in sensitivity to quantization, so the bit allocation between them matters too: giving bigrams the extra bit consistently works better, with four-bit bigrams and three-bit trigrams retaining 98.9%–99.4% while saving 77%–78%.

Validated on large models: half the memory, almost no lost accuracy

The real value shows up on DeepSeek-V4.1-Flash and Qwen3.8-Flash-Next. Four-bit tables save roughly half of the table memory, with a 1.26% perplexity increase for DeepSeek and almost no change for Qwen. At three bits the two diverge sharply: Qwen's perplexity rises only 0.52% while DeepSeek's climbs 5.06%.

The mixed-precision settings the paper settles on keep perplexity changes below 1.2% across three held-out datasets (WikiText-2, Penn Treebank, C4) while saving 47%–55% of table memory. On downstream tasks (HellaSwag, ARC-Challenge, WinoGrande) average accuracy falls by just 0.93% for DeepSeek and 0.27% for Qwen, with those configurations saving 46.7% and 55.0% of table memory respectively.

One detail is easy to miss: the optimal bit allocation is reversed between the two models. DeepSeek does better when the extra bit goes to the trigram or four-gram tables (4/5/4, 4/4/5) rather than to bigrams (5/4/4), while for Qwen the opposite holds and 4/3 beats 3/4. Mixed precision is therefore not something you can copy from a configuration table — it has to be measured per model.

The most interesting step: spending the savings on bigger tables

The final move reinvests the freed memory. Each table grows from 262k to 1M embeddings stored at two bits, so total table memory uses only 52%–54% of the original while delivering 22%–26% more improvement than the original 16-bit tables. Adding a short fine-tuning phase lifts the two-bit improvement from 122%–126% to 129%–145% at unchanged memory cost.

From this the authors propose a workable recipe: pack more embeddings into pre-training, then compress and fine-tune briefly. The boundaries are worth stating plainly: what is saved is table memory rather than the memory of the whole model, and the gains have only been validated on DeepSeek-V4.1-Flash and Qwen3.8-Flash-Next, so another model still needs its own measurement.

Why this matters

The memory bill for large models has been in constant motion: KV caches pushed to the limit, MoE making activated parameters far smaller than total parameters, and attention redesigned again and again. But one cost has been systematically ignored — lookup-style local memory. DeepSeek putting 196 billion parameters into Engram shows the approach genuinely works; and once something that effective grows large, what you save is real money.

The technique itself is unremarkable: a codebook and a scale, essentially the old trick from compressing embedding tables in recommendation systems, relocated to language models. What is worth borrowing is the experimental discipline — train one model with the tables and one without, take the difference as the table's genuine contribution, then see how much of that contribution survives compression. This is far cleaner than comparing perplexity directly, because the baseline is subtracted out.

More telling still: the original finding came from an autonomous agent the authors ran on the rStation multi-agent research platform, and the paper reproduces and extends it across more models and datasets. AI finding the direction, humans doing the verification — that division of labour is becoming a workable research pattern.

The practical advice for ordinary developers is simple. If you are running open-weight models and hitting memory pressure from a table like Engram, four bits is close to a no-brainer. If memory is genuinely tight, dropping to two bits and spending the savings on a larger table may beat switching to a smaller model altogether.

When lookup memory becomes cheap enough to add on demand, will the real bottleneck no longer be architecture but training data itself?