Training is continuous, deployment is discrete. Gradient descent needs a smooth loss landscape, so weights are modelled as real numbers; on actual hardware every weight is just a fixed bit pattern. That gap between the learned model and the executed object is where model compression has spent the last several years.

What is striking is that weight formats have been converging on the same direction. Group quantization lets a group of weights share one scale. Block floating point formats let a block share one exponent. Sparse formats let a zeroed weight terminate its description early. Codebook methods let a whole block point at one entry of a shared table. Every step amounts to "store one decision once and reuse it for many weights." But each format shares only one fixed kind of decision, at levels fixed in advance.

One tree turns every compression method into the same edit

Ammar Ali and Stamatis Lefkimmiatis of LEAD Lab at ITMO University propose TWIG (Tree-structured Weights with Inherited Gates). In their representation, every weight is the end of a path of small discrete decisions: a sign, a shift by a power of two, a zero gate, an index into a shared table. Weights that belong together share the upper part of their path, and each decision is inherited by every weight below it — stored once, applied once.

The paper's most elegant claim runs in both directions. Every format in use today is a shallow TWIG, at no extra cost in bits. Conversely, any trained network can be written as a TWIG. That means editing a single tree is enough: pruning replaces a leaf with a zero state, quantization truncates mantissa bits, low-rank factorization splits into a product of two trees, codebooks point a node at a shared subtree. The paper enumerates seven edits, and mainstream compression methods are essentially combinations of them. Paper: https://arxiv.org/abs/2610.31931

From 16 bits to 12.13, bit for bit

On LLaMA-3-8B, BF16 weights stored with one shared exponent per 64 weights, a 4-bit state and an 8-bit leaf per weight, average 12.13 bits per weight — 24% below 16 — and decode back to exactly the original BF16 values. Not an approximation: bit-exact. Roughly 0.01% of weights (those whose exponent lies 15 or more below the largest in their group, including exact zeros) are kept exactly in a short side list, and that cost is already included. The authors' CUDA kernel decodes the tree with byte-parallel integer operations, feeds the tensor cores directly, and runs at 97% of the bandwidth floor for the tree bytes.

Fewer bytes turn directly into tokens

LLM generation is bound by memory traffic, not arithmetic, so saved bytes are saved time. The paper reports end-to-end vLLM serving of Llama-3.2-1B and Qwen3.5-9B: 1.23x faster at batch size 1, 1.17–1.21x at batch size 8, and roughly 1.0x at batch size 32, where serving has become compute-bound. On an A100, matrix–vector products on layer 15 of LLaMA-3-8B run 1.05–1.51x faster than cuBLAS, of which the format alone bounds the gain at 16/12.13 ≈ 1.32x.

The official repository README gives a second set of numbers for Qwen3-8B on a single A100-40GB: GPU weights drop from 15.3 GiB to 12.8 GiB, KV cache per token falls from 144 KiB to 112.5 KiB, the number of tokens the KV cache holds under the same 0.85 GPU budget rises from 119K to 178K, and decode attention runs 1.21–2.46x faster than FlashAttention-2. The repository is Apache-2.0, and the vLLM plugin is enabled with a single flag: https://github.com/ammarali32/StateTree-Representation

No training, 3.25 bits beats GPTQ in the same pipeline

The paper runs a dynamic program across all of LLaMA-3-8B, letting each matrix pick from a menu of edits that mixes quantization, pruning and low-rank, with no rotation and no trellis-coded quantizer. At 3.25 bits per weight and with zero training, the result is perplexity 6.95 on WikiText-2, 10.93 on C4 and 9.76 on FineWeb; the strongest GPTQ configuration in the same pipeline gives 8.09, 12.72 and 11.81; the FP16 reference is 6.14, 9.44 and 7.85. In other words, 58% of GPTQ's remaining gap to FP16 is closed.

A compressed model can still be repaired

The question of how to choose the states, the paper translates into finding a ground state of a higher-order Boltzmann machine — precisely the problem class annealers and quantum optimizers are built for. They call the procedure Ising healing and run it with a quantum-inspired classical solver, ballistic simulated bifurcation. After compressing Qwen2.5-1.5B-Instruct to 2.44 bits per weight without training, 40 steps moved only 46,239 stored codes, about 0.004% of the model's codes; validation KL to the original model fell from 0.491 to 0.371, GSM8K rose from 13.95 to 21.83, and MMLU from 48.3 to 49.6. The paper flags its own caveats: GSM8K has 1,319 problems, the 95% interval is about ±2 points, and this is a single run.

Fewer adders for weights burned into silicon

When weights are fixed at fabrication, multipliers become expensive — nearly every weight would otherwise call for a dedicated constant multiplier. TWIG's decisions are all powers of two, so multiplication degenerates into shifting, and weight layers become circuits made almost entirely of wires and adders. The measurement uses five linear layers of LLaMA-3-8B with 128 FineWeb calibration sequences. At equal layer error, a flat tree needs 9–15% fewer adders than GPTQ on typical matrices at 3-bit and 4-bit accuracy, and 20–55% fewer on matrices with strong outliers. Counting weight digits alone, the tree still needs 6–7% fewer adders at 4-bit and 9–11% at 3-bit on typical matrices. One counterintuitive detail: shared exponents save bits but not additions, so the flattest tree turned out best on the adder count.

My read

The part worth dwelling on is not "24% saved" but the fact that this paper compresses formats, compression, circuits and discrete optimization — four things that have long lived in separate fields — into one object. They used to have separate vocabularies and separate toolchains, and could barely talk to each other; now they are four readings of the same tree. The other half of the value is deployability: the properties are proved, but the authors themselves say the measurements are "first evidence on selected models and layers," a full study of circuit area and energy for whole networks is left to a companion paper, the adder counts are a first-order proxy for circuit size rather than area itself, and the repository does not yet support MoEs.

The real ambition is the next step. Since the states are discrete to begin with, you can train directly in that space, so the object you train is the object you deploy. At that point the gap between training and deployment stops being an engineering patch job and becomes a representation question.

So the question is: if formats, compression, circuits and solvers turn out to be four readings of one object, is "model deployment" an engineering problem, or a representation problem?