Anyone who downloads GGUF quantized models for local inference knows the name bartowski: for years he has maintained his own fork of llama-quant.cpp, focused on better handling for MoE models. On September 10 he published a long post on the Hugging Face blog, announcing that the old quantization heuristics are being rewritten from scratch — based not on intuition but on roughly 96 hours and more than 1,000 controlled experiments.

The ceiling of the old rules

The upstream llama.cpp assignment code has gone largely untouched for years. It relies on model-agnostic heuristics: for example, use_more_bits checks whether the current layer falls in the first 1/8th, the last 1/8th, or every third layer in the middle, then decides which tensors get extra bits. That logic traces back to a handful of tests from several years ago. MoE models with more than 8 experts get no special handling at all (upstream only hard-codes the exact Mixtral 8-expert case), and the small but sensitive shexp tensors were long underestimated.

How the 1,000-experiment sweep ran

The core method is "degrade-one": set every tensor to q8_0 except one, which is crushed to q2_k, and measure the damage from that single tensor. The metric is the KL divergence (KLD, lower is better) between the quant's token probabilities and the bf16 model's, measured with llama-perplexity --kl-divergence on wikitext-2 at 512 context. The main sweep ran on Qwen3.5-0.8B and 4B, with cross-validation against Gemma 4, Granite 4.2, Ling, Muse, and other families. The machine was a Framework Desktop (AMD AI Max+ 395, 128GB), and the testing framework and scripts were written with Claude's help.

Three hard findings

The sensitivity patterns are remarkably clear. First, the embedding tensor (token_embd) dominates the entire sensitivity scale — crushing it costs about 8x the worst single weight tensor at 0.8B and 16x at 4B, though it recovers most of its performance from q4_k. Second, depth forms a U shape: the first and last layers of the model are the most compression-averse. Third, small attention projections are the most sensitive per bit (attn_v, attn_output, ffn_up, and ssm_out all stand out), while ffn_gate is basically never worth extra bits.

Names become trustworthy again

From this data the author shipped a solver: feed it a model shape, the prior.json table, and a target quant type, and it outputs the "optimal" tensor layout. The accompanying naming rules restore meaning to the file names — in a Q3_K_S, 90% of the tensors are genuinely Q3_K; _M means 70%; _L allows up to 50% bumped. Direct effects: Q3_K_M on Qwen3.5-4B shrinks by 15%; Q4_K_M comes in just over 5% smaller with a marginally worse KLD, meaning better value per bit. Every new model shape must first pass a "canary test": both the new method and the old heuristic produce Q4_K_M, Q3_K_M, and IQ2_XS, plotted on a KLD-per-bit curve — if the mapped quants fall above the curve beyond noise, the pipeline automatically falls back to the old heuristic. MiniCPM5-2B was caught this way.

So what

For local inference users the impact is immediate: on the same GPU, picking new quants by the same file size gets you higher quality per bit; when downloading bartowski's freshly released GGUFs, the new per-tensor layouts data on the model page is worth a look. The bigger takeaway: quantization layout, long a matter of feel, has been decomposed into a measurable, reproducible, and automatically verifiable engineering problem. Full methodology on the Hugging Face blog.