Quantizing a small local model to 4 bits and running it several times to average the answers are usually treated as separate levers. The problem is that they interact: quantization changes how sensitive a model is to the order of answer options, and averaging over reorderings is the standard remedy for exactly that. A paper submitted to arXiv on 10 November (arXiv:2611.14413) prices both in the same account, authored by researchers at Intel.
The setup: two ways to save, four benchmarks
The models are Qwen3-0.6B and Qwen3-1.7B. The benchmarks are ARC-Easy, ARC-Challenge, MMLU and HellaSwag, 300 items each and 1,200 per model pooled. Quantization is plain round-to-nearest with absmax scales, one fp16 scale per group of 64 weights, at 8, 6, 4 and 3 bits, with fp16 as the reference. Weights are rounded in memory; nothing is packed and no low-bit kernel runs. Embeddings, the tied output matrix and the norms stay in fp16. Every question is asked under eight option orders: the four cyclic shifts of the original order plus the four cyclic shifts of the reversed order.
Cost is expressed in one machine-independent unit: weight bytes read times forward passes. A pass costs 0.516 at 8 bits, 0.391 at 6 bits, 0.266 at 4 bits, 0.203 at 3 bits, and 1 at fp16.
The bit budget: 6 bits is the sweet spot
Six bits loses no accuracy against fp16. It actually scores 1.8 points higher on the 0.6B model and 0.9 on the 1.7B, but the authors say they did not find the reason and make no claim that 6 bits improves a model, so they read the result as "no loss". Eight bits costs 0.7 and 0.4 points. Four bits is where it collapses: the 0.6B model loses 14.3 points and the 1.7B loses 5.6. At 3 bits both models sit at chance, 25.4% and 26.6% against a 25% random baseline.
One 6-bit pass reads 0.391 of the fp16 weight bytes with no accuracy loss, so roughly six tenths of the weight traffic goes away for free.
At 4 bits the model answers by position
This is the most interesting finding. Slot skew measures how far the distribution of chosen answer letters deviates from uniform. For the 0.6B model it is 0.15 at fp16 and 0.07 for the 1.7B; at 4 bits the 0.6B jumps to 0.53 and the 1.7B to 0.12. Concretely, the 4-bit 0.6B model puts 78% of its answers in slot A. It is no longer answering the question, it is answering by position. The same-slot share rises from 0.24 to 0.70.
The consequence: when the correct option happens to sit in the favoured slot, the 4-bit model is right far more often than average, and elsewhere far less often. The range of accuracy across the four gold slots widens from 0.28 at fp16 to 0.76. A benchmark run in the original order averages this effect away, but a user whose own questions have a skewed answer position sees a completely different number.
The authors also ran a control: Gaussian noise at the same signal-to-noise ratio gives 0.15 and 0.37 on the 0.6B across two draws and 0.19 on the 1.7B across one. The control is inconclusive on whether rounding itself adds slot skew beyond equal-power noise.
An exchange rate: pricing bits in passes
With both levers in the same unit, the paper gives a decision rule that can be copied directly. If the budget is at least 0.391 fp16 passes, use 6-bit weights and spend the rest on K = floor(B / 0.391) orders, up to four, one pass each. Below that, use 4 bits and a single order.
The head-to-head score is 2 to 7. Across 10 cost-matched pairs, extra orders win in only 2 comparisons, always in cells against fp16 at 8 or 6 bits; more bits win in 7 comparisons for each model. The cells on the accuracy-cost frontier are (3 bits, 1 pass), (4 bits, 1 pass) and (6 bits, 1 to 4 passes). Neither fp16 nor 8-bit weights appear on it at all.
There is also a counter-intuitive conversion. From 4-bit weights, averaging two orders matches the accuracy of a single pass at about 4.7 bits, and averaging four matches about 5.6 bits. Lifting the weights from 4 to 5.6 bits multiplies weight bytes by 1.37, while running four orders multiplies them by 4. While there are still bits to give, bits are the cheaper currency.
There is one batching exception. When the several orders of one question are processed together, the weights are read once and four orders are nearly free in weight bytes. In that setting, averaging four orders at 4 bits recovers 12.6 points on the 0.6B model, as large as any gain in the paper's data. The authors' conclusion is blunt: a user who can batch should average.
Calibration gets fixed as a side effect
A byproduct the paper does not advertise heavily is practical. A quantized model is not only less accurate, it is also overconfident. The expected calibration error of a single 0.6B answer is 0.182 at fp16 and degrades to 0.332 at 4 bits; after averaging four orders it drops to 0.032 at fp16 and 0.087 at 4 bits. For the 1.7B the figures are 0.240 and 0.284 falling to 0.084 and 0.086.
Better still, whether the orders agree with each other is a label-free signal that indicates correctness better than the model's own confidence does. The AUROC of order agreement for the four-order answer is 0.70 on the 0.6B and 0.80 on the 1.7B at fp16, against 0.69 and 0.78 for the single-order top probability. At 4 bits the single-order top probability collapses to 0.56, close to uninformative, while order agreement still holds 0.65. After quantization, the model's own reported certainty is not to be trusted; whether it wavers across orders is the better health check.
The boundaries of the result
The authors list their own holes with unusual thoroughness. Both models come from the Qwen3 family, between 0.6B and 1.7B. Qwen3-4B was not run, because its fp16 weights need about 8 GB and the shared GPU had 6 GB free. Only the simplest quantizer was used, and the paper explicitly makes no comparison against named methods such as GPTQ or AWQ, so the 3-bit results are a statement about that simple quantizer. The cost unit counts weight bytes only; it ignores activation traffic, the KV cache, dequantization cost and the fact that prefill is often compute-bound on a GPU. The paper reports no latency at all, only relative units.
One account is easy to miss: the tied embedding and output matrix stay in fp16 and hold 26% of the 0.6B weights and 18% of the 1.7B. If an engine computes logits over the whole vocabulary on every pass, a 6-bit pass really costs 0.55 and 0.50, not 0.391.
The sample is also thin: 300 items per task and only one or two noise draws. More importantly, fragile questions do not stay fragile across bit widths. The per-question flip share of fp16 and 4-bit weights correlates at only 0.20 on the 0.6B, so a fragile-question list built at fp16 is a weak guide to where extra permutations would help a quantized model. The authors note that a title-level search on Hugging Face Papers found no earlier study combining quantization with option order, and immediately add that this is a statement about one index and its title-level matching, so a paper using other words may exist.
So what
What is useful here is not the number 12.6 but the conversion of "save memory" and "spend compute" from two unrelated engineering intuitions into a table you can calculate against. For anyone running a small model locally the answer is fairly clear: start from 6-bit weights and spend the remaining budget on averaging up to four orders, one pass each. If batching is available, 4 bits with four averaged orders is even better.
The deeper warning is that a 4-bit model does not become dumber, it becomes a model that answers by position. Running multiple-choice evaluation on a 4-bit small model returns a score carrying a systematic slot bias. What gets measured may be the model's preference for a letter rather than its capability, and that bias propagates into downstream filtering, labelling or triage.
How much room a local small model really has left may therefore not depend on squeezing out another bit, but on whether you are willing to pay a few extra forward passes for stability, and whether your inference engine can pack those passes into a single weight read.