Every model running on a single machine spends most of its time pulling weights out of memory, so for small models decoded one token at a time the number of bytes read per token is a decent primary cost measure. A paper from Intel Corporation asks what happens if you stop picking one weight precision for the whole generation and instead allocate precision per step.
The setup is deliberately simple. Weights are stored once at 8 bits, and every decoding step runs on the top 4 bits of each code. If the low-precision output distribution looks uncertain, the step is redone with all 8 bits and that answer is kept. The uncertainty rule is cheap because it reads the low pass's own output and nothing else: either the entropy of the distribution or the gap between its top two logits. Everything reported comes from one model (Qwen3-0.6B), one benchmark (GSM8K) and one simulated quantizer, with cost counted in weight bytes read per token. arXiv:2611.14418
A 4-bit floor with an 8-bit safety net
The implementation is plainer than the framing suggests. Every linear weight in the transformer blocks is quantized once, in memory, with round-to-nearest on an asymmetric 8-bit grid, and groups of 64 consecutive input weights share a scale and an offset. The high version dequantizes all 8 bits; the low version keeps the top 4 bits of each code and reconstructs at the midpoint of the dropped cell. Both come from the same stored byte, so no extra weights are held for the low model. The embedding table and the output head stay in bf16.
It is worth flagging that this is a simulation: the dequantized weights are bf16 tensors, the matrix products run in bf16, and no packed kernel exists, so no speedup is measured. Costs are computed from how many low and high passes a configuration makes, multiplied by the bytes each pass would read.
In replay, the gate looks like cheating
The two gate signals both come from the low pass's logits. Thresholds were set on a calibration run of 32 GSM8K training problems, chosen so the low-pass signal crosses on 3, 7, 15 and 30 percent of calibration steps. The entropy thresholds are 1.55, 1.16, 0.76 and 0.31 nats; the margin thresholds are 0.25, 0.50, 1.25 and 2.88.
The replay analysis feeds the all-high trajectory of 150 problems back in and runs the low pass at every position, recording entropy, margin, the KL from the high to the low distribution, and whether the two argmax tokens agree. Over 36724 steps the mean KL is 0.0203 nats and 3.79 percent of steps flip. The area under the ROC curve for predicting a flip is 0.928 for entropy and 0.959 for the margin, with intervals of 0.924 to 0.932 and 0.956 to 0.962. A random gate gives 0.5 by construction.
At a 15 percent escalation budget the entropy gate catches 78 percent of the argmax flips and 59 percent of the KL mass, the margin gate 94 percent and 55 percent, and random escalation 15 percent of each. An oracle that ranks steps by their true KL to the high pass reaches 84 percent and 85 percent. Read on its own, the gate looks like it had solved the problem.
There is one more detail that makes the setting concrete. A free-running all-low generation typically departs from the all-high one at the second token: across the 150 problems the initial departure has a median index of 2, and the departing token sits at the 87th entropy percentile of all steps. So the steps a gate really needs to catch are concentrated right at the start.
The live run is where it falls apart
The live decode is what has to answer the question. All-high scores 64.0 percent on the 150 problems and all-low scores 55.3 percent, a gap of -8.7 points with an interval of -16.7 to -0.7 and an exact McNemar p of 0.053. Even the loss from running entirely at low precision is borderline at this sample size.
The entropy gate at a realized 15.7 percent escalation rate (ent2) scores 62.7 percent at 0.874 of the all-high weight bytes per token. Its paired difference is -1.3 points to all-high with an interval of -8.0 to +5.3, and +7.3 points to all-low with an interval of +0.0 to +14.7 at p=0.080. The margin gate mar2 looks better: 66.0 percent at 0.868 of the bytes, +2.0 points to all-high and +10.7 to all-low at p=0.017. The paper itself holds that result down, noting that it is one of eight gate-versus-all-low comparisons with no correction applied, and that mar2 is the highest-scoring of the ten configurations, which inflates its own estimate.
The sample was never going to separate them
Among the 150 paired problems, 26 were right under all-high and wrong under all-low, and 13 went the other way, so the discordant share is 0.26, which the paper uses as the reference rate. At that share, the approximate half-width of a 95 percent interval for a paired difference is about 8.2 points. Telling a gate that recovers half of the observed 8.7-point gap apart from all-low would need a half-width below 4.3 points, which at this discordant share means roughly 532 problems. The replay rests on 36724 steps and is much tighter, but it measures agreement of single steps and produces no task accuracy at all.
The output head is where the bytes are
The byte table deserves its own look. The block weights are 440.4M parameters: 467.9 MB per pass at 8 bits, 247.7 MB at 4 bits. But the output head is 155.6M parameters held in bf16 and read at 311.2 MB per pass, roughly the same size as the blocks. So a whole pass is 779.1 MB at high precision and 558.9 MB at low, a ratio of 0.717, while the block ratio alone is 0.529. The head accounts for 40 percent of a high pass and 56 percent of a low pass at this model size.
The direct consequence is that escalating too much loses money. The gates ent3 and mar3, both at roughly a 30 percent escalation rate, read 1.027 and 1.014 times the all-high bytes per token. As the paper puts it, escalating a third of all steps with this form of the gate is a loss in the byte model whatever its accuracy. By contrast the live gate at 16 percent escalation saves 13 percent of the bytes before any implementation cost.
The other road is cheaper
In the same byte model, verified speculation with k=4 needs 0.845 of the all-high bytes per token with zero KL, accepting 4.58 tokens per round. The entropy gate at a 15 percent budget needs 0.867 and leaves a mean KL of 0.0084 nats. At these byte counts the verified scheme reaches zero KL for a similar price, so it is the better choice whenever the target can be run for verification and proposals are accepted at the rate seen here. The gate's advantage is that it needs no second pass over a block of proposals, so it never pays for a rejected block, and no rollback machinery, which this cost model prices at zero.
What the paper admits it did not run
The gaps are listed plainly. No live random control was run, so the comparison against random escalation exists in replay alone, as an analytic expectation. No thinking-mode phase gate was run, even though the driver contains the modes and the design is spelled out in the paper. There is no live KL measurement, no 1.7B model and no second benchmark. The weights are simulated round-to-nearest with no calibration data and no outlier handling, and the paper notes that a tuned 4-bit method such as GPTQ or AWQ would shrink the gap between low and high, leaving any gate less room to work with.
What this actually tells you
The value here is a null result reported carefully, next to a byte accounting that is easy to get wrong. Two lessons for anyone running a small model locally. For one thing, 4-bit is not a linear saving: at the 0.6B scale the output head is about as large as the block weights, so the real bytes may be sitting in the part nobody quantized. Second, being able to predict where two precisions disagree is not the same as improving task accuracy, and a step-level AUC of 0.93 turned into noise once 150 real problems were scored.
If you are planning to put an uncertainty gate into your own runtime, run the matched live random control before anything else. Until the gate and random escalation have been compared on live data, the curve in the byte model is a hypothesis, not a result.