Same questions, same weights, one change: the mixed-model state that the inference engine caches between decoding steps, moved from FP32 to bf16. On Qwen3.5-9B across 590 competition-math problems, accuracy fell from 85.6% to 70.8%. What moved was not weight precision and not model capability, but the precision of the state cache that gets rewritten on every decoding step — something most serving setups never log.

A new paper measures that effect as a paired comparison across six open hybrid reasoning models (arXiv:2611.12127, paper page, HTML full text). Recurrent mixers such as Mamba-2 and Gated DeltaNet read their state out, update it, and write it back at every decoding step. The engine computes in FP32 but stores according to the cache dtype, so every write-back is a rounding, and that rounding bias accumulates coherently over a long reasoning trace.

Five of six models think longer

The paper pairs FP32 against bf16 item by item on 590 problems: MATH-500 plus 30 each from AIME 2024, AIME 2025, and HMMT February 2025. Five of the six models produce longer outputs, with length ratios running from 1.070x to 1.265x, and every one of them hits the 32K token cap more often. The damage concentrates on Qwen3.5-9B: cap hits rise from 118 to 223 and accuracy drops from 85.6% to 70.8%. On LiveCodeBench, Nemotron-Nano-9B-v2 and Qwen3.5-9B also reason longer, losing 12.6 and 13.4 points of pass@1 respectively. The effect survives greedy decoding, so it is not a sampling artifact.

Where the loss lands matters just as much. Nearly all of Qwen3.5-9B's lost points come from items that hit the cap: on items both arms finish, accuracy is unchanged. But restrict attention to items that finished under both settings and the conclusion flips — on LiveCodeBench, Nemotron-Nano-9B-v2's bf16 samples that did finish scored higher pass@1 (85.1% against 65.0%). That kind of selection bias is enough to make an evaluation declare bf16 the winner.

More budget only closes half the gap

The paper reruns every capped item at a 128K budget. Qwen3.5-9B's overall accuracy gap narrows from 14.7 points to 6.9 (93.6% against 86.6%) but does not close: at 128K, 93 samples remain unfinished under bf16 against 44 under FP32, and 85% of those are exact repetition loops. bf16 produces two distinct failures — reasoning that costs extra tokens but still lands on an answer, and runs that never terminate. Nemotron-Nano-9B-v2 is the token-cost case: 30 of the 31 items that capped only under bf16 finish at 128K, so there the cost is mostly tokens rather than correctness.

Same-width stochastic rounding recovers it

If the problem were simply insufficient resolution, a wider format should help. The paper finds the opposite: at the same 16-bit width, switching to stochastic rounding (SR) removes the length increase in four of the five affected models and restores Qwen3.5-9B to 87.6%. Swapping Triton's default Philox generator for an integer hash tightens the match further — SR on Qwen3.5-9B matches FP32 on cap hits and accuracy (113 against 118; 85.6% for both) while running at 10,393 tokens/s end to end against 10,163 for a same-day bf16 RTN run, so memory and decoding speed are unchanged. That speed claim carries a caveat: the default Philox GDN kernel only reaches 30–60% of RTN throughput, and it is the integer hash that claws the throughput back.

Nemotron-3 is the one model SR does not rescue (Flog=0.38, with a residual bulk shift of roughly 1.05x still in place), and the cause is unknown. Head localization is stranger still: in the Nemotron-Nano-v2 lineage, applying SR to just 25% of the heads — an intermediate-decay band — removes the effect, while the single octile carrying the largest teacher-forced KL share (49% of the KL) shows no detectable effect on its own. Precision allocation cannot be decided by KL alone.

Why this matters now

This is not an academic nicety, because engine defaults are already on the wrong side of it. vLLM 0.33 falls back to the model dtype (usually bf16) for a config with no mamba_ssm_dtype field, while SGLang 0.5.21 falls back to FP32; a fine-tune that quietly drops that field changes behavior with no warning. NVIDIA quantified the same class of problem in the Nemotron 3 Super report: an FP16 SSM cache pushed LiveCodeBench completion tokens from 21,769 to 29,812, a 36.95% verbosity increase, and stochastic rounding brought them back to 21,392 (report). LeapQuant supplies the other side of the ledger: a bf16 state drops Qwen3.5-9B's AIME from 87.9% to 72.1% (paper), and the vLLM community RFC notes directly that --mamba-ssm-cache-dtype offers only float32, float16, and bfloat16, and that a bf16 state stored every step hurts long reasoning (RFC).

The fix already exists but has not shipped. SGLang's --enable-mamba-cache-stochastic-rounding currently requires --mamba-ssm-dtype float16, and a bf16 SR patch is not yet in the mainline (docs).

What to take away

The paper's real contribution is not a new quantization method but an evaluation rule: when you evaluate a hybrid model, report the state dtype, the cap rate, and the output length alongside accuracy. Reporting accuracy alone reports nothing. Once a serving stack defaults the state cache to bf16, what you measure is a function of the rounding bias, not the model.

One question to sit with: if you have ever dropped state-cache precision on a production reasoning model to save memory, was that included in your last accuracy-regression investigation?