DeepConf is a parallel-thinking method proposed in August 2025 by a team from Meta AI and UCSD. The original paper made a striking set of claims: up to 99.9% accuracy on AIME 2025, up to 84.7% fewer generated tokens compared with full parallel thinking, no extra training and no hyperparameter tuning, and drop-in compatibility with serving frameworks such as vLLM. (original paper)

The mechanism rests on a quantity called token confidence. At every step the model produces a probability distribution over its vocabulary; taking the top 20 candidates and averaging their negative log probabilities yields a per-token confidence value. Those values are averaged over a sliding window of 2048 tokens, and the minimum of that smoothed sequence across the whole chain becomes the chain’s "lowest group confidence". Low-scoring chains get dropped before the vote; chains whose score falls below a threshold get stopped early. It is a clean idea: use the model’s own internal uncertainty as its own quality inspector.

Treating confidence as a signal

An audit paper submitted on 27 October (arXiv:2610.36709, by Pragna Mallappa of Keshav Memorial Institute of Technology) takes a different route to the same computation. Token confidences are recorded token by token, so they form a discrete-time signal in exactly the sense an audio or radio waveform does. That means the standard tools of signal processing can be pointed at it. (full text)

The translation is straightforward. A sliding-window average is a boxcar filter with all-equal weights, hence a low-pass filter: it keeps slow changes and suppresses fast ones. Taking the minimum is a detector. DeepConf’s whole method decomposes into "filtering, then detection", two parts that can be tested separately. The paper also notes that the window length of 2048 was never derived from a measurement of the confidence signal; it is one point in a large space of filters.

How strong the length effect is

The experimental setup is substantial: DeepSeek-R1-Distill-Qwen-1.5B, a distilled reasoning model with 1.5 billion parameters; the first 200 problems of MATH500 plus the first 300 of GSM8K; 16 chains sampled per problem at temperature 0.6 with top-p 0.95 and a cap of 8192 generated tokens; vLLM on a single RTX 4090. The two runs generated 22.0 million tokens in 84 minutes of generation, 8000 chains in total.

The decisive comparison is AUROC, measured within a problem, which asks how often a score ranks a correct chain above an incorrect one. DeepConf’s 2048-token boxcar reaches 0.585 on MATH500, while simply ranking chains from shortest to longest reaches 0.715 — the length baseline beats the confidence score by 0.130, with a 95% interval of 0.019 to 0.257. On GSM8K the length baseline scores 0.636, statistically tied with DeepConf’s 0.625.

Now subtract length directly. Within each problem the paper subtracts the mean from both the score and the log of chain length, fits a single global slope across all problems, and computes the AUROC of what that slope fails to explain. DeepConf’s lowest group confidence falls to 0.486 on MATH500 (interval 0.372 to 0.607) and 0.520 on GSM8K (interval 0.463 to 0.575) — chance on both benchmarks. The plain mean confidence score behaves the same way.

The mechanism: a 1/f spectrum and a minimum operation

Why does length leak in? The paper measures the power spectrum of the confidence signal. The spectral slope is -0.80 for correct chains and -0.76 for incorrect chains on MATH500, and -0.70 and -0.68 on GSM8K. Both are close to the -1 of a 1/f process, meaning most of the variance sits in slow drifts. A low-pass filter removes almost none of that variance, so the longer the chain, the more chances the smoothed signal has to wander down — and the minimum over it keeps sliding lower.

The paper then removes all chain-to-chain differences by looking inside a single chain. As the prefix grows from 2048 to 6144 tokens, the lowest group confidence falls by 0.74 on MATH500 (0.65 to 0.84) and by 0.40 on GSM8K (0.33 to 0.48). Shuffling the tokens of the same chain — same values, flat spectrum — shrinks the fall to 0.15. Randomizing only the Fourier phases, which preserves the spectrum exactly, gives 1.10 and 0.75. The spectrum alone predicts a fall at least as large as the observed one. For contrast, averaging w samples of white noise shrinks the standard deviation by roughly the square root of w, about 45 times for w=2048, so the minimum barely moves; a 1/f signal behaves differently because its slow drift survives the filter.

A residual signal, but too weak to help

Some signal survives that length cannot explain: incorrect chains have flatter spectra, meaning relatively more of their variance sits in fast fluctuations. The slope difference is 0.041 on MATH500 (interval 0.018 to 0.064) and 0.020 on GSM8K (0.007 to 0.032). Resolved into bands, the band around 0.06 cycles per token — a period of about 18 tokens, roughly a short sentence or one step of an equation — carries more power in correct chains, with AUROC 0.64 (0.53 to 0.74) and 0.57 (0.52 to 0.62). The fastest band, with periods of two to three tokens, goes the other way, with AUROC 0.38 (0.27 to 0.49) and 0.39 (0.34 to 0.44). One reading is that correct chains work through their reasoning at a steadier rhythm, though the paper is explicit that this reading has not been tested.

The paper also measures the response to reflection words. When a reasoning model writes something like "Wait" or "Alternatively", confidence drops sharply — to about 1.3 standard deviations below the chain mean on MATH500 and 1.0 on GSM8K — and recovers over roughly 50 tokens. Incorrect chains contain more reflection words per thousand tokens: 6.7 against 5.8 on MATH500 (a difference of 0.9) and 7.7 against 6.9 on GSM8K (a difference of 0.8).

The problem is that this residual is too weak. A high-frequency energy detector requiring no training retains an AUROC of 0.554 on MATH500 and 0.549 on GSM8K; a learned filter bank reaches 0.601 on GSM8K but only 0.509 on MATH500. Adding the high-frequency score to the length score with equal weight after standardization does not push the AUROC above length alone on either benchmark.

Early stopping does not beat sampling fewer chains

The paper recasts early stopping as change detection. A few warmup chains fix the baseline mean and standard deviation of token confidence, then a classical CUSUM detector watches the standardized signal and stops the chain when it crosses a threshold. On MATH500 the conclusion is blunt: at half the full cost, simply sampling fewer chains reaches 90.0% against 88.4% for the detector; at 70% of the cost, sampling fewer chains reaches 90.8%, CUSUM 89.4%, and the 2048-token boxcar 88.7%. On GSM8K all methods lie within 0.6 points of each other at every cost between 40% and 70%, which is roughly two problems out of 300. A fixed token cap is clearly worse on both benchmarks, because it removes long chains that would have voted correctly.

What the paper does not claim

The boundaries matter. The study uses a single 1.5-billion-parameter distilled model and two math benchmarks, and only 31 MATH500 problems are mixed enough to offer both correct and incorrect answered chains, which leaves wide intervals. DeepConf’s original gains are reported for much larger models on harder competition problems, where chains are longer and the relation between length and correctness may differ. The paper’s own wording is that its results do not show confidence filtering fails in that setting; they show that on this data the score adds nothing beyond length. It also uses a single sampling temperature and a single value of k, and the early-stopping results are simulated on already-finished chains.

So what

The value here is not in overturning somebody’s method. It is in taking a widely reused engineering habit apart and checking it: a score that appears to measure how sure the model is may simply be measuring how long the chain is. The recommendation is concrete — work on confidence-based chain selection should report chain length as a baseline and report AUROC with length removed, since otherwise a length effect can easily pass for a confidence effect.

For anyone running inference in production, there is a plainer takeaway worth trying first: before deploying a complicated confidence scheduler, sample fewer chains and find out where the money was actually going.