Majority voting is no longer new: run the model several times on the same question, take the answer that appears most often. It is the cheapest way to lift a reasoning score, so people started attaching statistical certificates to it, using an anytime-valid e-process that accumulates evidence and stops once it crosses a threshold, with Ville's inequality supplying finite-sample Type I error control. Every construction in that line assumes the repeated samples for one question are conditionally independent. An arXiv preprint (not yet peer reviewed, https://arxiv.org/abs/2611.26592) argues that premise fails on real reasoning streams.

Correlation splits one null into two

The modelling borrows the random-effects and intraclass-correlation structure from clustered data: each certification unit draws a latent state Z once at the start, the increments are independent and identically distributed conditional on Z, so the marginal process becomes exchangeable and equicorrelated with correlation ρ. The trouble is that once ρ is above zero, a single null splits into two. The realized null concerns the belief that was drawn this time, the population null concerns the model's average behaviour, and the two coincide only when ρ=0.

The paper stresses it does not overturn existing certificates. For a single question with independently sampled rollouts the two nulls are the same and the earlier conclusions hold as they stand. The problem appears once correlation exists: the same construction then controls the realized null, while what an application actually wants is the population null.

Measured: the correlation is there

The runs use Claude Haiku 4.5 at temperature 1. On GSM8K, 300 prompts with 30 rollouts each, the intraclass correlation is ρ̂=0.435 with a bootstrap 95% interval from 0.208 to 0.619; on AIME, 150 problems, ρ̂=0.480 with an interval from 0.405 to 0.544. Both sit well away from zero. The more useful consequence follows from effective sample size: the design effect on the within-unit mean pushes the variance-equivalent sample size toward 1/ρ, which is about 2.3 on GSM8K and 2.1 on AIME. Drawing more rollouts for the same question stops buying information quickly.

What goes wrong: the certificate points at the wrong object

In the paper's own terms, the standard betting e-process stays anytime-valid for the realized null, but read as a population-null certificate, a suitable mixture crosses with asymptotic probability at least π=P(μ(Z)>0) and need not respect the nominal ε.

The gap shows up on the harder side of the real data. On AIME the estimated population margin is negative (θ̂=-0.15), suggesting the model is unreliable on most questions, and yet the single-unit rule crosses on 24% of prompts while the nominal level is 0.05, whereas the prompt-level remedy declines on the same data at a rate of 0.00. Binning by difficulty makes it clearer: the single-unit crossing rate is 0.06 in the (0.5,0.7] agreement bin and jumps to 0.73 in the (0.7,0.9] bin, with the drift θ crossing zero between exactly those two bins. In the synthetic study, where a correlated structure with ρ=0.09 and π=0.37 is planted, the single-unit rule still certifies the population null at a rate of 0.32, while the prompt-level remedy sits at 0.033 when θ=0.

The fix changes the unit, not the count

The remedy is to move the e-process up to the prompt level, where each question contributes a nearly independent summary and population validity is restored. The accompanying allocation result runs against intuition: under a pure rollout budget, the variance-minimising split is one rollout per prompt. In the paper's phrasing, the single-unit budget guarantee is vacuous for the population null, and sampling more cannot buy information about the population.

Where to push back

The paper lists its own limits. The correlation coming from a shared prefix within a question is not measured, and measuring it would need tree-of-thought or prefix-shared generation. The real data uses a single production model, Claude Haiku 4.5, and the paper says a reproducible open-model check on something like gpt-oss-20b or Qwen3-30B is the natural robustness test but could shift the magnitudes or even the qualitative pattern. There is no guarantee under arbitrary serial dependence. Type I error control is not power, so near the θ≈0 boundary certification may need many units. And this is still an arXiv preprint that has not been peer reviewed.

So what

For anyone building a reasoning system the implication is direct: if the pipeline is 'many rollouts per question plus a statistical certificate', the certificate may not be stamping what you think it is. Getting a population-level guarantee means spreading the budget across questions and aggregating at the prompt level, rather than piling more samples onto one question.

Other independent teams work on the certification side: CITE (https://arxiv.org/abs/2605.05873) targets anytime-valid certification of a prespecified target answer, and Certified Self-Consistency (https://arxiv.org/abs/2510.17472) uses a Martingale Majority Certificate for sequential stopping. Self-consistency itself comes from the ICLR 2023 paper (conference page https://iclr.cc/virtual/2023/poster/11718), which reports a +17.9% gain on GSM8K. The gains on this line are real; the statistical guarantee only counts once it moves to the right unit.