Self-consistency (SC) is one of the most common accuracy boosters at inference time: sample dozens of complete chain-of-thought trajectories for the same question, then majority-vote the final answers. The cost is equally direct — every extra vote requires generating a whole new trajectory from scratch, so token spend grows linearly with both the number of trajectories and their length. On MATH500, sampling 32 trajectories burns an average of 18,335 tokens per question.
Existing efficiency fixes almost all operate on the count: Adaptive Consistency stops early once answers converge, ESC applies window-based stopping rules, DSC allocates bigger budgets to harder questions. They answer "when to stop, how many to sample" — but each trajectory is still generated whole. A new paper, StepBack (arXiv:2611.08403), flips the question: reasoning errors usually occur at one intermediate step, so why should every candidate be rewritten from the beginning?
Locate the first bad step, then resample from there
StepBack runs in three stages. It first samples 16 draft trajectories and splits each into reasoning steps at blank lines. It then uses lightweight signals to find the "first suspicious step" in each draft — one of two options: a step-level mean negative log-likelihood (NLL) increase above 0.1, meaning the model's confidence suddenly dropped while writing that step; or a process reward model (PRM) score drop above 0.05, with Qwen2.5-Math-PRM-7B as the default scorer. Once the error step is located, the correct prefix is kept and only that step and its suffix are resampled; if no error is detected, only the final answer step is resampled. Candidates then go through standard majority voting, with the answer budget 16×2=32 strictly matched to SC@32.
Two engineering details stand out. The whole procedure is training-free: NLL uses token probabilities the model already produces, and the PRM only scores reasoning prefixes, so localization overhead averages just 7–11% of total wall-clock time. And the suffix continuations can be generated in parallel, since no rollback depends on the outcome of a previous one. Notably, the two thresholds (0.1 and 0.05) are shared across all datasets and models with no per-dataset tuning; held-out calibration over a wide grid matches or exceeds SC@32's average accuracy while saving 20–44% of tokens.
The ledger: seven benchmarks, three models
The evaluation spans five math benchmarks — GSM8K, MATH500, AMC2023, AIME24, OlympiadBench — plus two multi-hop QA sets, HotpotQA and 2WikiMultiHopQA, 4,166 questions in total, across Llama-3.2-3B, Qwen2.5-3B, and Qwen2.5-7B Instruct models.
The one-line result: at the same budget, accuracy holds or ticks up while tokens drop substantially. With the PRM signal, average accuracy gains on the three models are +0.36, +0.17, and +0.82 points while tokens fall 21.07%, 18.66%, and 22.80%. The NLL signal saves more — 27.74%, 27.15%, and 33.32% on average — staying within 0.2 points of SC@32, and actually +0.55 on Qwen-7B. The single best cell: Qwen-7B on GSM8K saves 41.26%, which is where the abstract's "up to" comes from. The PRM signal reduces tokens on all 21 model-dataset pairs (7.66–41.26%) and improves accuracy on 14 of them.
The paper also runs an upper-bound experiment: with an LLM annotator supplying the true first-error position (an oracle setting), 8 drafts × 4 suffixes on GSM8K saves 64% of tokens while accuracy rises from 91.21% to 91.89%. The reusable space in draft prefixes is much larger than what the practical method captures — the gap is localization error, not the idea.
Compounding with adaptive methods
AC, ESC, and DSC reduce how many candidates are consumed; StepBack reduces the cost of each candidate. The two are naturally orthogonal. Stacked together, tokens drop a further 22–24% on MATH500, 35–38% on GSM8K, and 6–7% on HotpotQA relative to each adaptive method alone, and accuracy improves in all six GSM8K comparisons by 1.1–2.2 points. The most aggressive combination, StepBack+DSC on GSM8K, uses about one-fifth of SC@32's tokens while scoring 0.9 points higher (92.12 vs 91.21).
The limitations section is candid: on harder competition benchmarks like AIME24 and OlympiadBench, gains are unstable, with Llama-3B losing points in places — some failures cannot be fixed by resampling a localized suffix and need a substantially different solution path. Swapping the PRM for the 1.5B Skywork model still works (+0.36 points, 23.8% token saving in the 8×4 setting), suggesting what matters is not the scorer's size but whether the signal points at the true bad step.
One appendix comparison deserves attention: with no localization at all — always resampling just the final answer step (the final-step rule) — HotpotQA gets both more accurate and cheaper, because prefix reuse alone delivers most of the token saving. The value of localization shows up only when suffix samples dominate the vote: at 4×8, PRM-signal beats final-step resampling by 2.06 points on average across the seven datasets.
So what
The real contribution is compressing the granularity of test-time compute allocation from the trajectory level to the step level: previous methods decided "should I sample one more", StepBack decides "how many steps of this trajectory are worth keeping". For teams billed by the token, this is a decoding strategy you can ship the same day — no weight changes, no training, and everything saved is pure margin.
The number worth remembering is the 64% upper bound: the distance between it and the realized 20–40% is pure localization error. Every improvement in step-level error localization — lighter signals, smaller PRMs — raises the price-performance of test-time compute another notch.
Paper and code: arXiv:2611.08403 · GitHub repository