Reinforcement learning has spent the last two years on a family of "dense rewards": instead of checking only whether the final answer is right, you score each step of the reasoning chain separately, on the theory that a model which gets partial credit for partial progress will make fewer wrong turns. The default way to evaluate such a method is to train it against outcome-only GRPO for the same number of updates and see which one ends up more accurate. That comparison counts steps but never counts the bill.
A paper submitted to arXiv on 10 November (arXiv:2611.13560, 19 authors at Case Western Reserve University, 59 pages with 10 pages of main text, 18 figures and 27 tables) takes that default path apart. Its conclusion fits in one line: once compute is counted, the ranking reverses.
The bill for dense rewards: 91% of tokens go to discarded drafts
The Monte-Carlo process reward works like this: cut a completion at five fixed token positions, which splits it into six segments, and at each cut sample four continuations from that prefix; the fraction of continuations that reach the correct answer estimates how promising the prefix is. The problem is that these continuations are only scored and then thrown away — they never enter training.
At k=4, those continuations consume 91% of all generated tokens. On the same token budget, outcome-only GRPO gets to run 584 updates; the dense reward gets 50.
GPU time is worse, because the ratio is not proportional. Continuations are produced by the inference engine in large batches, which is much cheaper per token than a training step. On the same token budget on an H200, the k=4 dense reward trains for 0.14 hours while outcome-only GRPO trains for 0.43. The paper's own breakdown: outcome-only GRPO spends 61% of its time on updates, while the Monte-Carlo reward spends 73% of its time generating continuations.
How the ranking flips
Per update, the dense reward is winning. Its gradient efficiency is 1.56x [1.28, 2.03] that of outcome-only GRPO per update, and it leads by 1.5 points [1.0, 2.1] at an equal number of updates.
Switch to an equal token budget and it falls 14.3 points behind [12.7, 16.0] (n=5, Holm-corrected p=5x10^-5), and 14.9 points behind [14.0, 15.8] on twelve fresh seeds. At equal training time it trails by 7.8 points [6.8, 8.8], in all twelve seeds. Longer completions (768 tokens) shrink the gap to 6.6 points but do not flip its sign; the deficit recurs on Qwen3 0.6B, 1.7B and 4B at 8.0, 9.1 and 10.0 points, and on GSM8K at 9.8 points.
The paper then takes the harder step: give the dense rewards the same training time as the baseline (1.29 A100 GPU-hours, 8 seeds, strict verifier). None of the four variants significantly beats the baseline. The k=4 Monte-Carlo reward trails by 3.3 points [2.4, 4.3]; the variant that also trains on its continuations trails by 5.2 [4.1, 6.3]; the VinePPO-style variant trails by only 1.1 [0.4, 1.7], a deficit that sits inside the +/-2 point equivalence margin (TOST p=.005); and the k=1 reward trails by 1.1 [-0.5, +2.8], neither significant nor shown to be equivalent.
Why the cheap validation method misleads
The paper also takes apart a more widely used substitute for a full comparison: the gradient test, which asks whether a reward's update points in the same direction as an independent estimate of the gradient of expected accuracy. The Monte-Carlo reward does align — mean cosine 0.84. But once you actually train with it, adding it at a low weight (alpha = 0.25, 0.5 or 1) is equivalent to outcome-only GRPO within +/-3 points, and adding it at a high weight (alpha = 5) loses 12.4 points [8.7, 16.1].
The subtler part is that the alignment never decays while training collapses. It stays at 0.969 to 1.000, while entropy climbs from 0.21 to 0.58 and the KL from the initial policy reaches about 30 times that of the low-weight runs. The reward is not pointing the wrong way; it is being over-optimized.
So the gradient test can rank, but it cannot size. The per-token efficiency test reproduces the ranking (k=1 at 0.45x, k=2 at 0.27x, k=4 at 0.15x outcome-only GRPO), yet it overestimates every one of the five rewards it rates by 0.8 to 1.8 points — an overestimate as large as the gains it is supposed to detect.
The half that does work: picking the right problems
The paper's second case study is the opposite of a dense reward. Prompt selection does not touch the reward at all; it spends one forward pass before generation to decide which problems to sample. Each iteration draws 24 candidate prompts, predicts each one's success with a ridge critic on the policy's hidden state, and trains on the 6 whose predictions are nearest 1/2, plus 2 chosen at random.
The effect is far better than the dense reward's: +1.7 points [1.2, 2.1] on Qwen3-0.6B over 54 seeds (a preregistered replication on 24 fresh seeds gave +1.2, Holm p=.003), and +0.8 [0.4, 1.1] on Qwen2.5-1.5B over 60 seeds. It holds on 1,543 MATH test problems outside every evaluation set (+0.8 and +1.1) and on GSM8K, which neither configuration trains on (+1.1 and +1.3, secondary). Most of it survives when training time is counted: +1.0 [0.6, 1.5] and +0.5 [0.2, 0.8].
The authors did not stop at the good news. They added a random-choice control over the same 24 candidates; random choice trails the filter by 1.7 points [0.5, 2.9] (p=.008), which shows the gain comes from which problems are chosen, not from moving through the prompt stream faster.
The size of the gain is stated just as carefully. It peaks at 2.6 points [2.2, 3.0] at three sixteenths of the budget and falls to 0.8 by the end of the budget; trained to twice the budget, the lead shrinks to 0.5 [-0.5, +1.6], inside the +/-2 point equivalence margin (TOST p=.004). In budget terms, the filter reaches outcome-only GRPO's final accuracy using 0.78 of the tokens.
The sharpest sentence in the paper may be the one about the baseline: doubling the constant learning rate raises outcome-only GRPO by 1.9 points [0.5, 3.3] (Holm p=.018), leaving it 0.03 points [-1.3, +1.2] from the filter (equivalence test p=.002). A large share of what looked like a new method's gain was a baseline that had not been tuned.
The boundaries the paper admits
The limitations section is where this kind of paper earns trust, and it is unusually candid:
- Four of the six models are from the Qwen family; Llama-3.2-1B and 3B were tested only with prompt selection and on one pool. The paper states plainly that it has shown no gain outside the Qwen family.
- The rule that the filter helps when the candidates' predicted success is below 1/2 was formed post hoc from five settings. It held in five of six preregistered tests, and the one predicted gain outside the Qwen family, on Llama-3.2-1B, did not appear (+0.4 [-0.1, +1.0]).
- The preregistration plan files are internal rather than in a public registry, and some primary contrasts rest on only 3 to 5 seeds.
- The paper also flags a trap that is easy to miss: a lenient verifier drifts. When the default verifier accepts the last number in a completion even if nothing is boxed, some runs gradually stop boxing their answers — up to 49 drift cases in 408 evaluations of one study. Retraining those configurations with a strict verifier eliminated the drift (0 cases in 2,196 evaluations) without costing accuracy.
So what
The most portable takeaway is not "dense rewards do not work." The authors themselves note that under an equal-training-time comparison the k=1 variant is neither significantly worse nor shown to be equivalent, so that conclusion is not clean. What is portable is a discipline for reading claims: when you see a new reward that makes a model gain X points, ask three questions first. Was the comparison per update or per token? When tokens were counted, did the validation rollouts get billed? And was the baseline's learning rate tuned?
The paper's own conclusion is carefully worded. A claim that a training intervention helps should state its token and GPU-hour budgets, its verifier, the models on which it replicated, and how its baseline was tuned. Leave one of those out, and the conclusion can be off by 14 points.
The paper states that it releases the training and evaluation settings, statistical methods, preregistration plans and per-run records: arXiv full-text HTML, alphaXiv abstract page.