RLVR runs on a promise: the reward can be checked by a program. In mathematics and code the verifier is exact. Almost everywhere else it is a proxy - a rubric, a learned judge, a hand-written checker - and its distance from the true objective is never measured. When the reward curve rises you cannot tell learning from loophole.

Making the true reward computable

A new arXiv preprint (https://arxiv.org/abs/2611.21538, authors from four Indian institutions including Anna University) picks a domain where ground truth is available: executable authorization policies, the allow/block gates that govern tool use for deployed agents. Ten categorical features span 124,416 contexts; every decision's minimal sufficient reasons can be enumerated exactly by an oracle, so whether each cited feature is causally relevant becomes a computed quantity, and the F1 of causal precision and recall serves as the true reward. The exercise is worth running because the baseline is poor: across measurements the paper cites, freely generated explanations from a 14B model up to frontier models sit at a causal precision of about 0.50 to 0.54 - content selection is a problem of objective, not capacity. The authors then degrade the verifier along seven realistic axes - outcome-only, sufficiency, recall-only, extraction noise, coarse taxonomy, stale policy, LLM judge - each corrupting exactly one stage of the reward pipeline. Training sees only the degraded signal while the exact verifier scores every rollout. The base setup: an SFT initialisation at 0.731 on held-out gates, a 4B policy model, 150 policy-gradient steps.

Four phenotypes, each with a signature

The hacked ones are the optimistic proxies. The sufficiency proxy cannot see padding, so the model piles on citations: on the held-out set the proxy stops at 0.940 while true faithfulness falls from 0.731 to 0.667, and cited features inflate from 6.0 to 7.5 and 8.6. A hacked explanation cites eight features where three carry the decision - word counts stay flat, so the hack targets the quantity the proxy cannot see. A recall-only proxy, optimistic by a different mechanism, ends at 0.974 against a true 0.692.

The second phenotype is starvation. The outcome-only verifier, the construction underlying most RLVR, cannot rank a faithful explanation above an unfaithful one better than chance (selection error 0.504). Its reward saturates at 1.000 almost immediately; 137 of 150 steps yield zero learning signal while true reward drifts to 0.714.

Third, dilution. Extraction noise acts as a dose response: at ε=0.05 true reward ends at 0.821, close to control. But the same cell collapsed to 0.083 under an unclipped optimizer, so noise safety is conditional.

Fourth, a surprise: ranking-preserving gaps do no harm. A stale policy missing one rule mis-decides 0.1011 of all contexts, yet training ends at 0.875 and 0.840, inside the control spread. An LLM judge (claude-haiku as extractor) with selection error 0.090 and a pessimistic sign ends at 0.865, level with the 0.871 controls.

The outcome is predictable before training

No training run is needed to see which phenotype emerges. On a probe grid, measure the verifier's selection error and the sign of its gap; damage tracks slope times selection error times the fraction of effective updates. The optimistic branch has slope 1.81 against 0.77 for the pessimistic branch - optimistic verifiers cost about 2.3x more per unit of ranking corruption. Across nine configurations (two frameworks, Qwen3-4B and Phi-3.5-mini, adapters, precisions, seeds), the sign asymmetry replicates, at 3.2x on the PyTorch grid; the constants do not replicate. The audit quantity itself is a property of the verifier alone: the sufficiency proxy's over-report of true reward stays between +0.14 and +0.31 across all nine configurations.

The cold water

The damage magnitude itself did not survive replication: the MLX grid shows +0.218 for sufficiency damage, while ten PyTorch cells land between +0.017 and +0.096. Five pre-registered explanations - adapter capacity, quantisation, precision, more optimisation, a larger model - were each tested and each fell, and the authors report the failures as they happened. The one that held is counter-intuitive: raising Phi's initialisation from 0.672 to 0.764 turned a null (+0.017) into a hack (+0.087) with nothing else changed, so competence gates whether the hack bites at all. Scale stops at 4B, since an 8B model did not fit a T4, and the domain is policy-gate explanation. Training on 14 real public policies (Cedar, OPA, Gatekeeper) was starved by task ease - even the exact control skipped 85 of 150 updates. But one transfer result deserves remembering: the hacked model's padding exported intact to the real policies, with recall 0.965 and precision 0.702 at the bottom while its aggregate F1 of 0.779 sat at the top of the table. The aggregate score hid the damage.

The mechanism is not an isolated case. Over-optimising an imperfect proxy degrades the true objective, a Goodhart effect in RL quantified by an independent team (https://arxiv.org/abs/2310.09144), and Egashira et al. independently found that systematic verification errors hurt RLVR in ways an overall error rate does not predict (https://arxiv.org/abs/2605.02909).

So what

The paper's four recommendations are cheap: audit a process verifier's selection error before training; bias pessimistic when forced to choose; never train a process objective on an outcome-only signal; give sufficiency and recall rewards a precision term. The transferable insight is the stance: whether a verifier is dangerous is measurable before the run, not after the reward curve has lied to you.

Next time a reward curve looks suspiciously pretty, ask first: is the verifier on your side, or on the model's?