Reinforcement learning lifted the reasoning ability of large models by a full step, but the most effective variant, RLVR (Reinforcement Learning with Verifiable Rewards), carries a hidden cost: every training prompt needs a verified correct answer. As models learn things beyond what humans can reliably grade, this annotation gets more expensive and scarcer.

The dead end of grading yourself

The industry's alternative is self-rewarding RL — letting the model score its own outputs. TTRL uses majority-vote self-consistency as the signal, Intuitor uses self-certainty, RENT uses predictive entropy. These methods share a structural flaw: the reward signal never leaves the model itself. Self-reinforcement amplifies existing biases, shrinks response diversity, and eventually drifts into homogeneous outputs or training collapse.

Co-RL cuts this knot: the reward always comes from a peer, never from yourself.

Three steps: sample, cross-check, update independently

The method works in three steps. First, each agent samples multiple completions for the same unlabeled prompt and reduces them to a single majority vote. Second, a completion earns reward 1 when its answer matches the peer's vote — an agent never contributes to its own supervision target. Third, GRPO updates each policy independently; agents share no parameters and exchange no gradients.

Beyond two agents, the votes pass along a directed ring. Three models — Qwen2.5-3B, Llama-3.2-3B-Instruct, and Qwen3-1.7B — train together in one ring, gaining 7.8%, 6.0%, and 8.2% respectively, each matching or beating its labeled reference.

Diversity is what makes a good teacher

Co-RL's most interesting finding concerns who qualifies as a teacher: two similar models make the same mistakes and reinforce the same wrong answers. The paper measured pre-training error overlap (Cohen's κ) across twelve base-checkpoint pairs: every different-family pair lands at κ ≤ 0.42, every same-family pair at κ ≥ 0.51, and the strip between them is empty. Crossing families drops the average overlap from 0.53 to 0.38, and downstream gains follow the same ordering.

Diversity enters through three channels: heterogeneous model families (differing in architecture, tokenization, and pretraining data), heterogeneous sizes, and rephrased training samples (one agent trains on the original MATH prompt, the other on a DeepSeek-V3 rewrite that keeps the answer unchanged). The "Different family+" setting reaches a 49.3% average across seven text benchmarks, above the 47.4% of the GT-Reward labeled reference.

The full scorecard: label-free matching supervised

With Qwen2.5-3B as the backbone, the full results: GSM8K 73.4→81.0, MATH500 56.6→66.6, AMC 28.9→36.1, HumanEval 39.0→65.8 (Different family+ setting). On the multimodal side, average gains of 2.3-7.2% across four VLM benchmarks.

Two ablations deserve attention. First, ensembling two TTRL-trained models scores below its own best single model (64.9%), while the Co-RL ensemble reaches 66.9% — the problem is not ensembling but training. Second, under CoMAS's own protocol, Co-RL gets 62.97% versus 58.94% with half the agents and no judge model.

Training dynamics are also clean: at four scales, self-rewarding baselines show reward collapse, sharp completion-length inflation, or mid-run divergence, while Co-RL stays stable throughout.

So what

For readers on the training side, the value is direct: when annotation cost for verifiable rewards squeezes your scaling plans, peer grading is a proven label-free alternative, with Apache-2.0 code already open (arXiv 2608.17253, GitHub). The more memorable takeaway is the κ gap: nothing lands between cross-family ≤ 0.42 and same-family ≥ 0.51 — "find peers whose mistakes differ from yours" is not folk wisdom but a measurable, pre-trainable quantity. While the industry debates whether synthetic data homogenizes models, Co-RL offers an operational answer: turn mutual disagreement itself into the supervision signal.