Post-training comes down to a ledger: after SFT you still want preference alignment, so where does the data come from? Hiring human annotators is expensive but long treated as the gold standard; having a model generate the data is cheap but suspected of self-deception. Past comparisons almost always held sample count fixed — take ten thousand pairs from each side and compare. A paper posted on arXiv:2611.16895 on November 12 by Shourik Banerjee and colleagues flips the question: pin the budget in dollars, let each side shop on its own, and see whose model ends up stronger.

Pin the dollars, then compare

Three open 7B-class base models — Olmo-3-7B, Llama-3.1-8B, Qwen-2.5-7B — all fine-tuned with SFT, then DPO, with LoRA. The human side draws on four public preference datasets: HelpSteer3, SHP, Anthropic HH, and TL;DR. The synthetic side builds two new sets structurally mirroring Anthropic HH and TL;DR: Llama-3.1-Tulu-3-8B-SFT generates the conversations, Persona Hub personas keep them diverse, and a Qwen-3-32B judge casts 32 votes per response pair, majority wins.

Costs are converted at public per-token rates: $0.0102 per synthetic preference pair ($0.0194 for summarization), while human annotation is estimated at about $1.30 per sample in replication terms — roughly a 130:1 unit-price gap on helpfulness. Shared budget tiers run $100, $200, $300, $400, and $1,000, with an extra $13k tier (10k samples) for human data. Evaluation uses a held-out 2,048-sample test set, a gpt-oss-20b judge, randomized candidate order; the paper also validates the judge — its agreement with two human annotators is on par with the agreement between those two humans.

Synthetic peaks early; human needs ten times more

The pattern holds across all three bases: synthetic data climbs fast within the first few hundred dollars, then saturates — the ceiling is set by data quality, not budget. Human data behaves oppositely: the same few hundred dollars buy only 76 to 769 examples, and trained models stay near the untrained baseline. Human data becomes competitive only at the $13k tier, more than ten times the most expensive synthetic tier, and even there it surpasses the synthetic peak only for Llama-3.1-8B and Qwen-2.5-7B — Olmo-3-7B falls short. On helpfulness, at every budget where both sources were trained, synthetic data matches or exceeds every human source.

Summarization is harsher: human TL;DR data remains below the synthetic ceiling at the $13k tier and never closes the gap at any tested budget. Qwen-2.5-7B traces a dip-then-recover curve — baseline win rate 0.41, dropping to 0.20 under small fine-tunes, recovering to 0.53 only at 12,000 samples (about $15,600), still below synthetic.

Two counterintuitive findings

First, mixing makes it worse. At a fixed $13k, every human/synthetic split from 95/5 to 99/1 lands below the better of the two pure sources, sometimes below both — on Llama-3.1-8B summarization, mixed splits score 0.52 to 0.56 against a pure-synthetic ceiling of 0.72 and pure-human 0.59. The root is cost structure: at a 130:1 price gap, a 90/10 dollar split already means a 14.4:1 sample ratio; the authors suspect the mixture dilutes whichever source would have performed better alone.

Second, human data can fail too. SHP preferences are inferred from Reddit scores rather than produced by annotators following guidelines; training on it drops all three bases below their untrained baselines, and domain restriction plus score filtering do not rescue it. The point is blunt: how a preference was produced, and whether the response format matches, matter at least as much as whether a human was involved — audit a human source before treating it as gold.

Boundaries and takeaways

The authors are explicit about limits: only 7-8B bases with LoRA, no full-parameter tuning; an all-open-weight pipeline whose conclusions may not transfer to proprietary frontier stacks; and a synthetic ceiling sealed by an 8B generator — a stronger generator could reshape the cost-performance curve. The $1.30 human price is a replication estimate, not a historical bill.

For practitioners the implication is direct: on a tight budget, start with synthetic data and reach the ceiling fast; reserve human annotation for settings where the higher ceiling justifies the cost, and vet the source before paying. The paper also offers a free early-warning signal — the premature self-termination rate: when Olmo-3-7B helpfulness win rate fell from 0.876 to 0.794 at the $1,000 tier, the share of responses ending mid-sentence rose from 5.5% to 17.5%. Treat it as a cheap checkpoint, the authors caution, not a stopping rule.

So — if your data budget loses a zero next year, which side do you cut first? The likely answer from this paper: cut the un-audited human source first.