What is the nastiest input in alignment training? Not compute — annotation quality. Preference optimization in the DPO family has become the mainstream alignment route, but the "human preferences" inside training pairs are not clean. A team from The Ohio State University opens their paper by naming the problem directly: preference data is often noisy due to annotation errors and ambiguity. Their work PORT, accepted at NeurIPS 2026, offers a response that adds almost no training overhead: do not clean the data, do not swap the model — just reassign weights per token inside the loss function.
Whole responses treated as right or wrong
Standard methods such as DPO, IPO, and SLiC-HF treat a pair of chosen-versus-rejected responses as the atomic unit. Sequence-level robust variants — cDPO, rDPO, β-DPO, Dr.DPO — still correct entire responses as uniformly right or wrong. Real noise, however, is fine-grained: the paper stresses that rejected responses may still contain informative reasoning steps, while preferred responses can include subtle errors. Handling a pair wholesale bundles good tokens together with bad ones.
Method: empirical CDF ranks within the batch
PORT's core signal is refreshingly cheap. Token log-probabilities are pooled across the batch to form a global empirical CDF, giving every non-padding token a percentile rank. The gap between the chosen token's rank and the rejected token's rank is a token-level noise signal: a rejected token with anomalously high probability marks a suspicious position. The gap is mapped to a loss weight w = γ·exp(ρ·Δ), with γ = 1.5 and ρ = 0.3 found by grid search on a held-out validation split. The whole computation rides on a single forward pass and introduces no extra backward-pass overhead. On the theory side, the authors prove that this weighting minimizes an upper bound on the gradient bias induced by noisy preference labels: samples with a higher noise rate or a larger gradient norm should get smaller optimal weights.
Results: the noisier the data, the bigger the edge
On AlpacaFarm with 0.3 of preference labels deliberately flipped, PORT on LLaMA2-7B reaches a Gold RM Score of 3.82 and a GPT Win Rate of 67.6% — beating the strongest sequence-level baseline β-DPO by 0.40 and 4.4 percentage points respectively, while vanilla DPO manages only 2.66 and 56.8%. On safety alignment (trained on Anthropic HH, evaluated on PKU-Beaver), PORT scores a Safety Rate of 0.55 and a win rate of 67.6%, the best across all baselines compared in the paper, which also observes that label-smoothing methods overall underperform reweighting-based ones. The trend matters more: as the flip probability rises from 0.2 to 0.4, DPO's safety rate drops from 0.45 to 0.38, while PORT only slides from 0.56 to 0.53 — the dirtier the data, the more token-level reweighting pays off.
Noise types get their own ablation. With the flip probability fixed at 0.4, under uniformly random label flips PORT posts a Safety Rate of 0.53 and a win rate of 65.9%, ahead of the strongest baseline Dr.DPO by roughly 0.05 and 4.1 percentage points; under the systematic noise regime, where flip probabilities are set by the reward margin under a Bradley-Terry model, it still holds 0.49 and 62.5%. Both regimes hold up, and the consistent margin across both noise types, the authors argue, is what validates the theoretical guarantee behind the CDF-based weighting. It is also worth noting that every baseline in the paper is re-implemented under identical training conditions, with an effective batch of 16 preference pairs, and results are reported as mean and standard deviation over 3 random seeds.
The edge narrows on Qwen3-4B: Gold RM 2.98 is still the highest among all methods, but the 53.6% win rate slightly trails the filtering-based SamS at 53.8%. The authors attribute this to LLaMA2-7B being trained on comparatively older data and, with higher capacity, being more prone to memorizing noisy labels. The mechanism check is the most convincing part: with Qwen2.5-0.5B trained for 30 steps on 128 HH-RLHF pairs, samples split by initial separation into high- and low-reliability groups — the reward model agrees with ground-truth ordering on 82.8% of high-reliability pairs but only 59.4% of low-reliability ones — DPO shows nearly identical loss trajectories for both groups, while PORT drives the high-reliability group down much faster. The gradient genuinely concentrates on clean samples.
So what
PORT's value is not leaderboard chasing; it moves data cleaning from a preprocessing step into the loss function. Teams with limited annotation budgets might stop agonizing over hiring more annotators and instead let training itself down-weight dirty labels. The paper is at arXiv:2611.12763, the code at osu-srml/PORT; the README marks it as the official implementation for NeurIPS 2026, keeps the usual two-stage recipe — SFT on the chosen responses, then PORT from the SFT checkpoint — and ships FSDP multi-GPU support with loaders for HH-RLHF and AlpacaFarm. Alignment data was never clean; admitting that and designing training around it is probably the more realistic path.