[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-port-token-reweighting-noisy-preference":3,"topics-all":38,"news-related-ef87ba13-75d5-41ff-a6d0-0a6e8b2cf30d":57},{"id":4,"title":5,"summary":6,"content":7,"original_url":8,"source_id":9,"tags":10,"translations":24,"news_slug":31,"published_at":32,"created_at":33,"modified_at":34,"is_published":35,"publish_type":36,"image_url":14,"view_count":37},"ef87ba13-75d5-41ff-a6d0-0a6e8b2cf30d","偏好标签三成翻错:PORT照拿67.6%胜率","偏好数据常因标注错误与歧义而不干净。俄亥俄州立团队提出PORT:把token的log概率在batch内换成经验CDF百分位,排名差映射为损失权重,只需一次前向传播。三成标签翻错的AlpacaFarm上,LLaMA2-7B拿到67.6%胜率,超最强序列级基线4.4个百分点,安全对齐同样领先,代码已开源。","对齐训练里最难缠的输入是什么?不是算力,是标注质量。DPO 一系的偏好优化已经成了对齐主流路线,但训练对里的\"人类偏好\"并不干净——俄亥俄州立大学团队的论文开篇就点名:偏好数据常因标注错误和歧义而充满噪声。这支团队被 NeurIPS 2026 接收的工作 PORT 给出一个几乎不增加训练开销的应对:不洗数据、不换模型,只在损失函数里按 token 重新分配权重。\n\n## 整条回复非黑即白,是老毛病\n\nDPO、IPO、SLiC-HF 这类标准方法把一对\"选中回复 vs 被拒回复\"当作原子单位;cDPO、rDPO、β-DPO、Dr.DPO 等序列级鲁棒变体,也只是把整条回复当成统一对错来修正。可真实的噪声是细粒度的:论文反复强调,被拒的回复里可能仍留着有效的推理步骤,被选中的回复也可能混着细微错误。整条统一处理,等于把好 token 和坏 token 一起打包。\n\n## 做法:batch 内的经验 CDF 排名\n\nPORT 的核心信号便宜得务实:把一个 batch 内全部非 padding token 的 log 概率池化,算出一个全局经验 CDF,每个 token 就有了百分位排名。选中 token 排名与被拒 token 排名之差就是 token 级噪声信号——被拒 token 的概率异常偏高,这个位置就可疑。排名差被映射为损失权重 w=γ·exp(ρ·Δ),论文取 γ=1.5、ρ=0.3,在验证集上网格搜索得到,整个计算只需要一次前向传播,不引入额外反传。理论侧,作者证明这种加权最小化噪声标签带来的梯度偏差的上界:噪声率越高、梯度范数越大的样本,最优权重越小。\n\n## 结果:噪声越大,优势越明显\n\n在 AlpacaFarm 上把 0.3 的偏好标签故意翻错:LLaMA2-7B 上 PORT 拿到 Gold RM 3.82、GPT 胜率 67.6%,比最强的序列级基线 β-DPO 分别高出 0.40 和 4.4 个百分点;原版 DPO 只有 2.66 和 56.8%。安全对齐上(Anthropic HH 训练、PKU-Beaver 评测),PORT 安全率 0.55、胜率 67.6%,在论文对比的全部基线里都拿到最好成绩;论文还观察到,标签平滑类方法整体不如重加权类方法。更关键的是趋势:翻转概率从 0.2 提到 0.4,DPO 的安全率从 0.45 掉到 0.38,PORT 只从 0.56 缓降到 0.53——数据越脏,token 级加权越占便宜。\n\n噪声类型也被单独拆开验证:固定 0.4 的翻转概率,随机标签噪声下 PORT 安全率 0.53、胜率 65.9%,领先最强的基线 Dr.DPO 约 0.05 和 4.1 个百分点;在按奖励边际决定翻转概率的系统性噪声下,也有 0.49 和 62.5%。两类噪声都稳。值得说明的是,论文的全部对比基线都在相同训练条件下复现,有效 batch 为 16 对偏好样本,结果报告 3 个随机种子的均值和标准差。\n\nQwen3-4B 上优势收窄:Gold RM 2.98 仍是所有方法中最高,胜率 53.6% 却略输过滤类方法 SamS 的 53.8%。作者的解释是 LLaMA2-7B 训练数据更旧、容量更大,更容易记住噪声标签,因此受益更明显。机制侧的对照实验更直观:Qwen2.5-0.5B 在 128 对 HH-RLHF 样本上训 30 步,按初始分离度把样本分成高\u002F低可靠两组,奖励模型对高可靠组的排序一致率有 82.8%,低可靠组只有 59.4%;DPO 对两组的损失轨迹几乎一样,PORT 则让高可靠组降得快得多——梯度确实集中到了干净样本上。\n\n## 所以呢\n\nPORT 的价值不在刷榜,而在把\"清洗数据\"这道预处理工序搬进了损失函数:标注预算有限的团队,与其纠结再雇多少标注员,不如让训练过程自己对脏标签降权。论文见 [arXiv:2611.12763](https:\u002F\u002Farxiv.org\u002Fabs\u002F2611.12763),代码在 [osu-srml\u002FPORT](https:\u002F\u002Fgithub.com\u002Fosu-srml\u002FPORT),README 注明这是 NeurIPS 2026 的官方实现,沿用先 SFT 后偏好训练的两阶段配方,支持 FSDP 多卡。对齐数据从来不是干净的;承认这一点再设计训练,可能才是更现实的路线。","https:\u002F\u002Farxiv.org\u002Fabs\u002F2611.12763","7437aeb9-930c-4866-a2e9-48003c1a792b",[11,15,18,21],{"id":12,"name":13,"slug":13,"description":14,"color":14},"1fcfaaf2-67de-43d3-9e35-5784852fec60","ai-safety",null,{"id":16,"name":17,"slug":17,"description":14,"color":14},"40269b40-7942-4650-9672-ed2e6524d37a","ai-technology",{"id":19,"name":20,"slug":20,"description":14,"color":14},"01598627-1ea6-4b27-a5d8-874971571a71","llm",{"id":22,"name":23,"slug":23,"description":14,"color":14},"b9bd9039-fcdb-41a8-b85b-fc1587def2b9","open-source",[25],{"id":26,"lang":27,"title":28,"summary":29,"content":30},"d483669d-923c-4f5b-821d-36fff518b06d","en","PORT: token reweighting survives 30% label flips","OSU's PORT reweights noisy preferences at token level via batch CDF ranks, one forward pass. At 0.3 flips it reaches 67.6% win rate on LLaMA2-7B.","What is the nastiest input in alignment training? Not compute — annotation quality. Preference optimization in the DPO family has become the mainstream alignment route, but the \"human preferences\" inside training pairs are not clean. A team from The Ohio State University opens their paper by naming the problem directly: preference data is often noisy due to annotation errors and ambiguity. Their work PORT, accepted at NeurIPS 2026, offers a response that adds almost no training overhead: do not clean the data, do not swap the model — just reassign weights per token inside the loss function.\n\n## Whole responses treated as right or wrong\n\nStandard methods such as DPO, IPO, and SLiC-HF treat a pair of chosen-versus-rejected responses as the atomic unit. Sequence-level robust variants — cDPO, rDPO, β-DPO, Dr.DPO — still correct entire responses as uniformly right or wrong. Real noise, however, is fine-grained: the paper stresses that rejected responses may still contain informative reasoning steps, while preferred responses can include subtle errors. Handling a pair wholesale bundles good tokens together with bad ones.\n\n## Method: empirical CDF ranks within the batch\n\nPORT's core signal is refreshingly cheap. Token log-probabilities are pooled across the batch to form a global empirical CDF, giving every non-padding token a percentile rank. The gap between the chosen token's rank and the rejected token's rank is a token-level noise signal: a rejected token with anomalously high probability marks a suspicious position. The gap is mapped to a loss weight w = γ·exp(ρ·Δ), with γ = 1.5 and ρ = 0.3 found by grid search on a held-out validation split. The whole computation rides on a single forward pass and introduces no extra backward-pass overhead. On the theory side, the authors prove that this weighting minimizes an upper bound on the gradient bias induced by noisy preference labels: samples with a higher noise rate or a larger gradient norm should get smaller optimal weights.\n\n## Results: the noisier the data, the bigger the edge\n\nOn AlpacaFarm with 0.3 of preference labels deliberately flipped, PORT on LLaMA2-7B reaches a Gold RM Score of 3.82 and a GPT Win Rate of 67.6% — beating the strongest sequence-level baseline β-DPO by 0.40 and 4.4 percentage points respectively, while vanilla DPO manages only 2.66 and 56.8%. On safety alignment (trained on Anthropic HH, evaluated on PKU-Beaver), PORT scores a Safety Rate of 0.55 and a win rate of 67.6%, the best across all baselines compared in the paper, which also observes that label-smoothing methods overall underperform reweighting-based ones. The trend matters more: as the flip probability rises from 0.2 to 0.4, DPO's safety rate drops from 0.45 to 0.38, while PORT only slides from 0.56 to 0.53 — the dirtier the data, the more token-level reweighting pays off.\n\nNoise types get their own ablation. With the flip probability fixed at 0.4, under uniformly random label flips PORT posts a Safety Rate of 0.53 and a win rate of 65.9%, ahead of the strongest baseline Dr.DPO by roughly 0.05 and 4.1 percentage points; under the systematic noise regime, where flip probabilities are set by the reward margin under a Bradley-Terry model, it still holds 0.49 and 62.5%. Both regimes hold up, and the consistent margin across both noise types, the authors argue, is what validates the theoretical guarantee behind the CDF-based weighting. It is also worth noting that every baseline in the paper is re-implemented under identical training conditions, with an effective batch of 16 preference pairs, and results are reported as mean and standard deviation over 3 random seeds.\n\nThe edge narrows on Qwen3-4B: Gold RM 2.98 is still the highest among all methods, but the 53.6% win rate slightly trails the filtering-based SamS at 53.8%. The authors attribute this to LLaMA2-7B being trained on comparatively older data and, with higher capacity, being more prone to memorizing noisy labels. The mechanism check is the most convincing part: with Qwen2.5-0.5B trained for 30 steps on 128 HH-RLHF pairs, samples split by initial separation into high- and low-reliability groups — the reward model agrees with ground-truth ordering on 82.8% of high-reliability pairs but only 59.4% of low-reliability ones — DPO shows nearly identical loss trajectories for both groups, while PORT drives the high-reliability group down much faster. The gradient genuinely concentrates on clean samples.\n\n## So what\n\nPORT's value is not leaderboard chasing; it moves data cleaning from a preprocessing step into the loss function. Teams with limited annotation budgets might stop agonizing over hiring more annotators and instead let training itself down-weight dirty labels. The paper is at [arXiv:2611.12763](https:\u002F\u002Farxiv.org\u002Fabs\u002F2611.12763), the code at [osu-srml\u002FPORT](https:\u002F\u002Fgithub.com\u002Fosu-srml\u002FPORT); the README marks it as the official implementation for NeurIPS 2026, keeps the usual two-stage recipe — SFT on the chosen responses, then PORT from the SFT checkpoint — and ships FSDP multi-GPU support with loaders for HH-RLHF and AlpacaFarm. Alignment data was never clean; admitting that and designing training around it is probably the more realistic path.","port-token-reweighting-noisy-preference","2026-11-10T17:30:00Z","2026-11-10T17:14:37.358425Z","2026-11-10T17:14:37.358433Z",true,"agent",194,[39,48],{"slug":40,"tag_slug":40,"title_zh":41,"title_en":42,"intro_zh":43,"intro_en":44,"id":45,"is_active":35,"created_at":46,"modified_at":47},"ai-for-science","AI for Science 2026：从 UniPert 到 GPT-Rosalind 的硬核进化","AI for Science 2026: from UniPert to GPT-Rosalind","生命科学、化学材料、物理世界模型——AI 正在从\"语言工具\"变成\"实验伙伴\"。本专题收录 AI 在三大科学方向的关键节点：UniPert 统一基因与化学扰动空间、GPT-Rosalind 端到端生命科学推理、达摩院 AI 智能体 28 小时找到 4 种超导新材料、Anthropic Claude Science 把工作台做成标准品。","From language tool to lab partner — AI is reshaping life sciences, chemistry\u002Fmaterials, and physical world models. This topic covers the key milestones: UniPert unifying genetic-chemical perturbation spaces, GPT-Rosalind's end-to-end life-sciences reasoning, DAMO's AI agent discovering 4 superconducting materials in 28 hours, and Anthropic's Claude Science workbench going mainstream.","988a4300-5fab-41c4-b5d8-63711a2dc757","2026-09-10T01:34:15.296649Z","2026-09-10T01:34:15.296663Z",{"slug":49,"tag_slug":49,"title_zh":50,"title_en":51,"intro_zh":52,"intro_en":53,"id":54,"is_active":35,"created_at":55,"modified_at":56},"h3-series","MiniMax H3 系列：从开源权重到 35 倍吞吐","MiniMax H3 Series: from open weights to 35x throughput","MiniMax H3 自 2026 年 8 月开源以来节奏密集：官方把生成、参考与编辑收回一个模型；ComfyUI 当天压进 RTX 3060；摩尔线程 3 小时完成国产 GPU 适配；fal 后训练版把吞吐拉到 35 倍；FastH3 蒸馏再砍推理成本。本专题持续追踪 H3 的发布—开源—蒸馏—部署全链路。","Since MiniMax open-sourced H3 in August 2026 the pace has been relentless: one unified omni-modal model, same-day ComfyUI support down to an RTX 3060, a 3-hour Day-0 port to Moore Threads GPUs, fal's post-trained H3 Max at 35x throughput, and FastH3 distillation cutting inference cost further. This topic tracks the full H3 chain — release, open weights, distillation, deployment.","83ef0daa-3c31-4cb3-86ed-e5ee58654d5f","2026-09-08T07:33:19.942193Z","2026-09-08T07:33:19.942209Z",{"items":58},[59,64,69,74,79,84],{"id":60,"title":61,"news_slug":62,"published_at":63},"ed8ef087-9dc6-4295-bbdd-d0d2c92977d3","GCC 拒绝 LLM 生成的实质性贡献:开源基础设施开始为 AI 代码划红线","gcc-rejects-llm-contributions-15-line-threshold","2026-07-30T03:30:00+00:00",{"id":65,"title":66,"news_slug":67,"published_at":68},"ac1b40ab-897b-4194-8631-4d873d39260e","英伟达、微软、IBM、OpenAI 罕见联名:反对美国政府限制开放权重模型","open-letter-against-open-weight-restriction","2026-07-26T10:00:00+00:00",{"id":70,"title":71,"news_slug":72,"published_at":73},"b604841d-e07b-4dfb-b090-ad19cebc5fe2","Codeberg 禁掉 vibe-coded 项目:LLM 算力成本正在拖垮开源基础设施","codeberg-vibe-coded-ban","2026-07-24T03:30:00+00:00",{"id":75,"title":76,"news_slug":77,"published_at":78},"2e7a2be3-762e-46eb-b582-af299c4ec173","模型越新，拒绝越难拆？76页审计：并不均匀","refusal-abliteration-generation-audit","2026-11-16T17:45:00+00:00",{"id":80,"title":81,"news_slug":82,"published_at":83},"f9174923-987b-4db8-a33c-d4c66e0498d7","水印不止辨真假:200 token 短文藏 24 比特溯源","codebook-watermark-short-text-attribution","2026-11-15T13:30:00+00:00",{"id":85,"title":86,"news_slug":87,"published_at":88},"251f5c18-2931-4f22-ab50-e19b7d5f58a9","AUROC 0.999 也可能零预警:监控器该怎么评","prewarn-token-level-monitor-early-warning","2026-11-15T05:20:00+00:00"]