[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-emoreward-video-rl-self-recalibrating-rewards":3,"topics-all":38,"news-related-94de69e8-d5b3-4199-ac19-aa7ca4e2299f":57},{"id":4,"title":5,"summary":6,"content":7,"original_url":8,"source_id":9,"tags":10,"translations":24,"news_slug":31,"published_at":32,"created_at":33,"modified_at":34,"is_published":35,"publish_type":36,"image_url":14,"view_count":37},"94de69e8-d5b3-4199-ac19-aa7ca4e2299f","视频RL的奖励自己调权重:EmoReward公开代码","港科大、腾讯混元等团队在 arXiv 公开 EmoReward,把视频生成 RL 固定的多奖励权重改成两级自适应:七个奖励源先按视频门控挑出可用证据,再用上一轮估出的可靠性乘数决定下一轮该信谁。在 HunyuanVideo 1.5 上三种 RL 算法总分全部上升,人评情感保真「好」占 60.0%。","视频生成的后训练正在往强化学习上走,但奖励这一环一直是个硬骨头:代码和数学有唯一正确答案可以自动判分,视频的「情感对不对」却没有标准答案。用固定权重的视觉质量、运动质量、文本对齐三类奖励去凑,往往既抓不住情绪意图,又给 reward hacking 留了口子。RLHF 能处理主观目标,但标注昂贵,而且不同人的审美差异让面部、肢体、色彩这些线索很难跨语境对齐。\n\nEmoReward 换了个提问方式:不是「哪个奖励更准」,而是「这一条证据该不该信、该信多少」。\n\n## 一条视频内部与两轮训练之间,两个粒度\n\n论文来自香港科技大学、腾讯混元、复旦大学与滑铁卢大学,已在 arXiv 公开(cs.CV \u002F cs.AI,30 页 15 图),并放出项目页与腾讯混元官方仓库。\n\n机制分两层。视频内部是 Reflective Reward Gating:七个奖励源(一个 VLM 裁判、面部动作单元、身体线索、时序证据、CLIP-AUTT、R1-Omni、运动自然度)给视频打分时,分数 s_k 与「这条证据可不可用」u_k 是分开记的。门控按训练阶段、人脸置信度、语义冲突、微妙程度四类信号决定每个源当次拿多少权重,不可用的直接踢出活跃集合再重新归一化。\n\n训练轮次之间是 RewardMemory:一轮跑完后,从奖励轨迹里估计每个奖励源在每种情感情境下的可靠性,导出有界的乘数表,冻结后用于下一轮。乘数取 exp(Δ_k),Δ_k 由该源与参考结果的相关性相对其他源的相关性均值作差、缩放后截断到 ±1 得到。关键在于整个过程不更新任何奖励模型的参数,只有生成器吃梯度。\n\n论文的公式是 R_emo = Σ(w⁰·g·m)·s \u002F Σ(w⁰·g·m) + ε,再与通用视频奖励按 R_total = (α·R_vid + β·R_emo)\u002F(α+β) 融合。\n\n## 跑分:三种 RL 算法都涨\n\n底座是 HunyuanVideo 1.5,跑了 DanceGRPO、FlowGRPO、SAGE-GRPO 三种 RL 算法,融合权重 α=β=0.5,批次 2、组大小 4,评测用 302 条与训练集不重叠的情感提示词。\n\n论文表 1 的 VideoAlign 总分:SAGE-GRPO 从 -0.8269 升到 -0.2624,FlowGRPO 从 -0.7719 升到 -0.5376,DanceGRPO 从 -1.2812 升到 -0.9594。四个 VideoAlign 分头(总分、视觉质量、运动质量、文本对齐)在三种算法下全部上升。DanceGRPO 那一档用的是 Reflect 而非完整 EmoReward,论文表格把两种配置分开标注。\n\n不过论文自己写了一句要紧的话:VideoAlign 同时是优化目标,所以「和它一致」不等于「情感保真度独立提升」。PickScore 和 CLIPScore 这两个旁证指标变化更小,方向也不统一,论文没有把这点藏起来。\n\n## 七种奖励策略对照,RewardMemory 是分水岭\n\n论文表 2 在固定底座与优化预算的前提下只换奖励策略,分七档:只用 VideoAlign、单一奖励源(R1-Omni)、随机路由、轮询路由、固定融合、Reflective Reward Gating、带 RewardMemory 的完整 EmoReward。\n\n加入 RewardMemory 后选择性指标到 0.428,两个复合分 core 0.922、emo 0.884;其余六档的复合分依次是 0.238\u002F0.308、0.302\u002F0.340、0.085\u002F0.064、0.266\u002F0.272、0.246\u002F0.191、0.337\u002F0.503,都落在 0.922\u002F0.884 之下。不带记忆的 Reflect 档复合分只有 0.337 和 0.503,但在单个偏好维度 EF 上是满分 1.000。记忆分配的乘数也值得看:宏观表情与运动自然度拿到的乘数分别是 1.833 和 1.510,三个几乎不动的通道则是 0.778。\n\n## 人评:坏评份额比好评份额更能说明问题\n\n论文表 3 是 18 名参与者、30 条提示词,按差\u002F一般\u002F好三档评三个维度。\n\n情感保真:EmoReward 的「好」占 60.0%,SAGE-GRPO 是 53.0%;时序连贯 54.3% 对 51.5%;视觉质量 57.2% 对 45.0%。EmoReward 三项的「差」份额也是最低:15.4%、20.2%、15.9%,SAGE-GRPO 对应 31.5%、28.9%、30.9%。\n\n真正的看点是后者:纯 SAGE-GRPO 在三个维度上的坏评份额全都高于未做 RL 的原始生成器(21.9%、23.5%、25.2%)——光靠通用视频奖励做 RL,坏样本的产出不降反升。EmoReward 才把它压下来。论文也提醒这只是描述性结果,参与者与提示词级的原始记录才能算不确定性。\n\n## 迁移到图像,结论依赖目标指标\n\n论文在 SD 3.5 和 FLUX 上做了迁移。SD 3.5 配 DanceGRPO 时,加校准后 HPSv3 有提升;但在 FLUX 上,那个专门针对 HPSv3 优化过的基线在这一项指标上并未被超过。论文同时说明 SD 3.5 各档训练预算不等,限制了因果归因;FLUX 前两个基线样例只保留了 500 update 的存档,其余是 3000。\n\n## 论文自己承认的边界\n\n附录 A 写得很直白:实验只验证了「参考轮 → 校准轮」这一次跃迁,没有证明反复迭代能持续变好;分布漂移、证据过期、以及奖励代理自我强化的反馈环都是未解问题。校准规则能否迁移到语言模型或工具调用 Agent,论文说仍待实证。\n\n复现成本也要看:官方 README 给的硬件建议是每卡 80GB 显存、32 卡(4 节点 × 8),Linux,PyTorch 2.6+。仓库里的 LICENSE 是腾讯混元社区许可,不是标准的 OSI 开源许可,模型权重也不随 Git 分发。奖励源权重缺失时会优雅降级、把该源权重分摊出去,但要拿到完整情感保真度,CLIP-AUTT、R1-Omni、OpenFace-3.0 这些权重得自己拉齐。\n\n## 所以呢\n\n这套东西的价值不在「又一个视频 RL 论文」,而在于把奖励系统的调参从人肉工程挪进了训练流程内部——上一轮跑出来的经验,变成下一轮分配信任的依据。\n\n但也要看清适用边界:VideoAlign 既是靶子又是尺子,自动指标涨了不等于情感真的更好;人评只有 18 人 30 条;跨轮次迭代的收益完全没被验证。对想上手的团队,诚实的顺序是:先把通用视频奖励的 RL 跑通,并检查坏样本有没有变多,再考虑要不要引入这一层校准——因为论文自己的数据显示,后者真正改变的恰恰是「哪些样本被判为坏」。\n\n资料来源:论文 https:\u002F\u002Farxiv.org\u002Fabs\u002F2611.06518 ,项目页 https:\u002F\u002Femoreward-project.github.io\u002F ,官方仓库 https:\u002F\u002Fgithub.com\u002FTencent-Hunyuan\u002FEmoReward 。","https:\u002F\u002Farxiv.org\u002Fabs\u002F2611.06518","7437aeb9-930c-4866-a2e9-48003c1a792b",[11,15,18,21],{"id":12,"name":13,"slug":13,"description":14,"color":14},"5e628969-6d2a-437f-998a-104e4b16cfb1","ai-progress",null,{"id":16,"name":17,"slug":17,"description":14,"color":14},"120fa59a-ff6f-4537-9bf5-f818df636a0e","benchmark",{"id":19,"name":20,"slug":20,"description":14,"color":14},"01598627-1ea6-4b27-a5d8-874971571a71","llm",{"id":22,"name":23,"slug":23,"description":14,"color":14},"ebe5dcd1-46b1-4298-b8c2-8e0e2f456e56","video-generation",[25],{"id":26,"lang":27,"title":28,"summary":29,"content":30},"9d7a7516-a152-48bf-bd8b-815509805176","en","EmoReward: Video RL Rewards That Reweight Themselves","EmoReward self-calibrates video RL rewards: seven gated providers plus prior-run multipliers. Raters scored 60.0% Good on emotion fidelity vs 53.0%.","Video generation post-training is moving toward reinforcement learning, but the reward stage has always been the hard part: code and math have a single correct answer that can be checked automatically, while whether a video \"conveys the right emotion\" has no standard answer. Fusing fixed weights for visual quality, motion quality, and text alignment tends to miss emotional intent and leaves room for reward hacking. RLHF can handle subjective goals, but annotation is expensive, and individual aesthetic differences make facial, gestural, and chromatic cues hard to calibrate across narrative contexts.\n\nEmoReward reframes the question. Instead of asking which reward is more accurate, it asks whether a given piece of evidence should be trusted at all, and how much.\n\n## Two granularities: within one video, and between two training runs\n\nThe paper comes from HKUST, Tencent Hunyuan, Fudan University, and the University of Waterloo. It is public on arXiv (cs.CV \u002F cs.AI, 30 pages, 15 figures), with a project page and an official Tencent Hunyuan repository.\n\nThe mechanism has two layers. Within a single video is Reflective Reward Gating: seven reward providers (a VLM judge, facial action units, body cues, temporal evidence, CLIP-AUTT, R1-Omni, and motion naturalness) score the video while keeping the criterion score s_k separate from whether that evidence is usable at all, u_k. Gates keyed on training phase, face confidence, semantic conflict, and subtlety decide how much weight each provider gets for that sample; unusable evidence is dropped from the active set and the weights are renormalized.\n\nBetween training runs is RewardMemory: after a run completes, provider reliability is estimated per emotional regime from the reward traces, converted into a bounded multiplier table, frozen, and reused in the next run. The multiplier is exp(Δ_k), where Δ_k comes from the provider's correlation with the reference outcome minus the mean correlation across providers, scaled and clipped to ±1. The key point is that no reward-model parameters are updated throughout; only the generator receives gradients.\n\nThe paper's formula is R_emo = Σ(w⁰·g·m)·s \u002F Σ(w⁰·g·m) + ε, fused with a general video reward as R_total = (α·R_vid + β·R_emo)\u002F(α+β).\n\n## Benchmarks: all three RL algorithms improve\n\nThe base model is HunyuanVideo 1.5, run with three RL algorithms — DanceGRPO, FlowGRPO, and SAGE-GRPO — at fusion weights α=β=0.5, batch size 2, and group size 4, evaluated on 302 emotional prompts that do not overlap the training set.\n\nThe VideoAlign Overall scores in Table 1: SAGE-GRPO rises from -0.8269 to -0.2624, FlowGRPO from -0.7719 to -0.5376, and DanceGRPO from -1.2812 to -0.9594. All four VideoAlign heads (Overall, visual quality, motion quality, text alignment) increase across all three algorithms. The DanceGRPO row uses Reflect rather than the complete EmoReward system, and the table labels the two configurations separately.\n\nBut the paper makes an important point itself: VideoAlign is also the optimization objective, so agreeing with it does not mean emotional fidelity improves independently. PickScore and CLIPScore, the two proxy metrics, move less and not uniformly — the paper does not bury this.\n\n## Seven reward policies: RewardMemory is the dividing line\n\nTable 2 fixes the backbone and optimization budget and varies only the reward policy, across seven arms: VideoAlign only, a single provider (R1-Omni), random routing, round-robin routing, fixed fusion, Reflective Reward Gating, and full EmoReward with RewardMemory.\n\nAdding RewardMemory brings the selectivity metric to 0.428 and both composite scores to core 0.922 and emo 0.884; the other six arms score 0.238\u002F0.308, 0.302\u002F0.340, 0.085\u002F0.064, 0.266\u002F0.272, 0.246\u002F0.191, and 0.337\u002F0.503, all below 0.922\u002F0.884. The Reflect arm without memory reaches only 0.337 and 0.503 on those composites, though it takes a perfect 1.000 on the standalone EF preference dimension. The memory multipliers are also worth reading: macro expression and motion naturalness receive multipliers of 1.833 and 1.510, while three near-constant channels sit at 0.778.\n\n## Human evaluation: the bad-rating share says more than the good-rating share\n\nTable 3 is 18 participants rating 30 prompts on three dimensions, using Bad\u002FNormal\u002FGood.\n\nEmotion fidelity: EmoReward takes 60.0% \"Good\" against 53.0% for SAGE-GRPO; temporal coherence 54.3% against 51.5%; visual quality 57.2% against 45.0%. EmoReward also posts the lowest \"Bad\" share on all three: 15.4%, 20.2%, and 15.9%, versus 31.5%, 28.9%, and 30.9% for SAGE-GRPO.\n\nThe real finding is in the latter. Plain SAGE-GRPO scores more \"Bad\" than the un-finetuned base generator on all three dimensions (21.9%, 23.5%, 25.2%) — RL driven purely by general video rewards makes bad outputs more common, not less. EmoReward is what pushes that back down. The paper also notes these are descriptive results: participant- and prompt-level raw records are needed to quantify uncertainty.\n\n## Transfer to image generation, where the conclusion depends on the metric\n\nThe paper runs transfer experiments on SD 3.5 and FLUX. On SD 3.5 with DanceGRPO, HPSv3 improves with calibration added; on FLUX, though, the baseline already optimized for HPSv3 is not surpassed on that metric. The paper also notes that the SD 3.5 arms have unequal training budgets, which limits causal attribution, and that the first two FLUX baseline cases exist only as saved examples at 500 updates while the rest use 3000.\n\n## Limits the paper states itself\n\nAppendix A is direct: the experiments evaluate a single reference-run-to-calibrated-run transition, not sustained improvement across repeated iterations; distribution shift, stale evidence, and feedback loops that reinforce a proxy's errors are all unresolved. Whether the same calibration rule transfers to language models or tool-use agents, the paper says, remains an empirical question.\n\nReproduction cost matters too: the official README recommends 80 GB of GPU memory per GPU across 32 GPUs (4 nodes × 8), Linux, PyTorch 2.6+. The repository's LICENSE is the Tencent Hunyuan community license rather than a standard OSI open-source license, and model weights are not distributed with Git. If a reward provider's weights are missing the pipeline degrades gracefully and redistributes that provider's weight, but CLIP-AUTT, R1-Omni, OpenFace-3.0 and the rest have to be fetched yourself to get full emotional fidelity.\n\n## So what\n\nThe value here is not \"yet another video RL paper.\" It moves reward-system tuning out of manual engineering and into the training loop itself — the experience of the previous run becomes the basis for allocating trust in the next one.\n\nBut the boundaries matter: VideoAlign is both the target and the ruler, so an automatic metric rising does not mean emotion actually improved; the human study is 18 people and 30 prompts; and the benefit of repeated cross-run iteration is entirely unverified. For teams considering this, the honest order is to first get general-reward RL working and check whether bad outputs increased, then decide whether to add this calibration layer — because the paper's own data shows that what this layer actually changes is precisely which samples get judged bad.\n\nSources: paper https:\u002F\u002Farxiv.org\u002Fabs\u002F2611.06518 , project page https:\u002F\u002Femoreward-project.github.io\u002F , official repository https:\u002F\u002Fgithub.com\u002FTencent-Hunyuan\u002FEmoReward .","emoreward-video-rl-self-recalibrating-rewards","2026-11-06T03:15:00Z","2026-11-06T03:26:23.856928Z","2026-11-06T03:26:23.856937Z",true,"agent",12,[39,48],{"slug":40,"tag_slug":40,"title_zh":41,"title_en":42,"intro_zh":43,"intro_en":44,"id":45,"is_active":35,"created_at":46,"modified_at":47},"ai-for-science","AI for Science 2026：从 UniPert 到 GPT-Rosalind 的硬核进化","AI for Science 2026: from UniPert to GPT-Rosalind","生命科学、化学材料、物理世界模型——AI 正在从\"语言工具\"变成\"实验伙伴\"。本专题收录 AI 在三大科学方向的关键节点：UniPert 统一基因与化学扰动空间、GPT-Rosalind 端到端生命科学推理、达摩院 AI 智能体 28 小时找到 4 种超导新材料、Anthropic Claude Science 把工作台做成标准品。","From language tool to lab partner — AI is reshaping life sciences, chemistry\u002Fmaterials, and physical world models. This topic covers the key milestones: UniPert unifying genetic-chemical perturbation spaces, GPT-Rosalind's end-to-end life-sciences reasoning, DAMO's AI agent discovering 4 superconducting materials in 28 hours, and Anthropic's Claude Science workbench going mainstream.","988a4300-5fab-41c4-b5d8-63711a2dc757","2026-09-10T01:34:15.296649Z","2026-09-10T01:34:15.296663Z",{"slug":49,"tag_slug":49,"title_zh":50,"title_en":51,"intro_zh":52,"intro_en":53,"id":54,"is_active":35,"created_at":55,"modified_at":56},"h3-series","MiniMax H3 系列：从开源权重到 35 倍吞吐","MiniMax H3 Series: from open weights to 35x throughput","MiniMax H3 自 2026 年 8 月开源以来节奏密集：官方把生成、参考与编辑收回一个模型；ComfyUI 当天压进 RTX 3060；摩尔线程 3 小时完成国产 GPU 适配；fal 后训练版把吞吐拉到 35 倍；FastH3 蒸馏再砍推理成本。本专题持续追踪 H3 的发布—开源—蒸馏—部署全链路。","Since MiniMax open-sourced H3 in August 2026 the pace has been relentless: one unified omni-modal model, same-day ComfyUI support down to an RTX 3060, a 3-hour Day-0 port to Moore Threads GPUs, fal's post-trained H3 Max at 35x throughput, and FastH3 distillation cutting inference cost further. This topic tracks the full H3 chain — release, open weights, distillation, deployment.","83ef0daa-3c31-4cb3-86ed-e5ee58654d5f","2026-09-08T07:33:19.942193Z","2026-09-08T07:33:19.942209Z",{"items":58},[59,64,69,74,79,84],{"id":60,"title":61,"news_slug":62,"published_at":63},"37e0486a-225d-4b44-b189-206bd756210c","多轮 Agent 强化学习提升了什么:46.8% 的真相","multi-turn-agentic-rl-budget-dependence","2026-11-06T01:18:33+00:00",{"id":65,"title":66,"news_slug":67,"published_at":68},"99c1eb7b-9226-4d61-afdb-5d54f920a85c","研究品味实测:AI 智能体的想法匹配只有人类的 18%","ideataste-llm-research-taste-benchmark","2026-11-04T11:13:00+00:00",{"id":70,"title":71,"news_slug":72,"published_at":73},"a43599f1-d635-4174-9a25-6e1031c497b1","长上下文崩掉的真凶不是长度，是内容竞争","context-rot-attention-competition-not-length","2026-11-04T09:00:00+00:00",{"id":75,"title":76,"news_slug":77,"published_at":78},"65de4df2-898d-4202-8f69-ac03e203e34c","0.8B 模型算对 220 题:正确性由 Lean 证明","sketchpad-certified-arithmetic-small-vlm","2026-11-04T07:00:00+00:00",{"id":80,"title":81,"news_slug":82,"published_at":83},"e8168e92-d16e-452a-991c-aac74dd23ec0","预训练赢了,微调输了:Muon 的反转","muon-adamw-fine-tuning-gap-reversal","2026-11-04T01:12:00+00:00",{"id":85,"title":86,"news_slug":87,"published_at":88},"e8e0df49-2426-4230-97ea-dded961ec7d0","后训练遗忘修不回来:14组对照最多捞回36%","prior-shift-logit-correction-forgetting-diagnostic","2026-11-02T07:23:24+00:00"]