Tencent Hunyuan open-sourced UniRL, a unified RL training framework for multimodal models. The framework comes with three new RL algorithms — DRPO, Flow-DPPO, and CPPO — designed specifically for multimodal tasks. The standout: UniRL is the first open-source framework to support end-to-end RL training of text + image + video models in a single loop.
The "unified multimodal RL" insight: training multimodal models with RL is hard because each modality has different reward signals. Text uses language-model rewards, image uses image-quality rewards, video uses temporal-consistency rewards. UniRL's fix: a "modality-agnostic reward" that combines all three into a single training signal, allowing the model to be trained end-to-end across modalities.
The three algorithms:
- DRPO (Diffusion Reward Policy Optimization): an RL algorithm for diffusion-based image and video generation. The reward is the quality of the generated sample (measured by a learned reward model).
- Flow-DPPO (Flow Dense Policy Preference Optimization): an RL algorithm for flow-matching models, which is the architecture used by Stable Diffusion 3 and Flux.
- CPPO (Cross-modal PPO): a PPO variant that handles text-image-video jointly, with cross-modal reward shaping.
The benchmark: on a set of multimodal RL tasks (text-to-image with human feedback, text-to-video with temporal consistency reward, multimodal reasoning), UniRL hits significant improvements over the base models. The biggest gain is on "human preference alignment" — the generated images/videos match human preferences more closely after UniRL training.
The bigger takeaway: "unified multimodal RL" is a significant new direction. The "separate RL for each modality" approach is wasteful, and the "unified" approach is significantly more efficient. For the industry, this means the next generation of multimodal models will be trained with unified RL, and the "modality-specific RL" pattern will be replaced.