[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-kling-multimodal-understanding-report":3,"topics-all":38,"news-related-ba84441d-5a19-48d4-b1cb-5d068f8dff40":67},{"id":4,"title":5,"summary":6,"content":7,"original_url":8,"source_id":9,"tags":10,"translations":24,"news_slug":31,"published_at":32,"created_at":33,"modified_at":34,"is_published":35,"publish_type":36,"image_url":14,"view_count":37},"ba84441d-5a19-48d4-b1cb-5d068f8dff40","可灵技术报告:视频生成质量是多模态理解的投影","可灵团队 11 月 3 日在 arXiv 发布技术报告,复盘 Kling 1.0 到 4.0 背后的多模态理解栈:瓶颈正从渲染转向语义控制;共享底座 Kling-MLLM 从 27B 到 397B-A17B,特化出字幕、提示增强、文本编码器与双奖励模型;字幕密度被视为数据效率的核心度量。","11 月 3 日,可灵(Kling)团队在 arXiv 发布技术报告《Behind Video Generation》,把 Kling 1.0 到 4.0 四代视频生成模型背后的「理解栈」一次性摊开:近 50 位研究者署名,从字幕、提示增强、文本编码器一路讲到奖励模型与评测。结论先行:视频生成的瓶颈正从渲染转向语义控制——通俗地说,从「画得像」转向「听得懂」。\n\n## 进步的单位,从帧变成了序列\n\n报告先梳理四年时间线:2023 年通用视频创作还停留在愿景,2024 年系统能出演示但不可控,2025 年头部模型已能产出可用的镜头级素材,2026 年前沿系统开始生成协调多镜头、持久主体、表演、镜头语言、对白、音乐与音效的完整短片。衡量进步的单位从帧、到片段、到镜头、再到完整序列。问题随之改变:序列级的质量是组合式的,约束相互作用、误差随时间累积,系统必须从异构输入里推断出潜在的创作意图,再表达成生成器能执行的条件。报告的核心论点是:**生成质量是多模态理解贯穿「数据-条件化-对齐-评测」整条链路的投影**。\n\n## 一个底座,全家特化\n\n链路起点是 Kling-MLLM:统一图像、视频、音频、音视频同步与交错输入的自回归多模态底座,家族包含 27B 稠密版与 35B-A3B、397B-A17B 两个 MoE 版,语言与视觉骨干从 Qwen3.5–Qwen3.8 系列检查点初始化。底座经任务后训练特化成字幕器、提示增强模型、文本编码器、奖励模型与自动评测器——视觉、时序与声学知识集中在底座复用,各任务自选精度与延迟的平衡点。音频塔分两档:小模型沿用 Qwen3-Omni 设计,16-kHz 单声道编码为每秒 13 个音频 token;大模型用双分支塔融合 Mel 与 waveform-VAE 特征,每秒 25 个。训练走四阶段课程:声学初始化、联合多模态、时序扩容、长上下文巩固。\n\n## 字幕是合同,奖励拆两个\n\nDiT 的训练样本量远少于语言模型,报告因此把「单位视频时长的描述密度」当作数据效率的核心度量:字幕是训练用的语义合同,并延伸出显式绑定引用的 reference-grounded captioning。提示增强分两层:basic PE 把用户意图翻译进字幕分布,最终阶段接入冻结的文本编码器与 DiT,做生成器在环强化学习,按视觉质量、运动质量、文本对齐三个奖励更新;agentic PE 在推理时先跑,负责故事修正与范例检索——因为生产环境里大量用户提示本身就是 AI 写的,长而流畅,却常自相矛盾。奖励模型也拆成两个:GRM 基于 Kling-MLLM-27B,输出可解释、带时空定位的批评,给出缺陷标签、时间区间、逐帧边界框与理由;DRM 单次 prefill 给多维分数。两者合计服务 DiT RL、PE RL、评测辅助、checkpoint 选择、数据过滤五个生产场景。\n\n## 所以呢\n\n文本编码器一节的判断值得同行细读:可靠的条件编码器来自感知训练,而非更强的语言能力;通用 QA 榜测的是自回归解码,不是 DiT 真正消费的隐藏状态;候选编码器可以纯文本预筛,再上昂贵的端到端训练。评测同理:单一聚合分诊断不了链路——同一个失败画面,可能源于字幕漏了监督、条件化丢了时序结构、奖励模型偏爱平滑但静止的运动,或评测器根本没测目标事件。这份报告的价值不在跑分,而在把「理解」从生成系统的事后评分器,变成贯穿生产链路的基础设施。[arXiv 报告原文](https:\u002F\u002Farxiv.org\u002Fabs\u002F2611.04916)","https:\u002F\u002Farxiv.org\u002Fabs\u002F2611.04916","7437aeb9-930c-4866-a2e9-48003c1a792b",[11,15,18,21],{"id":12,"name":13,"slug":13,"description":14,"color":14},"40269b40-7942-4650-9672-ed2e6524d37a","ai-technology",null,{"id":16,"name":17,"slug":17,"description":14,"color":14},"eaf35b67-8c08-4d6a-8567-90ee14f1175d","kling",{"id":19,"name":20,"slug":20,"description":14,"color":14},"499f4b56-819d-49a3-9609-33e775143b86","multimodal",{"id":22,"name":23,"slug":23,"description":14,"color":14},"ebe5dcd1-46b1-4298-b8c2-8e0e2f456e56","video-generation",[25],{"id":26,"lang":27,"title":28,"summary":29,"content":30},"5542a81a-d1f7-40d4-9a0a-9c53fa55639d","en","Kling report: the bottleneck moves from rendering to control","Kling team's Nov 3 arXiv report reviews the understanding stack behind Kling 1.0-4.0: the bottleneck shifts from rendering to semantic control.","On November 3, the Kling team published a technical report on arXiv, \"Behind Video Generation,\" laying out the multimodal understanding stack behind four generations of Kling video models. The central thesis: generation quality is the projection of multimodal understanding through the entire data-conditioning-alignment-evaluation chain, and the bottleneck is shifting from rendering to semantic control.\n\n## The unit of progress changed from frame to sequence\n\nThe report traces four years: 2023 was still aspirational, 2024 brought compelling but unreliable demonstrations, 2025 delivered production-usable shots, and 2026 frontier systems generate coherent short-form sequences that coordinate multiple shots, persistent subjects, performance, camera language, dialogue, music, and sound. The unit of progress moved from frame to clip, to shot, to the complete sequence. Sequence-level quality is compositional: constraints interact and errors accumulate, so the system must infer a latent creative specification from heterogeneous inputs and express it as conditions the generator can execute.\n\n## One backbone, many specialists\n\nKling-MLLM is a shared autoregressive foundation model unifying image, video, audio, synchronized audio-video, and interleaved inputs. The family spans a dense 27B variant and MoE variants at 35B-A3B and 397B-A17B, with LLM and vision backbones initialized from Qwen3.5-Qwen3.8 checkpoints. Task-specific post-training specializes the base into captioners, prompt-enhancement models, text encoders, reward models, and evaluators. Audio towers are scale-dependent: smaller variants follow the Qwen3-Omni design, encoding 16-kHz monaural audio as 13 audio tokens per second; larger variants fuse Mel and waveform-VAE features in a dual-branch tower at 25 tokens per second. Training follows a four-stage curriculum: acoustic initialization, joint multimodal training, temporal-capacity expansion, and long-context consolidation.\n\n## Captions as a contract, rewards split in two\n\nBecause a DiT trains on far fewer samples than a language model, the report treats description density per unit of video duration as the core measure of data efficiency: captioning is a semantic contract, extended by reference-grounded captioning with explicit bindings. Prompt enhancement is layered. Basic PE translates user intent into the caption distribution, and a final DiT-in-the-loop RL stage connects PE to the frozen text encoder and DiT, updating it with video-level rewards for visual quality, motion quality, and text alignment. Agentic PE runs before basic PE at inference, handling story refinement and retrieval of compelling examples, because many production prompts are themselves AI-written: long, fluent, and internally inconsistent. Reward modeling splits in two. A General Reward Model built on Kling-MLLM-27B produces explainable, spatiotemporally localized critiques with defect tags, temporal intervals, per-frame boxes, and rationales; a Decision Reward Model scores many dimensions in a single prefill. Together they serve five production scenarios: DiT RL, PE RL, auxiliary machine evaluation, checkpoint selection, and data filtering.\n\n## So what\n\nThe text-encoder section deserves a close read: a reliable conditioning encoder comes from perceptual training rather than stronger language ability; general QA benchmarks measure autoregressive decoding, not the hidden states a DiT actually consumes; and candidate encoders can be screened from text alone before costly end-to-end training. Evaluation should follow the task, because a single aggregate score cannot diagnose a chain where the same failure may originate in captions that omit supervision, conditioning that discards temporal structure, a reward model that favors smooth but static motion, or an evaluator that never tests the requested event. The value of this report is not benchmark scores but turning understanding from a post-hoc judge into production infrastructure. [arXiv report](https:\u002F\u002Farxiv.org\u002Fabs\u002F2611.04916)","kling-multimodal-understanding-report","2026-11-04T21:08:37Z","2026-11-04T21:08:40.670098Z","2026-11-04T21:08:40.670107Z",true,"agent",529,[39,49,58],{"slug":40,"tag_slug":41,"title_zh":42,"title_en":43,"intro_zh":44,"intro_en":45,"id":46,"is_active":35,"created_at":47,"modified_at":48},"image-generation","text-to-image","图像生成：从文生图到逐框编辑的模型竞赛","Image Generation: from text-to-image to frame-level editing","文生图已是红海，战场正转向编辑与可控性：FLUX.2\u002F3 用开源和量化把生图搬进消费级显卡，Qwen-Image、Seedream 等国产模型紧咬不放，GPT Image 2 则把推理能力带进像素。本专题追踪图像生成与编辑模型的发布、架构演进、加速量化与商业化数据，看「好图」如何变成「可用的图」。","Text-to-image is a crowded race, and the frontier has moved to editing and control: FLUX.2\u002F3 push open weights and quantization onto consumer GPUs, Qwen-Image and Seedream close in from China, and GPT Image 2 brings reasoning into pixels. This topic tracks model releases, architecture shifts, acceleration work, and the business numbers behind \"pretty\" becoming \"usable\".","52de7c59-7864-4569-8ae7-3d656d26a1a1","2026-11-20T07:20:23.534506Z","2026-11-20T07:20:23.534527Z",{"slug":50,"tag_slug":50,"title_zh":51,"title_en":52,"intro_zh":53,"intro_en":54,"id":55,"is_active":35,"created_at":56,"modified_at":57},"ai-for-science","AI for Science 2026：从 UniPert 到 GPT-Rosalind 的硬核进化","AI for Science 2026: from UniPert to GPT-Rosalind","生命科学、化学材料、物理世界模型——AI 正在从\"语言工具\"变成\"实验伙伴\"。本专题收录 AI 在三大科学方向的关键节点：UniPert 统一基因与化学扰动空间、GPT-Rosalind 端到端生命科学推理、达摩院 AI 智能体 28 小时找到 4 种超导新材料、Anthropic Claude Science 把工作台做成标准品。","From language tool to lab partner — AI is reshaping life sciences, chemistry\u002Fmaterials, and physical world models. This topic covers the key milestones: UniPert unifying genetic-chemical perturbation spaces, GPT-Rosalind's end-to-end life-sciences reasoning, DAMO's AI agent discovering 4 superconducting materials in 28 hours, and Anthropic's Claude Science workbench going mainstream.","988a4300-5fab-41c4-b5d8-63711a2dc757","2026-09-10T01:34:15.296649Z","2026-09-10T01:34:15.296663Z",{"slug":59,"tag_slug":59,"title_zh":60,"title_en":61,"intro_zh":62,"intro_en":63,"id":64,"is_active":35,"created_at":65,"modified_at":66},"h3-series","MiniMax H3 系列：从开源权重到 35 倍吞吐","MiniMax H3 Series: from open weights to 35x throughput","MiniMax H3 自 2026 年 8 月开源以来节奏密集：官方把生成、参考与编辑收回一个模型；ComfyUI 当天压进 RTX 3060；摩尔线程 3 小时完成国产 GPU 适配；fal 后训练版把吞吐拉到 35 倍；FastH3 蒸馏再砍推理成本。本专题持续追踪 H3 的发布—开源—蒸馏—部署全链路。","Since MiniMax open-sourced H3 in August 2026 the pace has been relentless: one unified omni-modal model, same-day ComfyUI support down to an RTX 3060, a 3-hour Day-0 port to Moore Threads GPUs, fal's post-trained H3 Max at 35x throughput, and FastH3 distillation cutting inference cost further. This topic tracks the full H3 chain — release, open weights, distillation, deployment.","83ef0daa-3c31-4cb3-86ed-e5ee58654d5f","2026-09-08T07:33:19.942193Z","2026-09-08T07:33:19.942209Z",{"items":68},[69,74,79,84,89,94],{"id":70,"title":71,"news_slug":72,"published_at":73},"d34bdd08-5fa9-4ce0-954d-17406b896756","牛津102页综述:视频世界模型的四级证据阶梯","video-world-model-evidence-ladder-survey","2026-10-26T11:13:04+00:00",{"id":75,"title":76,"news_slug":77,"published_at":78},"e840fad5-b3a1-47cd-ac68-679d5f635dc1","世界状态交给程序管:Programmable World Model 让视频模型只管渲染","programmable-world-model-persistent-state","2026-09-10T17:10:00+00:00",{"id":80,"title":81,"news_slug":82,"published_at":83},"faab4a6c-9cb0-4f2a-a5bf-1f122306008b","Wan-Streamer v0.2：分辨率 192p→640p，保住 200ms","alibaba-wan-streamer-v0-2","2026-07-10T16:15:00+00:00",{"id":85,"title":86,"news_slug":87,"published_at":88},"aeb60d9a-6639-4669-96a4-951aadad40cb","AI 视频工具进入「全场景」分化期:6 款主流产品的技术路线对比","ai-video-tools-comparison","2026-07-08T08:00:00+00:00",{"id":90,"title":91,"news_slug":92,"published_at":93},"2657cbe0-7743-43f2-9332-ee18b84b1229","Directing the World: 中国电信 TeleAI 把自回归视频世界模型推到\"组合控制\"","teleai-directing-the-world","2026-07-01T10:30:00+00:00",{"id":95,"title":96,"news_slug":97,"published_at":98},"efa5f558-e27f-4889-bf71-74cbef874ace","即梦 AI 把 Seedance 2.0 推上原生 4K:把超分这道工序从后期流水线里拿掉","seedance-2-0-native-4k-jimeng","2026-06-24T08:00:00+00:00"]