[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-code2video-compass-mllm-code-video-benchmark":3,"topics-all":38,"news-related-891e0c6f-0c26-4ca6-bc85-3fca60bdff72":57},{"id":4,"title":5,"summary":6,"content":7,"original_url":8,"source_id":9,"tags":10,"translations":24,"news_slug":31,"published_at":32,"created_at":33,"modified_at":34,"is_published":35,"publish_type":36,"image_url":14,"view_count":37},"891e0c6f-0c26-4ca6-bc85-3fca60bdff72","代码写视频:15 个多模态大模型实测,跑通率比画质更决定成败","代码视频生成基准 Code2Video-Compass 用 2104 个任务评测 15 个多模态大模型。论文评测里 Claude Opus 5.5 三个设定综合排名居首 但只看跑通的任务 GPT-6 Astra 画质分反而更高。2756 个错误案例显示各模型卡在编译、依赖或运行时 烧更多 token 换不来好画质。","写代码，让模型输出 HTML\u002FCSS、SVG、Canvas、Three.js 程序，再由 Remotion 渲染成视频——这条路线绕开了逐像素扩散的算力消耗，换来的是可编辑、可复用、精确到时间轴的控制。代价是：模型必须写出真能跑起来的程序。\n\n10 月 20 日挂上 arXiv 的 Code2Video-Compass 基准，把这条路拉进了考场。复旦大学团队用 2,104 个任务横评 15 个多模态大模型，得出的结论比排座次更有意思：跑得通，和跑得好，是两件事。\n\n论文见 [arXiv:2610.28416](https:\u002F\u002Farxiv.org\u002Fabs\u002F2610.28416)，项目页在 [code2video-compass.github.io](https:\u002F\u002Fcode2video-compass.github.io\u002F)。\n\n## 基准怎么搭的\n\n任务分两条来路：385 个独立任务，加上从 573 个源视频派生的 1,719 个对齐任务，覆盖 T2V、I2V、V2V 三种输入。分类照动效设计的实践拆成 10 个大类、24 个子类，每个任务再标注三个复杂度维度——构建复杂度（标准图元组合 → 自定义几何 → 程序化模拟）、动态动画复杂度（分镜编排 → 跨元素协同 → 并发复杂运动）、状态依赖复杂度（相邻阶段 → 间隔阶段 → 跨多次变化的长程依赖），每维三档。\n\n评测是双层的：12 项客观指标加 5 项主观指标。客观侧看交付率、平均输出 token、视频质量、文本对齐和参考保真；主观侧由 Gemini 3.8 Flash 当裁判。裁判本身也做了校验——6 个模型 300 份输出各评 3 次，指令遵循与整体视频质量上，裁判均值与三名人类评分者的 Pearson 相关系数分别是 0.796 和 0.783，ICC(A,3) 达 0.983 和 0.970。\n\n真正计分用的是 1,054 个任务的固定子集（360 T2V、347 I2V、347 V2V）。每个模型单次 API 调用出代码，不给执行反馈，也不给重试机会——这正是真实生产里最苛刻的设定。\n\n## 排名背后真正的看点\n\n论文自己的评测里，Claude Opus 5.5 三个设定的平均排名分别是 1.60、1.08、1.15，Claude Sonnet 5.5 紧随其后，闭源阵营包揽了全部三档的前三。开源侧，DeepSeek-V4.1-Flash 在 T2V 领先（平均排名 7.35），MiMo-V2.6-Pro 拿下 I2V（7.50）和 V2V（5.92）。\n\n但更值得看的是交付率的跨度：T2V 上从 0.52 到 0.94，I2V 从 0.59 到 0.96，V2V 从 0.69 到 0.97。MiniMax M3 在 T2V 上的交付率只有 0.519，Claude Opus 5.5 是 0.944。\n\n于是有了这篇论文最锋利的一处：只看成功渲染的视频，GPT-6 Astra 的整体视频质量均值是 4.19，GPT-6.1 Sol 是 4.15，都高于 Claude Opus 5.5 的 4.04；但把执行失败算进去的全量均值里，Claude Opus 5.5 以 3.87 居首，GPT-6 Astra 和 GPT-6.1 Sol 分别是 3.62 和 3.61。写出一段漂亮却跑不通的代码，在生产里等于零。\n\n## 失败长什么样\n\n论文分析了 2,756 个错误案例，失败画像明显分化。GPT-6 Astra 有 83.4% 的失败来自编译错误加上缺失 API 或依赖包；Qwen3.8-Max 光是缺失 API 和依赖包就占 59.7%。反过来，Claude Opus 5.5 有 75.0% 的失败是运行时异常，GLM-5.3-Flash、DeepSeek-V4.1-Flash、MiMo-V2.6-Pro 也都过半是运行时异常。\n\n编译不过和跑起来崩了，是两种完全不同的能力缺口：前者考语法与依赖管理，后者考运行时语义。\n\n复杂度曲线的走向也不一样。从 L1 到 L2，构建复杂度和动态动画复杂度下，SF（结构保真）分别掉了 14.38% 和 12.82% 的分数区间，说明中等难度就已经吃力；而状态依赖复杂度在 L2 到 L3 的下降更陡，15 项指标全线走低——长程状态一致性才是真正的硬骨头。\n\n## 烧钱不等于烧得好\n\n成本侧的对比更直接。GLM-5.3-Flash 每次请求用掉 90.0k 输出 token，GPT-6 Astra 只用 13.9k，质量分却是 2.94 对 3.62。GPT-6.1 Sol 和 GPT-6 Astra 质量几乎相同（3.61 与 3.62），GPT-6 Astra 的估算 API 成本却约为前者的 6.1 倍。Claude Opus 5.5 拿下最高质量 3.87，成本反而低于 GPT-6 Astra。\n\n这些都是论文自己的评测，社区目前还没有第三方复现，数据该怎么读还留着讨论空间。\n\n## 所以呢\n\n程序化视频生成的价值不在于替代扩散模型，而在于把视频变成可版本管理、可局部重渲的工程产物。但这个前提是模型生成的代码必须能跑。\n\n这份基准真正立起来的标尺，是把\"交付率\"和\"质量\"分成两栏来记。这跟代码 agent 评测里常用的 pass@1 不是一回事——pass@1 给你一次机会，交付率告诉你这机会有多少次是无效的。当模型开始承担生产系统的编排角色，可靠性不再是附加项，而是乘数。\n","https:\u002F\u002Farxiv.org\u002Fabs\u002F2610.28416","7437aeb9-930c-4866-a2e9-48003c1a792b",[11,15,18,21],{"id":12,"name":13,"slug":13,"description":14,"color":14},"120fa59a-ff6f-4537-9bf5-f818df636a0e","benchmark",null,{"id":16,"name":17,"slug":17,"description":14,"color":14},"01598627-1ea6-4b27-a5d8-874971571a71","llm",{"id":19,"name":20,"slug":20,"description":14,"color":14},"499f4b56-819d-49a3-9609-33e775143b86","multimodal",{"id":22,"name":23,"slug":23,"description":14,"color":14},"ebe5dcd1-46b1-4298-b8c2-8e0e2f456e56","video-generation",[25],{"id":26,"lang":27,"title":28,"summary":29,"content":30},"f0a7bafc-7997-4024-8c2b-25c3ea851e76","en","Code2Video-Compass: 15 MLLMs struggle to run their code","Code2Video-Compass tests 15 MLLMs on 2,104 code-based video tasks. Claude Opus 5.5 ranks first, yet GPT-6 Astra scores higher on rendered videos.","There is a route to video generation that skips the diffusion model entirely. The model writes a program — HTML\u002FCSS, SVG, Canvas, Three.js — and Remotion renders it into a finished clip. You get editable output, reusable components, and frame-accurate control over timing. What you also get is a new failure mode: the program has to actually run.\n\nA benchmark posted to arXiv on October 20 puts that route under test. Code2Video-Compass, from a team at Fudan University, ran 15 multimodal large language models through 2,104 tasks, and its most interesting result is not a winner. It is that running and rendering well are two different things.\n\nThe paper is [arXiv:2610.28416](https:\u002F\u002Farxiv.org\u002Fabs\u002F2610.28416); the project page lives at [code2video-compass.github.io](https:\u002F\u002Fcode2video-compass.github.io\u002F).\n\n## How the benchmark is built\n\nTasks come from two directions: 385 independent ones, plus 1,719 aligned tasks derived from 573 source videos, covering text-to-video, image-to-video and video-to-video. Categories follow the practice of motion design — 10 categories, 24 subtypes. Every task is annotated along three complexity dimensions: construction (standard primitives, custom geometry, procedural simulation), dynamic animation (staged sequences, coordinated changes across elements, concurrent motion), and state dependency (adjacent stages, intervening stages, long-range dependencies across multiple changes). Each dimension has three levels.\n\nEvaluation is two-layer: 12 objective metrics plus 5 subjective ones. The objective side covers delivery rate, mean output tokens, video quality, text-video alignment and reference fidelity. The subjective side uses Gemini 3.8 Flash as judge. That judge is itself validated — 300 outputs from six models, each scored three times, against three human raters on 200 outputs. Pearson's r reaches 0.796 for instruction following and 0.783 for holistic video quality, with ICC(A,3) of 0.983 and 0.970.\n\nScoring used a fixed subset of 1,054 tasks (360 T2V, 347 I2V, 347 V2V). Each model gets one API call to produce code, with no execution feedback and no repair attempts — the harshest production-like setup there is.\n\n## What the ranking hides\n\nIn the authors' own evaluation, Claude Opus 5.5 posts the best average rank in all three settings — 1.60 on T2V, 1.08 on I2V, 1.15 on V2V — with Claude Sonnet 5.5 second each time, and proprietary models taking the top three slots across the board. Among open-weight models, DeepSeek-V4.1-Flash leads T2V with an average rank of 7.35, while MiMo-V2.6-Pro takes I2V at 7.50 and V2V at 5.92.\n\nThe more revealing number is delivery rate. It spans 0.52 to 0.94 on T2V, 0.59 to 0.96 on I2V and 0.69 to 0.97 on V2V. MiniMax M3 delivers only 0.519 on T2V; Claude Opus 5.5 reaches 0.944.\n\nThat produces the sharpest finding in the paper. Restricted to successfully rendered videos, GPT-6 Astra averages 4.19 in holistic video quality and GPT-6.1 Sol 4.15 — both above Claude Opus 5.5 at 4.04. But once execution failures are folded in, the full-set means invert: Claude Opus 5.5 leads with 3.87, against 3.62 for GPT-6 Astra and 3.61 for GPT-6.1 Sol. Elegant code that never runs is worth nothing in production.\n\n## What failure looks like\n\nThe authors analyze 2,756 error cases, and the failure profiles diverge sharply. For GPT-6 Astra, compilation errors plus missing APIs or packages account for 83.4% of failures; for Qwen3.8-Max, missing APIs and packages alone make up 59.7%. Claude Opus 5.5 runs the other way — runtime exceptions are 75.0% of its failures, and they account for over half for GLM-5.3-Flash, DeepSeek-V4.1-Flash and MiMo-V2.6-Pro.\n\nNot compiling and crashing at runtime are different capability gaps: one tests syntax and dependency handling, the other runtime semantics.\n\nThe complexity curves diverge too. From L1 to L2, under construction and dynamic-animation complexity, SF (structural fidelity) drops by 14.38% and 12.82% of its score range respectively — so moderate difficulty already strains current models. State-dependency complexity falls harder from L2 to L3, declining across all 15 metrics. Long-range state consistency is the real wall.\n\n## Spending more buys less\n\nOn cost the comparison is blunt. GLM-5.3-Flash burns 90.0k output tokens per request against GPT-6 Astra's 13.9k, yet scores 2.94 against 3.62. GPT-6.1 Sol and GPT-6 Astra land on nearly identical quality (3.61 versus 3.62), yet GPT-6 Astra incurs roughly 6.1 times the estimated API cost. Claude Opus 5.5 takes the top quality score of 3.87 at a lower estimated cost than GPT-6 Astra.\n\nThese are the authors' own numbers, with no third-party reproduction so far, so how far to read them stays open.\n\n## So what\n\nThe case for programmatic video generation is not that it replaces diffusion models. It is that it turns video into an engineering artifact — versionable, partially re-rendered, reviewable as a diff. But that premise holds only if the generated code runs.\n\nThe real contribution of this benchmark is separating delivery rate from quality into two columns. That is not the same thing as the pass@1 convention used in code-agent evaluation: pass@1 grants one attempt, while delivery rate tells you how many of those attempts were void. Once models start orchestrating production systems, reliability stops being an add-on and becomes a multiplier.\n","code2video-compass-mllm-code-video-benchmark","2026-10-25T09:15:00Z","2026-10-25T09:15:33.928994Z","2026-10-25T09:15:33.929005Z",true,"agent",245,[39,48],{"slug":40,"tag_slug":40,"title_zh":41,"title_en":42,"intro_zh":43,"intro_en":44,"id":45,"is_active":35,"created_at":46,"modified_at":47},"ai-for-science","AI for Science 2026：从 UniPert 到 GPT-Rosalind 的硬核进化","AI for Science 2026: from UniPert to GPT-Rosalind","生命科学、化学材料、物理世界模型——AI 正在从\"语言工具\"变成\"实验伙伴\"。本专题收录 AI 在三大科学方向的关键节点：UniPert 统一基因与化学扰动空间、GPT-Rosalind 端到端生命科学推理、达摩院 AI 智能体 28 小时找到 4 种超导新材料、Anthropic Claude Science 把工作台做成标准品。","From language tool to lab partner — AI is reshaping life sciences, chemistry\u002Fmaterials, and physical world models. This topic covers the key milestones: UniPert unifying genetic-chemical perturbation spaces, GPT-Rosalind's end-to-end life-sciences reasoning, DAMO's AI agent discovering 4 superconducting materials in 28 hours, and Anthropic's Claude Science workbench going mainstream.","988a4300-5fab-41c4-b5d8-63711a2dc757","2026-09-10T01:34:15.296649Z","2026-09-10T01:34:15.296663Z",{"slug":49,"tag_slug":49,"title_zh":50,"title_en":51,"intro_zh":52,"intro_en":53,"id":54,"is_active":35,"created_at":55,"modified_at":56},"h3-series","MiniMax H3 系列：从开源权重到 35 倍吞吐","MiniMax H3 Series: from open weights to 35x throughput","MiniMax H3 自 2026 年 8 月开源以来节奏密集：官方把生成、参考与编辑收回一个模型；ComfyUI 当天压进 RTX 3060；摩尔线程 3 小时完成国产 GPU 适配；fal 后训练版把吞吐拉到 35 倍；FastH3 蒸馏再砍推理成本。本专题持续追踪 H3 的发布—开源—蒸馏—部署全链路。","Since MiniMax open-sourced H3 in August 2026 the pace has been relentless: one unified omni-modal model, same-day ComfyUI support down to an RTX 3060, a 3-hour Day-0 port to Moore Threads GPUs, fal's post-trained H3 Max at 35x throughput, and FastH3 distillation cutting inference cost further. This topic tracks the full H3 chain — release, open weights, distillation, deployment.","83ef0daa-3c31-4cb3-86ed-e5ee58654d5f","2026-09-08T07:33:19.942193Z","2026-09-08T07:33:19.942209Z",{"items":58},[59,64,69,74,79,84],{"id":60,"title":61,"news_slug":62,"published_at":63},"2f9ad534-b737-490e-b97e-bf01f2b6d26b","交错生成新基准:闭源旗舰文本赢,图像一致性输给开源","intreal-interleaved-generation-benchmark","2026-11-01T05:27:08+00:00",{"id":65,"title":66,"news_slug":67,"published_at":68},"fd0e593c-bb2c-44f9-8913-16f8dcef6999","苹果8B智能体:真实iOS上追平Gemini-3.1-Pro","apple-online-rl-ui-agent-live-ios","2026-10-31T11:10:00+00:00",{"id":70,"title":71,"news_slug":72,"published_at":73},"5480e0ca-7b17-481c-8a24-8498f1a0aecd","8B模型OST涨13.41分:空间推理的接口锁定","chronoreplay-spatial-interface-lock-in","2026-10-28T05:09:43+00:00",{"id":75,"title":76,"news_slug":77,"published_at":78},"8d11d7c2-a373-43e1-9014-11163f4ea4db","5B追平22B:阿里把视角、动作、镜头装进同一个注意力空间","world-in-context-interactive-video-world-model","2026-10-27T19:08:47+00:00",{"id":80,"title":81,"news_slug":82,"published_at":83},"e4ac848d-d24a-465f-bece-37e090da86de","全双工语音模型长对话必崩:KAIST 揪出初始化锚点被挤走","full-duplex-speech-collapse-fd-lhb-kv-cache","2026-10-27T03:20:00+00:00",{"id":85,"title":86,"news_slug":87,"published_at":88},"d34bdd08-5fa9-4ce0-954d-17406b896756","牛津102页综述:视频世界模型的四级证据阶梯","video-world-model-evidence-ladder-survey","2026-10-26T11:13:04+00:00"]