There is a route to video generation that skips the diffusion model entirely. The model writes a program — HTML/CSS, SVG, Canvas, Three.js — and Remotion renders it into a finished clip. You get editable output, reusable components, and frame-accurate control over timing. What you also get is a new failure mode: the program has to actually run.

A benchmark posted to arXiv on October 20 puts that route under test. Code2Video-Compass, from a team at Fudan University, ran 15 multimodal large language models through 2,104 tasks, and its most interesting result is not a winner. It is that running and rendering well are two different things.

The paper is arXiv:2610.28416; the project page lives at code2video-compass.github.io.

How the benchmark is built

Tasks come from two directions: 385 independent ones, plus 1,719 aligned tasks derived from 573 source videos, covering text-to-video, image-to-video and video-to-video. Categories follow the practice of motion design — 10 categories, 24 subtypes. Every task is annotated along three complexity dimensions: construction (standard primitives, custom geometry, procedural simulation), dynamic animation (staged sequences, coordinated changes across elements, concurrent motion), and state dependency (adjacent stages, intervening stages, long-range dependencies across multiple changes). Each dimension has three levels.

Evaluation is two-layer: 12 objective metrics plus 5 subjective ones. The objective side covers delivery rate, mean output tokens, video quality, text-video alignment and reference fidelity. The subjective side uses Gemini 3.8 Flash as judge. That judge is itself validated — 300 outputs from six models, each scored three times, against three human raters on 200 outputs. Pearson's r reaches 0.796 for instruction following and 0.783 for holistic video quality, with ICC(A,3) of 0.983 and 0.970.

Scoring used a fixed subset of 1,054 tasks (360 T2V, 347 I2V, 347 V2V). Each model gets one API call to produce code, with no execution feedback and no repair attempts — the harshest production-like setup there is.

What the ranking hides

In the authors' own evaluation, Claude Opus 5.5 posts the best average rank in all three settings — 1.60 on T2V, 1.08 on I2V, 1.15 on V2V — with Claude Sonnet 5.5 second each time, and proprietary models taking the top three slots across the board. Among open-weight models, DeepSeek-V4.1-Flash leads T2V with an average rank of 7.35, while MiMo-V2.6-Pro takes I2V at 7.50 and V2V at 5.92.

The more revealing number is delivery rate. It spans 0.52 to 0.94 on T2V, 0.59 to 0.96 on I2V and 0.69 to 0.97 on V2V. MiniMax M3 delivers only 0.519 on T2V; Claude Opus 5.5 reaches 0.944.

That produces the sharpest finding in the paper. Restricted to successfully rendered videos, GPT-6 Astra averages 4.19 in holistic video quality and GPT-6.1 Sol 4.15 — both above Claude Opus 5.5 at 4.04. But once execution failures are folded in, the full-set means invert: Claude Opus 5.5 leads with 3.87, against 3.62 for GPT-6 Astra and 3.61 for GPT-6.1 Sol. Elegant code that never runs is worth nothing in production.

What failure looks like

The authors analyze 2,756 error cases, and the failure profiles diverge sharply. For GPT-6 Astra, compilation errors plus missing APIs or packages account for 83.4% of failures; for Qwen3.8-Max, missing APIs and packages alone make up 59.7%. Claude Opus 5.5 runs the other way — runtime exceptions are 75.0% of its failures, and they account for over half for GLM-5.3-Flash, DeepSeek-V4.1-Flash and MiMo-V2.6-Pro.

Not compiling and crashing at runtime are different capability gaps: one tests syntax and dependency handling, the other runtime semantics.

The complexity curves diverge too. From L1 to L2, under construction and dynamic-animation complexity, SF (structural fidelity) drops by 14.38% and 12.82% of its score range respectively — so moderate difficulty already strains current models. State-dependency complexity falls harder from L2 to L3, declining across all 15 metrics. Long-range state consistency is the real wall.

Spending more buys less

On cost the comparison is blunt. GLM-5.3-Flash burns 90.0k output tokens per request against GPT-6 Astra's 13.9k, yet scores 2.94 against 3.62. GPT-6.1 Sol and GPT-6 Astra land on nearly identical quality (3.61 versus 3.62), yet GPT-6 Astra incurs roughly 6.1 times the estimated API cost. Claude Opus 5.5 takes the top quality score of 3.87 at a lower estimated cost than GPT-6 Astra.

These are the authors' own numbers, with no third-party reproduction so far, so how far to read them stays open.

So what

The case for programmatic video generation is not that it replaces diffusion models. It is that it turns video into an engineering artifact — versionable, partially re-rendered, reviewable as a diff. But that premise holds only if the generated code runs.

The real contribution of this benchmark is separating delivery rate from quality into two columns. That is not the same thing as the pass@1 convention used in code-agent evaluation: pass@1 grants one attempt, while delivery rate tells you how many of those attempts were void. Once models start orchestrating production systems, reliability stops being an add-on and becomes a multiplier.