Streaming video generation is moving from producing a clip toward accepting user instructions mid-generation. But once requests keep arriving during a live session, the hard question is no longer whether individual frames look convincing. It is whether a model can execute each change at the right moment while leaving already-generated content intact. The verdict from a new benchmark is blunt: high visual quality does not equal following instructions.
A gap nobody had measured systematically
The paper compares itself against more than a dozen existing video generation and interactive benchmarks, including VBench, VBench++, T2V-CompBench, TC-Bench, NarraStream-Bench, MBench, WorldMark, WBench, WBench-Editing, StreamAV-Bench, GenBench, THEval, Full-Duplex-Bench, VideoFDB and EditVerseBench. None of them covers all five interactive scenarios at once, so the authors position StreamEval as the first unified benchmark for interactive streaming video generation. The leaderboard is live at streameval.ai, the code is Apache 2.0, the benchmark data is CC BY 4.0, and roughly 3.1 GB of assets are distributed through Hugging Face and ModelScope.
300 cases, four dimensions, ten metrics
The benchmark contains 300 scripted test cases, each session lasting between 15 and 300 seconds. Every case specifies the initial assets, the multimodal requests that arrive incrementally, and explicit requirements about state changes and content preservation. Performance is scored along four dimensions: task execution, state consistency, visual fidelity and audiovisual coordination, instantiated through ten task-specific metrics on a 0 to 100 scale. Sessions are split across duration strata with 12 cases each, and every case is run three times per model, so longer sessions do not dominate the final score.
Where 37 models actually stand
In the avatar performance task, Vidu S2 Avatar leads with 99.60 on audiovisual synchronization and 97.25 on stability, for an overall of 81.71. Even so, its Following score reaches only 36.90, while the lowest value in that column belongs to Avatar-Forever 22B at 6.37. The paper highlights the point directly: some systems exceed 80 on identity retention and 96 on stability, and still score poorly on following instructions.
World generation is more divided. On overall score, MemFlow takes 33.36, just ahead of Rolling Forcing at 33.01. But on Following the order changes: Krea Realtime leads with 25.93, so the overall leader is not the instruction-following leader. HappyOyster 1.0 Adventure posts the best quality and stability at 58.34 and 56.67, yet its Following drops to 5.56 and continuity to 2.78. ShotStream reaches 33.80 on quality but falls to 20.55 on stability and 9.34 on continuity.
The weakest area is reference-based editing: among the three models reporting that metric, the highest edit success rate is 15.00 on the same 0 to 100 scale.
The part that is harder than looking better
The listed affiliations include Tsinghua University, NVIDIA, CMU, UC Berkeley, Princeton and the National University of Singapore, and the paper reached third place on the Hugging Face daily list after submission.
The value of this evaluation lies in what it refuses to reward. A model can score near the top on visual metrics while largely ignoring a request such as turning the camera left. The authors also acknowledge the boundary of their own work: automatic judging relies on predefined expected outcomes, and collecting human pairwise judgments to check agreement with the scoring rubric is the natural next step.
So the next time a realtime video model is demoed, it is worth asking one more question: is it playing back recorded footage, or is it actually listening? On the current leaderboard, the first is long since solved, while the second is still struggling to pass.
Source: arXiv:2610.28402, with code and leaderboard at github.com/StreamEval/StreamEval and https://streameval.ai