Scroll through demos of video generation models and the visuals keep getting more stunning. But ask a different question: given a reference image and an instruction, did the model actually complete the task? Most existing benchmarks cannot answer that. SemComp-Bench, which just topped Hugging Face Daily Papers on Aug 20 with 151 upvotes, is built precisely for this question (arXiv:2608.17426).

A New Test: Judge the Outcome, Not the Performance

The paper introduces Semantic Task Completion, an outcome-oriented video generation task. Success requires two things: achievement of the intended outcome, plus semantic grounding — the generated outcome must correspond to the reference image at the level of high-level semantics relevant to the task.

Counter-intuitively, the evaluation explicitly requires neither a complete sequence of intermediate steps nor conventional appearance consistency with the reference. As the paper submitter put it: "Can a video generator actually finish the task — not merely make a convincing video?"

Building the Dataset: A Nine-Stage Pipeline Centered on Outcomes

The accompanying SemComp-Data covers six domains. Each instance is a four-piece set: a reference image, a detailed instruction, a brief instruction, and an outcome-centric video clip. Data comes from real videos, and the official GitHub repo (Kelly372/SemComp-Bench) open-sources the full construction pipeline: title filtering, task classification, reference/outcome timestamp grounding, visual consistency checks, detailed bilingual instruction generation, clip extraction, instruction normalization, alignment-type annotation, and result-state description — nine stages that turn raw videos into standardized instances. Clip extraction supports two duration policies: 3–4 second short clips and 3–10 second long clips.

One licensing caveat: the original pipeline code is Apache 2.0, but splitting components adapted from Panda-70M are restricted to non-commercial research use, and the ImageBind portion follows CC BY-NC-SA 4.0 — the repository as a whole must not be described as commercially permissive.

OA and GR: Two Scores, Two Jobs

Scoring is delegated to a vision-language model that answers structured binary questions about the generated video, producing two scores: the OA Score for Outcome Achievement and the GR Score for Generation Reliability. Compared with human preference voting or a single end-to-end score, binary questions tie every lost point to one concrete item — far more interpretable.

The experimental conclusion on representative video generation models is blunt: achieving intended outcomes while maintaining task-relevant semantic grounding in reference images remains challenging. In other words, there is still a real gap between generating "beautiful frames" and generating "frames where the job got done".

Why It Matters

Video generation is moving from "can generate" to "is usable". Once the downstream scenario is a tutorial, an operation guide, or a product demo, users care not about texture and aesthetics but about whether the final step actually happened. By shifting the evaluation center of gravity from process expressiveness to outcome achievement, SemComp-Bench effectively hands the industry a new exam paper — and the next round of leaderboard races may have to be re-run on a different track.

In one sentence: when the benchmark starts grading task completion, the optimization target of video generation models has quietly been rewritten.