[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-semcomp-bench-video-task-completion":3,"news-related-b95b93e8-294a-4c5b-b53d-ce6ea07c1519":38},{"id":4,"title":5,"summary":6,"content":7,"original_url":8,"source_id":9,"tags":10,"translations":24,"news_slug":31,"published_at":32,"created_at":33,"modified_at":34,"is_published":35,"publish_type":36,"image_url":14,"view_count":37},"b95b93e8-294a-4c5b-b53d-ce6ea07c1519","SemComp-Bench 登顶 Hugging Face 日榜:视频生成开始考「任务做没做成」","arXiv 新基准 SemComp-Bench 今日登顶 Hugging Face 日榜(151 票):它不再问视频像不像参考图,而是让 VLM 出二值问答题,考视频生成模型能否真正「完成任务」,并给出 OA\u002FGR 双分数。实验显示主流模型在任务完成度上依然吃力。","刷到视频生成模型的 demo,画面一个比一个惊艳。但换一个问法:给它一张参考图、一句指令,它到底有没有把任务**做成**?过去的主流基准答不上来,而刚刚登顶 Hugging Face Daily Papers 日榜(151 票、8 月 20 日当日第一)的 SemComp-Bench,专门就是冲着这个问题去的([arXiv:2608.17426](https:\u002F\u002Farxiv.org\u002Fabs\u002F2608.17426))。\n\n## 换一种考法:只看结果,不看过场\n\n论文提出了一个新任务定义:Semantic Task Completion(语义任务完成)。它对「成功」的判定有两条——既要**达成预期结果**(outcome achievement),也要保持**语义接地**(semantic grounding):即生成结果与参考图之间,在任务相关的高层语义上对得上号。\n\n反直觉的地方在于,这套评价**明确不要求**两件事:不要求展示完整的中间步骤序列,也不要求和参考图保持传统意义上的外观一致性。用论文提交者的话说:「视频生成器能不能真正把任务做完——而不只是生成一段看起来像模像样的视频?」\n\n## 数据集怎么造:九步流水线,产物只考「结果帧」\n\n配套的 SemComp-Data 覆盖六个领域,每条实例由四件套组成:参考图 + 详细指令 + 简短指令 + **以结果为中心的视频片段**。数据来自真实视频,官方 GitHub 仓库(Kelly372\u002FSemComp-Bench)公开了完整构建管线:标题过滤、任务分类、参考\u002F结果时间戳定位、画面质量校验、双语言指令生成、片段抽取、指令归一化、对齐类型标注、结果状态描述——九个阶段逐步把原始视频洗成标准化实例。片段抽取支持两种时长策略:3–4 秒短片与 3–10 秒长片。\n\n需要留意许可证:管线原始代码是 Apache 2.0,但其中改编自 Panda-70M 的切分组件限于非商业研究用途,ImageBind 部分遵循 CC BY-NC-SA 4.0——仓库整体不能当作商业可用。\n\n## OA 与 GR:两个分数各管一件事\n\n打分环节交给一个视觉语言模型(VLM):它对生成视频回答**结构化的二值问题**,最后输出两个分数——**OA Score** 衡量结果达成度,**GR Score** 衡量生成可靠性。相比「人眼评比」或端到端打单一分,二值问答让每个失分点都能落到一道具体的题上,可解释性明显更好。\n\n在代表性视频生成模型上的实验结论很直白:既要完成任务、又要保住与参考图的任务相关语义接地,对现在的模型依然**很困难**。换句话说,生成「好看的画面」和生成「做成了事的画面」之间,还隔着一段不小的距离。\n\n## 为什么值得盯\n\n视频生成正在从「能生成」走向「能用」。一旦下游场景是教学演示、操作指引、产品展示,用户在意的就不是纹理和美学,而是「最后那一下到底做没做成」。SemComp-Bench 把评价重心从过程表现力挪到结果达成度,等于给全行业换了一张考卷——接下来各家的 leaderboard 军备竞赛,大概要换赛道重跑了。\n\n一句话:当基准开始考「任务完成」,视频生成模型的下一轮迭代目标,也就被悄悄改写了。","https:\u002F\u002Farxiv.org\u002Fabs\u002F2608.17426","7437aeb9-930c-4866-a2e9-48003c1a792b",[11,15,18,21],{"id":12,"name":13,"slug":13,"description":14,"color":14},"120fa59a-ff6f-4537-9bf5-f818df636a0e","benchmark",null,{"id":16,"name":17,"slug":17,"description":14,"color":14},"499f4b56-819d-49a3-9609-33e775143b86","multimodal",{"id":19,"name":20,"slug":20,"description":14,"color":14},"b9bd9039-fcdb-41a8-b85b-fc1587def2b9","open-source",{"id":22,"name":23,"slug":23,"description":14,"color":14},"ebe5dcd1-46b1-4298-b8c2-8e0e2f456e56","video-generation",[25],{"id":26,"lang":27,"title":28,"summary":29,"content":30},"39f3bd21-e4c6-4373-9d1b-6e227db4d4b8","en","SemComp-Bench Tops Hugging Face Daily Papers: Video Generation Now Gets Graded on Task Completion","SemComp-Bench, a new arXiv benchmark (2608.17426), tops Hugging Face Daily Papers with 151 upvotes. Instead of asking whether generated videos look like the reference, it uses a VLM asking structured binary questions and reports OA\u002FGR scores for outcome achievement and generation reliability. Experiments show leading models still struggle.","Scroll through demos of video generation models and the visuals keep getting more stunning. But ask a different question: given a reference image and an instruction, did the model actually **complete the task**? Most existing benchmarks cannot answer that. SemComp-Bench, which just topped Hugging Face Daily Papers on Aug 20 with 151 upvotes, is built precisely for this question ([arXiv:2608.17426](https:\u002F\u002Farxiv.org\u002Fabs\u002F2608.17426)).\n\n## A New Test: Judge the Outcome, Not the Performance\n\nThe paper introduces Semantic Task Completion, an outcome-oriented video generation task. Success requires two things: **achievement of the intended outcome**, plus **semantic grounding** — the generated outcome must correspond to the reference image at the level of high-level semantics relevant to the task.\n\nCounter-intuitively, the evaluation explicitly requires **neither** a complete sequence of intermediate steps nor conventional appearance consistency with the reference. As the paper submitter put it: \"Can a video generator actually finish the task — not merely make a convincing video?\"\n\n## Building the Dataset: A Nine-Stage Pipeline Centered on Outcomes\n\nThe accompanying SemComp-Data covers six domains. Each instance is a four-piece set: a reference image, a detailed instruction, a brief instruction, and an **outcome-centric video clip**. Data comes from real videos, and the official GitHub repo (Kelly372\u002FSemComp-Bench) open-sources the full construction pipeline: title filtering, task classification, reference\u002Foutcome timestamp grounding, visual consistency checks, detailed bilingual instruction generation, clip extraction, instruction normalization, alignment-type annotation, and result-state description — nine stages that turn raw videos into standardized instances. Clip extraction supports two duration policies: 3–4 second short clips and 3–10 second long clips.\n\nOne licensing caveat: the original pipeline code is Apache 2.0, but splitting components adapted from Panda-70M are restricted to non-commercial research use, and the ImageBind portion follows CC BY-NC-SA 4.0 — the repository as a whole must not be described as commercially permissive.\n\n## OA and GR: Two Scores, Two Jobs\n\nScoring is delegated to a vision-language model that answers **structured binary questions** about the generated video, producing two scores: the **OA Score** for Outcome Achievement and the **GR Score** for Generation Reliability. Compared with human preference voting or a single end-to-end score, binary questions tie every lost point to one concrete item — far more interpretable.\n\nThe experimental conclusion on representative video generation models is blunt: achieving intended outcomes while maintaining task-relevant semantic grounding in reference images **remains challenging**. In other words, there is still a real gap between generating \"beautiful frames\" and generating \"frames where the job got done\".\n\n## Why It Matters\n\nVideo generation is moving from \"can generate\" to \"is usable\". Once the downstream scenario is a tutorial, an operation guide, or a product demo, users care not about texture and aesthetics but about whether the final step actually happened. By shifting the evaluation center of gravity from process expressiveness to outcome achievement, SemComp-Bench effectively hands the industry a new exam paper — and the next round of leaderboard races may have to be re-run on a different track.\n\nIn one sentence: when the benchmark starts grading task completion, the optimization target of video generation models has quietly been rewritten.","semcomp-bench-video-task-completion","2026-08-20T13:30:00Z","2026-08-20T15:12:14.065433Z","2026-08-20T15:12:14.065441Z",true,"agent",79,{"items":39},[40,45,50,55,60,65],{"id":41,"title":42,"news_slug":43,"published_at":44},"9fa15063-58b1-4b87-a0cb-0e19ffc4dc6c","4B 参数横扫四大具身基准：开悟世界模型让小模型重新定义 SOTA","kaiwu-4b-world-model-sensetime-72x","2026-06-12T04:00:00+00:00",{"id":46,"title":47,"news_slug":48,"published_at":49},"4c7f5330-3aff-458a-9ef5-f04cc5585703","微信视觉团队开源 WeMM 嵌入模型:2B 反超 8B 前基线,9B 达 MMEB-v2 80.6","wemm-embedding-wechat-multimodal","2026-08-26T21:07:30+00:00",{"id":51,"title":52,"news_slug":53,"published_at":54},"ff0bc92a-295a-4707-be8d-76115fe9eeee","PerceptionBench 出炉:16 个前沿多模态模型,视觉感知无一及格","moonshot-perceptionbench-atomic-perception","2026-08-26T13:15:00+00:00",{"id":56,"title":57,"news_slug":58,"published_at":59},"b4214f43-353e-42e3-b48e-92dd4fc64290","京东开源 EchoWM 全模态世界模型:720p 音画同步,能跟着你走","jd-echowm-omnimodal-world-model","2026-08-25T23:10:00+00:00",{"id":61,"title":62,"news_slug":63,"published_at":64},"2874a2e5-beae-4627-8f6f-a34cf2cc8d7a","一段随手拍视频直出4D人体:4DAnyone用RCP+TCR破解多视角一致性,代码权重全开源","4danyone-monocular-video-4d-human","2026-08-20T17:59:53+00:00",{"id":66,"title":67,"news_slug":68,"published_at":69},"aad00b18-d354-48b5-ad21-62b53150b8c6","MiniMax H3 开源实测:你下载的权重,和 API 里跑的不是同一个模型","minimax-h3-local-vs-api-gap","2026-08-15T17:07:24+00:00"]