Video models have gotten good at recognizing what is in a frame, but a benchmark released in September by a University of Hong Kong team pours cold water on the field: cut the head and tail off a video, ask the model to pick the true missing middle from four candidate clips — a "video cloze test" humans pass casually — and 31 mainstream Video-LLMs get exposed (arXiv:2609.01515).

How the test works

TempCloze packs 1,521 videos from seven public sources, led by LVD-2M and EgoLife, favoring long-take and egocentric footage. Each video splits into a beginning (B), a missing middle (M), and an ending (E); the model gets B and E and must identify M among four candidates. The clever part is the distractors, all built from the same source video: Semantic distractors are three non-overlapping intervals, testing what should happen; Alignment distractors shift the true clip earlier, later, or wider, testing when it should happen; Progression distractors reverse, shuffle, or loop the clip, testing how it should unfold. Because prompts and answers are entirely visual, shortcuts from option wording and language priors are sealed off. Default evaluation samples 16 frames per clip, up to 96 per question.

The scoreboard: alignment collapses

Across 10 proprietary and 21 open-source models, the paper calls temporal alignment the main bottleneck: proprietary averages fall from 70.73% on Semantic and 67.72% on Progression to 48.13% on Alignment; open-source models manage just 26.54% against a 25% random line. Humans score 97%.

Seed1.8 (thinking mode) tops the proprietary board at 88.58 mean with 76.92 on Alignment; Qwen3.5-Plus follows at 85.74. The flagship collapses stand out: GPT5.4 averages 56.89 with Alignment at 37.41; Claude4.6-Sonnet posts 28.80 on Alignment, nearly halving its Semantic 55.69; Grok4.1 lands at 24.11 mean, below random. The within-family gap is dramatic: Seed1.6 scores just 17.83 on Alignment, a generation behind Seed1.8.

On the open side, only two Qwen3.5 models clear 50 on Alignment — 51.55 for 35B-A3B and 50.62 for 397B-A17B, the smaller cup edging the larger — while 397B-A17B also leads open models at 68.27 mean. Qwen3VL-32B-Instruct falls to 10.91, under half the random line: understanding content and ordering events fail together in the same family.

Why alignment is hard

Behavioral analysis on four representative models yields four findings: candidate reordering destabilizes choices; models lean on beginning context over ending context; denser or longer visual input dilutes decisive boundary cues; and test-time scaling brings model-dependent gains without closing the alignment gap.

So what

Accepted to EMNLP 2026 Findings, this work deserves a close read from anyone selling "video understanding." Two takeaways: average scores hide structural blind spots — GPT5.4 scores 68.11 on Semantic but 37.41 on Alignment, and Qwen3VL-32B-Instruct's 10.91 gets diluted in composite rankings. And for timing-sensitive deployments such as security, editing, and embodied AI, check alignment-style metrics separately instead of trusting an aggregate leaderboard. Video models still have one semester of temporal reasoning left to pass.