How well do multimodal LLMs actually understand video? OmniCompass, a benchmark submitted to arXiv on October 13 by the VCLab team at Hong Kong Polytechnic University (arXiv:2610.18829), offers a systematic checkup: 820 videos, 2,460 manually verified instances, and 50 multimodal models evaluated side by side. The verdict is sobering — open-source models still trail proprietary ones, and the audio-visual gap is especially wide. The paper also sits at the top of the October 16 Hugging Face Daily Papers list with 301 upvotes.

Not whether it saw, but when, where, and what

Existing video benchmarks cover only part of the evaluation space, which makes systematic diagnosis hard. OmniCompass splits evaluation into two tracks: OmniCompass-VL tests vision-language models on visual evidence, while OmniCompass-AV tests omni-modal models on joint audio-visual evidence. Each track runs three complementary tasks: temporal grounding (when a target appears), spatial grounding (where it is), and question answering (what happens). QA spans five perception plus three reasoning categories on the VL track, and three perception plus two reasoning categories on AV. The VL track holds 820 videos and 2,460 instances; a narrative- and audio-rich subset forms the AV track with 240 videos and 720 instances. Videos fall into short (under 90 seconds), medium (90 to 300), and long (over 300) buckets, spanning seven domains and 80.6 hours of content. The team reports roughly 1,800 human annotation hours on the project homepage, with every instance passing multi-round quality verification.

One target, two ways to ask

The sharpest design choice here is paired specification: the same target in the same video is referenced twice — once with text only, once with text plus a reference image. A Cross-Specification Consistency Score (CS) measures whether paired predictions are both correct and mutually consistent. The results expose a problem that average scores hide: models frequently give inconsistent predictions across the two phrasings — asked in words or shown an image, they do not always recognize the same thing.

The scoreboard: three weak spots

Evaluations of 50 MLLMs show three signals. First, open-source models lag proprietary counterparts, with the audio-visual gap especially wide. Second, precise temporal and spatial grounding remains challenging even for leading models. Third, inconsistencies across paired specifications are pervasive. In short, the industry still has a way to go from seeing to pointing precisely.

Open data, ready to use

The dataset is live on Hugging Face (VCLab-PolyU/OmniCompass), the project homepage omnicompass.netlify.app publishes the full 50-model leaderboard with consistency scores, and the paper is released under CC BY 4.0. For multimodal developers, this is a ready-made diagnostic: instead of chasing averages on general leaderboards, use the when/where/what triad to locate real weaknesses. As video understanding becomes the foundation for agents that operate video interfaces and embodied systems that read environments, pointing accurately may become the dividing line sooner than answering correctly.

Source: https://arxiv.org/abs/2610.18829