The next frontier for omni-modal large models is the real-time video assistant: a model that continuously watches the environment and guides the user toward a goal through multi-turn conversation. The awkward part has always been evaluation. Static video-QA datasets cannot simulate this setting, because the model's own responses dynamically change what the user does next — and therefore what the video contains.

OmniAssistBench, submitted to arXiv on Aug 21 (arXiv:2608.21360, paper) by a Nanjing University-led team with Nankai University and the University of Waterloo, offers a way out of that bottleneck.

Reverse-engineering internet videos into interactions

The team reverse-engineers existing internet videos: they deduce logical user goals from the source footage, then segment the videos into multi-turn clips that simulate continuous interaction. To handle diverging interaction paths — the same goal can be achieved through many methods — models receive predefined priors derived from the source video and must guide users along the exact same routes.

The dataset contains 685 open-ended QA pairs across 7 major task types and 16 fine-grained tasks, spanning sports, cooking, lectures, DIY and talk shows. The four-phase annotation pipeline demands roughly 4 expert hours per sample — more than 1000 expert person-hours in total by the paper's account. Scoring uses an LLM-as-a-Judge pipeline on a 0-5 scale, normalized to 0-100.

One design detail stands out: user questions are never typed as text. They are embedded directly into the video — mostly as TTS audio, sometimes as subtitles, handwriting, or picture-in-picture gestures. The model sees only the video itself. Multi-turn simulation also preserves temporal causality: each turn's clip starts where the previous one ended, with the user prompt embedded at the very end.

The leaderboard: 66.4 at the top, no real winners

Eleven models were evaluated. Gemini-3-Pro leads with 66.4/100, Gemini-2.5-Pro follows at 64.6, Doubao-Seed-2.0-lite takes third at 57.3, MiMo-V2-Omni scores 53.8 and Qwen3.5-Omni-Plus 51.6. The best open-source entrant, Qwen3-Omni-Instruct (30B-A3B), reaches only 51.2, while VITA-1.5 sits at the bottom with 24.6.

The team also filmed 3 real-world cases averaging 15 interaction turns (meeting simulation, blind-assisted navigation, and multi-person handicraft tracking). Here Gemini-2.5-Pro, second on the overall board, collapses to 44.8 — below Qwen3-Omni-Instruct at 53.8 — while Gemini-3-Pro holds first at 68.0. Offline leaderboard scores and genuine long-horizon interaction clearly diverge.

Three bottlenecks for the next generation

The paper identifies three recurring failure modes:

  • Visual prompts: models struggle to recognize hand gestures as user instructions; gesture-based prompt following broadly fails;
  • Context exhaustion: video and audio devour tokens so fast that, without long-term memory mechanisms, models exhaust the context window within minutes and then forget the user's goal;
  • Delayed response: models cannot tell when staying silent is the right move, frequently emitting loosely related video captions instead of waiting for the target event.

By the team's rubric, the 40-60 band means models "generally understand what to do but struggle to provide accurate and comprehensive answers" — which is where most of the leaderboard sits.

So what

OmniAssistBench draws a clear line between video understanding and video collaboration. The former is passive clip-watching QA; the latter demands active alignment of visual state, user goals and prior knowledge, plus a sense of timing. The Real Cases divergence is the warning worth heeding: the gap between offline benchmark scores and real long-horizon interaction is wider than aggregate numbers suggest. If your product bets on an AI video assistant, running it on this bench before shipping beats watching launch-day demos.