Most AI assistants on a video call still take the long way round: audio goes through speech recognition, frames go through captioning or description, and the stitched-together text is what the model actually reads. OmniVChat, a new paper carrying the Qwen org badge on the Hugging Face papers page, removes that pipeline entirely. The omni model receives the user's audio and video simultaneously and replies in text; the question lives inside the audio and video themselves, with no separate text prompt, no external captioning, and no ASR transcript. The paper's argument: direct audio-visual input reduces external latency and computation while preserving perceptual cues (arXiv: https://arxiv.org/abs/2609.21465).
Easy to define, hard to feed
The bottleneck is not architecture but material. The paper names two constraints. Real recordings are scarce — people rarely hand over footage of themselves talking to their own devices. And evaluation is hard: a good reply often depends on the user's surroundings, facial expressions, and nearby objects, while the same intent can be phrased countless ways, which makes keyword matching unreliable. The team's answer is generation-for-comprehension: synthesize the dialogues, then use them for both training and evaluation. The paper has 18 authors; the README's affiliation footnote lists CUHK, Alibaba Token Hub, Shanghai Jiao Tong University, Shanghai Innovation Institute, and Zhejiang University.
Four agents, 2,800 dialogues
OmniVChat-Studio, the multi-agent data engine, assembles the data: a Director handles text input and output, a Renderer turns accepted prompts into synchronized audio-visual clips, a Reviewer captions results and writes quality reports, and a deterministic Validator checks scripts against configured rules. Per the repo README, the OmniVChat-Bench benchmark holds 2,800 dialogues — 2,550 single-turn and 250 multi-turn — across five ability categories: Dialogue-State & Link Perception (DSLP, 900), Multimodal Entity Alignment (MEA, 900), Model Self-Awareness (MSA, 450), Anti-Hallucination (AH, 400), and Emotion Recognition (ER, 150), spanning 17 subcategories, 22 scenario domains, and 13,475 tiered rubric criteria. English accounts for 63.1% and Chinese 36.9%; the media amounts to 3,490 1080p mp4 files, about 28 GB, where the user's speech is the video's own audio track (GitHub: https://github.com/HarlandZZC/OmniVChat).
One rubric, two consumers
Training-side OmniVChat-RL compresses the reward into one formula — R(y) = r(y) + λ_fmt·f(y) + λ_eff·e(y) + λ_sty·s(y) — where r scores correctness, f format, e efficiency, and s style; README defaults are λ_fmt = 0.5, λ_eff = 0, λ_sty = 0, with reward range [0, 1.5]. A deliberate engineering choice: the benchmark scorer and the training reward share one rubric implementation — eval/score.py imports from reward/reward.py — so the yardstick has exactly one definition. The README also warns that training-reward means and benchmark scores live on different scales and are not directly comparable.
Sim-to-real is the point
The decisive evidence: after training Qwen3-Omni-Instruct with OmniVChat-RL on synthesized dialogues, the model improves on both OmniVChat-Bench and the human-recorded OmniVChat-Bench-Human. Capability built on synthetic data transfers to real-world dialogue training and evaluation — the sim-to-real point the first author highlighted in the HF paper-page comments.
So what
Two takeaways. For one, this benchmark scores whether a model knows it has no body (MSA) and whether replies stay grounded in the available audio-visual evidence (AH) — the actual gate for audio-visual assistants entering production, and a harder bar than eloquence. For another, once real data becomes the bottleneck, synthetic data graduates from training patch to part of the task definition: build the data, define the benchmark, train the model, all in one house. A question to leave you with: when the eval set itself is manufactured, how much discount should you apply to the score?