Consider an awkward setup: video arrives as a live stream, but the question may come a second later. By the time someone asks "what color was that person's jacket?", the frame is long gone. The core problem of streaming video understanding is not seeing clearly — it is deciding what to remember before knowing what will be asked. Most existing approaches compress video history into summaries, which blurs event transitions and cannot recover dropped details afterwards.

Researchers from VCIP at Nankai University, ByteDance, IQuest Research and the Institute of Automation at the Chinese Academy of Sciences have open-sourced DeltaStreamer (arXiv:2610.19849), which adds a deeply human move to this problem: looking back.

Record changes first, then revisit

DeltaStreamer is an agentic framework for streaming video understanding, built from three parts:

  • Explicit visual deltas: consecutive frames are compared, and what changed is retained alongside scene descriptions, making event transitions explicit.
  • Structured textual memory: observations, episode summaries and pinned facts are organized into a hierarchical DAG with entity indexing and pointers to source evidence.
  • Query-driven planning and replay: once a question arrives, the planner identifies what evidence the answer needs and guides retrieval and tool use; when memory falls short, it locates earlier scenes and replays historical frames to read back visual details that textual memory dropped.

In one line: observe the stream, remember what happened, and revisit the visual evidence when memory falls short. The system requires no additional training and runs with a supported local VLM checkpoint.

Scores and engineering details

Paper-reported overall scores: 66.07 on OVO-Bench and 59.53 on StreamingBench, improving over representative streaming video understanding methods — note these are paper-reported numbers, and the README states plainly that the open-source release has not yet completed full GPU inference regression. The engineering stack is refreshingly runnable: vLLM 0.19.1, ms-swift 4.2.0 and transformers 5.5.4; a Qwen3.6-27B backbone handles observation, planning and answering, SAM2.1 Hiera Large does object tracking, and Whisper base handles audio transcription. The repository ships an interactive demo with English and Chinese interfaces: upload a video, pick when to ask, and inspect the agent's observations, memory and plans. The license is CC BY-NC 4.0, so commercial use needs separate terms; training code and experiment artifacts are excluded — this release preserves the inference implementation, prompts and memory formulas.

Why it deserves attention

This work represents a different route from "end-to-end models with built-in memory": instead of betting the model can hold everything in context, it makes "what to remember and when to look back" an explicit policy layer. The cost is extra latency and storage from replay, but for livestream-style settings where a slightly delayed answer is acceptable, evidence-grounded accuracy is a good trade — the team followed this exact path with AI-Tongdan, a livestream companion app that even suggests danmaku comments. For teams working on video monitoring, livestream understanding or multimodal agents, this decomposition of change-aware memory plus targeted replay is worth a close read: when watching AI shifts from passive compression to active investigation, the bottleneck of streaming understanding may not be model size, but memory strategy.

References: arXiv:2610.19849 (https://arxiv.org/abs/2610.19849) · Code: https://github.com/BBBBchan/DeltaStreamer