arXiv 2606.19338 introduces RNG-Bench (Repeated Non-markovian Game Benchmark), a new multimodal benchmark from Shanghai AI Lab that evaluates LLMs and VLMs on "non-Markov games" — interactive scenarios where the optimal action depends on the full history of past interactions, not just the current state.
The benchmark design: 12 multi-round games (negotiation, cooperation, deception, etc.) where the Agent must maintain a mental model of the opponent's beliefs, intentions, and past actions. The games span visual (avatar gestures), text (chat negotiation), and multimodal (text + facial expression) modalities.
The result: GPT-5.4 scores 41.7% on RNG-Bench, Gemini 3.1 Pro scores 38.2%, Claude Opus 4.7 scores 36.5%. The numbers are surprisingly low — well below the 70%+ scores on standard multimodal benchmarks. The gap reveals a significant weakness: current LLMs/VLMs are "short-memory" — they can handle the current state well, but they struggle with "what happened 5 rounds ago."
The analysis: the authors identify two failure modes — (1) the model "forgets" opponent actions after 2-3 rounds; (2) the model fails to update its belief about the opponent when new information arrives. Both are "memory" and "belief-tracking" issues, not perception issues.
The bigger takeaway: RNG-Bench exposes the "in-context memory" gap. Current LLMs have a context window of 1M+ tokens, but they don't have a structured "memory" of past interactions — they just attend over the entire context. RNG-Bench shows that structured memory (episodic, semantic, belief-state) is the next big capability needed for truly interactive Agents.