Vision-Language Models Still Fail Visual Memory: MEMLENS Exposes the Long-Context Blind Spot

Model labs keep pushing context windows toward hundreds of thousands or millions of tokens, while Agent products layer on independent Memory modules. Both routes look viable, until a question smuggles in an image from a previous session and the system quietly falls apart.

MEMLENS, a newly public benchmark, tests exactly that. It is not another text-only long-context leaderboard. It folds multimodal evidence, multi-session continuity, temporal updates, and refusal ability into one suite, and forces models to answer questions whose evidence hides inside an image from an earlier turn.

What the benchmark actually measures

MEMLENS includes 789 questions covering five memory abilities: information extraction, multi-session reasoning, temporal reasoning, knowledge update, and the ability to refuse when evidence is missing. Context length is split into four buckets (32K, 100K, 128K, and 256K tokens), and a cross-modal token-counting scheme is used so difficulty is not silently biased toward the text side.

The team evaluated 27 vision-language models and 7 memory-augmented Agents, then ran a critical image-ablation study. On the 80.4% of questions whose evidence includes an image, removing those images drops two frontier LVLMs below 2% accuracy. The image is not decoration; the answer lives inside it.

Two routes, two different weak spots

LVLMs that just stuff the whole history into context often look stronger in the short context. The model sees the original image, and visual grounding is clean. As the conversation grows, performance falls. The reason is unsurprising: attention now has to hunt for evidence across a much wider pile of text and image fragments, and “having seen it” no longer means “still being able to find it.”

Memory-augmented Agents stay flatter as length grows. They do not collapse the way long-context models do. The cost shows up earlier, at write time. When the system summarizes, compresses, or stores structured memory, it keeps the outline of what happened but can lose the visual details, spatial cues, and local evidence that a later answer depends on. At retrieval time the Agent finds a relevant memory, but the image that would actually back the conclusion is already gone.

Multi-session reasoning makes the gap harsher. Most systems cap out below 30% on that slice. They may remember a snippet, but stitching information from different sessions and different times into a clean causal chain is a different problem. Memory is not “stored once, done forever.” It has to update, handle conflicts, and know when to say it does not have the evidence.

One path is no longer enough

The implication MEMLENS surfaces is direct: future multimodal Agents will need to combine long-context attention with structured multimodal retrieval. Long context holds the raw evidence, the memory layer organizes it across sessions, and the two need a shared evidence chain that can point back to the original image, the original audio, the original timestamp.

That, in turn, explains why a bigger window does not equal “the model has memory.” Window size answers whether the model can hold something; memory answers what should be kept, how to update it, and how to retrieve it at the right moment. The first is a capacity problem; the second is a data and decision problem.

My read: the real competition in the Memory layer is not about who writes the prettiest summary. It is about who can preserve evidence traceability. A system that says “the user has seen an image before” but cannot point to which image, when it appeared, and whether newer information has overwritten it, is closer to a talkative cache than to a reliable long-term memory.

As Agents move into companionship, customer support, and robotics, the cost of a wrong memory will outweigh the cost of one wrong answer. The next round of the model race may be worth tracking not by who has the longest context, but by who can remember more while hallucinating less.