arXiv 2605.27141 introduces VitaBench 2.0, a benchmark for evaluating LLM Agents on "long-term user modeling" — the ability to track, learn, and adapt to a specific user's preferences, habits, and context over weeks or months of interaction.
The benchmark structure: VitaBench 2.0 simulates 100 "users" with distinct personas, habits, and long-term goals. The Agent interacts with each user over 30 simulated days, and is evaluated on three tasks: (1) personalization — does the Agent adapt to the user's style? (2) memory — does the Agent remember key facts from earlier interactions? (3) anticipation — does the Agent proactively offer relevant help?
The result: even the leading model (Claude Opus 4.7 with full agent harness) scores only 41.2%. GPT-5.6 scores 38.7%. The gap to "useful" (70%) is huge. The most common failure: the Agent "forgets" user-specific context after 2-3 turns of conversation.
The diagnostic: the authors identify three failure modes — (1) context window overflow (the user history is too long); (2) memory retrieval noise (the wrong memory is recalled); (3) persona drift (the Agent gradually "forgets" the user's specific style and reverts to default).
The bigger takeaway: "long-term user modeling" is the next big Agent capability gap. Current Agents are good at "in-conversation" tasks, but they're bad at "across-conversation" personalization. This is the difference between a "useful chatbot" and a "personal assistant" — and the gap is huge.
For the industry, the takeaway is that "memory" and "personalization" are becoming first-class Agent capabilities, not afterthoughts. The next generation of Agent frameworks will need to invest heavily in long-term memory, persona tracking, and proactive anticipation.