Giving agents "long-term memory" has become a default move: systems like MemOS, A-Mem, and HippoRAG2 keep arriving, all selling "agents that remember you." But a new paper led by UIUC with collaborators from Tsinghua and Renmin University (on arXiv since Oct 25, already at 99 upvotes on the Hugging Face Daily Papers board) put 11 open memory systems on the same stage as the plainest possible RAG — and in most cases memory lost, expensively and fragilely. The paper names that net loss: memory tax. Full paper: arXiv:2610.34073.
How the experiment was run
Three benchmarks: LongMemEval-s (long-conversation memory QA), PersonaMem-v2 (user persona inference, 20% sample), MINT-Eval (multi-target interference, 10% sample); three backbones: Qwen3.5-4B, Qwen3.6-27B, and GPT-5.6-luna. The 11 memory systems fall into four families by how they organize storage: flat stores (A-Mem, MemOS, SimpleMem), hierarchical/lifecycle (MemoryOS, LightMem, OpenViking), graph-based (HippoRAG2, MAGMA, PlugMem), and trained policies (Mem-T, Memory-R2). The reference is naive RAG — chunked retrieval over raw history, no memory processing — in one-round and two-round variants. Retrieval budgets were aligned via Retrieval Scope, and Qwen3.6-27B served as the judge.
The scoreboard: losing across the board, at three orders of magnitude the cost
On LongMemEval-s, all 11 systems fall below two-round RAG on all three backbones: on the 27B backbone two-round RAG scores 81.4, the closest is A-Mem at 79.4, MemOS at 76.6; LightMem trails by 14.6 points on 4B, and OpenViking drops a full 40.4. On the other side, Memory-R2 beats RAG by 2.0 on PersonaMem-v2 and PlugMem by 3.3 on MINT-Eval — not a wipeout, and that is exactly what the paper dissects.
The cost side stings more: in the appendix cost table, A-Mem's offline build takes 117 hours on LongMemEval-s versus 0.11 hours for RAG — three orders of magnitude — with 98.4K LLM calls against 200 for two-round RAG. Pay a thousand times the build cost, get a lower score: that is the paper's negative net value.
Three sources of memory tax
The paper splits net value into memory benefit (what organization and distillation get right) and memory tax, then traces the tax to three sources. Faithfulness: tracing atomic facts from the raw corpus into memory units, OpenViking has no memory unit at all for 38% of facts, plus 13% major loss; the contrast, MAGMA, keeps 93.6% lossless — loss is a design choice, not fate. Accessibility: evidence that survives can still be unreachable — fragmented, diluted, or rephrased, memory systems recover fewer queries than RAG in the first retrieval round; the evidence is buried deeper. Robustness: with memory fixed and the QA backbone swapped, many memory systems wobble more than raw RAG — model-dependent intermediate operations weld fragility into the store.
The benchmarks share the blame
The paper also bills part of the account to evaluation sets: across a batch of long-term memory benchmarks, even vector-free IDF keyword matching scores above 50%, and hybrid retrieval reaches about 70% — the tasks are retrieval-friendly, the ceiling is low, and the tax becomes visible. The authors further generalize the benefit/tax lens to experiential memory (distilling workflows does beat RAG on web tasks, while several memory methods trail on tool use) and world-model memory (naively retrieving and re-injecting past observations already yields strong consistency): memory tax is not a conversational-memory quirk.
So what
The paper's real contribution is not "memory is useless" but an accounting frame: when you see any memory system's scorecard, first ask "what does naive RAG score, and how many build hours does it take?" Code and full results are open-sourced. For practitioners: if your workload is mostly factual recall, start with two-round RAG — it will likely suffice; save memory systems for tasks that truly need abstraction and preference distillation. Memory worth keeping is memory whose tax you can count.