Pushing 512 documents into the context window takes 19.9 seconds on a single H100. Handing the model 16 memory tokens instead takes 0.74 seconds. And on the fact-verification benchmark FactKG, the compressed version does not merely hold its ground: it climbs from 80.43% with full context to 84.33%. The method is called Memorilla. The paper is public on arXiv (paper: https://arxiv.org/abs/2610.21375), and both the code and the weights are released under Apache 2.0.

Not compression, a learned soft memory

The mainstream approach is retrieval: pick a few passages and paste them into the prompt. Memorilla inverts that. A frozen encoder encodes each document once and stores it in an embedding store. At query time, a set of learned memory slots reads both the question and the entire collection, then emits query-dependent memory tokens directly into the input embedding space of the language model.

Two properties matter. The LLM weights stay frozen throughout, and only the memory module is trained. And the token budget handed to the model is fixed, no matter how large the collection grows. Put differently, input-side cost decouples from document count: however many documents are stored, the model still sees 16 tokens. The team describes the design as treating memory as a modality. One of the authors frames it as: if RAG is episodic memory, Memorilla is semantic memory.

The numbers: where it wins, where it loses

Across eight benchmarks, the paper reports a highest mean score of 49.68 and leads on five of them:

  • PersonaMem-v2 reaches 44.60%, against 30.90% for the strongest baseline xRAG; full context manages only 27.92%, and it does so while using 1974 times more document tokens.
  • FactKG reaches 84.33%, versus 80.43% for full context, using 108 times fewer tokens.
  • TriviaQA exact match reaches 57.00% with 743 times fewer document tokens.
  • LaMP-7 scores 48.37 ROUGE-L and LaMP-4 scores 18.15.

There are losses too, and they deserve to be stated plainly: PubMedQA comes in at 59.00%, below LLMLingua-2 at 69.00%; PersonalizationV4 lands at 66.65%, slightly under MemoRAG at 67.88%; NarrativeQA reaches 19.31%, slightly under MemoRAG at 20.51. The team's own reading is that learned memory wins on questions that require fitting a whole body of material together, while retrieval wins when the answer is a name that can be found verbatim, and that on short sources simply reading the text directly is still better.

On latency and compression, with one H100-80GB at batch size 1 running Qwen3-8B, 512 documents (about 128K tokens) leave Memorilla steady at 0.74 seconds, while full context climbs from 0.75 seconds all the way to 19.9 seconds. Books and movie scripts from NarrativeQA are compressed by more than 4,000x, and the author gives a personal ceiling of 4,212x fewer tokens on social media. Averaged across the eight benchmarks, the method uses 394 times fewer document tokens.

The agent side: memory that learns itself

The same memory module was hooked into the TextWorld text-adventure environment. A frozen Qwen3-4B agent has to complete a quest across up to 65 rooms within 50 turns, its history compressed into 8 memory tokens, and only the memory module is trained, using GRPO reinforcement learning on the reward the game provides.

The results: average success rises from 18.6% to 24.2%, the share of games finished within three attempts rises from 25.0% to 37.5%, and the share of episodes ending before the turn limit rises from 29% to 42%. The paper makes a point of one control condition: untrained random memory falls below the baseline. The gain comes from learning, not from having a few more tokens in the prompt.

It fills the gap retrieval leaves

More interesting is that the two approaches do not overlap. On FactKG, RAG-50 retrieval reaches 78.4% and Memorilla reaches 84.3%, while an oracle union of the two reaches 94.3%: 15.9% of questions are solved by Memorilla alone, and 10.0% only by retrieval. The paper stresses that the 94.3% is an oracle upper bound rather than a measured hybrid, but it points the way. Memory handles reading the relationships between materials; retrieval handles finding a name exactly as written. One Hacker News reader summarized the mechanism as injecting something more like a vibe than quotable text, which lines up neatly with a method that is strong on concepts and weaker on exact-text retrieval.

Where the boundary sits now

Three limitations are written on the project page and are worth repeating. First, it loses fine-grained details such as exact names. Second, because memory is written directly into the input embeddings, the approach requires open model weights, and a memory module has to be trained for each LLM. Third, every experiment pairs a Qwen encoder with a Qwen decoder on text alone, so transfer to other model families and to other modalities is untested, and TextWorld covers a single game suite with episodes of at most 50 turns.

Training is not cheap either. The three stages take about 19 hours on 4 H100 80GB GPUs, downloading roughly 310 GiB of data and embeddings, and evaluating a checkpoint on all eight benchmarks takes about 15 minutes on one GPU. At the time of writing the repository had 339 stars and 26 forks, and the weights are published on Hugging Face under Apache 2.0.

So what

The most interesting part of Memorilla may not be those 16 tokens but what it does to the framing of memory. It moves memory from retrieval engineering to training engineering: what you want is no longer better recall, but a set of parameters that learn how to remember. The price is equally direct. You need open weights, you have to pay for training a module per model, and so far everything has been validated on text alone.

For teams working on long context and agent memory, the part most worth copying may not be the whole system but the fixed token budget: letting the model see the same small budget whether the collection holds 512 documents or half a million. The real question that follows is simpler. When memory becomes trainable, who trains it, and for whom.