For most of the past two years, memory in large language models has lived outside the model. RAG pushes history into a vector store, agent frameworks splice retrieved chunks back into the context every turn, and long-term memory systems periodically append historical fragments to the prompt. All of them share one premise: the model itself forgets, so memory has to be continuously fed in through an external pipeline. That premise carries a cost, because longer history means either a growing context window or repeated retrieval and repeated billing.

MemT-1.0 takes a different route. Submitted to arXiv on October 19, it argues that memory should not be an add-on feature but an intrinsic part of model computation, and the paper title states that goal directly: "Towards Memory-Native Evolution."

The Memory Block Sits Beside Attention

The architecture is not mysterious, but the placement is deliberate. MemT-1.0 runs a "Native Memory Block" in parallel with the backbone model's attention, and that block splits into two parts. The Local Memory Block maintains a dynamic, multi-channel memory matrix that evolves across interaction steps. The Hyper Memory Block holds trainable memory query, key, and value projections, along with erase, write, and forget gates plus an output gate.

At inference time the data flow is straightforward: memory attention reads the current memory state, normalizes and gates the readout, and adds it to the backbone attention output. At the end of each turn or segment, GDN-2 performs token-level memory updates for subsequent steps. The model never has to go back and reread the original context, because history has already been compressed into a continuously evolving state.

Building on Metis, MemT-1.0 uses Gated DeltaNet-2 for that token-level update and introduces Mixture-of-Storage (MoS) to scale native memory capacity.

Two Checkpoints: Nano on a 35B Backbone, Flash on DeepSeek

The official repository ships a model family with a wide gap between the two specifications:

  • MemT-1.0-Nano: built on Qwen3.6-35B-A3B, 36B total parameters, 3B activated, 451M trainable memory parameters, and a 10M dynamic memory state.
  • MemT-1.0-Flash: built on DeepSeek-V4-Flash, 288B total parameters, 17B activated, 3.61B trainable memory parameters, and a 336M dynamic memory state.

The comparison between memory state and total parameters is the part worth watching. Flash carries a 336M memory state against a 288B model body, which is negligible in relative terms. That maps directly onto the headline number in the paper abstract: at a history length of 128K tokens, MemT-1.0-Nano achieves approximately 1.86x end-to-end inference speedup and a 66x reduction in persistent storage relative to full-context KV cache. The repository phrases it as the persistent state remaining constant in size, and the mechanism is self-consistent: because the state does not grow linearly with history length, both the speedup and the storage reduction follow from that constancy.

Training runs in two stages, continued pre-training followed by post-training, supported by 34B tokens of data. For deployment, the Nano version serves through SGLang and the official example runs on two GPUs, while Flash has its own launch script.

The Honest Caveat: Not a Sweeping Win

Neither the abstract nor the repository dodges this point. The README's evaluation summary is notably restrained. Across 11 memory benchmarks, MemT-1.0-Flash and MemT-1.0-Nano rank first and second respectively among the evaluated parametric-memory methods. But strong textual-memory systems remain ahead overall, particularly on long histories that require detailed evidence retrieval.

That sentence carries more information than the ranking itself. "Parametric-memory methods" is a bounded comparison set: genuine RAG and long-context solutions were not pulled onto the same leaderboard, and it is precisely the latter that holds the advantage on detail retrieval over very long histories. So the accurate reading is that MemT-1.0 did best on the route of writing memory into weights and state. It has not won the wider memory landscape.

What This Route Actually Bets On

Memory internalization sounds like a pure engineering gain, but what it really wagers is a structural claim about what a model is. A persistent state that stays a fixed-size constant means history must be compressed into a container that does not inflate over time, which runs against the mainstream assumption that longer context is better context. The 1.86x speedup and 66x storage reduction are the dividends of that assumption holding.

The cost sits in the same document, because strong textual-memory systems still lead on detail retrieval over very long histories. In other words, if your task is fundamentally about pulling one exact line out of a hundred thousand tokens, internalizing memory into the model may not beat doing retrieval honestly. The scenarios that actually collect the dividend look different: very long interaction histories, highly repetitive experience, and tasks that never need to trace back to the original record verbatim. The paper's experience-guided agentic coding points in exactly that direction, storing retrieved experience in native memory and reading it through memory attention during execution, without spending task context on it.

The gap between the two specifications also makes sense in that light: a lightweight version validating that the mechanism works, and a heavy version testing the capacity ceiling.

Paper and code are both public, at arXiv:2610.26825, in the repository github.com/MemTensor/MemT-1.0, with weights under the MemTensor collection on Hugging Face. Worth noting alongside that: the author list includes Jiajun Shen, founder of DeepSeek, along with a long roster of names including Jiawei Liu and Tao Jiang, on a team of more than seventy.

So What

Attaching a memory store to a model and letting a model grow its own memory are two different technical philosophies. The first assumes the model is stable and that memory is an outsourceable engineering problem. The second assumes memory is part of model capability, and that it should be trained and evolved alongside the parameters. The second currently wins only on tasks with a particular shape. But if it holds, it does not merely optimize the cost structure of agents. It rewrites it.