[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-armt-associative-recurrent-memory-transformer":3,"news-related-b5638cab-a4d6-44ac-9230-32ed0a4cba9d":36},{"id":4,"title":5,"summary":6,"content":6,"original_url":7,"source_id":8,"tags":9,"translations":23,"news_slug":29,"published_at":30,"created_at":31,"modified_at":32,"is_published":33,"publish_type":34,"image_url":13,"view_count":35},"b5638cab-a4d6-44ac-9230-32ed0a4cba9d","ARMT 把「记忆」焊进 Transformer:用恒定显存换无限上下文","## ARMT 把「记忆」焊进 Transformer:用恒定显存换无限上下文\n\n长上下文是这两年 LLM 工程化最大的「显形成本」。标准 Transformer 注意力是 O(n²) 计算、O(n) 显存——上下文从 8K 推到 128K,推理成本要涨两个数量级。现有解法几乎都集中在 KV Cache 压缩(JLT、MosaicKV)或混合注意力(HydraHead、DeepSeek V4),都在「保留一段连续 KV」假设里打转。\n\n7 月 13 日挂在 arXiv 的 **2607.11614**(Kuzmin 等 11 位作者,MIPT + AIRI + MBZUAI 等机构合作)提出不太一样的角度:**Associative Recurrent Memory Transformer (ARMT)**——把联想式循环记忆直接插入 Transformer 某些层,实现**恒定显存**扩展上下文。\n\n### 核心贡献\n\n1. **两个领域特定长上下文数据集**,为窄域微调场景设计,比通用 needle-in-haystack 更接近真实工作负载;\n2. **完整训练配方**:持续预训练 + 合成长文数据 + 课程学习 + **选择性**把联想记忆接入部分层(不是全量替换);\n3. **系统实验**:ARMT 增强后能处理远超原生窗口的输入且不退化,对分布外长度泛化更好,且在原始窗口内**少用 30% FLOPs**。\n\n### 个人评论\n\nARMT 不是范式颠覆,更像「**在 Transformer 内部挖一个 LRU 缓存**」。精神上接近 SSM\u002FMamba,但 ARMT 显式选择「保留原主干 + 局部加记忆」的渐进路线,工程门槛低很多。\n\n**30% FLOPs 降幅**对已上线长上下文应用是直接成本收益;「**选择性插入层**」是关键工程洞察——全量替换会破坏表征局部性,稀疏插入在效率与质量间找到 trade-off;论文强调「窄域微调场景」,说明 ARMT 不是通用 1M 上下文基模,而是**面向垂直长文工作负载**(法律、医疗、代码库)的实用路径。\n\n接下来要看:ARMT 的「联想记忆」是否会和 KV Cache 压缩方法叠加——「**外存 + 内压**」组合可能是长上下文成本下降的下一站。\n\n从 ReContext 到 JLT 再到 ARMT,长上下文已经不是「能不能做」的问题,而是「**用多少成本做**」的问题。每篇都在把单位 token 推理开销往下压一档——这才是 LLM 真正走进生产环境的底层燃料。","https:\u002F\u002Farxiv.org\u002Fabs\u002F2607.11614","7437aeb9-930c-4866-a2e9-48003c1a792b",[10,14,17,20],{"id":11,"name":12,"slug":12,"description":13,"color":13},"5e628969-6d2a-437f-998a-104e4b16cfb1","ai-progress",null,{"id":15,"name":16,"slug":16,"description":13,"color":13},"0ef8513a-0a26-42f0-b6f9-5b6dadded45c","efficiency",{"id":18,"name":19,"slug":19,"description":13,"color":13},"01598627-1ea6-4b27-a5d8-874971571a71","llm",{"id":21,"name":22,"slug":22,"description":13,"color":13},"4f214978-cac1-4f39-aa4b-f92a0d0934b7","transformer",[24],{"id":25,"lang":26,"title":27,"summary":28,"content":28},"a590235f-5a4b-4189-bf7c-09e3782a5622","en","ARMT welds memory into Transformers for endless context","## ARMT welds 'memory' into the Transformer: trades constant memory for infinite context\n\nLong context has been the most \"visible cost\" in LLM engineering over the past two years. Standard Transformer attention is O(n²) compute and O(n) memory — pushing context from 8K to 128K means inference cost grows two orders of magnitude. Existing solutions almost all focus on KV cache compression (JLT, MosaicKV) or hybrid attention (HydraHead, DeepSeek V4), all circling the assumption of \"retaining a continuous KV segment\". Paper 2607.11614, posted to arXiv on July 13 (Kuzmin and 10 other authors, a collaboration between MIPT, AIRI, MBZUAI and others), takes a different angle: **Associative Recurrent Memory Transformer (ARMT)** — directly inserting associative-recurrent memory into certain Transformer layers, achieving **constant memory** context extension. ### Core contributions 1. **Two domain-specific long-context datasets**, designed for narrow-domain fine-tuning scenarios, closer to real workloads than the generic needle-in-haystack; 2. **A complete training recipe**: continual pretraining + synthetic long-text data + curriculum learning + **selectively** inserting associative memory into partial layers (not full replacement); 3. **Systematic experiments**: ARMT-enhanced can handle inputs far beyond the native window without degrading, with better out-of-distribution length generalization, and uses **30% fewer FLOPs** within the original window. ### Personal commentary ARMT isn't a paradigm revolution, more like \"**digging an LRU cache inside the Transformer**\". It's spiritually close to SSM\u002FMamba, but ARMT explicitly chooses the gradual path of \"keep the original backbone + locally add memory\", with much lower engineering threshold. **The 30% FLOPs reduction** is a direct cost win for already-shipped long-context applications; \"**selectively inserting layers**\" is the key engineering insight — full replacement would break representation locality, sparse insertion finds a trade-off between efficiency and quality; the paper emphasizes \"narrow-domain fine-tuning scenarios\", meaning ARMT isn't a general-purpose 1M-context base model, but a **practical path for vertical long-text workloads** (legal, medical, codebases). What to watch next: whether ARMT's \"associative memory\" can be stacked with KV cache compression methods — the \"**external storage + internal compression**\" combination may be the next stop for long-context cost reduction. From ReContext to JLT to ARMT, long context is no longer a \"can we do it\" question, but a \"**how cheaply can we do it**\" question. Every paper presses the per-token inference cost down another notch — that's the underlying fuel for LLMs truly entering production.","armt-associative-recurrent-memory-transformer","2026-07-23T00:00:00Z","2026-07-22T22:08:00.567068Z","2026-08-19T02:08:40.142862Z",true,"agent",104,{"items":37},[38,43,48,53,58,63],{"id":39,"title":40,"news_slug":41,"published_at":42},"3d922c00-afcb-4f1c-a6d5-8f9d6c10c642","从 Kimi Linear 到 Kimi K3:MoE 推理效率战里被忽略的架构升级","kimi-k3-latentmoe-kda-attnres-nope","2026-07-30T00:30:00+00:00",{"id":44,"title":45,"news_slug":46,"published_at":47},"d2844cbb-b70e-469c-95b8-71cee8d735a6","给 Transformer 装上「CNN 鼻子」:用 0.01% 的参数量换 benchmark 普涨","transformer-cnn-nose-0-01-percent","2026-07-22T12:10:00+00:00",{"id":49,"title":50,"news_slug":51,"published_at":52},"067a3f68-9bc8-4486-9715-5a391e537909","ETH 统一评测循环线性注意力：Kimi Delta Attention 损失最低","eth-zurich-clvr-kda","2026-07-12T04:00:00+00:00",{"id":54,"title":55,"news_slug":56,"published_at":57},"29774f38-c361-4dca-b11a-c14df2fc84d9","HiLS 把\"无限上下文\"从口号变成数学:让稀疏注意力首次跑赢 Full Attention","hils-hierarchical-landmark-sparse","2026-07-07T14:00:00+00:00",{"id":59,"title":60,"news_slug":61,"published_at":62},"5c53c383-9727-4880-95e9-9fa752132b01","把混合注意力推到 head 级：HydraHead 用 7:1 LA\u002FFA 比实现 3:1 层混的长上下文性能","hydrahead-7-to-1-la-fa-head-mixed-attention","2026-06-20T16:14:00+00:00",{"id":64,"title":65,"news_slug":66,"published_at":67},"217f417d-1b9c-475b-99f4-e21e7c909711","MHAR 把 Transformer 残差流从「单车道」拆成 H 条独立路由:子空间第一次有权自己挑历史层","multi-head-attention-residuals-mhar","2026-08-01T07:30:00+00:00"]