[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-kronlstm-kronecker-delta-rule":3,"topics-all":38,"news-related-fd29c2f1-175f-416e-bde2-95dd65ca3465":57},{"id":4,"title":5,"summary":6,"content":7,"original_url":8,"source_id":9,"tags":10,"translations":24,"news_slug":31,"published_at":32,"created_at":33,"modified_at":34,"is_published":35,"publish_type":36,"image_url":14,"view_count":37},"fd29c2f1-175f-416e-bde2-95dd65ca3465","KronLSTM 精准改写记忆:线性注意力长上下文新解","NX-AI 提出 KronLSTM:新的 Kronecker Delta Rule 让线性注意力的记忆改写只动该动的关联,矩阵范数全程有界、免额外归一化,长上下文基准上常常胜过 (Gated) DeltaNet;代码兼容 FLA,四个 1.5B 模型以 MIT 开源。","长上下文的军备竞赛里,线性注意力是省钱的那条路:把 softmax 注意力的二次复杂度压成线性,代价是记忆状态变成一个不断被改写的矩阵。问题在于\"怎么改\"——写入新东西时,旧记忆被扰动多少。NX-AI 团队的新论文 KronLSTM(arXiv:2610.26834,10 月 19 日提交,六位作者包括 Sepp Hochreiter)给出了一个更\"外科手术\"的答案。\n\n## 写入即干扰:线性注意力的老毛病\n\nNX-AI 在仓库 README 里用一张动画把三种写入行为摆在一起:传统 mLSTM 每写一个新值,会扰动矩阵里所有已存的值;Gated DeltaNet 会把与新 key 共享维度的旧值整批抹掉;而 KronLSTM 只修改与新 key-value 对重叠的那一小块,其余记忆原样保留。对长上下文来说这不是细节——上下文越长,写入次数越多,被反复扰动的旧记忆越难保真。\n\n## Kronecker Delta Rule:只改该改的\n\n论文的核心是给关联矩阵记忆设计的新更新规则。KronLSTM 把 xLSTM 家族扩展出一个记忆修正机制:更正目标关联时,只在矩阵空间的单一方向上修改记忆,相比 delta-rule 更新对已存信息的干扰更小。理论上,这套记忆动力学可以描述为一条收敛到稳态分布的马尔可夫链,记忆矩阵的期望范数在整个输入序列上保持一致有界——换句话说,不需要给记忆状态外挂额外的归一化或稳定化机制,它自己就是稳的。实验侧的论文口径是:与现有线性注意力架构(包括 (Gated) DeltaNet 家族)相比有竞争力、且常常胜出,长上下文基准上尤其明显。\n\n## 开源清单:FLA 兼容,四个 1.5B 模型\n\n工程侧给得相当完整。代码是 Triton 内核加 PyTorch 层,直接兼容 flash-linear-attention(FLA)生态,训练用 chunk_kronlstm(chunkwise WY\u002FUT 形式,与 GDN-2、KDA 同套路),解码用 fused_recurrent_kronlstm;仓库本身就是 FLA 的一个快照加上 KronLSTM 实现,作者表示未来会合入 FLA 主线。预训练模型有四个,全部 1.5B 参数、100B token 训练、Llama-2 tokenizer:两个纯 KronLSTM(24 层,8K 与 32K 上下文两档),两个混合版(20 层 KronLSTM + 4 层滑窗注意力,窗口 2048、不用位置编码)。训练配置也交代得细:8K 模型在 FineWeb-Edu 上练;32K 版从 75B token 处分支,再用 ProLong-64k 长文档混合喂 25B token。许可证 MIT,在 H100\u002FA100 上以 PyTorch 2.13 + Triton 3.7.1 开发测试。\n\n## 所以呢\n\n两点判断。其一,线性注意力的竞争已经从\"能不能压成线性\"进入\"记忆改写精度\"的深水区:今年 6 月 NVIDIA 的 Gated DeltaNet-2 把擦与写拆开,这次的 Kronecker Delta Rule 更进一步,连修正都限定在单一方向——架构层的微创新在回潮,谁的记忆更接近\"只动该动的\",谁的长上下文就站得住。其二,对从业者这是即插即用的:FLA 生态用户改个层名就能跑,作者在 HF 社区里预告了更快的内核正在路上。需要保持清醒的是,1.5B\u002F100B token 是研究级配置,\"常常胜出\"是论文自报口径,离生产级验证还有距离——但它给了长上下文这条赛道一个值得盯住的新基线。\n\n参考:arXiv:2610.26834;github.com\u002FNX-AI\u002FKronLSTM;huggingface.co\u002Fcollections\u002FNX-AI\u002Fkronlstm","https:\u002F\u002Farxiv.org\u002Fabs\u002F2610.26834","7437aeb9-930c-4866-a2e9-48003c1a792b",[11,15,18,21],{"id":12,"name":13,"slug":13,"description":14,"color":14},"0ef8513a-0a26-42f0-b6f9-5b6dadded45c","efficiency",null,{"id":16,"name":17,"slug":17,"description":14,"color":14},"01598627-1ea6-4b27-a5d8-874971571a71","llm",{"id":19,"name":20,"slug":20,"description":14,"color":14},"b9bd9039-fcdb-41a8-b85b-fc1587def2b9","open-source",{"id":22,"name":23,"slug":23,"description":14,"color":14},"4f214978-cac1-4f39-aa4b-f92a0d0934b7","transformer",[25],{"id":26,"lang":27,"title":28,"summary":29,"content":30},"1bd85c4b-3f05-4bf8-9299-287bd27bcee9","en","NX-AI's KronLSTM Redesigns Memory Writes in Linear Attention","KronLSTM writes memory in one direction only, keeps norms bounded without extra normalization, and ships FLA kernels plus four 1.5B models under MIT.","In the long-context arms race, linear attention is the budget route: it flattens the quadratic cost of softmax attention, at the price of turning memory into a matrix that keeps being rewritten. The question is how you rewrite it — how much old memory gets disturbed each time you write something new. A new paper from the NX-AI team, KronLSTM (arXiv:2610.26834, submitted October 19, six authors including Sepp Hochreiter), offers a more surgical answer.\n\n## Write-in Interference: The Old Problem of Linear Attention\n\nThe repository README lines up three write behaviors in one animation: in classic mLSTM, every new value written disturbs all stored values in the matrix; in Gated DeltaNet, old values sharing dimensions with the new key get erased in bulk; KronLSTM instead modifies only the small block that overlaps with the new key-value pair, leaving the rest of memory intact. For long contexts this is not a detail — the longer the context, the more writes, and the harder it is for repeatedly disturbed old memories to stay faithful.\n\n## The Kronecker Delta Rule: Touch Only What Must Change\n\nThe core contribution is a new update rule for associative matrix memories. KronLSTM extends the xLSTM family with a memory-revision mechanism: when correcting a targeted association, it modifies memory along a single direction in matrix space, reducing interference with previously stored information compared to delta-rule updates. In theory, the resulting memory dynamics can be described as a Markov chain converging to a steady-state distribution, and the expected norm of the memory matrix stays uniformly bounded across the entire input sequence — meaning no extra normalization or stabilization machinery is needed for the memory state; it is stable by construction. On the experimental side, the paper's own wording: KronLSTM is competitive with, and often outperforms, existing linear attention architectures including the (Gated) DeltaNet family, with particularly strong performance on long-context benchmarks.\n\n## The Open-Source Package: FLA-Compatible, Four 1.5B Models\n\nThe engineering side is thorough. The code ships as Triton kernels plus a PyTorch layer, directly compatible with the flash-linear-attention (FLA) ecosystem: training uses chunk_kronlstm (a chunkwise WY\u002FUT form, the same recipe as GDN-2 and KDA), decoding uses fused_recurrent_kronlstm. The repository is itself a snapshot of FLA with KronLSTM added in FLA's layout, and the authors say it is intended to be merged into FLA upstream. Four pretrained models are released, all 1.5B parameters, trained on 100B tokens with the Llama-2 tokenizer: two pure KronLSTM variants (24 layers, 8K and 32K context), and two hybrids (20 KronLSTM layers + 4 sliding-window attention layers, window 2048, no positional encoding). The training recipe is disclosed in detail: the 8K models train on FineWeb-Edu; the 32K variants branch from the 8K runs at the 75B-token mark and train the remaining 25B tokens at 32K context on a long-document mixture from ProLong-64k. The license is MIT; development and testing were done on NVIDIA H100 and A100 GPUs with PyTorch 2.13 and Triton 3.7.1.\n\n## So What\n\nTwo takeaways. First, linear attention competition has moved from \"can we make it linear\" into the deep water of memory-revision precision: NVIDIA's Gated DeltaNet-2 in June split erase from write, and the Kronecker Delta Rule goes one step further by restricting even the correction to a single direction — architectural micro-innovation is making a comeback, and whoever's memory comes closest to \"touch only what must change\" wins the long-context standoff. Second, for practitioners this is plug-and-play: FLA users can swap in the layer, and the authors have teased faster kernels on the way in the HF community discussion. A dose of realism: 1.5B parameters and 100B tokens is a research-scale configuration, and \"often outperforms\" is the paper's self-reported wording — production-grade validation is still ahead. Still, it gives the long-context track a new baseline worth watching.\n\nReference: arXiv:2610.26834 (abs page); github.com\u002FNX-AI\u002FKronLSTM; huggingface.co\u002Fcollections\u002FNX-AI\u002Fkronlstm","kronlstm-kronecker-delta-rule","2026-10-20T15:06:16Z","2026-10-20T15:13:53.550514Z","2026-10-20T15:13:53.550524Z",true,"agent",85,[39,48],{"slug":40,"tag_slug":40,"title_zh":41,"title_en":42,"intro_zh":43,"intro_en":44,"id":45,"is_active":35,"created_at":46,"modified_at":47},"ai-for-science","AI for Science 2026：从 UniPert 到 GPT-Rosalind 的硬核进化","AI for Science 2026: from UniPert to GPT-Rosalind","生命科学、化学材料、物理世界模型——AI 正在从\"语言工具\"变成\"实验伙伴\"。本专题收录 AI 在三大科学方向的关键节点：UniPert 统一基因与化学扰动空间、GPT-Rosalind 端到端生命科学推理、达摩院 AI 智能体 28 小时找到 4 种超导新材料、Anthropic Claude Science 把工作台做成标准品。","From language tool to lab partner — AI is reshaping life sciences, chemistry\u002Fmaterials, and physical world models. This topic covers the key milestones: UniPert unifying genetic-chemical perturbation spaces, GPT-Rosalind's end-to-end life-sciences reasoning, DAMO's AI agent discovering 4 superconducting materials in 28 hours, and Anthropic's Claude Science workbench going mainstream.","988a4300-5fab-41c4-b5d8-63711a2dc757","2026-09-10T01:34:15.296649Z","2026-09-10T01:34:15.296663Z",{"slug":49,"tag_slug":49,"title_zh":50,"title_en":51,"intro_zh":52,"intro_en":53,"id":54,"is_active":35,"created_at":55,"modified_at":56},"h3-series","MiniMax H3 系列：从开源权重到 35 倍吞吐","MiniMax H3 Series: from open weights to 35x throughput","MiniMax H3 自 2026 年 8 月开源以来节奏密集：官方把生成、参考与编辑收回一个模型；ComfyUI 当天压进 RTX 3060；摩尔线程 3 小时完成国产 GPU 适配；fal 后训练版把吞吐拉到 35 倍；FastH3 蒸馏再砍推理成本。本专题持续追踪 H3 的发布—开源—蒸馏—部署全链路。","Since MiniMax open-sourced H3 in August 2026 the pace has been relentless: one unified omni-modal model, same-day ComfyUI support down to an RTX 3060, a 3-hour Day-0 port to Moore Threads GPUs, fal's post-trained H3 Max at 35x throughput, and FastH3 distillation cutting inference cost further. This topic tracks the full H3 chain — release, open weights, distillation, deployment.","83ef0daa-3c31-4cb3-86ed-e5ee58654d5f","2026-09-08T07:33:19.942193Z","2026-09-08T07:33:19.942209Z",{"items":58},[59,64,69,74,79,84],{"id":60,"title":61,"news_slug":62,"published_at":63},"d6ec8624-ad4e-41ce-9566-22d1f0926d49","循环解码器+并行编码器:RLT长度外推翻盘","recurrent-looped-transformer-length-generalization","2026-10-08T23:30:00+00:00",{"id":65,"title":66,"news_slug":67,"published_at":68},"530e5aa5-2026-4c30-b4e6-421caca907b2","Transformer 提前罢工:13 个基座模型跟不住引用链,一个 rank-8 LoRA 修好","tiny-lora-frozen-transformer-chain-relay","2026-10-03T21:05:16+00:00",{"id":70,"title":71,"news_slug":72,"published_at":73},"926f89fc-5ed5-4170-bfac-931d3a31b6a4","腾讯混元开源 AngelSpec 投机解码框架：DFly 在 Hy3-A21B 上取得 1.98–2.40× 加速","tencent-angelspec-spec-decoding-hy3-dfly","2026-07-30T00:00:00+00:00",{"id":75,"title":76,"news_slug":77,"published_at":78},"c8ba186e-e7a6-40e1-9485-41aeb4de388e","Sebastian Raschka 发布 LLM 架构图谱：40+ 开源模型一站式横向对比","sebastian-raschka-llm-arch-gallery-40","2026-05-19T07:01:00+00:00",{"id":80,"title":81,"news_slug":82,"published_at":83},"3a419e1c-fb2b-467e-a5b6-a7188525bc03","TraceForge:推理痕迹自己造,学生反超教师真迹","traceforge-family-native-distillation","2026-10-22T15:10:42+00:00",{"id":85,"title":86,"news_slug":87,"published_at":88},"b369ce1a-f8a3-4aba-ba78-c73c097a413d","MemT-1.0：记忆变成模型内置能力，128K 历史存储降 66 倍","memt-1-0-memory-native-foundation-model","2026-10-20T11:07:23+00:00"]