[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-delta-ngram-forward-only-memory-writes":3,"topics-all":38,"news-related-261b6911-0300-4d6b-86dc-a64475e4aa5a":57},{"id":4,"title":5,"summary":6,"content":7,"original_url":8,"source_id":9,"tags":10,"translations":24,"news_slug":31,"published_at":32,"created_at":33,"modified_at":34,"is_published":35,"publish_type":36,"image_url":14,"view_count":37},"261b6911-0300-4d6b-86dc-a64475e4aa5a","Delta N-gram:给大模型记忆打可拆卸补丁","arXiv 论文 Delta N-gram 向大模型原生 n-gram 记忆表写入稀疏补丁:预训练表不动,410 万参数 writer 前向写入,清补丁即复原。0.6B 原型再写入后 16\u002F16 事实保持,梯度写入仅 2-4\u002F16,3.5 万次写入后漂移远小于梯度更新,Qwen3.8-Flash-Next 验证。","让一个大模型学会新知识,眼下有两条主流路:把文档塞进上下文,或者微调权重。前者贵且容易忘,后者动的是几十亿参数——写进一条新知识,可能在看不见的地方挤掉三条旧的。10 月 22 日提交到 arXiv 的单作者论文 Delta N-gram 给出了第三条路:不碰权重,直接往模型自带的 n-gram 记忆表里打一个可拆卸补丁。\n\n## 不动权重,只动一张表\n\n作者的前期分析发现,模型原生的哈希 n-gram 表是持续更新时对基座扰动最小的位置,但过去的写入方式需要反向传播,还会改动预训练表本身。Delta N-gram 把这两件事都拆掉了:预训练表永远不修改,新证据写进一个只覆盖被寻址行的稀疏补丁。写入由一个 410 万参数的 writer 完成,它对每个位置-哈希头事件产出一个目标值和一个门控,一条块平均 delta 规则保证每行只是旧值与门控目标的凸组合。writer 用原始文本加预训练目标离线训练,真正使用时只需要前向计算;把补丁清零,模型立刻回到原始状态。\n\n一句话:知识更新从「重训一层」变成「改一张可回滚的表」。\n\n## 数字说话:16\u002F16 对 2-4\u002F16\n\n效果对比相当直观。一个 0.6B、被训练来答题的原型,第二次写入后仍保持 16\u002F16 的事实;同场景下在线梯度写入只能保住 2-4\u002F16。在 Qwen3.8-Flash-Next 上,plain 版 writer 的记忆表现出文档特异性:对 64 个留出新闻事件,相对空记忆把续写 NLL 降低 0.068 nats\u002Ftoken,相对另一篇文档的记忆降低 0.024;中间插入 16 篇文章后仍剩 0.063 和 0.013。会边写边读当前记忆的 delta writer 更抗干扰,分心状态下保住 70% 的记忆,plain 版是 54%。\n\n稳定性也有交代:在前作的七任务链上连续写入 35000 次,两种 writer 都不崩,后期任务准确率只变化 -0.26 和 -0.52 个百分点,而梯度更新的对照组是 -7.29 个百分点。基座漂移同样更小。\n\n## 冷水也要泼\n\n论文没有回避缺陷:写入的「获取」收益为负——写得多不等于通用能力变强;所有样本都会读到的格式行会被推大到预训练范数的 4-12 倍,带来最高 32.8% 的一般输入漂移。好在机制可解释:把 8 行这样的格式行清零,就能去掉数学输入上超过 98% 的漂移,连带其 NLL 收益一起消失。可控、可诊断,这是比分数更值钱的性质。\n\n规模上也要清醒:原型 0.6B,主验证在 Qwen3.8-Flash-Next 上完成,离生产级模型还有距离。33 页论文、代码已开源,作者只有一个人。\n\n## 所以呢\n\n如果「记忆可拆卸」这条路线成立,模型更新的最小单位就从「一次训练」降到「一次前向写入」,回滚成本归零。下一个问题或许更尖锐:当补丁可以随时打、随时清,模型的「知识」边界,到底该算在权重里,还是算在表里?([论文原文](https:\u002F\u002Farxiv.org\u002Fabs\u002F2610.31125))","https:\u002F\u002Farxiv.org\u002Fabs\u002F2610.31125","7437aeb9-930c-4866-a2e9-48003c1a792b",[11,15,18,21],{"id":12,"name":13,"slug":13,"description":14,"color":14},"5e628969-6d2a-437f-998a-104e4b16cfb1","ai-progress",null,{"id":16,"name":17,"slug":17,"description":14,"color":14},"0ef8513a-0a26-42f0-b6f9-5b6dadded45c","efficiency",{"id":19,"name":20,"slug":20,"description":14,"color":14},"01598627-1ea6-4b27-a5d8-874971571a71","llm",{"id":22,"name":23,"slug":23,"description":14,"color":14},"b9bd9039-fcdb-41a8-b85b-fc1587def2b9","open-source",[25],{"id":26,"lang":27,"title":28,"summary":29,"content":30},"e56a47d6-60d2-4551-af7e-3b8b67d33226","en","Delta N-gram: Detachable Memory Patches for LLMs","Delta N-gram patches an LLM n-gram table forward-only via a 4.1M-parameter writer; a 0.6B prototype keeps 16\u002F16 facts vs 2-4\u002F16 for gradient writes.","Teaching a large language model new knowledge currently has two mainstream paths: stuff documents into the context window, or fine-tune the weights. The first is expensive and forgetful; the second touches billions of parameters, so writing in one new fact may quietly push out three old ones. A single-author paper submitted to arXiv on October 22, titled Delta N-gram, offers a third path: leave the weights alone and patch the model native n-gram memory table with a detachable overlay.\n\n## No Weight Edits, Just a Table\n\nThe author earlier analysis found that the model native hashed n-gram table is where continual updates disturb the base model least, but previous writes required backpropagation and modified the pretrained table itself. Delta N-gram removes both constraints: the pretrained table is never modified; new evidence goes into a sparse patch that only covers the rows it addresses. Writing is done by a 4.1M-parameter writer that emits a target and a gate per position-hash-head event, while a block-averaged delta rule keeps each row a convex combination of its old value and the gated targets. The writer is trained offline on raw text with the pretraining objective; at use time only forward passes are needed, and zeroing the patch instantly restores the original model.\n\nIn one sentence: knowledge updates shift from retraining a layer to editing a rollback-able table.\n\n## The Numbers: 16\u002F16 Versus 2-4\u002F16\n\nThe comparison is striking. A 0.6B prototype trained to answer questions keeps 16\u002F16 facts after a second write, while online gradient writes retain only 2-4\u002F16 in the same setting. On Qwen3.8-Flash-Next, the plain writer memory is document-specific: across 64 held-out news events it lowers continuation NLL by 0.068 nats\u002Ftoken versus empty memory and by 0.024 versus another document memory, still at 0.063 and 0.013 after 16 intervening articles. The delta writer, which reads its current memory while encoding, is more resistant to distraction, retaining 70% of its memory versus 54% for the plain version.\n\nStability is covered too: over 35,000 sequential writes on a seven-task chain, both writers remain stable, with later-task accuracy changing by only -0.26 and -0.52 percentage points, against -7.29 points for the gradient-update control group. Base-model drift is likewise smaller.\n\n## Cold Water Included\n\nThe paper does not hide the flaws: acquisition is negative, meaning more writes do not strengthen general capability; format rows read by every sample grow to 4-12 times their pretrained norm, causing up to 32.8% drift on general inputs. The redeeming part is interpretability: zeroing eight such rows removes over 98% of layer-2 drift on math inputs, together with their NLL gain. Controllable and diagnosable is worth more than a headline score.\n\nScale deserves sobriety as well: the prototype is 0.6B, main validation runs on Qwen3.8-Flash-Next, and production-grade models remain far away. The paper is 33 pages, the code is open-sourced, and the author list has exactly one name.\n\n## So What\n\nIf detachable memory holds up, the minimal unit of model updates drops from a training run to a single forward write, with rollback cost reduced to zero. The next question is sharper: when patches can be applied and cleared at any time, does the boundary of what a model knows live in its weights, or in its table? ([Paper](https:\u002F\u002Farxiv.org\u002Fabs\u002F2610.31125))","delta-ngram-forward-only-memory-writes","2026-10-25T23:50:00Z","2026-10-25T23:09:40.979699Z","2026-10-25T23:09:40.979718Z",true,"agent",339,[39,48],{"slug":40,"tag_slug":40,"title_zh":41,"title_en":42,"intro_zh":43,"intro_en":44,"id":45,"is_active":35,"created_at":46,"modified_at":47},"ai-for-science","AI for Science 2026：从 UniPert 到 GPT-Rosalind 的硬核进化","AI for Science 2026: from UniPert to GPT-Rosalind","生命科学、化学材料、物理世界模型——AI 正在从\"语言工具\"变成\"实验伙伴\"。本专题收录 AI 在三大科学方向的关键节点：UniPert 统一基因与化学扰动空间、GPT-Rosalind 端到端生命科学推理、达摩院 AI 智能体 28 小时找到 4 种超导新材料、Anthropic Claude Science 把工作台做成标准品。","From language tool to lab partner — AI is reshaping life sciences, chemistry\u002Fmaterials, and physical world models. This topic covers the key milestones: UniPert unifying genetic-chemical perturbation spaces, GPT-Rosalind's end-to-end life-sciences reasoning, DAMO's AI agent discovering 4 superconducting materials in 28 hours, and Anthropic's Claude Science workbench going mainstream.","988a4300-5fab-41c4-b5d8-63711a2dc757","2026-09-10T01:34:15.296649Z","2026-09-10T01:34:15.296663Z",{"slug":49,"tag_slug":49,"title_zh":50,"title_en":51,"intro_zh":52,"intro_en":53,"id":54,"is_active":35,"created_at":55,"modified_at":56},"h3-series","MiniMax H3 系列：从开源权重到 35 倍吞吐","MiniMax H3 Series: from open weights to 35x throughput","MiniMax H3 自 2026 年 8 月开源以来节奏密集：官方把生成、参考与编辑收回一个模型；ComfyUI 当天压进 RTX 3060；摩尔线程 3 小时完成国产 GPU 适配；fal 后训练版把吞吐拉到 35 倍；FastH3 蒸馏再砍推理成本。本专题持续追踪 H3 的发布—开源—蒸馏—部署全链路。","Since MiniMax open-sourced H3 in August 2026 the pace has been relentless: one unified omni-modal model, same-day ComfyUI support down to an RTX 3060, a 3-hour Day-0 port to Moore Threads GPUs, fal's post-trained H3 Max at 35x throughput, and FastH3 distillation cutting inference cost further. This topic tracks the full H3 chain — release, open weights, distillation, deployment.","83ef0daa-3c31-4cb3-86ed-e5ee58654d5f","2026-09-08T07:33:19.942193Z","2026-09-08T07:33:19.942209Z",{"items":58},[59,64,69,74,79,84],{"id":60,"title":61,"news_slug":62,"published_at":63},"34d309c3-5757-469a-85ff-b524c8865037","Queen 模型边下棋边解说:七轮自蒸馏涨 900 Elo","queen-chess-language-model-self-distillation","2026-10-05T23:04:34+00:00",{"id":65,"title":66,"news_slug":67,"published_at":68},"c361090b-291c-4357-9bd3-54845025f29d","CaRE-KD 让蒸馏学会看「教师脸色」:NeurIPS 2026 选中 IIT Delhi","care-kd-confidence-distillation-neurips-2609-36734","2026-10-05T03:00:00+00:00",{"id":70,"title":71,"news_slug":72,"published_at":73},"2dbc7c0f-083a-47d4-ba9a-8a7d66b22ae0","亚马逊八阶段配方:后训练让 GLM-4.5-Air 反超官方版","amazon-rufus-air-post-training-recipe","2026-09-25T23:12:24+00:00",{"id":75,"title":76,"news_slug":77,"published_at":78},"054e060c-e182-42a9-b3ed-229feb8ac0ac","2026 年的蒸馏长什么样:Hugging Face 拆解前沿模型三大范式","2026-distillation-three-paradigms","2026-07-09T12:00:00+00:00",{"id":80,"title":81,"news_slug":82,"published_at":83},"ab2d6e9e-8890-4ae7-b6ca-8febc831a279","HPLT MultiSynt\u002FMT：4.8 万亿 token 多语种数据集","hplt-multisynt-multilingual-dataset","2026-07-07T16:01:00+00:00",{"id":85,"title":86,"news_slug":87,"published_at":88},"93dfc6f4-e4a9-47a3-aa66-b4b9cee864e7","Beyond LoRA 不只是口号：HF 给 40+ PEFT 方法拍下公平基准，OFT 在图像任务上反超 LoRA","beyond-lora-hf-peft-benchmark-oft","2026-06-18T12:00:00+00:00"]