[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-aws-keysvalues-sparse-attention-finetuning":3,"news-related-cac485ab-a429-4ebc-88c6-a1f924f978ff":38},{"id":4,"title":5,"summary":6,"content":7,"original_url":8,"source_id":9,"tags":10,"translations":24,"news_slug":31,"published_at":32,"created_at":33,"modified_at":34,"is_published":35,"publish_type":36,"image_url":14,"view_count":37},"cac485ab-a429-4ebc-88c6-a1f924f978ff","AWS 开源 KeysAndValues:微调时就让模型学会“遗忘”,单张 A100 撑住 128K","arXiv 论文 2608.19920 提出带着稀疏注意力微调:训练阶段就让 KV 淘汰策略生效,模型权重与缓存策略共同适应,常反超精确注意力训练;单张 A100 40GB 可对 Qwen3-4B 做 128k 序列 LoRA 微调,配套开源库 KeysAndValues。","长上下文的真正瓶颈,从来不是模型“看不看得完”,而是 KV cache 撑不撑得住。过去的主流路径是:训练用精确注意力,推理时再套一层稀疏化的 KV 淘汰策略把缓存压小——训练与部署的内存机制并不一致。8 月 20 日提交到 arXiv 的论文《Learning how to Forget》(2608.19920)把这个顺序倒了过来:与其事后打补丁,不如让模型在微调阶段就“带着遗忘”学习。\n\n## 核心思路:让权重与淘汰策略共同适应\n\n论文由 Matthias Seeger 等五位作者完成,配套代码开源在 AWS 的 awslabs 组织下。方法的关键动作是:微调时让 KV cache 淘汰策略全程生效,权重主动适应“信息会被逐出”这一现实,与策略共同适应(co-adapt)。摘要报告的结果是,这种做法常常反超用精确注意力(序列并行)训练出来的模型。\n\n复现门槛也压得低:README 给出的示例基于 Qwen3-4B-Instruct-2507,单张 A100 40GB 显存即可在 Helmet 基准的 128k token 序列上做 LoRA 微调,缓存策略 h2o-torch-quantized8、槽位 16384;换到 8 张 A100 的 AWS p4d.24xlarge 实例上,全局 batch 可以到 32。\n\n## 把 H2O 从论文做进内核\n\n论文实验中表现最好的淘汰策略是 H2O(arXiv:2306.14048):按注意力权重的累计和对 KV 条目打分,分数最低者被逐出——README 的说法是,这在强意义上就是缓存世界里的 LRU。\n\n团队还点出一个被内核生态忽视的缺口:主流快速 SDPA 内核都不返回“沿 query 轴求和的注意力权重”,而这恰是 H2O 类策略打分的必需品。库里的解法是给 FlashInfer CUDA 内核补上该返回值,并用 Triton score-sum 内核补齐计算;配合 4\u002F8-bit 缓存量化与 CPU offload,README 明确说 4-bit 量化可把 KV 缓存的 GPU 显存需求降为原来的四分之一。\n\n## 限制也写得很清楚\n\nREADME 坦承:纯推理性能不敌 vLLM、SGLang,缺少多设备策略,定位是研究与评估;但对 H2O 这类高级缓存策略的支持比 vLLM 更好。另一个技术亮点在梯度计算:前向阶段把淘汰决策写进“重放日志”,反向阶段用重放缓存逐格回放、配合激活检查点,才让稀疏注意力下的长序列微调在显存上变得可行。\n\n## 所以呢\n\n这篇论文值得记住的不是某个跑分,而是一次方向修正:长上下文的下一程优化,可能不在“更大的窗口”,而在训练与部署的内存机制对齐——当淘汰策略成为训练的一部分,“遗忘”就从缺陷变成了可学习的能力。对做长文本落地的团队来说,单卡可复现的实验配置加全开源代码,意味着这条路线周末就能上手验证(论文:https:\u002F\u002Farxiv.org\u002Fabs\u002F2608.19920 ,代码:https:\u002F\u002Fgithub.com\u002Fawslabs\u002Fkeys_values )。","https:\u002F\u002Farxiv.org\u002Fabs\u002F2608.19920","7437aeb9-930c-4866-a2e9-48003c1a792b",[11,15,18,21],{"id":12,"name":13,"slug":13,"description":14,"color":14},"0ef8513a-0a26-42f0-b6f9-5b6dadded45c","efficiency",null,{"id":16,"name":17,"slug":17,"description":14,"color":14},"0a93ec8e-ea39-4693-81de-563ca8c173f7","inference",{"id":19,"name":20,"slug":20,"description":14,"color":14},"01598627-1ea6-4b27-a5d8-874971571a71","llm",{"id":22,"name":23,"slug":23,"description":14,"color":14},"b9bd9039-fcdb-41a8-b85b-fc1587def2b9","open-source",[25],{"id":26,"lang":27,"title":28,"summary":29,"content":30},"429251ac-4108-45af-9b3d-f1c9525ad29b","en","AWS KeysAndValues: Fine-tuning LLMs to Forget, 128K on One A100","An arXiv paper (2608.19920, submitted Aug 20) from the team behind AWS awslabs' KeysAndValues library proposes fine-tuning language models with sparse-attention KV eviction policies active during training, so weights co-adapt to cache eviction and often outperform exact-attention training. A single A100 40GB fine-tunes Qwen3-4B on 128k-token sequences; the open-source library ships H2O kernels that return summed attention weights, plus 4\u002F8-bit quantized KV buffers with CPU offloading.","The real bottleneck for long context is not whether a model can read a long input, but whether the KV cache fits in memory. The dominant recipe so far: train with exact attention, then bolt a sparse KV eviction policy on top at inference time — so the memory mechanics differ between training and deployment. A paper submitted to arXiv on August 20, \"Learning how to Forget\" (2608.19920), flips that order: instead of patching after the fact, let the model learn while forgetting during fine-tuning.\n\n## Core idea: let weights co-adapt with the eviction policy\n\nThe paper is authored by Matthias Seeger and four colleagues, with the companion code open-sourced under AWS's awslabs organization. The key move: keep the KV cache eviction policy active throughout fine-tuning, so the weights adapt to the reality that information gets evicted — co-adapting with the policy. The abstract reports this often outperforms models trained with exact attention (sequence parallelism).\n\nThe reproduction bar is low: the README's example fine-tunes Qwen\u002FQwen3-4B-Instruct-2507 with LoRA on 128k-token sequences from the Helmet benchmark on a single A100 GPU with 40 GB of memory, using the h2o-torch-quantized8 cache policy with 16,384 slots; on an 8xA100 AWS p4d.24xlarge instance, the global batch reaches 32.\n\n## Engineering H2O into kernels\n\nThe leading eviction policy in the paper's experiments is H2O (arXiv:2306.14048): score KV entries by the cumulative sum of attention weights and evict the lowest scorer — which, per the README, is in a strong sense the LRU strategy from general caching.\n\nThe team also flags a gap the kernel ecosystem has overlooked: mainstream fast SDPA kernels do not return attention weights summed over the query axis, exactly what H2O-style scoring needs. The library's fix adds that return value to FlashInfer CUDA kernels and fills the computation with a Triton score-sum kernel; combined with 4\u002F8-bit KV buffer quantization and CPU offloading, the README states 4-bit quantization cuts KV cache GPU memory requirements to a quarter.\n\n## Limitations are stated plainly\n\nThe README admits raw inference is not competitive with vLLM or SGLang, multi-device strategies are missing, and the library targets research and evaluation — though its support for advanced KV cache strategies like H2O is better than vLLM's. Another highlight is gradient computation: eviction decisions are logged into a replay log during the forward pass, then replayed through replay caches cell by cell during backward, paired with activation checkpointing, which is what makes long-sequence fine-tuning under sparse attention memory-feasible at all.\n\n## So what\n\nThe lasting idea here is not a benchmark number but a course correction: the next leg of long-context optimization may lie not in bigger windows but in aligning memory mechanics between training and deployment. Once eviction is part of training, forgetting stops being a defect and becomes a learnable skill. For teams shipping long-context workloads, a single-GPU reproducible setup plus fully open code means this route can be tested over a weekend (paper: https:\u002F\u002Farxiv.org\u002Fabs\u002F2608.19920 , code: https:\u002F\u002Fgithub.com\u002Fawslabs\u002Fkeys_values ).","aws-keysvalues-sparse-attention-finetuning","2026-08-26T05:20:00Z","2026-08-26T05:20:24.324961Z","2026-08-26T05:20:24.324970Z",true,"agent",36,{"items":39},[40,45,50,55,60,65],{"id":41,"title":42,"news_slug":43,"published_at":44},"c94bdf86-5de9-49fe-8c98-0f5c47611bfe","SGLang v0.5.18 发布:大模型冷启动提速 2.38 倍,710 个 PR 都改了什么","sglang-v0-5-18-cold-start-2-38x","2026-08-24T23:15:00+00:00",{"id":46,"title":47,"news_slug":48,"published_at":49},"92433e6b-113a-4ada-af77-fbb8995a9850","LFM2.5-DSpark 开源:300M 草稿模型让端侧推理快 2.87 倍,输出零损耗","lfm2-5-dspark-draft-models","2026-08-21T21:10:00+00:00",{"id":51,"title":52,"news_slug":53,"published_at":54},"926f89fc-5ed5-4170-bfac-931d3a31b6a4","腾讯混元开源 AngelSpec 投机解码框架：DFly 在 Hy3-A21B 上取得 1.98–2.40× 加速","tencent-angelspec-spec-decoding-hy3-dfly","2026-07-30T00:00:00+00:00",{"id":56,"title":57,"news_slug":58,"published_at":59},"894787b7-8799-4c22-8e20-5e04db8f7181","DeepSeek V4 GA 把分时电价塞进大模型 API:CSA + HCA 撑起 1M 上下文,峰值翻倍只是商业答案","deepseek-v4-ga-time-of-use-pricing","2026-07-24T02:00:00+00:00",{"id":61,"title":62,"news_slug":63,"published_at":64},"518d0be2-1220-4920-98fd-dee9df27a43d","VIDRAFT VKUE 把同一份 34.7B 稀疏 MoE 权重从 B200 跑到裸 CPU","vidraft-vkue-cpu-moe","2026-07-12T10:00:00+00:00",{"id":66,"title":67,"news_slug":68,"published_at":64},"f8a33525-3846-46e2-9f88-05be407dbe2f","Liquid AI 用 Antidoom 把推理模型的 doom loop 压到 1%：一次训练只改 1 个 token","liquid-ai-antidoom"]