[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-llm-inference-in-flash-cim-ssd":3,"topics-all":38,"news-related-b571067a-9fa8-42bf-9431-98f26ac78e03":57},{"id":4,"title":5,"summary":6,"content":7,"original_url":8,"source_id":9,"tags":10,"translations":24,"news_slug":31,"published_at":32,"created_at":33,"modified_at":34,"is_published":35,"publish_type":36,"image_url":14,"view_count":37},"b571067a-9fa8-42bf-9431-98f26ac78e03","伯克利把LLM推理搬进SSD:KV缓存压缩15倍","UC Berkeley等机构提出把LLM推理搬进Flash存内计算:整数化推理加字典KV压缩,缓存流量降15倍,256K上下文能耗省6.8倍。","LLM 推理的瓶颈正在从算力转向内存带宽：RAG、推理时算力扩展和长上下文应用让请求序列越来越长，而内存容量和通信带宽的增速跟不上负载复杂度——这就是业界常说的\"内存墙\"。自回归解码每生成一个 token 都要读一遍不断增长的 KV 缓存，数据搬运本身成了延迟和能耗的大头。\n\nUC Berkeley、ICSI 与 LBNL 的研究团队 9 月 14 日在 arXiv 提交论文《LLM Inference in a Flash!》，给出的答案是把推理搬进 Flash 存内计算（Compute-in-Flash）设备，再用算法为这套硬件\"量身裁剪\"。作者包括 Michael W. Mahoney、Yakun Sophia Shao、Kurt Keutzer、Amir Gholami 等人（[arxiv.org\u002Fabs\u002F2609.16161](https:\u002F\u002Farxiv.org\u002Fabs\u002F2609.16161)）。\n\n## 两个瓶颈，两套算法\n\nFlash 存内计算把计算单元搬到存储阵列旁边，靠 SSD 的大容量和高内部读带宽绕开内存墙。但直接跑 LLM 有两道坎：这类设备不支持高精度浮点运算，且写入寿命有限——KV 缓存恰好是高频动态写入，正好踩在短板上。\n\n团队的应对是两条算法线并行。其一是端到端整数化量化：从线性层到 Softmax、RMSNorm、SiLU 等非线性算子全部用整数近似（多项式\u002F移位实现），彻底消除浮点计算。在 Llama-3.1-8B 上，完整整数流水线的 WikiText-2 困惑度为 7.5803，与 FP16 基线 7.5454 只差 0.0349；论文特别提到，不加 INT16 重量化的裸整数非线性近似会让激活直接发散成 NaN。\n\n其二是基于稀疏字典编码的 KV 缓存压缩：为每层的 K 和 V 各训练一个 32768 原子的过完备字典，常驻 Flash 只读；每个 KV 向量表示为少数字典原子的线性组合（稀疏码），写入量大幅缩减，正好避开写入寿命限制。配合局部窗口（128 token）加查询感知的两级稀疏（先 3 个原子粗筛、对前 10% token 精修到 16 个原子），Llama-3.1-8B 的 LongBench 平均分 43.65 对基线 44.07，动态 KV 缓存流量压缩 15 倍，Qwen-2.5-7B 上 44.23 对 44.44，同样保住 99% 分数。\n\n## 为什么这套设计偏爱 Flash\n\n论文有一个诚实的讨论值得注意：同样的字典压缩搬到 GPU 上并不划算——字典得反复从 HBM 搬运，1K 上下文时反而比不压缩更慢（0.96 倍）。只有当字典常驻 CIM 阵列、投影和 gather 在数据旁边就地完成时，收益才兑现。系统级建模显示，对 256K 长上下文，CIM-SSD 方案对比 NPU+DRAM 基线延迟降 4.4 倍、能耗降 6.8 倍；1K 短上下文也有 3.1 倍延迟和 2.7 倍能耗收益。\n\n## 所以呢\n\n这项工作的真正启示不是\"15 倍压缩\"这个数字，而是算法与硬件的协同设计范式：与其在 GPU 内存层级里继续挤水分，不如让推理算法迁就新型存储硬件的物理约束——整数化迁就算力约束，只读字典迁就写入寿命。当 HBM 供给紧张、边缘端长上下文需求膨胀时，这类\"为 Flash 重写推理\"的路线值得持续关注。当然，论文也坦承局限：只覆盖了解码阶段，计算密集的 prefill 如何映射到 CIM 仍是未解问题。","https:\u002F\u002Farxiv.org\u002Fabs\u002F2609.16161","7437aeb9-930c-4866-a2e9-48003c1a792b",[11,15,18,21],{"id":12,"name":13,"slug":13,"description":14,"color":14},"7ac06d8e-b074-4147-abfc-ffaa4c6b8744","ai-efficiency",null,{"id":16,"name":17,"slug":17,"description":14,"color":14},"0a93ec8e-ea39-4693-81de-563ca8c173f7","inference",{"id":19,"name":20,"slug":20,"description":14,"color":14},"01598627-1ea6-4b27-a5d8-874971571a71","llm",{"id":22,"name":23,"slug":23,"description":14,"color":14},"b49648f9-963e-4082-8684-3d085b7358fe","quantization",[25],{"id":26,"lang":27,"title":28,"summary":29,"content":30},"5107a87c-3756-4f4f-85bd-626a8de09ef2","en","Berkeley Runs LLM Inference Inside Flash SSDs","Berkeley runs LLM inference on Flash SSDs: integer-only execution plus dictionary KV compression cuts cache traffic 15x and saves 6.8x energy at 256K context.","The bottleneck of LLM inference is shifting from compute to memory bandwidth: retrieval-augmented generation, inference-time compute scaling, and long-context applications push requests toward ever longer sequences, while memory capacity and communication bandwidth fail to keep pace — the well-known \"memory wall.\" Autoregressive decoding must re-read the growing KV cache for every token generated, so data movement itself becomes the dominant cost in both latency and energy.\n\nA team from UC Berkeley, ICSI, and LBNL submitted a paper titled \"LLM Inference in a Flash!\" to arXiv on September 14, proposing to move inference onto Flash compute-in-memory (Compute-in-Flash) devices, with algorithms tailored to that hardware. Authors include Michael W. Mahoney, Yakun Sophia Shao, Kurt Keutzer, and Amir Gholami ([arxiv.org\u002Fabs\u002F2609.16161](https:\u002F\u002Farxiv.org\u002Fabs\u002F2609.16161)).\n\n## Two Bottlenecks, Two Algorithms\n\nFlash compute-in-memory moves computation next to the storage array, exploiting SSD capacity and high internal read bandwidth to bypass the memory wall. But running LLMs directly hits two obstacles: these devices lack support for high-precision floating-point operations, and their write endurance is limited — while the KV cache is exactly a high-frequency dynamic write workload, landing squarely on that weakness.\n\nThe team's response is two parallel algorithm tracks. The first is end-to-end integer-only quantization: from linear layers to nonlinear operators such as Softmax, RMSNorm, and SiLU, everything runs in integer approximations (polynomial\u002Fshift-based), eliminating floating-point computation entirely. On Llama-3.1-8B, the full integer pipeline achieves a WikiText-2 perplexity of 7.5803 versus an FP16 baseline of 7.5454 — a gap of just 0.0349; the paper notes that naive integer nonlinear approximations without INT16 requantization make activations diverge into NaN.\n\nThe second is dictionary-based KV cache compression built on sparse dictionary coding: a separate overcomplete dictionary of 32,768 atoms is trained per layer for keys and values, resident read-only in Flash; each KV vector becomes a linear combination of a few dictionary atoms (sparse codes), slashing write traffic and sidestepping the endurance limit. Combined with a 128-token local window and query-aware two-level sparsity (3 atoms for coarse scoring, refined to 16 for the top 10% of tokens), Llama-3.1-8B scores 43.65 average on LongBench against a 44.07 baseline while compressing dynamic KV cache traffic by 15×; Qwen-2.5-7B holds 44.23 versus 44.44, likewise retaining 99% of the score.\n\n## Why This Design Loves Flash\n\nOne honest discussion in the paper stands out: the same dictionary compression is not worthwhile on GPUs — the dictionary must be streamed repeatedly from HBM, making it slightly slower than no compression at 1K context (0.96×). Benefits only materialize when the dictionary stays resident in the CIM array and projections\u002Fgathers execute near the data. System-level modeling shows that at 256K long context, the CIM-SSD design delivers 4.4× latency and 6.8× energy savings versus an NPU+DRAM baseline; even at 1K short context, it achieves 3.1× latency and 2.7× energy gains.\n\n## So What\n\nThe real insight of this work is not the \"15× compression\" headline but the paradigm of algorithm-hardware co-design: instead of squeezing water out of the GPU memory hierarchy, reshape inference algorithms to match the physical constraints of novel storage hardware — integer execution bends to compute constraints, read-only dictionaries bend to write endurance. As HBM supply tightens and edge-side long-context demand grows, this \"rewrite inference for Flash\" route deserves attention. Admittedly, the paper also candidly notes its limits: it covers only the decode phase, and mapping the compute-bound prefill to CIM remains an open question.","llm-inference-in-flash-cim-ssd","2026-09-19T21:10:00Z","2026-09-19T21:10:10.475328Z","2026-09-19T21:10:10.475344Z",true,"agent",73,[39,48],{"slug":40,"tag_slug":40,"title_zh":41,"title_en":42,"intro_zh":43,"intro_en":44,"id":45,"is_active":35,"created_at":46,"modified_at":47},"ai-for-science","AI for Science 2026：从 UniPert 到 GPT-Rosalind 的硬核进化","AI for Science 2026: from UniPert to GPT-Rosalind","生命科学、化学材料、物理世界模型——AI 正在从\"语言工具\"变成\"实验伙伴\"。本专题收录 AI 在三大科学方向的关键节点：UniPert 统一基因与化学扰动空间、GPT-Rosalind 端到端生命科学推理、达摩院 AI 智能体 28 小时找到 4 种超导新材料、Anthropic Claude Science 把工作台做成标准品。","From language tool to lab partner — AI is reshaping life sciences, chemistry\u002Fmaterials, and physical world models. This topic covers the key milestones: UniPert unifying genetic-chemical perturbation spaces, GPT-Rosalind's end-to-end life-sciences reasoning, DAMO's AI agent discovering 4 superconducting materials in 28 hours, and Anthropic's Claude Science workbench going mainstream.","988a4300-5fab-41c4-b5d8-63711a2dc757","2026-09-10T01:34:15.296649Z","2026-09-10T01:34:15.296663Z",{"slug":49,"tag_slug":49,"title_zh":50,"title_en":51,"intro_zh":52,"intro_en":53,"id":54,"is_active":35,"created_at":55,"modified_at":56},"h3-series","MiniMax H3 系列：从开源权重到 35 倍吞吐","MiniMax H3 Series: from open weights to 35x throughput","MiniMax H3 自 2026 年 8 月开源以来节奏密集：官方把生成、参考与编辑收回一个模型；ComfyUI 当天压进 RTX 3060；摩尔线程 3 小时完成国产 GPU 适配；fal 后训练版把吞吐拉到 35 倍；FastH3 蒸馏再砍推理成本。本专题持续追踪 H3 的发布—开源—蒸馏—部署全链路。","Since MiniMax open-sourced H3 in August 2026 the pace has been relentless: one unified omni-modal model, same-day ComfyUI support down to an RTX 3060, a 3-hour Day-0 port to Moore Threads GPUs, fal's post-trained H3 Max at 35x throughput, and FastH3 distillation cutting inference cost further. This topic tracks the full H3 chain — release, open weights, distillation, deployment.","83ef0daa-3c31-4cb3-86ed-e5ee58654d5f","2026-09-08T07:33:19.942193Z","2026-09-08T07:33:19.942209Z",{"items":58},[59,64,69,74,79,84],{"id":60,"title":61,"news_slug":62,"published_at":63},"3cecce90-70b9-4bb3-b9b7-93e6b0c05105","D-Quant 用熵编码压 KV:2.26bit 近无损","d-quant-entropy-coding-kv-cache","2026-09-20T17:10:42+00:00",{"id":65,"title":66,"news_slug":67,"published_at":68},"b667e52f-ec7d-4ca4-8d9e-1db81e1a5616","DeepSeek论文:890字节KV缓存的三层架构账","deepseek-v41-flash-kv-cache-paper","2026-09-18T15:10:00+00:00",{"id":70,"title":71,"news_slug":72,"published_at":73},"63042acc-9bec-4d6e-960b-8352ed541bb5","bartowski 用 1000 组实验重写 GGUF 量化排布规则","gguf-quant-layout-maps","2026-09-10T21:05:00+00:00",{"id":75,"title":76,"news_slug":77,"published_at":78},"178aa5e5-2a4f-4a87-a97c-0da16295d96f","EMNLP 2026 OCGQuant:用通道配对治 NVFP4 陪葬误差,Qwen3-1.7B 接近 FP16","ocgquant-nvfp4-outlier-companion-grouping","2026-09-10T09:15:00+00:00",{"id":80,"title":81,"news_slug":82,"published_at":83},"34b087d5-f502-4a46-9599-6f014fe0cf89","llama.cpp v0.4.0 发布:权重懒加载进主线,小内存也能碰 75B 级 MoE","llama-cpp-v0-4-0-lazy-tensor-loading","2026-09-07T13:10:00+00:00",{"id":85,"title":86,"news_slug":87,"published_at":88},"988bbfb8-672a-4c6c-98f7-3a170b6bd8b3","Macaw 把 LFM2.5 装进 1.5GB:4-bit 端侧 LLM 跑 Mac 控制工具链","macaw-lfm25-15gb-edge-mac-agent","2026-08-24T06:00:00+00:00"]