[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-sparsedecoding-decoding-aware-pruning":3,"topics-all":38,"news-related-dab2e487-cfba-4d75-a93f-eff133476348":57},{"id":4,"title":5,"summary":6,"content":7,"original_url":8,"source_id":9,"tags":10,"translations":24,"news_slug":31,"published_at":32,"created_at":33,"modified_at":34,"is_published":35,"publish_type":36,"image_url":14,"view_count":37},"dab2e487-cfba-4d75-a93f-eff133476348","剪枝校准对不上解码分布?西湖蚂蚁让稀疏解码提速48%","西湖大学与蚂蚁的 SparseDecoding 点破 LLM 剪枝盲区：校准用自然语料，解码却吃自生成 token，分布错位拖垮剪枝模型长文质量。改用自回归生成校准后，2:4 稀疏 Qwen3-14B 写作分从 1.85 升至 4.18；bitmask SpMV 内核让 A100 解码提速最高 1.48 倍。","大模型推理的瓶颈一半在解码：batch 为 1 的自回归生成是访存受限的，每吐一个 token 都要把全部权重读一遍显存。Hessian 引导的免训练剪枝是老牌解法——把不重要的权重置零，要读的参数少了，解码自然变快。但西湖大学与蚂蚁 10 月 8 日提交 arXiv 的 SparseDecoding 论文（arXiv:2610.12327）指出一个被忽视的错位：剪枝时校准用的数据，和模型实际解码时吃的数据，根本不在一个分布上。\n\n## 校准数据错在哪\n\n传统做法（SparseGPT 一系）用预收集自然语料（如 C4）算 Hessian 评估权重重要性。问题是：解码阶段模型输入的是自己生成的 token，不是自然语料。论文实测发现，剪枝后模型的激活分布偏差在解码最初几步快速增长，所有层在第 12 步左右就达到平台期的 90%——剪枝时对齐的分布与真实解码时的分布脱节，长文本生成质量随之崩坏。\n\nSparseDecoding 的解法直白：校准激活不再取自固定语料，而是让 dense 模型在任务提示上自回归生成，只收集 decode 步的激活、丢弃 prefill 部分。两种配置都用 1M 校准 token、同一个 SparseGPT 后端，唯一变量是校准数据来源。\n\n## 内核是另一半答案\n\n只改校准还不够。论文的第二个观察更扎心：现有能真正提速的稀疏 kernel（如 cuSPARSELt 的 2:4 Sparse Tensor Core 路径）主要优化稀疏矩阵-矩阵乘（SpMM），适合 prefill 和大 batch；而 batch-1 解码的核心运算是稀疏矩阵-向量乘（SpMV）。实测该路径 prefill 提速 1.31-1.46 倍，解码阶段却只有 dense 吞吐的 0.85-0.87 倍——比不剪枝还慢。\n\n团队因此自己写了 N:M SpMV kernel：用 bitmask 索引（每 32 列打包成一个 32 位掩码词，50% 稀疏下每词恰好 16 个置位位）替代逐非零元的列索引，配合固定步长遍历。效果是 2:4 到 16:32 四种 N:M 模式吞吐几乎一致，A100 上端到端解码提速 1.34-1.48 倍（GPT-Fast 框架，context 512，50 次预热 + 50 次测量取中位）。\n\n## 数字说话\n\n质量侧的增益比速度侧更醒目。WritingBench 长文基准（1-10 分）上，2:4 稀疏下 C4 校准的 Qwen3-14B 只剩 1.85 分，SparseDecoding 校准拉回 4.18 分；Qwen3-32B 从 2.99 涨到 5.31。Llama 系增益温和（Llama-3.1-8B 从 1.34 到 1.52），Qwen 系是断崖式修复。代码生成 ClassEval 上，2:4 稀疏的 Qwen3-32B Pass@1 从 9.0% 涨到 24.0%，八个模型-稀疏组合全部领先 4 到 15 分；C4 校准在 Llama-3.1-8B 和 Qwen3-14B 上直接归零，SparseDecoding 都能救回非零分。\n\n## 所以呢\n\n这篇论文的价值不在 1.48 倍这个数字，而在它把「剪枝校准的分布偏移」这个盲区摆上了台面：社区调了几年的剪枝算法，校准数据却一直停留在静态语料，与部署时模型自回归生成的真实分布脱节。对要长文本输出又要压部署成本的团队（客服、写作助手、代码补全），「让模型先自己写一段，再拿这段去剪」是个几乎零成本的正确动作。而 SpMV 内核那半则提醒另一件事：半结构化稀疏的收益从来不是剪完就自动到账，Tensor Core 的 2:4 神话只覆盖 prefill——解码阶段的稀疏红利，得自己写 kernel 去拿。\n\n参考：arXiv:2610.12327","https:\u002F\u002Farxiv.org\u002Fabs\u002F2610.12327","7437aeb9-930c-4866-a2e9-48003c1a792b",[11,15,18,21],{"id":12,"name":13,"slug":13,"description":14,"color":14},"2d9c2fb0-2be5-4ad1-aedb-e9747addf355","compression",null,{"id":16,"name":17,"slug":17,"description":14,"color":14},"0ef8513a-0a26-42f0-b6f9-5b6dadded45c","efficiency",{"id":19,"name":20,"slug":20,"description":14,"color":14},"0a93ec8e-ea39-4693-81de-563ca8c173f7","inference",{"id":22,"name":23,"slug":23,"description":14,"color":14},"01598627-1ea6-4b27-a5d8-874971571a71","llm",[25],{"id":26,"lang":27,"title":28,"summary":29,"content":30},"312d85c0-94b7-4823-aad2-b5564cbce644","en","Westlake-Ant SparseDecoding: Decoding-Aware Pruning Speeds LLMs","SparseDecoding prunes LLMs on self-generated tokens: Qwen3-14B writing 1.85→4.18, A100 decoding 1.48x faster via a bitmask SpMV kernel.","Half of the bottleneck in large-model inference sits in decoding: batch-one autoregressive generation is memory-bound, so every emitted token requires a full pass over the weights in GPU memory. Hessian-guided training-free pruning is the classic remedy — zero out unimportant weights, read fewer parameters, decode faster. But SparseDecoding, a paper submitted to arXiv on October 8 by Westlake University and Ant Group (arXiv:2610.12327), points out an overlooked mismatch: the data used to calibrate pruning and the data the model actually consumes during decoding do not come from the same distribution.\n\n## Where Calibration Goes Wrong\n\nThe traditional pipeline (the SparseGPT line) computes Hessians from pre-collected natural corpora such as C4 to score weight importance. The problem: during decoding the model consumes its own generated tokens, not natural text. The paper measures that the activation discrepancy of a pruned model grows rapidly over the first few decoding steps, with all layers reaching 90% of their plateau by around step 12 — the distribution aligned at pruning time drifts away from real decoding, and long-form generation quality collapses accordingly.\n\nThe SparseDecoding fix is straightforward: instead of fixed corpora, calibration activations are collected from the dense model's autoregressive generation on task prompts, keeping only decode-step activations and discarding prefill. Both configurations use 1M calibration tokens and feed the same SparseGPT backend, so the only variable is where calibration data comes from.\n\n## The Kernel Is the Other Half of the Answer\n\nFixing calibration alone is not enough. The paper's second observation cuts deeper: existing sparse kernels that deliver real speedups — such as cuSPARSELt's 2:4 Sparse Tensor Core path — mainly optimize sparse matrix-matrix multiplication (SpMM), which suits prefill and large batches. Batch-one decoding is dominated by sparse matrix-vector multiplication (SpMV). Measured, the 2:4 Tensor Core path speeds prefill up by 1.31-1.46x yet delivers only 0.85-0.87x of dense throughput during decoding — slower than not pruning at all.\n\nThe team therefore wrote their own N:M SpMV kernel: bitmask indexing (every 32 input columns packed into one 32-bit mask word, exactly 16 set bits per word at 50% sparsity) replaces per-nonzero 32-bit column indices, paired with fixed-step traversal loops the compiler can unroll. The result: all four N:M patterns from 2:4 to 16:32 hit nearly identical throughput — each retains 50% of weights and identical metadata memory traffic — for 1.34-1.48x end-to-end decoding speedup on A100 (GPT-Fast framework, context length 512, median of 50 warmup and 50 measured iterations).\n\n## Numbers That Matter\n\nThe quality-side gains are more striking than the speedup. On the WritingBench long-form benchmark (1-10 scale), C4-calibrated 2:4-sparse Qwen3-14B scores just 1.85, while SparseDecoding calibration restores 4.18; Qwen3-32B climbs from 2.99 to 5.31. Llama models see modest gains (Llama-3.1-8B from 1.34 to 1.52), while the Qwen family gets a cliff-edge repair. On the ClassEval code-generation benchmark, 2:4-sparse Qwen3-32B Pass@1 rises from 9.0% to 24.0%, and SparseDecoding leads across all eight model-sparsity pairs by 4 to 15 points. C4 calibration collapses to zero on Llama-3.1-8B and Qwen3-14B; SparseDecoding recovers nonzero scores in both cases.\n\n## So What\n\nThe paper's real contribution is not the 1.48x figure but putting \"calibration distribution shift in pruning\" on the table: the community has spent years tuning pruning algorithms while calibration data stayed on static corpora, disconnected from the autoregressive distribution models face in deployment. For teams that need long-form output under deployment-cost pressure — customer service, writing assistants, code completion — \"let the model write a passage itself, then prune on that passage\" is a near-zero-cost correct move. The SpMV half carries its own lesson: semi-structured sparsity gains never arrive automatically. The 2:4 Tensor Core story only covers prefill — the sparse dividend in decoding must be claimed by writing your own kernel.\n\nReference: arXiv:2610.12327 (https:\u002F\u002Farxiv.org\u002Fabs\u002F2610.12327)","sparsedecoding-decoding-aware-pruning","2026-10-10T19:09:23Z","2026-10-10T19:10:22.430791Z","2026-10-10T19:10:22.430800Z",true,"agent",98,[39,48],{"slug":40,"tag_slug":40,"title_zh":41,"title_en":42,"intro_zh":43,"intro_en":44,"id":45,"is_active":35,"created_at":46,"modified_at":47},"ai-for-science","AI for Science 2026：从 UniPert 到 GPT-Rosalind 的硬核进化","AI for Science 2026: from UniPert to GPT-Rosalind","生命科学、化学材料、物理世界模型——AI 正在从\"语言工具\"变成\"实验伙伴\"。本专题收录 AI 在三大科学方向的关键节点：UniPert 统一基因与化学扰动空间、GPT-Rosalind 端到端生命科学推理、达摩院 AI 智能体 28 小时找到 4 种超导新材料、Anthropic Claude Science 把工作台做成标准品。","From language tool to lab partner — AI is reshaping life sciences, chemistry\u002Fmaterials, and physical world models. This topic covers the key milestones: UniPert unifying genetic-chemical perturbation spaces, GPT-Rosalind's end-to-end life-sciences reasoning, DAMO's AI agent discovering 4 superconducting materials in 28 hours, and Anthropic's Claude Science workbench going mainstream.","988a4300-5fab-41c4-b5d8-63711a2dc757","2026-09-10T01:34:15.296649Z","2026-09-10T01:34:15.296663Z",{"slug":49,"tag_slug":49,"title_zh":50,"title_en":51,"intro_zh":52,"intro_en":53,"id":54,"is_active":35,"created_at":55,"modified_at":56},"h3-series","MiniMax H3 系列：从开源权重到 35 倍吞吐","MiniMax H3 Series: from open weights to 35x throughput","MiniMax H3 自 2026 年 8 月开源以来节奏密集：官方把生成、参考与编辑收回一个模型；ComfyUI 当天压进 RTX 3060；摩尔线程 3 小时完成国产 GPU 适配；fal 后训练版把吞吐拉到 35 倍；FastH3 蒸馏再砍推理成本。本专题持续追踪 H3 的发布—开源—蒸馏—部署全链路。","Since MiniMax open-sourced H3 in August 2026 the pace has been relentless: one unified omni-modal model, same-day ComfyUI support down to an RTX 3060, a 3-hour Day-0 port to Moore Threads GPUs, fal's post-trained H3 Max at 35x throughput, and FastH3 distillation cutting inference cost further. This topic tracks the full H3 chain — release, open weights, distillation, deployment.","83ef0daa-3c31-4cb3-86ed-e5ee58654d5f","2026-09-08T07:33:19.942193Z","2026-09-08T07:33:19.942209Z",{"items":58},[59,64,69,74,79,84],{"id":60,"title":61,"news_slug":62,"published_at":63},"c83af54b-79ed-445c-9482-07d98c26c36b","BeaconKV:长推理会回头看,只压最近窗口的 KV 缓存注定丢东西","beaconkv-beacon-query-kv-cache-compression","2026-09-09T11:25:00+00:00",{"id":65,"title":66,"news_slug":67,"published_at":68},"bce9fc16-d31a-49be-b17b-f144619a58e2","LatentPress:上下文压成软令牌直读，7.7 倍压缩反超原文，训练仅动 0.1% 参数","latentpress-soft-token-context-compression","2026-09-05T19:06:09+00:00",{"id":70,"title":71,"news_slug":72,"published_at":73},"d31bc388-b6c7-41a5-a6e9-6f00657c7616","加GPU还是压KV缓存？arXiv论文：压缩省钱1.2到2倍，但36B是道坎","tensor-parallelism-vs-kv-compression-cost","2026-08-30T17:10:00+00:00",{"id":75,"title":76,"news_slug":77,"published_at":78},"d2c56430-f9a7-4844-bfab-a8651279a70c","ResKV 不再把 KV 缓存压缩等同于删词：给被淘汰的信息留一份残差账本","reskv-residual-kv-cache-compression","2026-08-03T10:43:23+00:00",{"id":80,"title":81,"news_slug":82,"published_at":83},"c32d3160-4e07-4128-890f-4e135aac2cce","CompactifAI 把 Llama 3.3 70B 砍到一半:Multiverse 在 Intel Xeon 6 上跑出 1.9 倍吞吐","compactifai-llama-3-3-70b-intel-xeon","2026-07-26T04:00:00+00:00",{"id":85,"title":86,"news_slug":87,"published_at":88},"5f745fe5-ea5d-453a-8b08-7dac524d1ac2","ACL 2026 综述 sKis：KV 缓存优化重塑为 LLM serving 系统学","acl-2026-skis-kv-cache","2026-07-12T18:15:00+00:00"]