[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-llm-distillation-offline-top-k-chunked-kl":3,"news-related-183fb3be-e062-47e7-9591-7c2372e116c1":38},{"id":4,"title":5,"summary":6,"content":7,"original_url":8,"source_id":9,"tags":10,"translations":24,"news_slug":31,"published_at":32,"created_at":33,"modified_at":34,"is_published":35,"publish_type":36,"image_url":14,"view_count":37},"183fb3be-e062-47e7-9591-7c2372e116c1","LLM 蒸馏的显存瓶颈不只在教师模型：离线 Top-K 与分块 KL 把长上下文训练装回单卡","一篇新论文系统比较在线与离线知识蒸馏，并将融合分块 KL 损失用于稀疏 Top-K 教师分布。作者报告，离线蒸馏在单张 H200 上可保持近似训练损失，同时降低显存并提升吞吐；分块 KL 则避免完整词表 logits 常驻显存，让 32K 上下文的紧凑模型恢复训练能够在单卡完成。","# LLM 蒸馏的显存瓶颈不只在教师模型：离线 Top-K 与分块 KL 把长上下文训练装回单卡\n\n小模型要在低延迟、低成本或本地环境中部署，常见做法不是从零训练，而是先压缩大模型，再用知识蒸馏恢复能力。但蒸馏本身也可能昂贵：在线方案需要让教师与学生同时驻留显存，每一步还要重新运行教师前向计算。新论文《Efficient Knowledge Distillation for LLMs》把问题拆成两个独立瓶颈：教师模型的重复计算，以及语言模型输出头产生的完整词表 logits。\n\n## 第一刀：把教师移出训练循环\n\n作者先计算教师模型在每个 token 上的 Top-100 概率并缓存，之后学生只读取缓存训练。实验对象是从 Llama 3.1 8B Instruct 得到的约 3.2B 学生模型。在单张 H200、8K 上下文的配置中，离线方案与在线蒸馏给出近似的训练损失曲线；同时峰值显存从约 103GB 降到 78GB，单步时间从 25.9 秒降到 18.5 秒，吞吐从 237 提升到 331 TFLOP\u002Fs。论文把收益解释得很直接：教师只运行一次，缓存还能供多组消融实验重复使用。\n\n这里的关键并不是“离线一定优于在线”，而是作者用同一目标函数验证了一个工程交换：把持续占用 GPU 的计算，换成可复用的稀疏教师分布。对于需要反复试验蒸馏配方的团队，这个交换会直接改变实验成本。\n\n## 第二刀：不要生成完整词表 logits\n\n即使教师离开显存，学生输出头仍会产生随“序列长度 × 词表大小”增长的 logits 张量。作者提出融合分块 KL：按序列块完成输出投影、归一化与稀疏教师项计算，并在反向传播时重算局部结果，因此不需要保存完整 logits。代价是短上下文下会增加一些重算，但显存峰值转为随序列长度线性增长。\n\n在同一张 H200 的真实训练配置中，8K 上下文下，稠密 KL、仅前向分块和融合分块三种实现的峰值显存分别约为 78GB、62GB 和 58GB。到了 32,768 token，稠密方案估算接近 250GB、无法装入单卡；融合分块方案约为 128GB，因此能在 141GB 的 H200 上运行。这也是论文最有现实意义的结果：限制紧凑模型“长上下文恢复训练”的，未必是 Transformer 主体，而可能是最后一层输出与损失计算。\n\n论文还用一个只包含输出投影的受控实验隔离了这个机制。在 32K token 下，融合分块 KL 的峰值显存为 5.45GiB，稠密 KL 为 85.2GiB；在 256K token 下，融合方案使用 11.6GiB，而仅前向分块方案达到 134.2GiB。作者明确提醒，这组数据不是端到端 LLM 训练速度，不能和真实训练结果直接混用。\n\n## 真正值得拿走的结论\n\n这项工作没有声称发明新的蒸馏算法，而是给出一份可复现的工程配方：**先缓存教师 Top-K 分布，解除教师常驻；再把 KL 损失与输出投影分块融合，解除完整词表 logits 常驻。**消融结果还显示，只有中间层特征损失会让学生性能崩塌，logit 级 KL 是核心；在此基础上再加入隐藏状态特征损失，MMLU 与 GSM8K 才有小幅稳定提升。\n\n边界也很清楚：真实模型实验主要围绕一组 8B 教师与约 3.2B 学生展开，硬件和软件栈集中在 H200、Megatron-Bridge 与 ModelOpt，迁移到其他架构和设备仍需验证。论文与实现链接见[原文](https:\u002F\u002Farxiv.org\u002Fabs\u002F2608.03796)。\n\n所以，这篇论文最值得关注的不是一个更高分的学生模型，而是一条更实际的提醒：做小模型，不只要压缩参数，也要压缩训练过程里那些本来不必完整存在的中间张量。","https:\u002F\u002Farxiv.org\u002Fabs\u002F2608.03796","7437aeb9-930c-4866-a2e9-48003c1a792b",[11,15,18,21],{"id":12,"name":13,"slug":13,"description":14,"color":14},"7ac06d8e-b074-4147-abfc-ffaa4c6b8744","ai-efficiency",null,{"id":16,"name":17,"slug":17,"description":14,"color":14},"40269b40-7942-4650-9672-ed2e6524d37a","ai-technology",{"id":19,"name":20,"slug":20,"description":14,"color":14},"0a93ec8e-ea39-4693-81de-563ca8c173f7","inference",{"id":22,"name":23,"slug":23,"description":14,"color":14},"01598627-1ea6-4b27-a5d8-874971571a71","llm",[25],{"id":26,"lang":27,"title":28,"summary":29,"content":30},"904d5ee3-8403-457d-9878-9a13ad6311e3","en","LLM distillation's memory fix: offline Top-K and chunked KL","A new paper examines two practical bottlenecks in knowledge distillation for compact language models: repeatedly running the teacher and materializing full vocabulary-sized student logits. Its offline Top-K target cache preserves nearly identical training loss while reducing memory and improving throughput, while a fused chunked KL loss enables 32K-context recovery training on a single H200 GPU.","# The Hidden Memory Bottleneck in LLM Distillation: Offline Top-K Targets and Chunked KL Bring Long-Context Recovery Back to One GPU\n\nCompact language models deployed under strict latency, cost, or on-premises constraints are often not trained from scratch. A larger model is compressed, and knowledge distillation is then used to recover the resulting student's capabilities. That recovery stage can itself be expensive: online distillation keeps both the teacher and student in GPU memory and reruns the teacher's forward pass at every training step. The paper *Efficient Knowledge Distillation for LLMs* separates the expense into two bottlenecks: repeated teacher computation and the full vocabulary-sized logits produced by the student's language-model head.\n\n## First cut: remove the teacher from the training loop\n\nThe authors precompute and cache the teacher's Top-100 probabilities for every token. The student subsequently trains against this sparse cache instead of a live teacher. Their main setup uses an approximately 3.2B-parameter student derived from Llama 3.1 8B Instruct. On one H200 GPU with an 8K context, offline and online distillation produce nearly identical training-loss curves. Meanwhile, peak memory falls from about 103GB to 78GB, iteration time drops from 25.9 seconds to 18.5 seconds, and throughput rises from 237 to 331 TFLOP\u002Fs.\n\nThe practical point is not that offline distillation must always be superior. It is that the same teacher targets can be computed once and reused across many ablations. This trades recurring GPU computation and residency for a sparse, reusable target cache. For teams that need to test many recovery recipes, that trade can materially change the cost of experimentation.\n\n## Second cut: stop materializing full vocabulary logits\n\nEven after the teacher leaves GPU memory, the student's output head still creates a logits tensor that scales with sequence length multiplied by vocabulary size. The authors introduce a fused chunked KL loss. It processes the output projection, normalization, and sparse teacher terms in sequence chunks and recomputes local values during the backward pass, avoiding storage of the full logits tensor. The trade-off is additional recomputation at shorter contexts, but peak memory becomes linear in sequence length.\n\nIn real-model training on the same H200, the dense KL, forward-chunked, and fused-chunked implementations consume about 78GB, 62GB, and 58GB respectively at 8K context. At 32,768 tokens, the dense version is estimated to approach 250GB and cannot fit on one GPU. The fused implementation peaks at about 128GB and therefore fits within the H200's 141GB capacity. This is the paper's most operationally significant result: the component blocking long-context recovery for a compact model may not be the Transformer body, but the final output projection and loss.\n\nThe paper also isolates this mechanism in a controlled benchmark containing only the vocabulary output projection. At 32K tokens, fused chunked KL uses 5.45GiB of peak memory compared with 85.2GiB for dense KL. At 256K tokens, the fused implementation uses 11.6GiB, while the forward-chunked version reaches 134.2GiB. The authors explicitly caution that this is not an end-to-end LLM training benchmark; it isolates loss-kernel scaling and should not be mixed with the real-model throughput numbers.\n\n## What practitioners should take away\n\nThe work does not claim a new distillation algorithm. Instead, it offers a reproducible systems recipe: **cache sparse Top-K teacher targets to remove the live teacher, then fuse and chunk the KL loss with the output projection to remove persistent full-vocabulary logits.** Additional ablations show that an intermediate feature loss alone causes the student to collapse, while logit-level KL is indispensable. Adding a hidden-state feature loss on top of logit KL produces a small but consistent improvement on MMLU and GSM8K.\n\nThe boundaries are equally important. The real-model study focuses on one teacher-student pair—an 8B instruction-tuned teacher and an approximately 3.2B student—and the implementation is evaluated primarily on H200 GPUs with Megatron-Bridge and ModelOpt. Transfer to other architectures, accelerators, and frameworks remains to be validated. The paper and released implementation are linked in the [original source](https:\u002F\u002Farxiv.org\u002Fabs\u002F2608.03796).\n\nThe broader lesson is straightforward: building a smaller model is not only about compressing parameters. It also requires eliminating intermediate tensors that never needed to exist in full in the first place.","llm-distillation-offline-top-k-chunked-kl","2026-08-05T20:08:13Z","2026-08-05T20:09:44.061997Z","2026-08-05T20:09:44.062011Z",true,"agent",147,{"items":39},[40,45,50,55,60,65],{"id":41,"title":42,"news_slug":43,"published_at":44},"2a312a41-8284-4429-92a8-d68559800cbe","神经符号AI新突破：能耗降低100倍，机器人任务准确率显著提升","tufts-neuro-symbolic-ai-100x-energy-95pct","2026-05-18T16:01:00+00:00",{"id":46,"title":47,"news_slug":48,"published_at":49},"93637a71-d655-4aae-a7e0-f0e9a6383228","vLLM V0 迁移 V1：强化学习训练为何要把推理正确性放在首位","vllm-v0-v1-migration-servicenow-correctness","2026-05-07T04:10:00+00:00",{"id":51,"title":52,"news_slug":53,"published_at":54},"f7eaae9c-a0b0-437e-a2f9-77c8fd1bf59e","vLLM 2026 Q2 RL路线图：推理引擎为何要成为RL训练的一等公民","vllm-2026-q2-rl-roadmap-fp4-rdma","2026-05-07T01:00:00+00:00",{"id":56,"title":57,"news_slug":58,"published_at":59},"5da227db-f53d-4b07-a0c6-4ea16e04cd4d","CURE 用不确定性焦点做「block-parallel 投机解码」：端到端 2.66–3.49×、接受长度涨 4.2–7.5%","cure-block-parallel-speculative-decoding","2026-08-08T02:00:00+00:00",{"id":61,"title":62,"news_slug":63,"published_at":64},"e3f049f5-2f0d-48d2-8e88-246ef006fa16","LoopMTP 给循环 Transformer 装上前瞻路标：固定参数下让每一轮都做不同的事","loopmtp-latent-multi-token-loop-guidance","2026-08-04T13:13:09+00:00",{"id":66,"title":67,"news_slug":68,"published_at":69},"217f417d-1b9c-475b-99f4-e21e7c909711","MHAR 把 Transformer 残差流从「单车道」拆成 H 条独立路由:子空间第一次有权自己挑历史层","multi-head-attention-residuals-mhar","2026-08-01T07:30:00+00:00"]