[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-compression-guide-quant-distill-on-device":3,"topics-all":36,"news-related-b3fce899-9a5e-4c04-a582-2c6c444d33a7":55},{"id":4,"title":5,"summary":6,"content":6,"original_url":7,"source_id":8,"tags":9,"translations":23,"news_slug":29,"published_at":30,"created_at":31,"modified_at":32,"is_published":33,"publish_type":34,"image_url":13,"view_count":35},"b3fce899-9a5e-4c04-a582-2c6c444d33a7","压缩决策指南：量化、蒸馏与端侧部署的工程权衡","当 AI 应用撞上 100ms 延迟天花板，云端大模型的物理限制就成了无法绕开的瓶颈。近日一篇系统梳理 LLM 压缩决策的技术文章引发开发者关注，文章指出量化、蒸馏和端侧部署三条路径虽目标相似，但工程成本、质量表现和适用场景差异显著，团队往往缺乏清晰的决策框架。\n\n量化是最快速的压缩路径。FP16 已成实际基准，INT8 精度损失通常低于 1%，AWQ\u002FGPTQ 等高级方法通过识别敏感权重比朴素 INT4 表现更好。但 INT4 在 Agent 和工具使用工作负载上真实任务成功率下降 10–15%，代码生成和多步推理退化明显。NVIDIA Hopper\u002FBlackwell 架构上 FP8 是务实选择，吞吐量接近 INT8，质量接近 FP16。\n\n蒸馏需要完整训练流程换取任务专项速度。研究表明知识密集型任务（事实召回、实体提取）能在蒸馏中存活，而复杂推理、指令遵循链和多语言任务大幅退化。这意味着蒸馏适合狭窄稳定任务域，而非通用聊天。\n\n压缩顺序研究证实剪枝→蒸馏→量化顺序产生最佳大小缩减与能力保留平衡。先剪枝去除冗余结构，再蒸馏重建专项能力，最后量化提取最终效率收益。在蒸馏之前应用量化会使质量损失叠加。\n\n端侧部署方面，Jetson Orin INT8 量化 8B 模型每 token 达 8–12ms，Apple Silicon 通过 llama.cpp 有竞争力运行 7B 模型。驱动因素通常是硬性延迟要求、数据主权或规模化成本。\n\n文章最核心的建议是：在压缩任何模型之前先构建任务专项评估集。MMLU 和 HumanEval 衡量广泛能力，但你的产品功能有特定任务分布——在 MMLU 上得分低 2% 的模型，如果任务恰好压到压缩退化的能力，可能在实际用户查询上差 15%。赢下通用基准的模型，不一定是最能承受压缩的那个。\n\n对于 AI 工程团队而言，压缩不是一次性判断，而是随模型改进、硬件演进而持续进行的工程权衡。从量化开始、用蒸馏补足、端侧部署兜底，这套组合拳的关键在于评估先行——基准测试套件和蒸馏训练流水线在被需要之前就已构建完毕，而非在截止日期压力下临时添加。","https:\u002F\u002Ftianpan.co\u002Fzh\u002Fblog\u002F2026-04-17-model-compression-quantization-distillation-on-device","edd2e36a-855e-4e24-a09b-3037b9154dc8",[10,14,17,20],{"id":11,"name":12,"slug":12,"description":13,"color":13},"2d9c2fb0-2be5-4ad1-aedb-e9747addf355","compression",null,{"id":15,"name":16,"slug":16,"description":13,"color":13},"0ef8513a-0a26-42f0-b6f9-5b6dadded45c","efficiency",{"id":18,"name":19,"slug":19,"description":13,"color":13},"0a93ec8e-ea39-4693-81de-563ca8c173f7","inference",{"id":21,"name":22,"slug":22,"description":13,"color":13},"b49648f9-963e-4082-8684-3d085b7358fe","quantization",[24],{"id":25,"lang":26,"title":27,"summary":28,"content":13},"64dfc748-0469-4350-9cf8-e10511ed0fe3","en","The compression playbook: quantization, distillation, edge","TianPan published a comprehensive decision guide for model compression, covering quantization, distillation, and on-device deployment. The guide helps practitioners navigate the trade-offs between model size, inference speed, and quality for different deployment scenarios.","compression-guide-quant-distill-on-device","2026-05-31T08:10:00Z","2026-05-31T16:06:20.619356Z","2026-08-19T02:08:40.142862Z",true,"agent",144,[37,46],{"slug":38,"tag_slug":38,"title_zh":39,"title_en":40,"intro_zh":41,"intro_en":42,"id":43,"is_active":33,"created_at":44,"modified_at":45},"ai-for-science","AI for Science 2026：从 UniPert 到 GPT-Rosalind 的硬核进化","AI for Science 2026: from UniPert to GPT-Rosalind","生命科学、化学材料、物理世界模型——AI 正在从\"语言工具\"变成\"实验伙伴\"。本专题收录 AI 在三大科学方向的关键节点：UniPert 统一基因与化学扰动空间、GPT-Rosalind 端到端生命科学推理、达摩院 AI 智能体 28 小时找到 4 种超导新材料、Anthropic Claude Science 把工作台做成标准品。","From language tool to lab partner — AI is reshaping life sciences, chemistry\u002Fmaterials, and physical world models. This topic covers the key milestones: UniPert unifying genetic-chemical perturbation spaces, GPT-Rosalind's end-to-end life-sciences reasoning, DAMO's AI agent discovering 4 superconducting materials in 28 hours, and Anthropic's Claude Science workbench going mainstream.","988a4300-5fab-41c4-b5d8-63711a2dc757","2026-09-10T01:34:15.296649Z","2026-09-10T01:34:15.296663Z",{"slug":47,"tag_slug":47,"title_zh":48,"title_en":49,"intro_zh":50,"intro_en":51,"id":52,"is_active":33,"created_at":53,"modified_at":54},"h3-series","MiniMax H3 系列：从开源权重到 35 倍吞吐","MiniMax H3 Series: from open weights to 35x throughput","MiniMax H3 自 2026 年 8 月开源以来节奏密集：官方把生成、参考与编辑收回一个模型；ComfyUI 当天压进 RTX 3060；摩尔线程 3 小时完成国产 GPU 适配；fal 后训练版把吞吐拉到 35 倍；FastH3 蒸馏再砍推理成本。本专题持续追踪 H3 的发布—开源—蒸馏—部署全链路。","Since MiniMax open-sourced H3 in August 2026 the pace has been relentless: one unified omni-modal model, same-day ComfyUI support down to an RTX 3060, a 3-hour Day-0 port to Moore Threads GPUs, fal's post-trained H3 Max at 35x throughput, and FastH3 distillation cutting inference cost further. This topic tracks the full H3 chain — release, open weights, distillation, deployment.","83ef0daa-3c31-4cb3-86ed-e5ee58654d5f","2026-09-08T07:33:19.942193Z","2026-09-08T07:33:19.942209Z",{"items":56},[57,62,67,72,77,82],{"id":58,"title":59,"news_slug":60,"published_at":61},"c32d3160-4e07-4128-890f-4e135aac2cce","CompactifAI 把 Llama 3.3 70B 砍到一半:Multiverse 在 Intel Xeon 6 上跑出 1.9 倍吞吐","compactifai-llama-3-3-70b-intel-xeon","2026-07-26T04:00:00+00:00",{"id":63,"title":64,"news_slug":65,"published_at":66},"3cecce90-70b9-4bb3-b9b7-93e6b0c05105","D-Quant 用熵编码压 KV:2.26bit 近无损","d-quant-entropy-coding-kv-cache","2026-09-20T17:10:42+00:00",{"id":68,"title":69,"news_slug":70,"published_at":71},"ff6f65e1-28b2-4a48-b317-7870072ecfa9","VC-Attention低比特注意力:视频生成提速1.59倍","vc-attention-low-bit-video-attention","2026-09-17T13:30:00+00:00",{"id":73,"title":74,"news_slug":75,"published_at":76},"e91b3add-4f1d-48d3-ab7a-bd2c6e8c1765","QuIP 崩、OPTQ 降级:Kashin-DCT 在 4-bit 量化压力测试里活了下来","kashin-dct-2bit-llm-quantization","2026-09-12T15:10:00+00:00",{"id":78,"title":79,"news_slug":80,"published_at":81},"178aa5e5-2a4f-4a87-a97c-0da16295d96f","EMNLP 2026 OCGQuant:用通道配对治 NVFP4 陪葬误差,Qwen3-1.7B 接近 FP16","ocgquant-nvfp4-outlier-companion-grouping","2026-09-10T09:15:00+00:00",{"id":83,"title":84,"news_slug":85,"published_at":86},"c83af54b-79ed-445c-9482-07d98c26c36b","BeaconKV:长推理会回头看,只压最近窗口的 KV 缓存注定丢东西","beaconkv-beacon-query-kv-cache-compression","2026-09-09T11:25:00+00:00"]