[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-lfm25-qad-quantization-aware-distillation-edge":3,"news-related-b2d34fba-93ee-469b-9f64-6a7568f89745":38},{"id":4,"title":5,"summary":6,"content":7,"original_url":8,"source_id":9,"tags":10,"translations":24,"news_slug":31,"published_at":32,"created_at":33,"modified_at":34,"is_published":35,"publish_type":36,"image_url":14,"view_count":37},"b2d34fba-93ee-469b-9f64-6a7568f89745","Liquid AI 用量化感知蒸馏,把 LFM2.5 4-bit 精度拉回 97%","Liquid AI 公布 LFM2.5 系列 Q4_0 GGUF:通过量化感知蒸馏,QAD 把 PTQ 损失掉的 BF16 精度挽回 96.5%-97.4%,在 MacBook、NucBox、Snapdragon 手机和树莓派上吞吐超过 Q5_K_M,直接对位 230M 到 2.6B 四个边缘尺寸。","## 为什么 Q4_0 是边缘部署的\"硬骨头\"\n\n要在 16GB 内存的 MacBook 或一台树莓派上跑一个能聊天的 LLM,模型权重必须被压到 4-bit 以下。GGUF 生态里最常见的 4-bit 格式是 Q4_0 和 Q4_K_M:前者最快但最糙,后者略好但吞吐略慢。Q5_K_M 通常比 Q4_K_M 再涨 1-2 个百分点的能力,代价是文件大 25%。这意味着**任何在 Q4_0 上做的精度恢复,都可以\"白嫖\"到一份比 Q4_K_M 更快、且接近 Q5_K_M 精度的权重。**\n\n传统做法是 post-training quantization(PTQ),把训好的 BF16 权重直接 round 到 4-bit。这个过程的精度损失在 LFM2.5-230M 这种小模型上尤其严重,因为权重之间的余量本来就薄。PTQ 的 Q4_0 GGUF 在 GPQA Diamond、MMLU-Pro、IFEval、IFBench、Multi-IF、BFCLv4 等六到七项任务上的平均,只能拿到 BF16 大约 80-90% 的水平,240M 和 350M 的相对损失明显比 2.6B 大。\n\n## Liquid AI 的解法:训练时就教学生\"接受 4-bit\"\n\nLiquid AI 在 8 月 19 日释出的更新里,把 Q4_0 的 4 个尺寸(230M \u002F 350M \u002F 1.2B-Instruct \u002F 2.6B)用 **Quantization-Aware Distillation(QAD)**重训了一次。核心思路:让一个高精度的 teacher 模型在训练阶段就把\"被量化过\"的 student 模型教到位——student 不是事后被 round,而是全程在模拟 4-bit 量化的反向传播里学。发布的 GGUF 和原生 Q4_0 是同一个文件格式、同一个 tensor 布局、同一条 llama.cpp 路径,所以**没有吞吐损失**:QAD Q4_0 用的是原生 Q4_0 的速度,精度却向 Q5_K_M 看齐。\n\n## 数字到底差多少\n\nLiquid AI 在四个真机后端上跑了完整 benchmark 套件:\n\n- **MacBook Pro(Apple M5 Max)**:GPU 推理,256 token prompt prefill 后生成 100 token。\n- **NucBox EVO-X2(AMD Ryzen AI Max+ 395)**:同上。\n- **Samsung Galaxy S26 Ultra(Qualcomm Snapdragon 8 Elite Gen 5)**:Arm CPU 推理。\n- **Raspberry Pi 5(Broadcom BCM2712)**:Arm CPU 推理,小模型用 32 token decode-only 测吞吐。\n\n恢复率定义:BF16→PTQ Q4_0 的精度差,QAD 关闭了多少百分比。四个尺寸的结果是 **70.6%(230M)\u002F 73.4%(350M)\u002F 65.5%(1.2B)\u002F 48.4%(2.6B)**;折算成\"QAD Q4_0 保留 BF16 多少\"是 **97.1% \u002F 96.5% \u002F 97.4% \u002F 96.6%**。也就是说,即使是被 PTQ 砸得最狠的 230M,QAD 也能把 Q4_0 的精度抬到跟 Q5_K_M 在 run-to-run 方差内打平。1.2B 比 Q5_K_M 差不到 0.5 分。\n\n在 1.2B 和 2.6B 上,QAD Q4_0 比标准 Q4_K_M 快 **3-14%**,精度持平;在 230M 和 350M 上,它比 Q5_K_M 快 **4-33%**,精度打平。它也追平了 Unsloth 之前发布的 UD-Q4_K_XL——一个被社区视为 4-bit SOTA 的外部 PTQ 检查点。\n\n## 这件事为什么值得看\n\nQAD 不是新概念,但工业界把它当成\"Liquid AI 的工程输出\"而不是论文概念来跑完整 benchmark 套件,而且和 PTQ、Q5_K_M、Q4_K_M、UD-Q4_K_XL 做了**同一组对照实验**,而不是只比\"自己 vs 自己\"——这件事的边际信息量其实很高。它把\"Q4_0 = 廉价但是不能上线\"这个长期共识推翻了。\n\n对边缘部署的实际意义是:在 16GB MacBook、Ryzen AI Max+ NUC、Snapdragon 8 Elite 手机和树莓派上,**Q4_0 现在能跑出 Q5_K_M 的精度,而体积和速度都更优**。GGUF 仓库已经在 Hugging Face 公开,文件命名带 `QAD-Q4_0` 后缀,llama.cpp 直接拉。LLM 推理里,文件大小和吞吐通常比 benchmark 分数更影响能不能上线——这次的收益正好打在这条线上。\n\n来源:Liquid AI 官方博客《LFM2.5 Q4_0: Quantization-Aware Distillation for Edge Deployment》(https:\u002F\u002Fwww.liquid.ai\u002Fblog\u002Fqad),以及 Hugging Face 团队博客《LFM2.5 Q4_0 Checkpoints from Quantization-Aware Distillation》(https:\u002F\u002Fhuggingface.co\u002Fblog\u002FLiquidAI\u002Fqad)。两个发布均在 2026 年 8 月 19 日公开,benchmark 套件与硬件配置完全一致。","https:\u002F\u002Fwww.liquid.ai\u002Fblog\u002Fqad","511bb1e6-a31f-4dc1-929b-9a7582e67447",[11,15,18,21],{"id":12,"name":13,"slug":13,"description":14,"color":14},"0ef8513a-0a26-42f0-b6f9-5b6dadded45c","efficiency",null,{"id":16,"name":17,"slug":17,"description":14,"color":14},"01598627-1ea6-4b27-a5d8-874971571a71","llm",{"id":19,"name":20,"slug":20,"description":14,"color":14},"b9bd9039-fcdb-41a8-b85b-fc1587def2b9","open-source",{"id":22,"name":23,"slug":23,"description":14,"color":14},"b49648f9-963e-4082-8684-3d085b7358fe","quantization",[25],{"id":26,"lang":27,"title":28,"summary":29,"content":30},"a3377d06-c6ae-4d08-a304-cca5d497d26b","en","Liquid AI uses quantization-aware distillation to claw LFM2.5 4-bit accuracy back to 97%","Liquid AI shipped Q4_0 GGUF checkpoints for the LFM2.5 family (230M \u002F 350M \u002F 1.2B \u002F 2.6B) trained with Quantization-Aware Distillation. QAD recovers 96.5%-97.4% of BF16 accuracy, matches Q5_K_M quality within run-to-run variance, and beats Q4_K_M by 3-14% throughput on MacBook, NucBox, Snapdragon 8 Elite phones, and Raspberry Pi 5.","## Why Q4_0 is the edge-deployment hard case\n\nTo run a chat-capable LLM on a 16GB MacBook or a Raspberry Pi, weights have to drop below 4 bits. In the GGUF ecosystem the most common 4-bit formats are Q4_0 and Q4_K_M: the former is the fastest but the roughest, the latter slightly better at slight throughput cost. Q5_K_M typically adds another 1-2 accuracy points over Q4_K_M, at the cost of 25% more file size. That means **any accuracy recovery done on top of Q4_0 effectively gives you something faster than Q4_K_M with Q5_K_M-like accuracy for free.**\n\nThe traditional approach is post-training quantization (PTQ), where you round trained BF16 weights directly to 4 bits. The accuracy hit is particularly severe on small models like LFM2.5-230M, where weight headroom is thin to begin with. PTQ's Q4_0 GGUF averages only about 80-90% of BF16 on the six-to-seven task suite covering GPQA Diamond, MMLU-Pro, IFEval, IFBench, Multi-IF, and BFCLv4, and the relative loss is clearly worse at 230M and 350M than at 2.6B.\n\n## Liquid AI's fix: teach the student to accept 4-bit during training\n\nIn the August 19 release, Liquid AI retrained the Q4_0 variants of all four sizes (230M \u002F 350M \u002F 1.2B-Instruct \u002F 2.6B) with **Quantization-Aware Distillation (QAD)**. The core idea: a high-precision teacher model trains a \"quantized-aware\" student during the training loop — the student is not rounded after the fact but learns end-to-end through simulated 4-bit gradient backpropagation. The released GGUFs share the same file format, tensor layout, and llama.cpp path as native Q4_0, so **there is no throughput penalty**: QAD Q4_0 runs at native Q4_0 speed while matching Q5_K_M accuracy.\n\n## What the numbers actually look like\n\nLiquid AI ran the full benchmark suite on four real hardware backends:\n\n- **MacBook Pro (Apple M5 Max)**: GPU inference, 100 generated tokens after a 256-token prompt prefill.\n- **NucBox EVO-X2 (AMD Ryzen AI Max+ 395)**: same protocol.\n- **Samsung Galaxy S26 Ultra (Qualcomm Snapdragon 8 Elite Gen 5)**: Arm CPU inference.\n- **Raspberry Pi 5 (Broadcom BCM2712)**: Arm CPU inference; small models use a 32-token decode-only throughput run.\n\nRecovery is defined as the percentage of the BF16→PTQ Q4_0 quality gap that QAD closes. Across the four sizes the results are **70.6% (230M) \u002F 73.4% (350M) \u002F 65.5% (1.2B) \u002F 48.4% (2.6B)**; translated to \"QAD Q4_0 retains how much of BF16\", that is **97.1% \u002F 96.5% \u002F 97.4% \u002F 96.6%**. Even on 230M — the model PTQ hit hardest — QAD lifts Q4_0's accuracy to within run-to-run variance of Q5_K_M. On 1.2B it trails Q5_K_M by less than 0.5 points.\n\nOn 1.2B and 2.6B, QAD Q4_0 runs **3-14% faster** than standard Q4_K_M at parity accuracy. On 230M and 350M, it runs **4-33% faster** than Q5_K_M at parity accuracy. It also ties Unsloth's UD-Q4_K_XL, which the community has been treating as the 4-bit SOTA external PTQ checkpoint.\n\n## Why this is worth looking at\n\nQAD is not a new idea, but industry releasing it as an engineering output (rather than a paper concept) and running the full benchmark suite against PTQ, Q5_K_M, Q4_K_M, and UD-Q4_K_XL in the **same controlled comparison** — rather than just \"ours vs ours\" — gives this release high marginal information. It overturns the long-standing assumption that Q4_0 means \"cheap but not deployable\".\n\nThe practical edge implication: on a 16GB MacBook, a Ryzen AI Max+ NUC, a Snapdragon 8 Elite phone, and a Raspberry Pi, **Q4_0 now runs at Q5_K_M accuracy while being smaller and faster**. The GGUFs are public on Hugging Face, with filenames suffixed `QAD-Q4_0` and consumed by llama.cpp out of the box. In LLM inference, file size and throughput usually decide shippability more than a benchmark number — and this release lands exactly on that axis.\n\nSources: Liquid AI's official blog post \"LFM2.5 Q4_0: Quantization-Aware Distillation for Edge Deployment\" (https:\u002F\u002Fwww.liquid.ai\u002Fblog\u002Fqad), and the Hugging Face team blog \"LFM2.5 Q4_0 Checkpoints from Quantization-Aware Distillation\" (https:\u002F\u002Fhuggingface.co\u002Fblog\u002FLiquidAI\u002Fqad). Both went live on August 19, 2026, with identical benchmark suites and hardware configurations.","lfm25-qad-quantization-aware-distillation-edge","2026-08-20T11:00:00Z","2026-08-20T09:07:26.342384Z","2026-08-20T09:07:26.342395Z",true,"agent",109,{"items":39},[40,45,50,55,60,65],{"id":41,"title":42,"news_slug":43,"published_at":44},"48e1c261-a40a-4c71-9cba-450a459e6ad3","4-bit 模型反超全精度:QAH 把量化从性能税变成第二次蒸馏","quantization-aware-healing-hypernova-60b","2026-08-25T17:20:00+00:00",{"id":46,"title":47,"news_slug":48,"published_at":49},"c197245e-0a8a-4028-9ad2-6547bdc01be5","BitNet 团队把 1.58-bit 量化推进到 embedding:检索向量也可以\"训练时就压\"","bitnet-1-58bit-embedding","2026-07-18T03:00:00+00:00",{"id":51,"title":52,"news_slug":53,"published_at":54},"93981299-c75e-4140-ad2c-ae68fb5d27ce","CAT-Q：512 样本把 235B LLM 压到 1.58-bit，成本降 10 万倍","cat-q-1-58-bit-512-samples-icml-oral","2026-06-26T20:25:00+00:00",{"id":56,"title":57,"news_slug":58,"published_at":59},"cac485ab-a429-4ebc-88c6-a1f924f978ff","AWS 开源 KeysAndValues:微调时就让模型学会“遗忘”,单张 A100 撑住 128K","aws-keysvalues-sparse-attention-finetuning","2026-08-26T05:20:00+00:00",{"id":61,"title":62,"news_slug":63,"published_at":64},"c94bdf86-5de9-49fe-8c98-0f5c47611bfe","SGLang v0.5.18 发布:大模型冷启动提速 2.38 倍,710 个 PR 都改了什么","sglang-v0-5-18-cold-start-2-38x","2026-08-24T23:15:00+00:00",{"id":66,"title":67,"news_slug":68,"published_at":69},"988bbfb8-672a-4c6c-98f7-3a170b6bd8b3","Macaw 把 LFM2.5 装进 1.5GB:4-bit 端侧 LLM 跑 Mac 控制工具链","macaw-lfm25-15gb-edge-mac-agent","2026-08-24T06:00:00+00:00"]