[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-edge-llm-compression-raspberry-pi":3,"news-related-ddb7bc6c-6b6e-4797-ab76-d1aeab5a3002":38},{"id":4,"title":5,"summary":6,"content":7,"original_url":8,"source_id":9,"tags":10,"translations":24,"news_slug":31,"published_at":32,"created_at":33,"modified_at":34,"is_published":35,"publish_type":36,"image_url":14,"view_count":37},"ddb7bc6c-6b6e-4797-ab76-d1aeab5a3002","压缩得好≠部署得好:树莓派实测边缘 LLM,LoRA恢复模型100题押97个同答案","arXiv 新研究在 GPU、CPU、树莓派上实测三种边缘 LLM 压缩方案:量化在问答保分,剪枝在分割砍体积,但 LoRA 恢复的模型 100 题押 97 个同答案仍拿 71% 准确率——压缩能制造能力假象,部署前必须测平衡准确率。","一台树莓派上跑大模型,问答接近满分——但这份能力可能不是模型真有,而是压缩过程制造出来的错觉。8 月 16 日提交到 arXiv 的论文《Large Models for Small Devices》(编号 2608.15693)用一组跨硬件实测,给边缘 AI 部署泼了一盆冷水:压缩得好的模型,不一定部署得好。\n\n## 三档硬件上的对照实验\n\n研究团队(Subhransu Das、Jiaming Cheng 等 9 位作者)先调研了 20 多篇在真实受限硬件上报告过压缩结果的工作,提炼出部署指南,然后自己动手做对照实验:3 个约 1B 参数的 LLM 家族,配合 6 个图像分割模型,分别施加结构化剪枝、GGUF 量化、LoRA 恢复三种处理,再部署到 GPU、CPU 和树莓派三类平台,覆盖问答和分割两类任务。\n\n结论第一层很直白:**没有任何单一技术在所有任务上通吃**。\n\n- **问答任务,量化完胜**:Qwen3.5 0.8B 在 Q5_K_M GGUF 量化下拿到 93.85 的 SQuAD F1 和 92 的 EM;而同精度的结构化剪枝,在 1% 的剪枝率下就损失了 16 个 F1。\n- **分割任务,排名反转**:默认量化根本不减少参数量和 MACs,而剪枝能把模型体积砍掉近 80%,mIoU 却几乎不变。\n\n## 剪枝反而让模型变胖\n\n更反直觉的是文件体积:剪枝本该让模型更小,但它会破坏 k-quant 超块的对齐结构,最终部署产物反而膨胀 21% 到 49%。叠加剪枝后模型倾向生成更长、格式更不合规的输出,树莓派上的推理延迟最高被抬到 3.4 倍。换句话说,你以为在省资源,实际可能在浪费资源。\n\n## 最狠的发现:压缩能制造能力假象\n\n论文里最值得警惕的一幕是:一个经 LoRA 恢复的模型变体,输出格式完全可解析,在 BoolQ 上还有 71% 的 strict 准确率——看起来一切正常。但拆开预测分布,100 个预测里有 97 个被押到了同一个类别,平衡准确率只有 52.6%,几乎等于瞎猜。\n\n这意味着压缩带来的损伤可以不是「明显变笨」,而是「静默坍缩成单类别预测器」。常规 benchmark 只看 accuracy 时,这类退化完全会被漏检——模型看起来还在答题,实际上只是在无脑押同一个答案。团队用 neural-flow 图分析和 prefill\u002Fdecode 两阶段延迟分解,解释了这些现象背后的机理。\n\n## 所以呢\n\n对要在边缘设备上部署模型的团队,这篇论文给的操作建议浓缩成三句话:先定任务再选压缩方案,NLP 优先量化,结构敏感的 CV 场景再评估剪枝;评测不能只看 accuracy,必须加平衡准确率这类对预测分布敏感的指标;最终一定要在目标硬件上实测延迟,而不是看压缩率的纸面数字。论文实验代码和产物已在 GitHub 开源(github.com\u002FArnavvvkumar\u002Fdeployment),原文见 [arXiv:2608.15693](https:\u002F\u002Farxiv.org\u002Fabs\u002F2608.15693)。\n\n在所有人都在谈把大模型塞进小设备的当下,这篇论文提醒的是另一件事:塞进去不难,难的是确认塞进去的还是原来那个模型。","https:\u002F\u002Farxiv.org\u002Fabs\u002F2608.15693","7437aeb9-930c-4866-a2e9-48003c1a792b",[11,15,18,21],{"id":12,"name":13,"slug":13,"description":14,"color":14},"7ac06d8e-b074-4147-abfc-ffaa4c6b8744","ai-efficiency",null,{"id":16,"name":17,"slug":17,"description":14,"color":14},"2d9c2fb0-2be5-4ad1-aedb-e9747addf355","compression",{"id":19,"name":20,"slug":20,"description":14,"color":14},"01598627-1ea6-4b27-a5d8-874971571a71","llm",{"id":22,"name":23,"slug":23,"description":14,"color":14},"b49648f9-963e-4082-8684-3d085b7358fe","quantization",[25],{"id":26,"lang":27,"title":28,"summary":29,"content":30},"d97a4690-d28d-4465-a358-69287ff94dd6","en","Compresses Well Is Not Deploys Well: Raspberry Pi Tests Expose Edge LLM Illusions","A new arXiv study benchmarks structured pruning, GGUF quantization, and LoRA recovery across GPU, CPU, and Raspberry Pi. Quantization wins for QA, pruning wins for segmentation — but one LoRA-recovered model sent 97 of 100 predictions to a single class while keeping 71% accuracy, showing compression can fabricate competence.","A large model running on a Raspberry Pi aces its question-answering benchmark — yet that competence may be an illusion manufactured by compression itself. A paper submitted to arXiv on August 16, titled \"Large Models for Small Devices\" (2608.15693), pours cold water on edge AI deployment with a cross-hardware empirical study: a model that compresses well does not necessarily deploy well.\n\n## Controlled Experiments Across Three Hardware Tiers\n\nThe research team (Subhransu Das, Jiaming Cheng, and seven co-authors) first surveyed more than twenty recent works that reported compression results on real constrained hardware, distilled practical deployment guidelines from them, and then ran their own controlled study: three roughly 1B-parameter LLM families alongside six image segmentation models, each subjected to structured pruning, GGUF quantization, or LoRA recovery, then deployed across GPU, CPU, and Raspberry Pi platforms, covering both question answering and segmentation tasks.\n\nThe first-layer conclusion is blunt: **no single technique wins across all tasks**.\n\n- **For QA, quantization dominates**: Qwen3.5 0.8B reaches 93.85 SQuAD F1 and 92 EM under Q5_K_M GGUF quantization, while structured pruning at the same precision costs 16 F1 even at a 1% pruning ratio.\n- **For segmentation, the ranking reverses**: default quantization leaves parameters and MACs unchanged, whereas pruning cuts model size by nearly 80% at near-constant mIoU.\n\n## Pruning Can Make the Model Bigger\n\nThe more counterintuitive finding concerns file size. Pruning is supposed to shrink models, but it breaks the super-block alignment of k-quant formats, inflating the deployed artifact by 21% to 49% instead. Combined with the pruned model's tendency toward longer, less format-compliant outputs, Raspberry Pi latency can rise up to 3.4x. In other words, what looks like saving resources may actually be wasting them.\n\n## The Sharpest Finding: Compression Can Fabricate Competence\n\nThe most sobering moment in the paper: a LoRA-recovered model variant stays fully parseable and holds 71% strict accuracy on BoolQ — everything looks normal on the surface. But inspect the prediction distribution, and 97 of 100 predictions land in a single class, with balanced accuracy at just 52.6%, barely better than guessing.\n\nThis means compression damage need not manifest as \"visibly dumber\" — it can silently collapse the model into a single-class predictor. Conventional benchmarks that only look at accuracy will completely miss this kind of degradation: the model appears to keep answering, while in reality it mindlessly bets on the same answer every time. The team explains these effects through neural-flow graph analysis and prefill\u002Fdecode-level latency decomposition.\n\n## So What\n\nFor teams deploying models on edge devices, the operational advice condenses to three lines. Pick the compression method after fixing the task: quantization first for NLP, and pruning deserves evaluation only for structure-sensitive CV workloads. Never evaluate with accuracy alone — add distribution-sensitive metrics like balanced accuracy. And always measure latency on the target hardware rather than trusting paper compression ratios. The experiment code and artifacts are open-sourced on GitHub (github.com\u002FArnavvvkumar\u002Fdeployment), and the full paper is at [arXiv:2608.15693](https:\u002F\u002Farxiv.org\u002Fabs\u002F2608.15693).\n\nAt a moment when everyone talks about squeezing large models onto small devices, this paper reminds us of something else: squeezing them in is the easy part — the hard part is confirming that what ends up on the device is still the same model.","edge-llm-compression-raspberry-pi","2026-08-23T13:30:00Z","2026-08-22T17:08:22.678454Z","2026-08-22T17:08:22.678462Z",true,"agent",42,{"items":39},[40,45,50,55,60,65],{"id":41,"title":42,"news_slug":43,"published_at":44},"031e715e-c9d6-4855-83da-0515f33f0e3c","POCKET：35B MoE 1-bit 跑进 iPhone，27 tok\u002Fs","pocket-35b-moe-iphone-edge","2026-07-28T04:00:00+00:00",{"id":46,"title":47,"news_slug":48,"published_at":49},"c32d3160-4e07-4128-890f-4e135aac2cce","CompactifAI 把 Llama 3.3 70B 砍到一半:Multiverse 在 Intel Xeon 6 上跑出 1.9 倍吞吐","compactifai-llama-3-3-70b-intel-xeon","2026-07-26T04:00:00+00:00",{"id":51,"title":52,"news_slug":53,"published_at":54},"7f723663-8405-43aa-b31e-73efd714fa97","KV-Cache Grafting：冻结权重，Gemma-4-12B AIME 80%→93.3%","byte-exact-kv-cache-grafting","2026-07-17T06:20:00+00:00",{"id":56,"title":57,"news_slug":58,"published_at":59},"3af7d9f7-9cb3-43a3-a338-00d716c8053e","JoLT 用 Tucker + JL 残差把 KV 缓存压到 1\u002F3：让长上下文 LLM 推理不再被显存卡脖子","jolt-tucker-jl-kv-cache","2026-07-15T02:18:00+00:00",{"id":61,"title":62,"news_slug":63,"published_at":64},"5a5b1531-e1b2-469b-8064-772223231183","KronQ：Kronecker Hessian 拆掉 GPTQ 的 2-bit 墙","kronq-kronecker-hessian-gptq","2026-07-13T16:02:00+00:00",{"id":66,"title":67,"news_slug":68,"published_at":69},"53d67819-1d5c-4c1a-8dbb-1f49d9c76304","BiSCo-LLM 把 LLM 量化推过 2-bit 墙：Lookup-free 球面编码 + 类别恢复蒸馏，告别 VQ 码本","bisco-llm-2bit-quantization","2026-07-10T08:00:00+00:00"]