A large model running on a Raspberry Pi aces its question-answering benchmark — yet that competence may be an illusion manufactured by compression itself. A paper submitted to arXiv on August 16, titled "Large Models for Small Devices" (2608.15693), pours cold water on edge AI deployment with a cross-hardware empirical study: a model that compresses well does not necessarily deploy well.

Controlled Experiments Across Three Hardware Tiers

The research team (Subhransu Das, Jiaming Cheng, and seven co-authors) first surveyed more than twenty recent works that reported compression results on real constrained hardware, distilled practical deployment guidelines from them, and then ran their own controlled study: three roughly 1B-parameter LLM families alongside six image segmentation models, each subjected to structured pruning, GGUF quantization, or LoRA recovery, then deployed across GPU, CPU, and Raspberry Pi platforms, covering both question answering and segmentation tasks.

The first-layer conclusion is blunt: no single technique wins across all tasks.

  • For QA, quantization dominates: Qwen3.5 0.8B reaches 93.85 SQuAD F1 and 92 EM under Q5_K_M GGUF quantization, while structured pruning at the same precision costs 16 F1 even at a 1% pruning ratio.
  • For segmentation, the ranking reverses: default quantization leaves parameters and MACs unchanged, whereas pruning cuts model size by nearly 80% at near-constant mIoU.

Pruning Can Make the Model Bigger

The more counterintuitive finding concerns file size. Pruning is supposed to shrink models, but it breaks the super-block alignment of k-quant formats, inflating the deployed artifact by 21% to 49% instead. Combined with the pruned model's tendency toward longer, less format-compliant outputs, Raspberry Pi latency can rise up to 3.4x. In other words, what looks like saving resources may actually be wasting them.

The Sharpest Finding: Compression Can Fabricate Competence

The most sobering moment in the paper: a LoRA-recovered model variant stays fully parseable and holds 71% strict accuracy on BoolQ — everything looks normal on the surface. But inspect the prediction distribution, and 97 of 100 predictions land in a single class, with balanced accuracy at just 52.6%, barely better than guessing.

This means compression damage need not manifest as "visibly dumber" — it can silently collapse the model into a single-class predictor. Conventional benchmarks that only look at accuracy will completely miss this kind of degradation: the model appears to keep answering, while in reality it mindlessly bets on the same answer every time. The team explains these effects through neural-flow graph analysis and prefill/decode-level latency decomposition.

So What

For teams deploying models on edge devices, the operational advice condenses to three lines. Pick the compression method after fixing the task: quantization first for NLP, and pruning deserves evaluation only for structure-sensitive CV workloads. Never evaluate with accuracy alone — add distribution-sensitive metrics like balanced accuracy. And always measure latency on the target hardware rather than trusting paper compression ratios. The experiment code and artifacts are open-sourced on GitHub (github.com/Arnavvvkumar/deployment), and the full paper is at arXiv:2608.15693.

At a moment when everyone talks about squeezing large models onto small devices, this paper reminds us of something else: squeezing them in is the easy part — the hard part is confirming that what ends up on the device is still the same model.