Background: Small LLMs on phones, but no open ledger

For the past two years, every small-model team (LFM2, Qwen3.5, Granite-4, and others) has been telling the same story: their model can deliver usable quality on a phone or laptop. The actual gap between "runs" and "runs well," though, has never had a public ledger. Model cards ship fp16 benchmark scores; device vendors ship marketing videos. The independent middle layer was missing.

Artificial Analysis, partnering with Liquid AI, has now published Pipette, a benchmark that pulls apart the deployment stack into five variables — model, quantization, runtime, device, workload — and measures them in the open (artificialanalysis.ai).

The timing matters. Late August, Google told Android developers to cut memory usage by February 2027 because AI data centers are soaking up DRAM. Microsoft reported an internal engineer burning $28,000 of AI in 28 days. Both stories point to the same pressure: as cloud inference gets more expensive, demand for "can the inference move to the device?" is real. But device inference is not "install an APK and you're done" — quantization, runtime, KV cache budget, and context length each decide whether a model actually works on your phone.

How the benchmark works

Eligible models are those that fit in 8 GB after quantization, including the KV cache at 8K context. Tests use Q4_K_M or smaller GGUF quantizations, served through llama.cpp. Devices covered at launch: iPhone 17 Pro (12 GB), Galaxy S26 Ultra (12 GB), MacBook Pro M5 Max, with AMD Ryzen AI Max+ 395 + Radeon 8060S coming soon. The intelligence suite averages five evaluations with equal weight: BFCL (640-task tool-calling subset), IFBench (294 prompts), AA-Omniscience (6,000 questions on factual knowledge + hallucination resistance), GPQA Diamond (198 graduate-level science questions), MATH-500 (500 math problems). Inference is measured at a fixed 1,024-token prompt plus 256-token response, recording end-to-end latency, prefill and decode throughput, and peak RAM. Each evaluation runs five repeats at pass@1, and inference runs gate on a platform-specific thermal/idle check before timing to reduce environmental noise.

The first release covers 23 models: Liquid AI's LFM2.5 family, Qwen3.5/3.6, Gemma 4 E2B/E4B/12B, IBM Granite 4.0/4.1, Mistral Ministral 3, Llama 3.2 1B/3B, TII Falcon-H1R-7B, Allen AI Olmo-3-7B-Think, Ornith-1.0/1.5-9B, Nanbeige4.2-3B, InclusionAI Ling-3.0-tiny, OpenBMB MiniCPM5-1B, NVIDIA Nemotron Nano 9B v2, AI9Stars G9v3 3B, plus Prism ML's 1-bit Bonsai-27B and Ternary-Bonsai-27B as the extreme low-bit cases (Liquid AI announcement, 8/24).

Four counterintuitive findings

MoE on a phone does not necessarily save memory. LFM2.5-8B-A1B is a Mixture-of-Experts model; only 1.5B of its 8.5B parameters activate per token. On the Galaxy S26 Ultra at 2,048 input tokens and Q4_K_M, it decodes 2.4× faster than Qwen3.5-4B and 2.6× faster than Ministral-3-3B-Instruct-2512. But its peak RAM still hits 5.29 GiB. Every expert weight has to live in memory, so "sparse activation" does not equal "small model." This is worth remembering whenever anyone pitches MoE as a small-model substitute.

Speed and quality are independent axes. MiniCPM5-1B and LFM2.5-1.2B-Instruct are both in the ~1B range, quantized to Q4_K_M. On the iPhone 17 Pro at 2,048 input + 256 output, MiniCPM5-1B finishes in 3.47 seconds versus 4.12 seconds for LFM2.5-1.2B-Instruct — MiniCPM is 15.8% faster. But the same quantized artifacts run on MATH-500, and LFM2.5-1.2B-Instruct scores 9.0 points higher. Neither dominates the other. Deployment now requires an explicit priority: latency-sensitive (MiniCPM) or math-sensitive (LFM). No silver bullet.

Architectural refinements matter more than parameter count for long-context behavior. Granite-4.0-H-350M and Granite-4.0-350M share the same 350M parameter count and Q4_K_M quantization. On the Galaxy S26 Ultra, scaling input from 256 to 4,096 tokens, the H variant keeps 78.4% of its decode throughput; the regular variant keeps only 33.8%. The extra architecture work buys more long-context headroom than shrinking parameters does.

Two models with nearly identical throughput can flip rankings depending on the task. Granite-4.1-8B and Ministral-3-8B-Instruct-2512, both at Q4_K_M, run on the M5 Max at 2,048 input tokens with only 2.4% difference in decode throughput and 1.2% difference in peak RAM. But on IFBench, Granite leads by 7.3 points; on GPQA Diamond, Ministral leads by 14.0 points. Similar numbers in the inference column do not mean interchangeable models in production — the workload distribution picks the winner.

Why this matters

Pipette covers the gap between "model card scores" and "what actually happens on your phone" with an open, reproducible methodology. Three things that were previously hard to argue now have data: same-parameter architectures degrade at very different speeds as context grows; MoE's "small activation, large memory" pattern on devices punctures the intuition that "fewer parameters means less"; and quantization-induced quality loss is highly task-dependent — you cannot extrapolate from one benchmark to the whole field. Liquid AI CEO Ramin Hasani's framing — "on-device behavior is a property of the deployed system, not the model in isolation" — has now been turned into something you can measure.

The next six months of industry moves will likely orbit this leaderboard. AMD's Ryzen AI Max+ 395 and Radeon 8060S are not yet on it; NPU paths are excluded for now because per-vendor kernel coverage isn't consistent. Expect the device-LLM selection logic to shift from "check the QPS benchmark table" to "find the Pareto curve for your device, your quantization, your target context length." That is the most concrete thing this release gives the whole industry.