The same model behaves like a different animal on a different phone. In their measurements, llama.cpp GPU prefill is 5.4x faster than CPU prefill on the OnePlus 15, yet 7.4x slower on the OnePlus 8 Pro; Qwen3-4B at 8-bit decodes 12.6-13.8 tokens/s on the OnePlus 15 but only 0.7-4.8 tokens/s on the 8 Pro. The hard part of on-device language models was never whether it can run - it is that every phone needs its own tuning pass. The Facebook app alone ships to more than ten thousand distinct phone and tablet models, and roughly two new open foundation models appear every week. Hand-tuning each task on each device is a luxury only the largest companies can staff.

How the factory runs

The team's answer, Mobile AI Factory, takes a task specification and a target phone, and produces a fine-tuned, quantized small language model sized for that device. The core is a loop: each iteration, an orchestrator LLM (Claude Sonnet 5) picks one intervention - tune hyperparameters (LoRA rank, learning rate, epochs), mine more real data from the task's sources, or have a locally hosted teacher (Qwen3.6-35B-A3B) generate synthetic examples aimed at the failure categories hurting accuracy most. Fixed rules do the janitorial work: any change that lowers accuracy is rolled back to the best model so far, and a contamination check removes rows that also appear in the held-out test set.

The search space is bucketed. Nine base models (SmolLM2, Gemma 3, Qwen3, Qwen3.5; 135M to 4B parameters) times three precisions (4-bit, 8-bit, 16-bit) give 27 combinations, grouped into five buckets by on-disk size and searched from smallest up. Accuracy goals adapt too: the initial target is the teacher's five-shot score with a floor of 0.8, and if it is reached too easily the orchestrator raises it and keeps pushing. The crucial component is the phone profiler: once per phone, it measures prefill and decode speed, peak memory, and energy across llama.cpp and MNN runtimes on CPU and GPU - 364 configurations across four phones - and reuses the profile across tasks, so the search only considers models the phone can actually run.

The scorecard

Across ten tasks and four Android phones (from a current flagship down to the Redmi 9A, whose GPU the inference engines cannot use for lack of OpenCL): on unseen test sets, the factory's small models beat both Claude Sonnet 5 and DeepSeek v4 Flash given five examples on five tasks, by 0.024 to 0.342, and match them on three more. The stricter the output format, the bigger the gap: on biomedical entity extraction, a 517 MB model scores 0.828 where five-shot Claude manages 0.652.

The search itself is frugal: on entity extraction, bucketed search lands within 3 points of exhaustively training all 27 variants while using 19x less GPU time. When the Galaxy S23's resource limits rule out the factory's best model for call summaries, it returns an alternative within 0.013. After the first fine-tune, search adds a median of 8 points on the smallest models and 2 on the largest; in seven of the nine runs where a raised goal triggered more training, the model improved by 3.6 points.

Spec-sheet reasoning fails

The control experiment is the kicker: given the full hardware and model specifications but no measurements, Claude Opus 5.5 misses the app's requirements in 25% of easy and moderate cases and 60% of hard ones - misjudging runtime memory and the right backend, neither of which appears on a spec sheet. The best configuration changes across phones, and reasoning cannot substitute for measurement.

So what

In their contributions list, the authors call this, to their knowledge, the first agentic system that builds a fine-tuned, quantized SLM for a given task and a given phone - priority is theirs to claim; the engineering facts stand on their own. The real wall for on-device AI is hardware fragmentation, not model capability, and the way through is turning one-model-per-phone into an automated pipeline. For independent developers, that idea is worth more than any single new checkpoint. One caveat: the paper says code is available on GitHub, but as of writing the repository still returns 404 - reproduction fans will have to wait.

Paper: https://arxiv.org/abs/2611.13509