Let the small model answer first, escalate to the large model only when it wavers — small-to-large routing is a standard recipe for cutting inference cost. A paper submitted to arXiv on October 16 put the idea through a systematic multilingual checkup: 3 model families, 4 languages, 3 task types, 72 configurations, and 42,288 predictions. The verdict: confidence-based routing is strongly task-dependent, and it nearly fails on knowledge-and-reasoning tasks. (arXiv 2610.23249)
How the experiment was built
The team (Bangladesh University, Georgia Southern University, and collaborators) picked three instruction-tuned families — Qwen3, Gemma 3, and Llama 3.2 — each with a 1B–1.7B small model paired with a 3B–4B large model, escalating only within the same family. Languages cover English, Bangla, Hindi, and Urdu; tasks are Belebele (reading comprehension), MMLU-ProX-Lite (knowledge and reasoning), and SIB-200 (topic classification). The confidence signal is the probability margin between the top two candidate answers, and four routing policies are compared: global confidence, random routing, equal-quota, and an oracle upper bound.
Two hard findings
First, task type decides whether routing lives or dies. On reading comprehension and topic classification, confidence identifies escalations worth making (mean correction-detection AUC of 0.612 and 0.705); at a 50% escalation budget it beats random routing by 1.40–4.67 and 2.54–5.63 percentage points. But on MMLU-ProX-Lite the mean AUC is just 0.510, essentially chance. The stated reason is blunt: in the highest-confidence quintile, small models still get reasoning questions wrong at high rates (Qwen3 63.45%, Gemma 3 76.89%, Llama 3.2 86.17% errors), only 15.59–21.89% of those confident errors get corrected by the same-family large model, and both models fail together on 63.79–67.33% of examples.
Second, "who pays for efficient inference" has no fixed answer. Under global confidence routing at a nominal 50% budget, the gap between the most- and least-escalated language reaches 30.33 percentage points (Qwen3 on Belebele). English is the least-escalated language in 5 of 9 settings (mean escalation 43.98%); switching to equal-quota routing pulls it back to 50% and actually raises English routed accuracy by 1.35 points — removing the language allocation gap costs almost nothing in aggregate accuracy.
So what
Two practical takeaways: treating confidence as a task-agnostic routing signal will bite you — run an AUC check per task type before shipping; and in multilingual products, an equal-quota constraint is a nearly free fairness insurance. The efficiency gains are real: at 25% escalation, wall time drops 27.5%–58.2% and energy drops 26.3%–62.6%. Caveats: the study covers only 1B–4B open-weight models and multiple-choice tasks — larger models, MoE, and free-form generation remain untested. Code is on GitHub. (GitHub)