Should a transit kiosk run a large language model? Dutch studio Continker offers the most complete evidence chain yet: a 955-case benchmark, posted to arXiv on Sep 9 (2609.10016), with code, dataset, and fine-tuned models all open-sourced under Apache 2.0.

What it tests

MetroLLM-Bench puts a language model in the kiosk's seat: it reads an operations policy written in prose, then handles routing, fare calculation, disruptions, accessibility, and adversarial input through tool calls, ending each case in a machine-checkable terminal state. Six real metro systems are covered — Atlanta, Doha, San Francisco, Taipei, Chicago, Beijing — spanning 37 to 414 stations and three fare models, across eleven case categories. Scoring has two tiers: Tier 1 packs 14 deterministic components covering route, fare, and tool-call correctness, needing no API key; Tier 2 adds 8 semantic components, six judged by Claude Haiku. The 955 cases are split 75/25 with 238 strictly held out for reporting. Twenty-six models from six vendors entered; twenty-three are ranked.

The headline numbers

On the held-out set, a 2.6GB Qwen 3.5 4B student (PEFT-tuned, Q4_K_M quantized) scores 91.32 on Tier 1 — above both GPT-5.6 tiers (luna at medium reasoning 90.63, sol at maximum 90.00) and within 0.05 points of GPT-5.4 full at maximum effort. The more sobering line is the control: a pure rule-based deterministic baseline reaches 84.6. The LLM's net edge over rules is roughly 6.7 points, concentrated in policy adaptation, compound scenarios, accessibility, and temporal reasoning — the parts rules cannot fully enumerate. Only two proprietary models crack the composite top six, both OpenAI; Muse Glimmer 30B leads at 92.03.

The diminishing returns of PEFT

Perhaps the most valuable output is not the leaderboard but the curve: fine-tuning gains decay monotonically as the base model grows. The 2B base gains +7.03 after PEFT, 4B gains +2.00, 9B +1.65, and 27B actually drops -0.91 — the stronger the base, the less an adapter can help, and past a point it hurts. Bootstrap CIs on the full 955-case matrix certify both the 4B gain (+1.72) and the 27B regression (-1.09). Reproduction is deliberately cheap: an M2 Max runs the 15-case probe at about 62 tok/s in 7GB of RAM, and the README offers three tiers from llama.cpp serving to full replication.

Two caveats

First, community discussion on Hugging Face already flags that the benchmark mostly scores final states; recovery behavior after tool-call failures — a timed-out fare lookup, a schedule API returning garbage, a card reader needing a second tap — has no separate scoring tier, and real kiosks die exactly there. Second, the Tier 2 LLM judge agrees with human annotators only 82% of the time, and the paper itself leans comparative claims on deterministic Tier 1. Serving configuration alone moves the Qwen 3.5-vs-3.8 comparison by 2.7 points — deployment details weigh as much as model choice.

The takeaway: on narrow, rule-dense vertical tasks, a 2.6GB local model is already good enough, and sovereign-deployment buyers (transit, healthcare, government) can treat this benchmark as a negotiating baseline. But the boundary of "matching the frontier" is also in the numbers — step outside the metro sandbox and the edge may not travel.

Refs: arXiv:2609.10016 (https://arxiv.org/abs/2609.10016); GitHub: continker/metrollm-bench (https://github.com/continker/metrollm-bench)