Handing 10,000 database questions a day to a frontier API runs about $70,000 a year at list price. That is a typical bill structure for mid-sized companies: the questions themselves are routine, what is expensive is that every single one of them calls the most expensive model you can buy.

The paper that tackles this takes an unintuitive route. Instead of swapping in a stronger model, the UC Irvine team puts a cheap gate in front of the expensive one (paper: https://arxiv.org/abs/2611.17780).

A 7B model as the gate

LuthorSQL is a 7B non-autoregressive text-to-SQL model. The framing borrows Kahneman's split between a fast, automatic System 1 and a slow, deliberate System 2: the model only escalates a question to System 2 when it is unsure.

It does not write SQL token by token. A single forward pass produces scores for every slot of a bounded query-plan grammar, a deterministic compiler then turns those plans into executable SQL, up to 32 candidate plans get compiled and run, and a validator trained on hard negatives picks the final answer. The paper describes each piece as targeting a familiar failure mode of small models: constraining output to discrete slot decisions prevents syntax errors and hallucinated string literals, and inferring table joins from the foreign-key graph takes complex join clauses off the network's plate.

The backbone is a LoRA adapter over OmniSQL-7B, with both adapters sitting on the same frozen weights, so LuthorSQL is one 7B model with two small adapters rather than an ensemble of LLMs. The design draws on TypeSafe's Jev.

What 68.03% actually means

On BIRD's official hidden test set of 1,789 questions, LuthorSQL reaches 68.03% execution accuracy at 0.84 s per question on one GPU. With 8-bit weights it needs about 9 GB at the same accuracy and runs on Apple silicon, roughly 30 s per question on a base 24 GB M3 laptop.

There is a reference point you can check independently. BIRD's own single-model leaderboard (https://bird-bench.github.io/) lists the 32B Arctic-ExCoT at 68.19 on the test set and 68.32 on dev. The paper's single-adapter variant lands exactly on 68.19.

Cross-engine transfer is the other selling point. Because the model predicts an engine-agnostic query plan before compiling to SQL, the same weights transfer without retraining. Across all 1,534 BIRD dev questions: SQLite 60.17%, Snowflake 59.32%, MySQL 57.63%, DuckDB 57.43%, PostgreSQL 57.37%, BigQuery 57.11%. All of it inside a 3.1-point band.

The line item that matters

The cascade rule is simple: escalate to Claude Opus 5.5 only when the validator score falls below a threshold of 3.0. The paper picks that threshold on the DECISION split, where System 1 answers 45.4% of the questions on its own and the combined system scores 78.39%, 0.50 points below always calling Opus.

The savings come from registered experiments. On 227 new questions over databases published after Opus's training cutoff, calling Opus directly through the API, the cascade cuts API cost by 33.3% at equal accuracy, an interval of 26.5% to 39.1%, with accuracy at 95.15% versus 95.59% and p = 1.0. System 1 answers 102 of the 227 locally.

The control numbers are more modest. Three runs on 348 held-out BIRD questions saved 24.8%, 32.9% and 32.5%, 30.1% pooled. On 693 KaggleDBQA and Spider-Realistic questions the saving was 19.9%, with accuracy within 1.2 points, and the paper notes none of those differences were significant.

The less comfortable half

The paper also does something that matters more to me than the savings.

Given only a question and a database name, with no schema and no database tool, Claude Opus 5.5 generated executable queries that returned the gold result on 79-95% of instances across four public splits, and reproduced 48-92% of the gold SQL queries verbatim, down to dataset-specific alias conventions and backtick formatting.

The authors do not stop at calling that cheating. They add two controls. Rename every identifier in the test databases and accuracy does not collapse. Move to new questions over databases published after the training cutoff and accuracy holds at 95.6-97.4%, with Opus alone at 97.4% and the cascade at 94.3%. In other words the tool-assisted generalization is solid; the contamination sits mostly in the layer that recalls answers verbatim.

A separate line of evidence comes from Oracle AI's SPENCE paper (https://arxiv.org/abs/2604.17771), which probes contamination syntactically by generating 10 paraphrases of each question and watching whether accuracy degrades. Across the older benchmarks Spider, SParC and CoSQL, Kendall's tau clusters between -0.6 and -0.95, clearly negative, meaning accuracy falls as paraphrases drift; the newer BIRD sits near zero at -0.46 to -0.11. The two papers measure different things, one asking whether the model can recall the gold answer and the other whether it survives a rewording, but both point at the same thing: a public NL2SQL score cannot be read directly as generalization.

What the paper admits

The limitations section is unusually blunt. The grammar expresses 84.6% of BIRD dev gold queries, so questions needing CTEs, correlated subqueries, window functions or unseen templates usually go wrong, which is one reason to route them to an agent. The cascade has not yet been run on BIRD's hidden test set. The held-out tests are small, spanning 8, 27 and 8 databases, so intervals are wide and results vary between runs, and RESERVED's first registered run failed its registered bars. On BIRD dev the agent is much stronger on confident questions, so the threshold of 3.0 lost 2.93 points there. Costs are list prices applied to logged usage rather than actual invoices. The GitHub repository is currently marked as a private pre-release with the license undecided.

So what

The part worth keeping is probably not the 33.3%, but the fact that the paper also took the leaderboard scores of large models apart. When a model fits in 9 GB and runs on a laptop, yet can still reproduce a gold query verbatim, the evaluation method itself has become part of model capability. You have to check for yourself how much of a score was actually reasoned out.