One kind of failure in structured output is especially nasty: the model does not emit garbage, it emits a syntactically perfect tree that contains a topping the customer never ordered. The pipeline checks the format, checks the fields, runs the business logic, and the wrong order still goes through. Nothing downstream can see it. The paper calls this the silent error rate.

A paper submitted to arXiv on 12 November (arXiv:2611.17169, from the IDM Lab at ITMO University) puts numbers on it. It is not about how to train a stronger model. It is about a completely free design variable: what format you ask the model to write in.

Six formats, one tree

The experiment uses the PIZZA pizza-ordering semantic parsing benchmark: a training pool of 2,456,446 utterances generated from a grammar, a 348-utterance development set, and a 1,357-utterance test set written by people. The authors picked two sub-billion-parameter models, Qwen3.5-0.8B and the largely convolutional LFM2.5-350M, fine-tuned them with LoRA, and ran 300 configurations in total (6 formats x 5 training pool sizes x 5 seeds x 2 models).

The design is clean: six encodings, all lossless, each decoding back to the identical tree, ranging from 44.3 tokens per order down to 10.9. What matters more are the two matched-length control pairs the authors kept. exr is the dataset's native s-expression and json is a bracket notation of about the same length. dsl is a terse symbolic grammar using sigils such as +cheese, and words has the same structure with the markers written as ordinary words, extra cheese. The latter two differ by less than a token; the only difference is whether a marker is a symbol or a word.

Length is not the mechanism, surface form is

The intuition is that fewer output tokens leave less room for error. But when the paper regresses accuracy on output tokens, the slope is positive (removing tokens lowers accuracy) in 10 of 10 model-size cells, never more than 1.1 points per ten tokens, and it separates in only one cell. The sharper counterexample is exr-short: it only renames exr's constructors to single letters, cutting length from 44.3 down to 32.3 tokens, and Qwen loses 10.6 points of exact match, separating at every pool size. Meanwhile tagged, at half the length of exr, matches it on Qwen. Length does not predict accuracy.

The real contest is in the matched-length pair: words beats dsl in all 10 model-size cells, separating in 9 after Holm correction, leading by 7.8 points on Qwen at full pool (interval 6.1 to 9.5) and 5.4 on LFM2. What is counterintuitive is that on LFM2 words actually emits more tokens than dsl (12.0 against 11.4) and is still more accurate. Longer and more accurate means it is not winning by being short, it is winning by being words.

Against the native format the authors stay deliberately restrained: 87.5% against 85.8% exact match on Qwen is a match, not a win, and after Holm correction it does not separate on Qwen. They write it as matching the native format's accuracy with 4.1x fewer tokens.

Where the errors come from: negation

The paper also changes the evaluation lens. The usual metric is exact match, but what a deployment should actually watch is the silent error rate, the share of outputs that parse cleanly and are still wrong. The budget splits four ways: correct + silent + unparseable + refused = 1. Only the last two are visible to the caller, so they can be retried or handed to a human. Silent errors reach the customer.

On Qwen the words format has a 4.5% silent error rate against 10.3% for the native s-expression, a ratio of 2.3; on LFM2 it is 10.0% against 17.9%, a ratio of 1.8. These are the paper's own measurements on sub-billion models, and whether they hold on larger models or other tasks is still open. More interesting is what changed in the errors. On Qwen, negation accounts for only 5% of words' silent errors against 19% to 34% for the other formats. Expressed as a rate per order, Qwen makes a silent negation error on 2.0% of orders with exr and 0.2% with words. The reason is visible in the output: words writes no onions the way customers say it, while the other formats translate it into a NOT construct or a sigil. One more detail: words invents entities the customer never mentioned 5.5x fewer times on Qwen.

The bill comes due

The paper does not stop at the good numbers. The guardrail trained into every run (refusing off-menu input, applying in-utterance self-corrections) refuses held-out negatives 95.9% to 99.0% of the time on Qwen, and exact match on corrections runs 62.9% to 75.3% on Qwen. The classic cheese failure, where a customer says a pizza with cheese, wait no cheese, only tomato sauce, is fixed in 60 of 60 full-pool runs.

But strict refusal has a price, and it is priced explicitly: 2.1% to 5.4% of real test orders are refused on Qwen, and 6.3% to 10.1% on LFM2. Reading them, they are genuine orders in human phrasing that the synthetic grammar never produces. Almost every refused order contains a word the training grammar never uses. An order with a greeting, with please or with thanks (289 of the test orders) is refused 6.1% of the time by Qwen words against 2.6% for other orders. Format rescues you from hallucination; it does not rescue you from politeness it never saw.

Latency is pure profit: words answers 3.7x faster than exr on Qwen and 4.4x on LFM2, and GPU memory barely moves, between 1,704 and 1,806 MiB. The appendix adds a device demonstration: Q8_0 weights, a 795 MiB file, 350 MiB resident, running on an iPad through llama.cpp with Metal offload, at a median of 167 ms per order. The authors note this is a demonstration on ten orders, not an evaluation.

Boundaries, and whether it travels

The honest limits: one dataset; both models are under a billion parameters and the paper itself says the effect may shrink with scale; the guardrail data was generated by the authors; and the exported LFM2 GGUF agrees with its trained run on only 93% of orders.

But the mechanism has independent corroboration. The EMNLP 2024 paper 'Let Me Speak Freely?' (on ACL Anthology) reports, from a different team and a different set of models, that restricting output to structured formats degrades reasoning, and that stricter constraints degrade it more. This paper's related-work section lists 'forcing an LLM into JSON costs reasoning accuracy' as an existing result. The LREC 2026 paper 'Format Matters' (on HAL) measures output format as a first-class hyperparameter across 13 open-weight models, 4 slot-filling and 3 NER benchmarks, and finds statistically significant swings of 2 to 46 F1 points. Three different teams, three different sources, the same conclusion: output format is not a display detail, it changes how the model behaves.

So what

Output format is one of the few levers you can pull without retraining, without more data, and without paying anything at inference time. But measure your own silent error rate before touching the format. If all you track is exact match, then exact match is the metric that will mislead you.