There is a default consensus in the engineering community: wrap a model in a hard constraint like "the output must be valid JSON" and the model gets dumber.

That belief is widespread, and earlier work had reported accuracy drops in structured-output settings. A new paper submitted to arXiv on October 26 (arXiv:2610.35328) takes the question apart and redoes it properly. The conclusion: under a fully apples-to-apples setup — same prompt, same expected output format — constrained decoding does not inherently harm reasoning capability. The degradation people observed mostly stems from non-equivalent prompting and output formats, not from decoder-level masking itself.

The authors come from Télécom Paris (Institut Polytechnique de Paris) and Nokia Bell Labs, and the code is open-sourced as litelines (GitHub).

Splitting "constraint" into three types, not on/off

The core of the paper is a three-level taxonomy of constrained decoding:

  • Format constraints (FC): enforce output structure only — valid JSON, all required fields present.
  • Validity constraints (VC): on top of format, mask out semantically invalid answers, for instance dropping options that are not in the candidate set.
  • Reasoning constraints (RC): go further and constrain the reasoning trajectory itself, such as forcing the model to emit "let's think step by step" before answering.

The theoretical result: as long as one precondition holds — that the output constraint does not exclude the correct answer (Assumption 1 in the paper) — masking at the decoder level will not hurt performance. Gaining a benefit, though, requires an additional condition.

Eleven open-source models, four families, eleven tasks

The experimental scale is not small: 11 open-source models spanning four model families, of which 10 (Qwen3-1.7B, Qwen3-4B, Qwen3-8B, Qwen3.5-2B, Qwen3.5-4B, Qwen3.5-9B, Qwen3.5-27B, Llama3-8B, Gemma2-9B, SmolLM3-3B) were used for task solving, with Qwen3-32B serving as the extractor model in the two-step pipeline.

The datasets span two difficulty tiers. The standard tier is GSM8K and Last Letter Concatenation. The reasoning-intensive tier is AIME 2026 plus LastLetter-Extended, built by the authors by extending Last Letter from four names to eight. Both were released or constructed after every model under test, which reduces the potential for data leakage.

Three decoding strategies are compared: JSON (adding a reason field to the output), NL-to-Format (free-form reasoning first, then a second model extracts the format), and InWriting (single-pass free reasoning with a trigger token switching into constrained mode).

The numbers: taking the "degradation" apart, piece by piece

On the Last Letter task with plain-text answers, once FC+VC was applied, the constrained versions consistently beat the unconstrained ones: Qwen3.5-9B went from 86.8 to 89.9, Qwen3.5-2B from 62.9 to 68.7, Qwen3.5-4B from 84.1 to 86.9, and Qwen3-1.7B from 60.7 to 66.1.

Switching to JSON format, the gap persists: Qwen3.5-9B moves from 89.6 to 91.6.

The paper further decomposes where the gains come from, and finds that validity constraints deliver more consistent gains than format constraints, up to 17.4%. The reason is not mysterious: validity constraints inject semantic information about the correct answer, whereas format constraints merely bend the output shape into the required form — and when the model already complies, there is almost no room left to gain.

Reasoning constraints behave unevenly. Forcing the output to begin with "let's think step by step" generally helps on GSM8K and AIME 2026; but Qwen3.5-4B actually drops 6.6% on AIME 2026. The paper's conclusion is notably restrained: reasoning constraints are not a universal remedy. Their benefit depends on the task and the model, and specific constraints may exclude other valid reasoning paths.

Parse rate: the one unconditional benefit

If the accuracy gains still require conditions, parse rate does not.

On LastLetter-Extended, the parse success rate of unconstrained JSON output was 0 for Llama3-8B, 0 for Gemma2-9B, 0.7 for Qwen3-1.7B, 60.4 for Qwen3.5-9B, and 82.4 for Qwen3.5-27B. With format constraints applied, those numbers become 100, 100, 96.2, 99.3, and 93.8.

In other words, if your pipeline has a hard requirement that the model's output must be directly parseable, constrained decoding addresses precisely that. And it holds for large models too — Qwen3.5-27B is no exception.

Token cost: constraints are not always more expensive

Intuition says constrained decoding should burn more tokens, but the tables do not fully support that. On GSM8K, Qwen3-1.7B's JSON output averaged 221.1 tokens unconstrained, and dropped to 129.2 with FC+VC; on Last Letter the same model went from 192.0 to 73.4.

What is genuinely expensive is NL-to-Format — it requires an extra LLM call for extraction, which the paper states explicitly in the conclusion brings token usage close to double. The compromise is InWriting: apply the constraint only to the final output, and the paper reports fewer than 15 additional tokens in their experiments.

The boundaries the paper admits to

The authors do not oversell it, and the limitations section is specific:

  • Validity constraints were evaluated primarily on Last Letter and LastLetter-Extended, with only a preliminary content-filtering experiment on IMDb; extensions to chemistry (enforcing valid molecular structures) and content filtering remain future work.
  • Making reasoning constraints work requires prior knowledge of which trajectories are beneficial — information that is usually unavailable in real deployments.
  • The experiments use relatively simple JSON schemas; tasks demanding both extended reasoning and complex output structure are explicitly left to future work.

There is also a fun detail: the paper's own AI use statement says the authors used free versions of ChatGPT and Claude to draft and edit parts of the manuscript for readability, while all research questions, experiments, and code are their own.

What I make of it

Setting aside the question mark in the title, the most directly useful implication for engineering practice is not about accuracy at all — it is about methodology.

When you suspect "adding a constraint cost us points," do not blame the decoder first. The paper's causal ablations point at non-equivalent prompting and output formats: comparing a score from "unconstrained plus natural-language prompt" against "constrained plus JSON prompt" compares two different task setups, so the difference need not come from the constraint itself. The paper's own recipe is to fix the prompt and the expected output format, then run constrained against unconstrained.

The second point: the most certain benefit of constrained decoding lies on the engineering side, not in the score. Lifting parse rate from 0 to 100 removes an entire layer of error-handling code from a downstream pipeline.

The implementation is worth a look too. litelines is the accompanying implementation; pip install litelines installs it, the only dependency is outlines-core, it plugs into Hugging Face transformers, it hooks into generation as a logits processor, and it exposes three controls: allow_preamble, max_preamble_tokens, and trigger_token_ids.

So the question is left for you: in your own project, how many of the times you concluded "the model got dumber" were really the prompt's fault?