Mathematical modeling is a competition problem, and also the interface layer of real work.

A deliberately under-specified real-world problem shows up, and you have to set the assumptions yourself, pick a model, write the code, run the solver, and hand back a conclusion that holds up. If any link in that chain breaks, the answer is worthless. For the past two years, agent evaluation has mostly measured the tail end of that chain — whether the code runs, whether the number is right.

The new TB-MathModel puts the whole chain inside the evaluation. The tasks themselves are not exotic: the 2024 CUMCM A dragon-dance problem, where the agent reads the statement, looks at the figures, writes code, and returns one JSON artifact carrying the final speed peak. What is awkward is that the evaluation itself stumbled before the models did.

The paper records that 12 evaluated models reported the correct peak of 1.246266 m/s, yet were held to 49.82% because the reference answer itself was wrong. Two other models matched that erroneous reference value and scored 100%. Here is the error: the human Outstanding-award paper maximized over integer seconds and stepped over the true continuous-time peak at t=14.48 s, producing 1.406276 m/s; multiplied by the 1.604793 amplification ratio, that becomes 2.2568 m/s, above the 2 m/s cap stated in the problem. This "correct answer" was physically inadmissible.

The team dug the problem out themselves and corrected the anchor. The paper writes it plainly: the models were right and our anchor was wrong.

Turning modeling into something a machine can score

TB-MathModel carries 18 real contest modeling problems: CUMCM (Chinese) and MCM (English), problems A, B and C for each of 2023 through 2025, nine each. The verbatim statements, official attachments, and figures cropped from the PDFs all ship into a Docker container. The agent has to read the statement, open the spreadsheets, look at the figures, write code, run the solver, and return one JSON artifact. The container is capped at 2 CPUs, 8 GB of memory, no GPU, a 4-hour timeout, with the network disabled.

The scoring method is the point. The paper lays three existing approaches side by side: Mamo has 1,209 tasks but only checks solver fragments; ModelingBench with 68 tasks and MM-Bench with 111 tasks both grade reports with an LLM judge, which the paper charges with "rewarding fluent prose without establishing that the reported numbers are correct or even admissible". TB-MathModel scores only the numeric core of the final question, using an executable verifier.

Two design choices matter more than the leaderboard does.

The first is direction-aware, one-sided scoring. Every metric explicitly declares whether it is higher_is_better, lower_is_better, target_value, or exact_value, and meeting or beating the anchor never scores below 1.0. Anchor-based scoring has a built-in pathology: copying the paper scores well, while beating the paper scores badly. One-sided scoring closes that hole.

The second is a panel of 99 human Outstanding-award papers rather than one designated "correct answer". Of the 99, 94 supply the quantity being scored; each task gets between 1 and 18 papers with a median of 4, and one paper equals one vote. Mathematical modeling admits no single right answer, so promoting one team's choices to ground truth is itself the error.

Feasibility handed to Lean 4 to grade

The sharper layer is feasibility verification.

The outer layer is a programmatic gate covering 18/18 tasks, sorting hard errors into six decidable invariant forms: domain and range, integer and binary, ordering, exact identity, finite domain, and partition or sum. The heliostat field task, for instance, requires a rated power of at least 60 MW and mirror mounting heights between 2 and 6 meters.

The inner layer is Lean 4: it re-proves the same invariants as theorems, generating 624 kernel-checked obligations across the 18 tasks, with no Mathlib, and with all values scaled to integers so there is no floating-point tolerance. The trust audit (#print axioms) reports only propext and Quot.sound, with no sorryAx and no native_decide.

This is not using Lean 4 to solve anything — solving is the model's job. It is using Lean 4 to audit the benchmark itself: every decision the gate makes needs a machine-checkable theorem behind it.

The gate hard-invalidated 7 cells in total, concentrated on the heliostat task with 4, plus one each on smoke-screen, Juneau, and Olympic.

Expensive does not always win, and paying more can hurt

Across seven multimodal entries, three seeds per task. O-Eval is the headline metric; Multi-BO is the gain index against the DeepSeek-Flash baseline, where 1.000 means level with the baseline and 1.132 means closing 13.2% of the remaining gap to the human panel:

  • GPT-6 Astra + codex: 64.58% (±2.93), Multi-BO 1.132, $246.41
  • Fable 5.1 + claude-code: 63.11% (±1.33), Multi-BO 1.081, $272.80
  • DeepSeek-Flash + opencode: 59.81% (±1.25), Multi-BO 1.000, $2.54
  • GPT-5.5 xhigh + codex: 59.75% (±1.55), Multi-BO 0.998, $99.37
  • Kimi K3 + opencode: 58.23% (±1.12), Multi-BO 0.953, $23.26
  • GLM-5.3 + opencode: 57.78% (±1.30), Multi-BO 0.941, $28.15
  • Qwen3.8 Flash + opencode: 52.66% (±1.17), Multi-BO 0.803, $8.30

Two conclusions follow. First, the $2.54 DeepSeek V4.1 Flash run holds 59.81% at roughly one hundredth the leader's cost, trailing by about 4.8 points — that cost-performance ratio is more worth watching than any score comparison. Second, pushing DeepSeek-Flash to its maximum reasoning tier made things worse: the multi-reference score fell to 48.11% and Multi-BO dropped from 1.000 to 0.864. A longer deliberation budget is not monotone on these tasks.

There is also a set of numbers aimed squarely at the question of whether the board measures the model or the harness: with the model fixed and only the harness varied across four options, O-Eval moved by at most 0.07 percentage points in total (59.81 to 59.88). The board is measuring the model.

The authors also undercut their own headline, which is to their credit. Per-seed deviations reach 2.93 percentage points, so the gap between the top two sits inside noise and does not constitute a significant difference. And the live leaderboard in the repository (https://github.com/wxj630/terminal-bench-math-modeling) has already flipped once — on single runs, Fable 5.1 + claude-code at 64.62% actually edges out GPT-6 Astra + codex at 61.47%. The repository itself notes that a one-to-two point gap is inside seed noise, and that no model clears 65%.

So what

The real contribution of TB-MathModel is not another leaderboard. It is moving "verifiable" out of the contest problems and into the evaluation infrastructure: the gate checks constraints, Lean 4 checks the gate, and the human panel checks the bias of any single anchor.

The model layer keeps getting stronger. But as long as evaluation is an LLM judge reading a report, or aligning against one designated "correct answer", what you are actually measuring is fluency and transcription. The genuinely hard layer — compressing a messy real-world situation into a model that is computable, verifiable, and defensible — remains the least adequately measured one.

Its words on the Hugging Face paper page (https://huggingface.co/papers/2611.08237) are blunt: agents that can handle real-world problems are still a long way off, and the road there needs exactly a uniform environment, a verifiable reward, and an agent-trajectory corpus that compounds. According to AI Weekly (https://aiweekly.co/alerts/tb-mathmodel-gpt-6-astra-tops-contest-bench-at-6458-o-eval), the authors come from The Hong Kong University of Science and Technology (Guangzhou), IDEA Research and DataArcTech.

This benchmark paid its own tuition first: it was the one that was wrong. An evaluation willing to say that out loud is the only kind worth measuring models with.