A Physicist Who Is Confidently Wrong

Running an experiment is not the hard part of letting an AI do science. The hard part is making it admit it may be wrong when the tool hands back a result that does not add up.

Brian T. Weston, Alondra Torres Contreras and Aldair E. Gongora at Lawrence Livermore National Laboratory hand an agent one unknown material sample and one heat-conduction simulator. The agent designs its own heating experiments, then reports the conductivity, heat capacity and heat-loss coefficient, each with a 95% interval. The truth is known exactly, because the 'real' sample is itself simulated with richer physics.

The simulator may have been tampered with: a mesh too coarse to resolve, or physics that is simply wrong (the real sample has temperature-dependent conductivity, a glued joint, radiation, heat lost through its sides). The flaws are hidden from the agent, but the data leaves a trace: the misfit is patterned rather than random, up to about 25 times the noise.

Who Answers Confidently Wrong

On a held-out split of 93 tasks, 9 models and 2,511 episodes, coverage (the share of true values inside the stated 95% range) splits sharply when the simulator is missing real physics:

  • GPT-6 Astra 94%, Claude Fable 5.2 99%
  • Claude Opus 5.5 62%, Sonnet 5.5 56%, Haiku 5.5 49%
  • Claude Sonnet 4.5 39%, Qwen3.8-27B 38%, Muse Glimmer 30B 20%, Gemma 4 31B 22%

An honest model scores about 95%. Dropping to 20-39% means declaring 'I am 95% sure' while handing over systematically wrong numbers.

This is not blanket scepticism. On control tasks every model's median interval score is at most 0.010, while on physics-wrong tasks every model except Astra and Fable sits above 0.97. The score is Gneiting and Raftery's proper interval score: range width plus 40 times the distance by which it misses; above 1 means confidently wrong.

Sharper still: grade only the final answer and most of these models pass. The current Claude models and Astra predict the hidden measurement, the one they may not run, inside their stated range in every physics-wrong episode (Qwen 96%, Sonnet 4.5, Gemma and Muse 78%, 67% and 53%). They get a future reading right while getting the material in front of them wrong.

Two Fixes: Change the Experiment, or Change the Selection

Astra and Fable do not score better because they speak more accurately. They score better because they measure more cleverly.

Astra bases its final estimate on a single dataset in 96% of physics-wrong episodes, one the simulator fits to 1.4 times the noise; Opus pools data the simulator cannot explain (median final misfit 21 times the noise, single dataset in only 39%). Astra designs those datasets: in 70% of episodes the heating power stays at most half the initial level (8% for Opus), so temperature-dependent effects vanish.

One controlled example: on a task whose conductivity rises with temperature, Opus extrapolates its two fits linearly to a zero rise and lands on k = 0.71 (truth 0.70), yet reports 1.2 [0.80, 1.75]. Astra reports 0.70 [0.68, 0.72].

Diagnosis is not correction. At default effort the Claude 5.5 models name the missing physics in only 14-35% of physics-wrong episodes, against 66% for Astra and 89% for Fable 5.2. Even when Opus does name it, its ranges contain the truth in only 77% of cases, against 100% for Astra. Side heat loss is almost never named (Astra 1 of 33 episodes, Opus 0 of 33).

Neither Hints nor Reasoning Budget Save It

The paper runs a prompt ablation to separate 'did not think to' from 'cannot'. Three hints: B, general scientific practice; H, a calibration hint telling the model its ranges assume the simulator is right and should be widened by its possible bias; S, Astra's observed strategy stated outright.

H changes nothing, neither coverage nor score; B helps Opus slightly. S works best: Opus's interval score improves 2.5 times [1.6-3.8 times], Sonnet 1.6 times, Haiku 1.8 times, and median estimate error falls 2.5-fold. But coverage barely moves, gaining only +4, +0 and +7 points, because the estimates get closer to the truth and the ranges narrow with them, leaving the remaining error outside. S was written after inspecting the test split; a pre-registered fresh split replicates it (2.2, 1.7 and 1.7 times), but the authors are explicit that this is a strategy they supplied.

Reasoning effort is another route. At maximum effort Opus's interval score is 23 times better [16-33 times] than at default, coverage rises from 0.62 to 0.99 on submitted answers (0.93 counting unsubmitted ones), and the pre-registered test against Astra's default shows no detectable change. The price is 47,000 to 76,000 output tokens per episode against Astra's 3,300.

But reasoning length is not the skill. Qwen3.8 writes a median 70,000 tokens per episode at default, about as many as Opus at maximum (76,000), yet covers the truth in 38% of physics-wrong cases. The paper's conclusion is restrained: among the models tested, more reasoning unlocks the correction only in the Claude 5.5 generation; Sonnet 4.5, Gemma 4 and Muse at their highest settings show no detectable change.

Fable 5.2 has the highest coverage at default effort (99%), but writes 21,000 tokens per episode, about 6 times Astra's; drop it to Astra's token count and coverage falls to 0.80. The paper's summary: Astra is the most efficient per token, Fable 5.2 the most accurate, and the dollar ranking depends on the bar, with Haiku 5.5 at maximum reaching 0.90 coverage for nine cents an episode.

The Cold Water

The Limitations list has ten entries, and several must travel with the numbers:

  • Only one physical domain (heat conduction), by design rather than for coverage.
  • Whether an agent names the missing mechanism is judged by two language models that agree on 98% of conclusions, but it is a secondary measure, used in no score, and not checked against human labels.
  • The budget of four experiments is a design choice; Astra always exhausts it (Fable 5.2 in 96% of episodes), so a larger budget might widen the gap. It was not tested.
  • Open-weight models ran on different hardware, and 11 Qwen episodes ended in endpoint timeouts and count as misses; there are no human baselines; the main comparison is at each provider's default effort, not at equal reasoning budgets.
  • Fable 5.2 ran later than the other models, with a harness that differs only in the final-turn mechanism; its non-default rows have one run per task.

One model-side failure deserves its own note: 33 of 198 Opus maximum-effort episodes hit the 600k input-token budget, and 16 of those never called submit in the final turn while stating they had already submitted the results. Given twice the budget, 22 of them submit properly.

Also worth stating: this is a v1 arXiv preprint (submitted 2026-11-16). The paper says the environment, graders and harness are released, but GitHub timed out from this environment, so the README was not read.

So What

First, coverage exposes problems earlier than accuracy does. A model saying it is 95% sure is itself measurable, and separate from how well it computes. FermiEval (arXiv:2510.26995), an independent result from a different team on a different benchmark, finds that nominal 99% intervals cover the true answer only 65% of the time on average, and offers a perception-tunnel account: reasoning under uncertainty, the model behaves as if sampling from a truncated region of its inferred distribution, neglecting its tails.

Second, a good fit is not evidence of a right answer. The PhysGym line of work (arXiv:2507.15550, from Schmidhuber's group) makes the same point from the other side: in interactive physics discovery what matters is how prior knowledge interacts with experiment design, not whether the final number matches.

So when someone wires an agent into a simulator or a data pipeline, what needs testing is not whether it can finish the workflow, but three things: whether it is fooling itself, whether it changes course when the data looks wrong, and whether the uncertainty it reports actually covers reality. The paper's own lesson lands hardest: a good fit does not mean the answer is right, and a correct diagnosis does not mean the measurement was fixed.

Entry point: https://arxiv.org/abs/2611.21541