The industry habit of measuring revision by counting how often a model changes its answer has a serious flaw, and a two-pass audit makes the accounting exact. When a dissenting voice is introduced as a domain expert, models flip more often, but the flips are far less selective than a high initial accuracy demands, so the net effect is a loss.

The work is When Expert Disagreement Hurts: Prestige and Selectivity in LLM Decision Revision (https://arxiv.org/abs/2611.16694) by Yupeng Tang and Mingfeng Lin of Georgia Institute of Technology, listed on its own pages as a NeurIPS 2026 paper. The question it asks is narrow: with the evidence and the model's first answer held fixed, how much does the label attached to a disagreeing source change the decision, and does that change help or hurt?

1,989 items, evidence locked, only the label changes

The audit uses a two-pass probe. In pass one the model gives a Yes or No answer to a question with attached evidence. That exact answer is reused across three pass-two branches: neutral reflection, anonymous disagreement, and disagreement attributed to a domain expert. All three share the same question, the same evidence and the same initial answer, so any difference can only come from how the source is described.

The benchmark has 1,989 binary items assembled from three expert-annotated sources: PubMedQA medical questions with the ambiguous Maybe labels dropped (890), SciFact scientific claims with binary labels kept (299), and ContractNLI legal hypotheses with Neutral dropped (800). The final label split is 1,379 Yes against 610 No. Five models run the full benchmark: GPT-4o-mini, Claude-Sonnet-4, Gemini-2.0-Flash, Llama-3.1-8B and Mistral-7B.

Ninety percent accuracy demands nine times the selectivity

The paper's criterion is exact. Let initial accuracy be a, let the model revise correct answers at rate h and wrong answers at rate b. Revision raises accuracy only when b/h > a/(1-a). At 90% initial accuracy that threshold is nine: the model has to revise a wrong answer nine times as often as a right one merely to break even.

Measured behaviour runs the other way for every model. The paper plots each model's selectivity b/h against its break-even point, and the ratios are 0.25 for GPT-4o-mini, 0.19 for Claude-Sonnet-4 and 0.25 for Gemini-2.0-Flash. The two open-weight models, Llama-3.1-8B and Mistral-7B, sit at 0.82 and 0.81, roughly twenty percent short of the line. The three proprietary models would need to become three to five times more selective to stop losing accuracy.

The accuracy cost lands directly in the tables. Under expert-labeled disagreement Claude-Sonnet-4 turns 19.9% of correct answers wrong while fixing only 3.8% of wrong ones, a 16.1 point drop. GPT-4o-mini loses 11.0 points and Gemini-2.0-Flash 6.8 points; the two open-weight models lose 2.3 and 1.9 points. For the three proprietary models the paired tests give p at or below 2e-20.

One comparison is especially telling. GPT-4o-mini's reversal rate on initially correct answers is 10.0% under anonymous disagreement and 16.9% under the expert label, on the same items with the same answers. Only the identity of the objector changed.

More reversals do not automatically mean lower accuracy

The paper separates the label's effect on reversal from its effect on accuracy, and this is the most important layer of the analysis. For GPT-4o-mini the label moves the reversal rate by +6.90 points and accuracy by -5.19 points. Llama-3.1-8B shows +5.18 and -3.77. But Claude-Sonnet-4 reverses more often (+2.62) with a small, uncertain positive accuracy effect (+0.30, 95% CI [-1.01, 1.61]), and Gemini-2.0-Flash reverses less (-1.86) while becoming more accurate (+2.37). The loss therefore cannot simply be blamed on the expert label. For Claude-Sonnet-4 and Gemini-2.0-Flash it comes from disagreement itself.

A stricter control follows. The anonymous and expert-labeled sources are given the identical two-sentence rationale with no filler, so the two prompts differ only in the sentence describing the source. The label still raises the reversal rate by 4.82, 5.62 and 4.42 points for GPT-4o-mini, Claude-Sonnet-4 and Gemini-2.5-Flash, and lowers accuracy by 4.82, 3.21 and 2.81 points, with only GPT-4o-mini's accuracy interval excluding zero.

Stated reliability turns out to carry weight on its own. In the padded control suite, adding the clause that the expert may be wrong lowers the reversal rate for all five models tested, significantly for four, with Claude-Sonnet-4 falling from 7.8% to 1.4%. Adding a claimed track record raises it for all five, significantly for two.

A prompt-level fix is not enough

The paper tests three mitigation paths with limited success. An evidence gate that instructs the model to revise only when the excerpt itself supports the change cuts correct-to-wrong transitions from 171 to 76 across three proprietary models on the answer-only subset. Against argued disagreement it helps less: Claude-Sonnet-4 drops from 24.7% to 20.2% (p = 0.013) and GPT-4o-mini from 28.1% to 24.6% (p = 0.077), while Gemini-2.5-Flash and the four newer models show no significant change.

The paper also looks at the token-score level for the mechanism. For the two open-weight models, 63.2% of Llama-3.1-8B reversals and 26.8% of Mistral-7B reversals happen without the Yes/No log-probability changing sign. The answer is pushed across the decision boundary by a graded score shift rather than by a genuine reversal of the model's conclusion.

This is not an isolated result. A separate team from Amazon, Georgia Tech and Illinois Institute of Technology, in Easier to Mislead Than to Correct: Harmful and Beneficial Revision in LLM Conformity (https://arxiv.org/abs/2606.01637), ran a comparable study over four open-weight models and seven QA datasets and reached a consistent direction: all-wrong peers push harmful revision from 15.6% to 62.9% (+47.3 points), while all-correct peers raise beneficial revision only from 32.7% to 51.5% (+18.8 points). Authority labels again make models more likely to pick the endorsed answer regardless of whether it is correct. That work also reports that generic reasoning interventions such as chain-of-thought and reflection do not reliably suppress harmful revision.

So what

Multi-agent pipelines now routinely treat another agent's disagreement as a correction signal, and some attach role labels such as expert or team lead. This paper puts a price on that design. Models do revise more often under a prestigious label, but the selectivity of those revisions falls far short of what a high initial accuracy demands, and once the direction of the disagreement is wrong the loss exceeds the gain.

Claude-Opus-4.8 clears the break-even line on its own, with a ratio of 1.33, and its accuracy does not fall, because it hardly reverses at all. That points to what is actually scarce in disagreement-based pipelines: not a model that listens harder, but one that can hold its answer when the evidence has not changed. The paper's recommendation follows directly. Do not evaluate a revision pipeline by reversal counts or by final accuracy alone. Report drift, disagreement sensitivity, both effects of the source label, and the selectivity of revision, per model and per protocol.

The protocol and code are released at https://github.com/yupengtang/when-expert-disagreement-hurts.