Packing 25 items into one prompt and asking the model to return 25 JSON verdicts in a single pass is the standard cost-saving trick in LLM review pipelines. It rests on an assumption nobody writes down explicitly: that a verdict depends only on the item itself, not on the neighbours sharing the prompt or on the item's position inside it. A paper from the Technical University of Munich put that assumption to the test, and the result is not flattering.

The title says it plainly: judged in company (arXiv:2610.40241). Across three tasks - harm detection in social-media comments, short-answer grading, and review star ratings - the authors ran five open-weight models of 4B to 9B parameters over 2,000 items each, batching 25 items per prompt. The finding: reshuffling the same 2,000 items into different random batches flips 11% to 22% of the decisions, while AUROC barely moves.

The money saved costs you judgement

First, why this practice spread. Every request repeats the same instruction block, so stitching 25 items into one prompt cuts token cost by more than 80%. A prior study cited in the paper (Pipal et al., 2026) reports batches of 25 to 100 items with up to ten coded variables per prompt, saving more than 80% of tokens while losing less than 2 points of accuracy. Vendor documentation referenced by the paper recommends batching for moderation that need not happen in real time. When both vendors and academia say so, practitioners following it is unsurprising.

The problem is that the condition this practice depends on was never written down: verdicts must be separable. The paper's definition is blunt - for any item, in any context, the level assigned must be the same. No decoder-only model enforces that. When the model writes the verdict for item 12, the other 24 texts and the 11 verdicts it has already written are all sitting in its context.

How the controls were built

Each task contributes 2,000 items. Harm uses the Measuring Hate Speech corpus with a five-level scale, counting a hit at level 2 (insults or harassment aimed at a person) and above; grading uses SciEntsBank, labelled incorrect, partial, or correct; stars uses Yelp reviews, counted as a hit at four stars or above. The five models are Qwen3-4B-Instruct-2507, Qwen3-8B with thinking disabled, Gemma-4-E4B-it, Granite-3.3-8B-Instruct, and GLM-4-9B-0414, run in bf16 with greedy decoding, one RTX 5090 per model.

The controls are the interesting part. Beyond reshuffling, the authors ran three baselines: renaming the item keys, rewording the instruction, and generating the same prompts one at a time to isolate pure numerical noise. Those baselines move 4.2% to 8.4% of decisions, and numerical noise alone accounts for only 0.5% to 2.1%. A new random partition moves 11% to 22%, which is 6.5 to 13.3 points above the largest baseline.

Renaming keys and rewording instructions are surface perturbations. Changing which items share a prompt is a different thing entirely.

Why accuracy hides it

The most worrying part: AUROC for batched grading differs from single-item grading by only -0.033 to +0.010, a shift too small for any conventional validation threshold to catch. Worse, the operating point drifts. On the harm task, Gemma-4 flags 49% of comments when grading one at a time but only 31% when batching; Granite drops from 59% to 43%. The same comments, flagged at rates that differ by more than ten points.

The paper supplies the mechanism. Under a threshold model, variance contributed by context lowers the AUROC of scores in proportion to its share, but flips decisions in proportion to its square root. "Small effect on scores" and "large effect on decisions" can hold at once, because the flip rate depends on how densely the scores cluster near the threshold.

Scaling up does not rescue it. In the appendix, four models from 14B to 32B parameters still show 6% to 15% of decisions changing across batches, 4.2 to 9.1 points above baseline.

Taking it apart: order, slot, company

The paper decomposes flip rates exactly into three components. The order of the neighbours accounts for 0.42 to 0.69 of the latent dissimilarity, the item's slot for 0.09 to 0.32, and membership - which items share the prompt - for 0.01 to 0.25 (mean 0.15). On the decision scale, order accounts for 0.62 to 0.80 of the flips while the membership main effect accounts for only 0.00 to 0.14.

So why does reshuffling bite so hard? Because order and slot can at least be averaged away by voting, while who shares the batch cannot. The authors ran a harsher experiment: grade every batch in nine different orders and take the majority vote, and 52% to 60% of the flips survive. Replace the batch with fifteen independent random batches and take the majority of those, and 8.4% to 17.8% of decisions still disagree with single-item grading.

Mechanically, the paper intervenes mid-generation: swapping in another batch's verdicts changes 7% to 23% of decisions, and swapping in another batch's texts changes 9% to 26%. Two channels act at once - the model is pulled around both by neighbouring texts and by the verdicts it has already written itself.

The stars task is a counterexample. With AUROC 0.96 against ground truth, the written rating changes for 16% to 21% of reviews, yet only 2.9% to 3.9% of decisions change. The more ambiguous the task, the larger the effect.

What to do instead

The recommendation is direct: grade ambiguous items one per prompt and recover most of the cost with a cached prefix, and whenever items are batched, report agreement with single-item grading.

What is worth remembering is really the finding that majority voting does not save you. Approaches like BatchPrompt have long used voting over permutations to damp position effects, which looks like a general fix, but the paper measures it as nearly ineffective against the membership layer. That is precisely the layer practitioners trip over most easily: who gets grouped with whom in one prompt is usually decided by whatever business logic is convenient, not by deliberate evaluation design.

This points at a larger question. How many LLM-as-a-judge conclusions survive a re-run? An independent study (arXiv:2610.31172, spanning 27 languages, 9 prompting strategies, 4 open-weight judges, 13 datasets, and over a million pointwise scores) finds that prompting strategies are not transferable, that the choice of judge model dominates the choice of prompting strategy, that a strategy's effect can even reverse across judges, and that this instability worsens in low-resource languages while aggregate reporting conceals it.

The two papers probe different layers, yet land in the same place: underneath the aggregate number sit two layers of noise - the choice of judge model, and how the context was assembled during judging. Many "evaluation scores" today are better described as artefacts of one particular sampling run than as a property of the system being evaluated.

So the next time you read a leaderboard or a review report, look past the score and check one thing: were these verdicts produced one at a time, or in a batch? If the latter, how much of the score came from the system under test and how much from the composition of the batch is something the report itself probably cannot tell you.