The standard way to evaluate a retrieval system today is to let a large language model act as judge: one API call per document, producing a relevance score from 0 to 3. The cost is the obvious problem, rising linearly with the size of the document pool. The less obvious problem is that the grading logic is buried inside a closed model, so when a judgment is wrong you cannot even tell what it was looking at. A recent paper proposes a counterintuitive alternative: have an LLM write a program, and let the program do the judging (https://arxiv.org/abs/2611.24849).
From one call per document to one program per topic
The method is called ExecJudges, authored by researchers at Emory University, the University of Waterloo and Stanford University. It runs in two stages. First, Claude Opus 5.5 acts as a program writer and translates the official TREC scoring guidelines into Python code, writing three independent programs per topic; the final label is the mean of their grades, rounded down. Second, those programs stop calling any model and score every document on their own, using only lexical matching, standard NLP parsing libraries and pool-level statistics.
That second stage is the whole point. The programs do deterministic string operations, take under a second per topic, and need no GPU. According to the paper, the generated programs have a median of 66 to 75 lines and two functions, holding 6 to 8 regular expressions and one to three toolkit calls. On the Robust04 topics, 92% of the programs do only one thing: find the single sentence that satisfies the topic conditions.
Where it matches the Sonnet judge, and where it does not
Evaluation covers seven TREC collections, with trained NIST human assessors as the reference and ordinal Krippendorff's alpha as the measure. Without human labels, ExecJudges differs from the listwise Claude Sonnet 5.5 judge only within the confidence intervals on DL 2019 to 2021; extended to DL 2019 to 2022, the gap stays under 0.054. The paper's own summary is that the two agree with NIST at the same level. But on DL 2023, Robust04 and TREC-COVID, the program version falls clearly behind the judge version.
Feed it 16 human labels per topic and the picture flips. In the ExecJudges-L variant, agreement with NIST beats the topic-specific classifier given the same 16 labels on all seven collections, and on DL 2020 and DL 2021 it exceeds the Sonnet judge given the same labels by 0.117 and 0.037. Across 142 development and pilot topics, it is ahead of the classifier on 84 of them, with a sign test at p<0.001.
Where the money actually goes
Writing the programs is a one-time cost of $0.17 to $0.49 per topic: $0.17 to $0.24 without labels, $0.20 to $0.49 with 16 labels. Grading each document afterwards costs nothing. The break-even point is roughly 200 to 350 documents per topic; below that, the program version is actually more expensive than calling a judge per document, and above it the savings grow with pool depth, because the judge's bill tracks document count while the program's does not.
Human labeling is dearer still. NIST's own public page puts it plainly: a typical TREC track usually involves a team of six contractors working for 2-4 weeks, and those contractors need to be trained and monitored. On the judge side, the paper's description is an expensive and often GPU-dependent model that must evaluate every document. The topic classifier is a different cost structure: no API cost, but 13 seconds of training per topic on a GPU. The program route occupies neither end.
The cold water
Three objections deserve to be stated plainly.
First, this is still a variant of letting a model help write the answers. NIST's Ian Soboroff published a position paper back in September 2024 whose title is a flat instruction not to use large language models to make relevance judgments, on the grounds that letting an LLM write the judging criteria sets that LLM as a ceiling on performance, and that the same model is often part of the systems being measured (https://arxiv.org/abs/2409.15133). A different route drops the LLM from the loop entirely: Gienapp and colleagues train topic-specific classifiers whose rankings correlate with ground truth at Spearman rho above 0.94, needing only 128 human judgments per topic (https://arxiv.org/abs/2510.04633). ExecJudges does not overturn the first objection. It reduces the number of calls from one per document to one per topic, and the ceiling is still there.
Second, readable programs are targetable programs. The paper admits that a system builder who reads a released ExecJudges program could tune a ranker to its rules, that this risk was not tested, and that usability testing of the generated code was not performed either. That runs against the intuition that explainability buys trust: the moment you publish the rules, the game moves to a different field.
Third, the method depends on closed frontier models. In the main configuration, all LLM judges and all but one program writer come from the closed Anthropic Claude family. Swap the writer for the open-weight GPT-OSS-120B and agreement drops to 0.486, 0.393 and 0.344, all significantly below the main configuration at p<0.01. The savings from having AI write the code are, for now, tied to closed frontier models.
One detail is easy to skip: the program version grades too low. On DL 2019 and 2020, 26% and 28% of the passages that NIST graded 3 received a 0 from the program, while the Sonnet judge gave a 0 to only 24% and 17% of them. Adding human labels cuts that share to 1.2%, 8.1% and 1.3%, and the paper notes these errors are highly concentrated, with four topics per year accounting for 90% to 95% of them. Programmatic grading errors do not spread evenly; they clump, which makes them harder to notice than random noise.
The turn worth watching
Cost aside, the real contribution of ExecJudges is moving the evaluation logic out of model weights and into a few dozen readable lines of code. Every previous judgment was an unauditable model call; now it is a set of regular expressions and conditions a person can scan and edit. Two examples from the paper: for the California whale-watching topic on Robust04, a program counts a passage as relevant only if it mentions Californian places at least half as often as other places; for the hydroxychloroquine topic on TREC-COVID, only passages describing study designs earn a grade of 2. The scoring criteria are laid out on paper.
That is black-box grading replaced by white-box grading. But white-box grading has its own cost: once the rules are public, they simultaneously become a target for deliberate optimization. The paper did not test that side, and offers no answer.
So what is the takeaway? If you build evaluation systems, the part of this paper worth copying is not the gimmick of having AI write code, but the idea of making the scoring criteria explicit. If you read leaderboards, ask one more question about the numbers: were they produced by a black-box model, by a white-box program, or by humans? Those three kinds of score have no business sitting in the same table.