Can we still trust benchmark scores? The industry has wrestled with this for years: a model may have effectively memorized public benchmark questions during training, so a high score no longer proves real capability. A 2026 study covering 60 LLM benchmarks found roughly half already show signs of this contamination-driven saturation, and NIST flagged the same risk in a February 2026 report. On August 27, Google DeepMind offered a new answer: lock both sides' secrets inside a hardware-encrypted enclave and run an exam where neither party can cheat.
The structural deadlock in AI evaluation
Third-party evaluation of closed models has long offered only two bad options: hand the test questions to the vendor — run them through the provider's API, where the company can see every prompt — or hand the model weights to the evaluator, exposing the crown jewels. Zero-logging protocols and contractual NDAs are the current patches, but they ultimately rely on organizational trust. The 2026 Singapore Consensus on Global AI Safety Research Priorities — a report spanning 13 countries and 100 contributors — explicitly named this gap as a priority research challenge: evaluation infrastructure should ideally be double-blind, where the evaluator cannot access model parameters and the developer does not know the exact evaluations, because reliance on organizational trust and procedural controls is insufficient for the stakes of frontier AI oversight.
What happened inside the enclave
The pilot's technical foundation is Google Cloud's Confidential Space: a single NVIDIA H100 80GB confidential GPU paired with Intel TDX host memory encryption. The flow has several stages. DeepMind loads Gemini 2.5 Flash-Lite's weights and inference code; the evaluators load AILuminate safety benchmark prompts and scoring code. Both travel into the enclave over encrypted connections — weights land in hardware-encrypted GPU memory, prompts in encrypted host memory. Before anything is sent, remote attestation verifies the enclave is running the software both parties agreed on. Throughout the run, the vendor cannot read the prompts and the evaluators cannot read the weights. Only the bounded, pre-agreed metrics leave the machine, and the temporary environment is then decommissioned.
The cast is worth noting: MLCommons supplied a never-before-used set of AILuminate prompts; OpenMined produced and adapted the execution software; the Singapore AI Safety Institute and AVERI (the AI Verification and Evaluation Research Institute) acted as evaluators, decrypting outputs and grading them against AILuminate criteria to produce qualitative and small-scale quantitative assessments. AVERI's full pilot report is on its site (http://averi.org/ourwork/averi-pilot-report-the-worlds-first-double-blind-eval).
Significance and limits
The big picture: this moves the trust basis of AI evaluation from contract clauses into cryptography. Cryptography can prove which code touched which secret; it cannot prove the test asked the right questions, that the scoring policy is correct, or that the model is therefore safe — a boundary the participants themselves acknowledge. The report also notes that the biggest practical obstacle is no longer raw enclave overhead, but legal agreements, code review, output policy, and coordination between organizations — the human problems.
So what? As regulation, procurement, and deployment decisions lean increasingly on third-party evaluation, verifiable neutral exam rooms will become infrastructure. Next time you see "model X sets a new record," it's worth asking one more question: was that exam closed-book?