Google DeepMind has demonstrated what it calls the first double-blind evaluation of a proprietary frontier-class AI model, designed to address a problem that's inflating performance scores across the industry. In a pilot test released Thursday, the company evaluated its Gemini 2.5 Flash Lite model against private benchmarks from MLCommons and the Singapore AI Safety Institute using a setup that keeps the model's internal workings hidden from evaluators while simultaneously preventing Google from seeing the test questions. Rather than promoting a new Gemini score, Google is highlighting how the tests were conducted, with neither party gaining access to the other's protected information.

The evaluation framework relies on Google Cloud Confidential Space combined with an NVIDIA H100 Confidential GPU and Intel TDX host memory encryption. Google DeepMind submits Gemini along with its weights and inference code, while evaluators provide their benchmark prompts and evaluation code; both are transmitted through encrypted connections into a secure enclave where testing occurs without either side accessing the other's protected assets. Model weights sit in hardware-encrypted GPU memory, and evaluation prompts are stored in encrypted host memory as well. Once testing finishes, evaluators receive the permitted results, and the temporary environment is destroyed. Before private data is sent, remote attestation confirms the enclave is running the agreed-upon software. OpenMined's PySyft manages code approval, allowing both Google and the evaluator to review code beforehand and block sensitive evaluation portions from making external connections. The pilot ran Gemini 2.5 Flash Lite on a single H100 80GB Confidential GPU, though researchers are already examining clusters of H100 and B200 GPUs linked via encrypted connections to eventually test models too large for a single GPU.

The technical report cites earlier research that found signs of benchmark leakage in roughly half of the 31 models tested, and another study published this year that found contamination can inflate scores, especially for larger models. The report notes that compute overhead sits at less than 5% and points instead to legal agreements and code reviews between organizations as some of the larger obstacles to running these evaluations. According to the paper, researchers eventually want to make attestation far less hands-on, comparing the goal to the HTTPS lock icon in a browser, where security checks happen without the user dealing with the underlying hashes and keys.

The push for double-blind testing stems from a growing challenge: expanding datasets and public benchmarks are making it harder to determine whether a model is being tested on material it hasn't encountered before, and keeping benchmarks private presents difficulties with closed models. With a closed model, evaluators typically must run their questions through the provider's API, meaning the company can view a test that was supposed to remain private, whereas running it independently would require the provider to hand over its model weights. While the system reduces how much the two sides need to trust each other, it doesn't eliminate trust entirely from the equation. Some of it simply shifts elsewhere, including to the hardware itself, and there's still some trust involved in the hardware, including the assumption that the cloud provider and the hardware maker aren't collaborating to bypass the protections. Google still participates in verification because, although the Confidential Space guest OS is open source and its build process has been externally validated, individual builds depend on private signing keys and can't be independently reproduced, and Google's own services are used to sign and verify the attestation report. MLCommons points out that the benchmark itself still needs to be carefully managed, no matter how well the questions are protected. If this approach gains traction, developers could have another way to look beyond the benchmark score, with some proof that the company behind the model didn't see the test before the results emerged. For organizations evaluating AI systems, the technique offers a path around a persistent dilemma—though the infrastructure required means adoption will hinge on whether the effort justifies the marginal gain in assurance.