Google DeepMind pilots the world's first double-blind AI evaluations

Google DeepMind has piloted what it calls the world's first double-blind evaluation of a proprietary, frontier-class AI model, using confidential-computing safeguards so that neither the AI lab nor the outside evaluator can see the other's secrets during testing.

What's new

DeepMind partnered with the Singapore AI Safety Institute, the privacy-tech nonprofit OpenMined, the evaluation group AVERI, and the industry benchmarking consortium MLCommons to test a Gemini Flash Lite model against confidential benchmarks in what it describes as a privacy-preserving environment. The setup relies on Confidential Space, part of Google Cloud's Confidential Computing portfolio, to cryptographically enforce a two-way blind: "The evaluator cannot see the Gemini model weights, and Google cannot see the evaluator's test prompts."

DeepMind frames this as resolving a structural conflict in third-party AI evaluation. Normally, an external evaluator either has to trust the lab's self-reported results, or the lab has to hand over its model weights to an outside party, or the evaluator has to reveal its test questions to the lab in advance, any of which can taint the result or leak sensitive material. "Double-blind evaluations eliminate this compromise. By using Confidential Space within Google Cloud's Confidential Computing portfolio, we can cryptographically verify that both the external evaluation data and the proprietary model remain private to their respective owners," the company said.

The pilot used a smaller Gemini Flash Lite model rather than a top-tier frontier model, suggesting this is being treated as a proof of concept for the mechanism rather than a public verdict on a specific model's capabilities or safety.

Context

Independent evaluation of frontier AI systems has become one of the more contested pieces of AI governance: labs are increasingly asked to open their models to outside scrutiny — whether from government AI safety institutes, academic groups, or industry consortia — but doing so has typically required either trusting the evaluator with proprietary weights or trusting the lab not to game a known benchmark. Confidential computing, which uses hardware-enforced encrypted environments to keep data private even from the infrastructure operator, has been used elsewhere for sensitive data processing; DeepMind is applying the same primitive specifically to the lab-versus-evaluator trust problem in AI benchmarking.

The list of partners is notable: a national government safety institute (Singapore's), a privacy-engineering nonprofit built around techniques like this (OpenMined), an AI evaluation specialist (AVERI), and the industry group that maintains widely used ML benchmarks (MLCommons) — spanning government, nonprofit, and industry evaluation infrastructure in one pilot.

Why it matters

If the mechanism holds up, it gives AI safety institutes and other outside evaluators a way to test frontier models against confidential benchmark suites without the lab ever seeing the test questions, and without the evaluator ever touching the model weights — addressing benchmark contamination and IP leakage in one move. That's directly relevant to ongoing debates about how governments and independent bodies should verify safety claims from AI labs without simply taking their word for it, or forcing labs to expose commercially sensitive model internals to get that verification. A successful pilot on a smaller model gives DeepMind, and potentially other labs watching this experiment, a template to point to the next time it's asked how outside evaluation of a genuinely frontier model could work without a trust conflict on either side.