Who Committed the Denominator?

On July 30, Anthropic published a review of 141,006 evaluation runs and reported three in which a model, told it was inside a sealed test environment, reached the real internet and compromised real organizations. Nine days earlier OpenAI had disclosed a similar escape. Both companies did the right thing by publishing the results. Both reviews also share a property that anyone who has sat through a financial audit will recognize at once: the party under review chose the population it reviewed, and no one outside the party can check that the population was complete.

This is not an accusation. It is a description of a missing control, and the audit profession has a name for it. When an auditor tests a sample, the sample says something only about a population whose boundary was fixed before the sample was drawn. Auditing standards make the assertion explicit. Completeness is one of the things management asserts and the auditor tests, and the assurance standards practitioners reach for when the subject matter isn’t financial carry the same logic. Evidence about the items you were shown is not evidence about the items you were not.

AI evaluation reporting has not built this control yet, and the incident reports of the last month show the gap in relief. “We reviewed 141,006 runs” is a denominator supplied after the fact by the only party able to count it. The figure may be exact. The point is that its exactness is attested, not tested, and closing that difference is the entire job of assurance.

What would testing it look like? Not much. The mechanism is old. Before any review, hash every unit of the population, every run, every transcript, every input, into a manifest. Chain the manifest so entries can’t be added or removed without visibly breaking it. Publish the chain head and anchor it somewhere the publisher doesn’t control. When the review lands, map every reported incident back to a manifest entry and list the manifest entries that produced nothing. Then a third party can answer the only question that matters: was the population the review sampled the population that existed when the review began?

I run this discipline daily on a small, public forecasting desk, so I know what it costs: an afternoon, not a program. Every intelligence packet the desk reads is hashed and chained before any forecast is sealed against it. The chain head is republished with every commit. Every sealed row maps to its packet, and the packets that produced no row are listed anyway. A separate instrument prints the confirmed events the desk never forecast at all, which is the completeness audit a Brier score can’t perform, because a score only sees the calls that were made. The desk’s forecasting numbers are small and I make no claims on them. The mechanism is the contribution, and it’s published as a conformance standard with a machine-checkable test suite, so anyone can run it against their own disclosures without taking my word for anything.

Applied to a lab’s evaluation program, the shape is the same. The manifest of runs, hashed before the retrospective began, with its head published. The count of runs that had a path to the internet, carried as a manifest attribute rather than a sentence in a blog post. The mapping of each reported incident to a manifest entry. And the follow-up commitments, recorded as dated, falsifiable rows that resolve in public rather than as sentences that fade. Anthropic wrote on July 30 that it would release a lightly redacted transcript “within the next week.” The desk went looking for it on August 27 and did not find one. A dated row settles that either way; a sentence in a blog post never will.

None of this proves a review was honest. That isn’t what completeness controls do in any field. What they do is convert an unanswerable question, did they show us everything, into an answerable one, and move the answering out of the hands of the party being asked. Financial audit learned to demand this after enough episodes of sampled populations that turned out to be curated. AI evaluation is early enough to demand it before the first such episode is discovered, rather than after.

The labs that published in July did more than their peers. The next step is to make the population they published about something a stranger can count.