Six numbers.
Five were ours.

Why the ratio is the finding

Published comparisons put the same B2B data vendor at 42.4% in one study and 78% in another. Neither publishes a methodology. Harness design moves a vendor number further than vendor quality does. This is a worked example of how far.

In case 4 the affirmations kept looking correct: a dermatologic oncologist affirmed for "Educational Methods" had in fact co-authored medical education research. Correct answers, scored as errors.

What survivesn=1,000

A provider is bad at saying no

Expertise claims put to Exa, with every false claim verified against OpenAlex /works before the provider was asked.

It confirms real expertise 96% of the time and affirms claims about topics the person has never published in roughly one time in six. Topic distance halves that rate without removing it. Asked whether an organometallic chemist has published on historical studies on Spain, it still says yes one time in twelve. No generous reading of the question explains those 21 affirmations.

Calibration

Index freshness

An inverted U, not a half-life

Reflection of an attested job change, by time since the filing became public. 100 observations per bin.

Non-monotone, so no half-life exists and the pre-registered estimator was withdrawn. It had produced a "median lag" of 7,360 days by forcing a shape the data does not have. The fall after three years is most likely our oracle ageing out: executives leave public companies and SEC stops seeing them.

We falsified our own hypothesis

A capability constraint does disambiguate

D029 argued a people-search surface cannot be handed a biographical constraint: every documented filter describes a person’s current state, so where a name collides the caller cannot pass the one fact that would resolve it. We wrote down the falsification condition first, then ran it. It fired.

Three bands, three intervals excluding zero, n=80 each, compared within task. Two things we did not predict: naming the past employer captures nearly all the benefit — adding the attested date and role buys nothing measurable once names collide — and a colliding name with no context is 0 of 80, not merely harder but unresolvable.

What survives is the claim’s scope. It was about indexes with structured filters, and an answer engine is a different surface. On the filter-based one, both renderings scored 0 of 12 and nothing is claimed.

A false answer usually cites nothing real

The provider returns citations with every answer. For all 399 affirmations we resolved those citations against OpenAlex and asked a mechanical question: is any of them a work this author actually wrote? No model read the evidence. It cost $0 — the citations were already cached.

The strongest objection to our headline was that the provider might be reading “expertise in T” loosely as “works near T” — wrong question, not wrong answer. The control arm now fails two separate tests: it is the arm no generous reading explains, and the arm citing nothing verifiable at all, zero of twenty-one.

But the middle of that table is why the objection survives. Roughly one in four adjacent false affirmations is grounded in the person’s real work, just on another topic. The defence holds for near misses and collapses only at the extreme. We are reporting the half that argues against us.

False merges run backwards

We predicted that confidently-wrong identities would rise as names get more common. They fall.

n=80 per band. On a colliding name the provider mostly cannot name anyone we can resolve, so the outcome is a miss rather than a wrong identity — it never gets the chance to be confidently wrong. The risk sits where nobody stratifies for it: the easy, unique-name cases where an answer comes back and looks clean.

This hypothesis sat unmeasured for the life of the project, filed as “strata empty by construction”. It was never a budget problem. A random draw had put 251 of 299 observations into one band; the stratified draw built for a different experiment answered it for $0.

A trained policy, and three reward hacks it would have found

Reading the environment before writing a training loop found five defects. Three were exploits an optimiser finds and the baselines never do: a free reward atom worth about +0.33 an episode, a confidence knob where stating 0.49 saved penalty and cost nothing, and a turn cap that did not bind.

The published baselines were wrong. Corrected, never_verify falls from 0.0455 to −0.2292 and always_deep_verify from 0.0277 to −0.2574. Both are now beaten by doing nothing: at a 37% solve rate, answering is negative expected value.

It learned when to answer and not how sure to be. It abstains on 41.7% of tasks, lifting precision from the 37% base rate to 64%, then states confidence 1.0 on every answer it gives. That is the same failure we measured in the provider, reproduced by our own policy in an environment that prices exactly it.

The retraction record2 formal

The log, the leak probe, the integrity probe and the claim gates were all committed before any measurement existed. That ordering is checkable in the git history, and it is the only reason five artefacts were caught rather than published.

Limits

One provider. Every surviving number describes Exa; nothing here supports a claim about the category. Part of the 16.8% may be prompt interpretation: the confirmatory run removed contaminated negatives but cannot remove adjacent ones, so only the control's 8.4 points are error under every reading. Publication record is not expertise: engineers, practising clinicians and lawyers leave no bibliographic trace and are invisible here. Nothing is trained.

Artifacts

Both pre-registrations were hash-published before their studies ran, and are byte-identical to their first publication. Every measurement regenerates from the replay cache without an API key.