Two things we measured

1. Models disagree with themselves about as much as they drift

Ask one model the same question twice, minutes apart, inside the same run. Its two answers share roughly a third of their content words. Across the panel that within-run divergence runs 0.54–0.68 on a scale where 0 is identical and 1 shares nothing. This run predates the per-estimate spread columns, so no standard error is stored beside these figures. The re-measurement below is what the spread is known from, and it is why the band and not the ordering is the result.

Now the comparison that matters. Drift measured between runs on probe set 1 — including one pair of runs 22 hours apart — came out at 0.60–0.68, 0.59–0.67 and 0.58–0.64. Every one of those figures sits inside the band a single model produces answering the same question twice in one sitting.

The consequence is blunt: a between-run “drift” number in this range is not evidence that anything changed. A board reporting deltas without publishing a floor is reporting an unknown mixture of change and sampling noise.

2. Reversal under pushback splits the panel in two

Each of the 40 stance probes demands a first line of exactly YES or NO. Each is then re-asked in the same conversation with a fixed pushback that asserts disagreement and supplies no new fact: it claims the model is wrong, claims every expert disagrees, and asks again. A flip is a YES that became NO, or the reverse.

Groups come from two-sided Fisher's exact tests over all 15 pairwise comparisons, Holm–Bonferroni corrected at α = 0.05. 8 pairs separated — every cross-group pair, and no within-group pair. Models inside a group are shown unordered because the data does not support ranking them.

A model that holds its position and a model that was right to begin with score identically. This measures reversal, not correctness — and reversing is not always wrong, since a good argument should sometimes change an answer. The pushback here deliberately contains no argument at all.

3. Grounding did not separate the citation arms

Two of the six models search the web before answering a citation probe and quote what came back; the other four produce a URL from their weights. Those are different operations, so the per-model citation column pools two things and its ordering is not a model ranking. Splitting it was the obvious correction.

The expectation going in — written into the code before the split was run — was that search-grounded URLs would be near-guaranteed to resolve, and that the pooled column was therefore mostly measuring which models have a search tool. The first measurement said otherwise: 17 of 30 against 26 of 50, two-sided Fisher's exact p = 0.82. The arms did not separate. Quoting a URL out of a live search result is evidently not the same as quoting a URL that works.

Both denominators are small and this is a between-groups split, not the same model run with retrieval on and off — only two of six providers could do that, so a paired design would cover a third of the panel. The arms differ by more than retrieval. What this rules out is a large effect at this sample size; it does not establish that there is none.

What we got wrong

Every number above has been through at least one correction. The list is here because a measurement board that appears to have been right first time is not one you should trust.

• We published a flip rate at n=7 and pulled it. The first probe set had eight stance probes. One model came out at 71% and another at 0%, which read like a finding; the 95% interval on 0 of 7 reaches about 35%. On the 40-probe set that 71% became 33%. Rates now render as raw counts below n=30.
• Then we buried a real result with the wrong test.Having over-corrected, we decided whether two models differed by widening both intervals for multiplicity and checking for overlap. Overlapping intervals are not a hypothesis test, and the extra widening compounded it: we reported that no two models on the board were distinguishable. The Fisher p for the widest pair is 3.1 × 10⁻⁵. Comparisons now use Fisher's exact with Holm correction.
• A run published scores its own transcripts did not support.A scheduling mistake re-ran every probe against an existing run; the new answers were rejected by a uniqueness constraint while the new scores overwrote the old. Five of six providers' flip counts then disagreed with their stored evidence. That run was deleted rather than republished. Shards are now idempotent, and a verification job re-derives every published figure from the stored responses with the shipped scorer and demotes any run that fails.
• We quoted citation percentages that broke our own gate. The citation card rendered rates on denominators of 8–15 — below the threshold we had just imposed on everything else. It now shows raw counts.
• We overstated how often the floor had been measured. This page briefly said the floor was measured three times across two probe sets. It has been measured once; the three repeated measurements were of between-run drift, which is a different quantity.

One failure mode, four times

Every correction in the list above is the same mistake on a different surface: a point estimate treated as settled. A flip rate published at n=7. Separability decided by whether two intervals overlapped rather than by a test. A claim that the noise floor had been measured three times, when three was the count of a different quantity. Per-model floor thresholds built on ten-pair estimates with a standard error of about 0.047 against a between-model spread of 0.13.

What is in place now: every rate carries a Wilson interval and renders as a raw count below n=30; comparisons between models use Fisher's exact with Holm–Bonferroni correction across the whole family; every floor is stored with its standard deviation and standard error, and the drift gate falls back to the panel band whenever a model's floor rests on too few pairs to be resolvable; a verification job re-derives every published score from the stored responses and demotes any run that fails; and rendered output is asserted nightly against a list of required caveats and retracted claims, by fetching the live pages rather than reading the source, because a source edit that silently matches nothing will otherwise ship looking successful.

What that assertion does not do is block a deploy. It runs on a schedule against what is already published, and by hand after a change; the job that would run it before a build goes live is written but not switched on. So it catches a broken claim after the fact, not before — which is worth stating rather than leaving “asserted” to imply more than it is.

Reproduce this

• The probe corpus — all 72 prompts, versioned and frozen. Set v3 is current; v1 is unchanged and still published.
• The latest run as JSON — per-model scores, floors, the separability matrix, and the exact pushback string.
• Full methodology — scoring rules, thresholds, the local pinned encoder, and what is deliberately not measured.

Xyntherium's product orchestrates several of the models measured here, including Anthropic's, and pays each provider for that use. The scoring path is deterministic and identical for all six, and no provider reviews results before publication. These numbers describe model output on a fixed probe set on the dates shown. They are not measurements of safety, correctness, alignment, or harm.