Open benchmark
How often does a system release an answer its own sources do not support?
Each item is a question, an AI answer and the passages a retriever returned, with a label built by construction: one planted change to a supported answer, so no model under test ever decides the label. Headline numbers are on the held-out split, which nothing was tuned on.
Supported means supported by the supplied evidence, not true in the world.
verify-bench v1.1, held-out, by rules version
deterministic layer (code only) · n = 820 · pre-registration sha256 4c8cb6050ea4…
One held-out run per rules version, each after its rules were frozen. Every number is k/n with a 95% Wilson interval. What changed in each version, and why, is in the rules changelog. What the items are, how each was built and how the split was locked: how the test works. These items have been scored once per rules version, so they are no longer untouched. On 300 fresh answers, scored once, code-only Verify let more flawed answers through (8 of 220), and other checkers were run beside it: Verify compared with other checkers.
| Metric | rules 1.0.0 | rules 1.1.0 | rules 1.2.0 | rules 1.3.0 | rules 2.0.0 | rules 2.1.0 | rules 3.0.0 | rules 4.0.0 |
|---|---|---|---|---|---|---|---|---|
| False-supported rate (lower is better) | 2.0%1.1%–3.4% · 12/612 | 0.3%0.1%–1.2% · 2/612 | 0.3%0.1%–1.2% · 2/612 | 0.3%0.1%–1.2% · 2/612 | 0.3%0.1%–1.2% · 2/612 | 0.3%0.1%–1.2% · 2/612 | 0.3%0.1%–1.2% · 2/612 | 0.3%0.1%–1.2% · 2/612 |
| Unsupported release rate (lower is better) | 3.9%2.2%–6.7% · 12/308 | 0.7%0.2%–2.3% · 2/308 | 0.7%0.2%–2.3% · 2/308 | 0.7%0.2%–2.3% · 2/308 | 0.7%0.2%–2.3% · 2/308 | 0.7%0.2%–2.3% · 2/308 | 0.7%0.2%–2.3% · 2/308 | 0.7%0.2%–2.3% · 2/308 |
| Qualifier-loss detection | 92.2%86.9%–95.5% · 142/154 | 98.7%95.4%–99.6% · 152/154 | 98.7%95.4%–99.6% · 152/154 | 98.7%95.4%–99.6% · 152/154 | 98.7%95.4%–99.6% · 152/154 | 98.7%95.4%–99.6% · 152/154 | 98.7%95.4%–99.6% · 152/154 | 98.7%95.4%–99.6% · 152/154 |
| Qualifier loss named | 87.7%81.5%–92.0% · 135/154 | 94.2%89.3%–96.9% · 145/154 | 94.2%89.3%–96.9% · 145/154 | 94.2%89.3%–96.9% · 145/154 | 94.2%89.3%–96.9% · 145/154 | 94.2%89.3%–96.9% · 145/154 | 94.2%89.3%–96.9% · 145/154 | 94.2%89.3%–96.9% · 145/154 |
| Supported recall | 67.8%61.2%–73.8% · 141/208 | 67.8%61.2%–73.8% · 141/208 | 67.8%61.2%–73.8% · 141/208 | 67.8%61.2%–73.8% · 141/208 | 68.8%62.2%–74.7% · 143/208 | 68.8%62.2%–74.7% · 143/208 | 69.2%62.7%–75.1% · 144/208 | 69.2%62.7%–75.1% · 144/208 |
| Contradiction detection | 94.0%90.6%–96.2% · 265/282 | 94.0%90.6%–96.2% · 265/282 | 94.0%90.6%–96.2% · 265/282 | 94.0%90.6%–96.2% · 265/282 | 94.0%90.6%–96.2% · 265/282 | 94.0%90.6%–96.2% · 265/282 | 94.0%90.6%–96.2% · 265/282 | 94.0%90.6%–96.2% · 265/282 |
| Abstention rate | 32.1%29.0%–35.4% · 263/820 | 32.1%29.0%–35.4% · 263/820 | 32.1%29.0%–35.4% · 263/820 | 32.1%29.0%–35.4% · 263/820 | 32.2%29.1%–35.5% · 264/820 | 32.2%29.1%–35.5% · 264/820 | 32.1%29.0%–35.4% · 263/820 | 32.1%29.0%–35.4% · 263/820 |
By category (rules 4.0.0)
A category is published at n ≥ 80; below that it is preliminary and its interval is wide.
| Category | Status | False-supported | Supported recall | Qualifier-loss detection | Contradiction detection | Abstention |
|---|---|---|---|---|---|---|
| adversarial n=29 | preliminary | 0.0%0.0%–12.1% · 0/28 | 0.0%0.0%–79.3% · 0/1 | n/a | 14.3%4.0%–39.9% · 2/14 | 93.1%78.0%–98.1% · 27/29 |
| contradicted n=142 | published | 0.0%0.0%–2.6% · 0/142 | n/a | n/a | 96.5%92.0%–98.5% · 137/142 | 3.5%1.5%–8.0% · 5/142 |
| numeric n=126 | published | 0.0%0.0%–3.0% · 0/126 | n/a | n/a | 100.0%97.0%–100.0% · 126/126 | 0.0%0.0%–3.0% · 0/126 |
| qualifier_loss n=154 | published | 1.3%0.4%–4.6% · 2/154 | n/a | 98.7%95.4%–99.6% · 152/154 | n/a | 4.5%2.2%–9.1% · 7/154 |
| supported n=207 | published | n/a | 69.6%63.0%–75.4% · 144/207 | n/a | n/a | 29.9%24.1%–36.5% · 62/207 |
| unsupported n=144 | published | 0.0%0.0%–2.6% · 0/144 | n/a | n/a | n/a | 100.0%97.4%–100.0% · 144/144 |
| unverifiable n=18 | preliminary | 0.0%0.0%–17.6% · 0/18 | n/a | n/a | n/a | 100.0%82.4%–100.0% · 18/18 |
Known limitations
- Short-sentence negation. A sentence under five tokens that flips its source’s polarity (“Monthly plans are refundable.” against “Monthly plans are not refundable.”) is below the template rule’s minimum length, so code alone does not decide it.Code only: Unverifiable; never supported. Model-assisted: Never supported: a model that calls it supported is refused by R-SUPPORT-POLARITY-CONFLICT (rules 1.2.0). Pinned by packages/verify-policy/test/short-negation.test.ts.
- Paraphrased answers. Code settles verbatim support, verbatim qualifier loss and one-value, negation or modality changes against a source sentence. A reworded answer that drops a condition is not caught by code unless the claim’s supporting passage is declared.Code only: Unverifiable (no false support). Model-assisted: Model-assisted qualifier analysis, checked in code. Pinned by packages/verify-policy/test/evaluate.test.ts.
- Arithmetic code recognises. Percent change, a percentage discount, unit price × count, a stated total of figures joined by “and”, a stated difference (“so … 240,000 more”) and a monthly price over a year (rules 1.3.0). Other computations are not settled by code.Code only: A correct figure outside that set is unverifiable; a wrong one is not caught by code. Model-assisted: The model’s arithmetic is recomputed in code over values in the cited evidence; a figure it cannot establish is unsupported. Pinned by packages/verify-core/test/rules-1.3.test.ts.
- Split conditional answers. When model-assisted claim splitting cuts a conditional answer in two (“if it’s unworn with tags attached, it’s returnable within 30 days”), the half without the condition is checked on its own and reported as dropping it, although the answer as a whole states it. The result errs toward revise, never toward release.Code only: Not affected: code checks the answer as written. Model-assisted: A condition the answer states in another clause can be named as dropped (noise, not a release). Pinned by tests/verify/split-claim-noise.test.ts.
- Benchmark coverage. The v1.1 public-source items are U.S. regulatory text, mostly verbatim supported items and negation flips (docs/benchmark-methodology-v1.1-addendum.md §8). English and text only.Code only: — Model-assisted: — Pinned by packages/verify-bench/test/dataset-v11.test.ts.
- Two held-out dropped-condition items. Two held-out qualifier-loss items (2/154) are still released as supported by the deterministic layer at rules 1.1.0, 1.2.0, 1.3.0, 2.0.0 and 2.1.0. Published as losses.Code only: Released as supported (a loss). Model-assisted: Not measured yet (model methods not run). Pinned by packages/verify-bench/results/.
Held-out split
verify-bench-v1-alpha · run 641fd424 · 2026-10-20 · pre-registration sha256 060dc63d0cbd…
| Metric | verify-core, code only (reference) | Single judge: claude-sonnet-5-5 | Verify v1 default |
|---|---|---|---|
| False-supported rate (headline) | 0.0%0.0%–3.3% · 0/112 · lower is better | 8.0%4.3%–14.6% · 9/112 · lower is better | 0.0%0.0%–3.3% · 0/112 · lower is better |
| Unsupported release rate | 0.0%0.0%–6.3% · 0/57 · lower is better | 15.8%8.5%–27.4% · 9/57 · lower is better | 0.0%0.0%–6.3% · 0/57 · lower is better |
| Supported recall | 58.4%47.3%–68.8% · 45/77 | 100.0%95.2%–100.0% · 77/77 | 58.4%47.3%–68.8% · 45/77 |
| Qualifier-loss detection | 100.0%87.1%–100.0% · 26/26 | 53.8%35.5%–71.2% · 14/26 | 100.0%87.1%–100.0% · 26/26 |
| Contradiction detection | 54.5%38.0%–70.2% · 18/33 | 100.0%89.6%–100.0% · 33/33 | 54.5%38.0%–70.2% · 18/33 |
| Abstention rate (unverifiable or error) | 55.6%48.4%–62.5% · 105/189 · lower is better | 2.6%1.1%–6.0% · 5/189 · lower is better | 55.6%48.4%–62.5% · 105/189 · lower is better |
| Exact verdict accuracy | 56.1%49.0%–63.0% · 106/189 | 82.0%75.9%–86.8% · 155/189 | 56.1%49.0%–63.0% · 106/189 |
| Latency p50 / p95 | 0.0s / 0.0s | 1.2s / 1.9s | 4.7s / 5.2s |
| Provider cost per check | 0.000¢ | 0.319¢ | 0.000¢ |
| Items / errors | 189 / 0 | 189 / 3 | 189 / 0 |
95% Wilson intervals. Errors (no usable output) count as abstentions. Item-level verdicts for claim-level systems: an answer counts as supported only if every claim is.
In this run, 7 of 480 Single judge: claude-sonnet-5-5 items had no usable output and count as abstentions.
In this run, 417 of 480 Verify v1 default items ran partially: a model call failed, so Verify used sentence-level claims or left claims unchecked. Their verdicts are included above.
Monitoring schedule: the code-only reference runs every night; the model-based methods run on the held-out split on Sundays (UTC), within a daily spend cap. This replaces the nightly schedule in the pre-registration from 21 October 2026, for cost. The pre-registered evaluation itself is unchanged.
Where Verify loses
- · Median latency: verify-core, code only (reference) is faster (0.0s vs 4.7s).
- · Supported recall: Single judge: claude-sonnet-5-5 100.0% vs Verify v1 default 58.4%.
- · Contradiction detection: Single judge: claude-sonnet-5-5 100.0% vs Verify v1 default 54.5%.
- · Abstention rate (unverifiable or error): Single judge: claude-sonnet-5-5 2.6% vs Verify v1 default 55.6%.
- · Exact verdict accuracy: Single judge: claude-sonnet-5-5 82.0% vs Verify v1 default 56.1%.
- · Median latency: Single judge: claude-sonnet-5-5 is faster (1.2s vs 4.7s).
Pre-registered comparisons (two-sided Fisher’s exact, Holm–Bonferroni, held-out)
- verify_default:false_supported 0/112 vs single_judge:false_supported 9/112: p=0.0033, adjusted 0.0033 — separable at 0.05
- verify_default:supported_recall 45/77 vs single_judge:supported_recall 77/77: p=0.0000, adjusted 0.0000 — separable at 0.05
Development split
Rules and prompts were developed against this split, so these numbers are optimistic. They are shown for completeness, not as the result.
| Metric | verify-core, code only (reference) | Single judge: claude-sonnet-5-5 | Verify v1 default |
|---|---|---|---|
| False-supported rate (headline) | 0.0%0.0%–2.2% · 0/168 · lower is better | 6.5%3.7%–11.3% · 11/168 · lower is better | 0.0%0.0%–2.2% · 0/168 · lower is better |
| Unsupported release rate | 0.0%0.0%–4.5% · 0/81 · lower is better | 13.6%7.8%–22.7% · 11/81 · lower is better | 0.0%0.0%–4.5% · 0/81 · lower is better |
| Supported recall | 62.6%53.8%–70.6% · 77/123 | 95.1%89.8%–97.7% · 117/123 | 53.7%44.9%–62.2% · 66/123 |
| Qualifier-loss detection | 100.0%91.6%–100.0% · 42/42 | 66.7%51.6%–79.0% · 28/42 | 100.0%91.6%–100.0% · 42/42 |
| Contradiction detection | 31.5%20.7%–44.7% · 17/54 | 100.0%93.4%–100.0% · 54/54 | 33.3%22.2%–46.6% · 18/54 |
| Abstention rate (unverifiable or error) | 56.4%50.6%–61.9% · 164/291 · lower is better | 2.7%1.4%–5.3% · 8/291 · lower is better | 54.3%48.6%–59.9% · 158/291 · lower is better |
| Exact verdict accuracy | 55.0%49.2%–60.6% · 160/291 | 79.4%74.4%–83.6% · 231/291 | 52.6%46.8%–58.2% · 153/291 |
| Latency p50 / p95 | 0.0s / 0.0s | 1.3s / 4.5s | 4.8s / 5.5s |
| Provider cost per check | 0.000¢ | 0.323¢ | 0.060¢ |
| Items / errors | 291 / 0 | 291 / 4 | 291 / 0 |
By category
| Category (all splits) | verify-core, code only (reference) | Single judge: claude-sonnet-5-5 | Verify v1 default |
|---|---|---|---|
| supported | 63.7% exactreleased 63.7% · n=190 | 100.0% exactreleased 100.0% · n=190 | 57.9% exactreleased 57.9% · n=190 |
| unsupported | 0.0% exactwrongly released 0.0% · n=48 | 100.0% exactwrongly released 0.0% · n=48 | 0.0% exactwrongly released 0.0% · n=48 |
| contradicted | 71.7% exactwrongly released 0.0% · n=46 | 100.0% exactwrongly released 0.0% · n=46 | 71.7% exactwrongly released 0.0% · n=46 |
| unverifiable | 100.0% exactwrongly released 0.0% · n=44 | 13.6% exactwrongly released 0.0% · n=44 | 100.0% exactwrongly released 0.0% · n=44 |
| qualifier_loss | 80.6% exactwrongly released 0.0% · n=67 | 53.7% exactwrongly released 29.9% · n=67 | 85.1% exactwrongly released 0.0% · n=67 |
| adversarial/multi_hop | 0.0% exactwrongly released 0.0% · n=7 | 71.4% exactwrongly released 0.0% · n=7 | 0.0% exactwrongly released 0.0% · n=7 |
| adversarial/wrong_date | 9.1% exactwrongly released 0.0% · n=11 | 100.0% exactwrongly released 0.0% · n=11 | 9.1% exactwrongly released 0.0% · n=11 |
| adversarial/wrong_scope | 0.0% exactwrongly released 0.0% · n=6 | 50.0% exactwrongly released 0.0% · n=6 | 0.0% exactwrongly released 0.0% · n=6 |
| adversarial/wrong_entity | 10.0% exactwrongly released 0.0% · n=10 | 100.0% exactwrongly released 0.0% · n=10 | 20.0% exactwrongly released 0.0% · n=10 |
| adversarial/cherry_picked | 0.0% exactwrongly released 0.0% · n=9 | 88.9% exactwrongly released 0.0% · n=9 | 0.0% exactwrongly released 0.0% · n=9 |
| adversarial/stale_superseded | 0.0% exactwrongly released 0.0% · n=11 | 100.0% exactwrongly released 0.0% · n=11 | 0.0% exactwrongly released 0.0% · n=11 |
| adversarial/contradictory_sources | 100.0% exactwrongly released 0.0% · n=11 | 0.0% exactwrongly released 0.0% · n=11 | 100.0% exactwrongly released 0.0% · n=11 |
| adversarial/citation_wrong_sentence | 10.0% exactwrongly released — · n=10 | 40.0% exactwrongly released — · n=10 | 10.0% exactwrongly released — · n=10 |
| adversarial/arithmetic_missing_input | 0.0% exactwrongly released 0.0% · n=10 | 80.0% exactwrongly released 0.0% · n=10 | 0.0% exactwrongly released 0.0% · n=10 |
External baselines
- aws_bedrock_contextual_grounding — not run: credentials absent (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, BEDROCK_GUARDRAIL_ID)
- google_vertex_check_grounding — not run: credentials absent (GOOGLE_CLOUD_PROJECT, GOOGLE_APPLICATION_CREDENTIALS_JSON)
- azure_groundedness_detection — not run: credentials absent (AZURE_CONTENT_SAFETY_ENDPOINT, AZURE_CONTENT_SAFETY_KEY)
- vectara_hhem — not run: credentials absent (VECTARA_API_KEY)
No result is shown for a baseline that was not run. None is estimated.
Calibration
Bins fixed in advance on Verify’s internal score; precision measured on dev and checked on held-out (ECE 0.002). Validated: API results carry calibrated_confidence labelled "calibrated on benchmark verify-bench-v1-alpha@2026-10-20".
A calibrated confidence is a measured hit rate on labelled benchmark items, never a probability that a claim is true.
The dataset
480 published items across policies, plan and pricing pages, product docs and help-centre articles, all about fictional organisations and written for this benchmark. Categories: supported (verbatim and paraphrase), contradicted (one planted number, date, entity or negation), qualifier loss, unsupported, unverifiable, and nine adversarial categories.
48 items are in the review queue awaiting a human label and are excluded from every score above.
Known limitations: items were written with help from Claude-family models, which overlap with some methods tested; answers are single sentences; English and text only; small samples per adversarial category give wide intervals.
Download the dev split (JSON) · construction procedure, scripts and the held-out split ship with the open-source @xyntherium/verify-bench package.
Challenge an item
Think a label is wrong or ambiguous? Challenged items go to a person and out of the scores until resolved.
Also from Xyntherium: the Drift Board, nightly measurements of how six models’ behaviour changes. Methodology: docs/benchmark-methodology.md and the v1.1 addendum, pre-registered before the first run. Every rules change: rules changelog.