Verify benchmark

Verify compared with other checkers

We gave 7 checkers the same 300 new answers, each with the passages it should be based on, and counted what each let through. 220 of the answers are flawed: they change a number, reverse a rule, drop a condition, or say something the passages never say. 80 are fully supported by their passages. Each checker ran once, on 2026-10-19.

How the items are built: /verify/benchmark/method · Verify’s own benchmark: /verify/benchmark

Read this with one caveat in mind: we build Verify, and we built this test around the kinds of flaw Verify was designed to catch. That favours Verify. Why, and by how much.

The result

Verify, code only, let through 8 of 220 flawed answers (3.6%). The next lowest was AI judge: GPT-6.1 Sol with 52 of 220 (23.6%). The difference is answers that drop a condition: Verify stopped 72 of 80; the AI judges stopped between 5 and 28. On changed numbers, reversed rules and text the passages never mention, every checker except AgentClaimGuard stopped all or nearly all.

On supported answers Verify did worse: it let through 77 of 80, while every AI judge let through all 80. And Verify does not always decide: it sent 66 of 300 answers to a person. Every other checker decided every answer.

Verify 2.1.0 failed our pre-registered 2% false-approval bar on the fresh set: 8 of 220 (3.6%, CI 1.9–7.0%). We set that bar before this run, and it has not been changed. Verify’s full path, which adds an AI model to the rules, has not been run on this set yet.

Decision (2026-10-19): continue; next version must pass a new pre-registered bar on a new fresh set. That bar is no more than 2% of flawed answers let through and no more than 30% of all answers sent to a person, on 300 new answers built and fixed before the next version runs on them.

Rules 2.2.0 was run once against that bar on 2026-10-20 and failed it: it let through 6 of 220 flawed answers (2.7%, CI 1.3–5.8%) on the second fresh set, where the limit was 4. It sent 68 of 300 answers to a person, within the 30% limit. It was not released; Verify still runs rules 2.1.0, which lets through the same 6 of 220 on that set. The next attempt needs a third fresh set and a new bar, both fixed before it runs.

What each checker let through

Fewer flawed answers let through is better; more supported answers let through is better. The small grey range under each figure is its 95% interval: with this many items, the rate a much larger test would find is likely inside it. Where two ranges overlap, this test cannot tell the two checkers apart.

Cost: provider tokens at list prices on 2026-10-19 (USD per million input/output tokens: GPT-6.1 Sol $2/$10, Claude Opus 5.5 $4/$20, Claude Sonnet 5.5 $2/$10, Gemini 3.1 Pro $2/$12). $0 means it runs on your own computer. Time for the AI judges and RAGAS includes the network round trip; Verify, the NLI model and AgentClaimGuard ran locally.

What each checker stopped, by kind of flaw

“Stopped” means not let through: refused, sent back to be fixed, or sent to a person.

Regulations and support policies, separately

Half of the answers come from U.S. regulations, half from support policies of made-up companies. Flawed answers let through, by half:

Where Verify loses

• It let through 8 flawed answers, all of them answers that drop a condition, which is more than the 2% we set as Verify’s failure line. Five are the same sentence at five made-up companies: a furniture warranty answer that leaves out “with proof of purchase”. Two come from the air-travel disability rules (14 CFR 382) and one is an appointment answer that leaves out “after verifying their insurance”.
• One of those 8 is, we believe, labelled wrongly. The regulation says a battery need not be removed “if [it] has been labeled… as non-spillable, or if [the wheelchair] can be loaded… in an upright position”. The answer keeps only the second “if”. A rule that holds when either condition is met still holds when the second one is, so the passage supports the answer. We found this with a check of every dropped-condition item that does not look at any checker’s output (it flags a dropped condition joined by “or”), after seeing Verify’s result. No other item is flagged by that check. The label stays as committed; without that item Verify let through 7 of 219 (3.2%), still over 2%.
• It let through 77 of 80 supported answers. The AI judges let through all 80. The 3 it missed write a number as a word (“ten days” where the regulation says “10 days”); Verify did not match them and sent them to a person.
• It sent 66 of 300 answers to a person instead of deciding: all 60 answers whose content is not in the passages, 3 reversed rules, and the 3 number-as-word answers. None was let through, but someone has to read each one. The AI judges named the problem in all 60 of those “not in the passages” answers.
• It matches text; it does not interpret it. An answer the passage supports, but in different words, or that combines two passages, is likely to go to a person. Every answer in this set is one sentence close to a passage, which suits code-only Verify; longer free-form answers are not tested here.

Why this test favours Verify

• We build Verify, and we built this test, with the same method as Verify’s own benchmark. The four kinds of flaw are the ones Verify was designed to catch. No part of Verify was changed for, or developed on, these items.
• Dropped conditions are 80 of the 220 flawed answers. That is where the checkers differ most, so the mix decides much of the ranking. A test with fewer of them would narrow the gap.
• The AI judges used one short prompt, written before this test existed, and were not tuned. A prompt that asks specifically about conditions and exceptions might let fewer through.

Limits of this test

• The intervals are too narrow for the support-policy half. Its 150 answers come from 25 sentence templates, and answers from the same template tend to pass or fail together, as Verify’s five furniture-warranty misses show. The 95% intervals treat every answer as independent.
• The regulation half leans on two rules. 70 of its 150 answers come from the air-travel disability rules (14 CFR 382) and 45 from the overtime pay rules (29 CFR 778). The rest come from deposit accounts (12 CFR 1030), consumer leasing (12 CFR 1013), health breach notification (16 CFR 318), pay-per-call services (16 CFR 308), unfair fees (16 CFR 464) and bus travel (49 CFR 374).
• Labels come from how each answer was built, not from people reading them, so mistakes like the one above are possible. The answers are short, one sentence each, and some begin awkwardly where a clause was cut (“Or if…”).
• The items are not published, so others cannot repeat this run. Their fingerprint (SHA-256 8e61e9971e022dbcb119f9567f07d1689056cd0efe5117217e1f925b42bc35fc) was recorded in our repository before any checker ran. The items stay private because together with benchmark v1.2’s new test set they are the fresh set Verify’s failure line is measured on.

How each checker was set up

Every checker received the same question, answer and passages, and nothing else: no web search, no tools.

Verify 2.1.0, code only

@xyntherium/verify-core with rules 2.1.0, no model. Let through = released as “allow”; sent to a person = “human review”.

AI judges

One call per answer with the prompt below, unchanged since Verify’s benchmark registered it in advance, and the question, answer and passages. Let through = the judge answered “supported”. Models called through each provider’s own API, with default settings and no tools.

You are a grounding judge for a retrieval-augmented generation system. Given a question, an AI-generated answer, and the retrieved context passages, decide whether the answer is grounded in the context.

Labels:
- supported: the context fully supports the answer.
- contradicted: the context contradicts the answer.
- unsupported: the context does not contain the information in the answer.
- unverifiable: the answer cannot be checked against the context.

Use only the context. Reply with ONLY a JSON object: {"verdict":"supported|contradicted|unsupported|unverifiable","reason":"one sentence"}

RAGAS faithfulness

RAGAS 0.4.3’s Faithfulness metric with its own prompts: it splits the answer into statements, then asks whether each follows from the passages. The model is Claude Sonnet 5.5, the same as the registered judge, so the difference is the method. Let through = every statement follows (a score of 1.0), a rule fixed before the run.

Open NLI model

MoritzLaurer/mDeBERTa-v3-base-mnli-xnli, the model family eleata-verify-mcp says its hosted service uses. The passages are cut into windows of whole sentences; the answer is let through if any window “entails” it.

AgentClaimGuard 0.4.3

Its built-in RAG policy, set up as its quickstart shows: the answer is one claim that cites the supplied passages. AgentClaimGuard checks that each claim cites evidence of the required type; its own documentation says it does not judge whether the evidence supports the claim. Every answer here cites its passages, so it let every answer through. This is the tool working as designed, not a defect: it is meant to run alongside a checker that reads the evidence.

Checkers we did not run

None of these needs a sales call. Each needs something this run did not have:

• Verify, full path (rules plus an AI model; the default for API checks): needs a production account key, which this run did not use. To be added.
• AI judge: Gemini 3.1 Pro: the provider’s API quota was exhausted partway through (144 of 300 answers had no reply), so there is no complete result. To be rerun.
• AWS Bedrock Guardrails (contextual grounding check), Azure AI Content Safety (groundedness detection), Google Vertex AI (check grounding): each needs an account with that cloud provider, which we have not set up.
• eleata-verify-mcp: its hosted service needs a paid key. We ran the open model it says it uses instead (above), which is not the same as its service.
• groundcheck-mcp: it checks claims against a live web search and has no way to accept passages you supply, so it does a different job from this test, which is about the passages an answer was given. Half of these answers are about made-up companies that no search would find.

Supported means supported by the supplied evidence, not true in the world. Results describe whether the supplied passages support an answer, not whether it is true. Scored 2026-10-19.