How we tested
KYP benchmark · November 2026
In short
We gave the same frontier AI model (Claude Opus 5.5) to six approaches, KYP and five common ways of using AI on documents, and asked each one to find the problems in real renewable energy permit documents. The answer key was what government regulators actually wrote back to the developers.
- On the 13 cases KYP had never seen, KYP found more of the regulator’s problems than every other approach, in every case (13 of 13).
- KYP found about 70% of the regulator’s problems on average; the best alternative found about 37%.
- KYP found about 80% of the critical and high-severity problems; the best alternative about 56%.
- Every KYP finding pointed to the exact sentence on the page it cites. Frontier AI’s did 62% of the time.
1. The question
When a developer applies for a permit, the authority writes back listing what is wrong or missing in the application. That is what a buyer, lender or adviser needs to find before signing. So we asked: given the developer’s own documents, how much of what the regulator found can each approach uncover? We did not decide what counts as a problem; the regulator did.
2. The cases
20 public renewable energy permit cases. Each pairs the developer’s document (an environmental impact study, a scoping report or a permit application) with the authority’s written response to it. The approaches read only the developer’s document.
| Country | Authority | Cases | Developer pages |
|---|---|---|---|
| Italy | Ministry of the Environment, technical commission | 7 | 1,651 |
| United Kingdom | Planning Inspectorate | 4 | 1,498 |
| United States | California Energy Commission; New York renewable siting office | 6 | 862 |
| Spain | Regional environmental authority (Castilla y León) | 3 | 1,100 |
| Total | 5 authorities, decisions from 2020 to 2026 | 20 | 5,111 |
Projects cover solar, agrivoltaic, onshore wind and battery storage, over 4 GW in total, in Italian, English and Spanish. Documents run from 55 to 454 pages.
The 20 cases were used in three groups:
- 7 development cases. Used to improve KYP. Excluded from every result on this page.
- 7 final-exam cases. Picked at random before any development work, then run once on the final version of KYP.
- 6 new cases. Added after development was finished. Their answer keys were written and frozen before any approach ran on them.
All results below come from the 13 final-exam and new cases.
3. The answer key
Each authority response was turned into a list of items under written rules fixed before any run: one item per problem, a severity (critical, high, medium or low), and whether the problem is visible in the developer’s document. Only visible ("findable") problems are scored, because no approach can find what is not on the page.
| Items | |
|---|---|
| Problems listed by the authorities | 979 |
| Visible in the developer’s documents | 673 |
| Visible and critical or high severity | 241 |
Three independent reviewers checked a random quarter of the items against both documents; every requested fix was made, then each answer key was frozen with a fingerprint and never edited. The answer keys were built and checked by AI reviewers, not yet by a human expert.
4. The six approaches
All six used the same AI model. Each rival was set up as a competent team would set it up, and a separate review checked that none was weakened and none used KYP’s instructions or the answer keys.
| Approach | What reaches the AI model |
|---|---|
| KYP | Every page, in order, with memory of what was already found; a check of what a permit file should contain; quotes verified on their page |
| Frontier AI | The whole document at once, in a 1-million-token window |
| RAG | The passages a search ranks as most relevant |
| Advanced RAG | Several search questions, keyword and meaning-based search, and an AI reranker |
| Agentic RAG | The AI decides what to look up, step by step |
| GraphRAG | A map of the document’s entities, combined with meaning-based search |
5. How we scored
- Problems found: the share of the visible problems each approach found. A match on the problem’s key terms counts; every critical or high problem an approach missed on that basis was then read by hand by a separate reviewer, in case it was found in other words.
- Strict re-score: a tougher count where a reviewer must agree the finding really raises the authority’s point.
- Useful findings: blind raters judged a sample of each approach’s findings (30 per case) without knowing which approach wrote them.
- Quotes on the page: whether each quoted piece of evidence appears word for word on the page it cites.
6. Results
All 13 cases KYP had never seen
| Approach | Problems found |
|---|---|
| KYP | ~70% |
| Frontier AI | ~37% |
| Advanced RAG | ~35% |
| GraphRAG | ~34% |
| RAG | ~33% |
| Agentic RAG | ~31% |
Averages of the per-case results. KYP found more than the best of the five alternatives in all 13 cases; if the approaches were equally good, that would happen by chance about once in 8,000 tries. On critical and high problems KYP averaged about 80%, against about 56% for the best alternative. Every approach ran on all 13 cases.
The 6 new cases
| Approach | Problems found | Critical and high | Useful findings | Quotes on the page |
|---|---|---|---|---|
| KYP | 59.9% | 76.6% | 88.9% | 100% |
| Frontier AI | 29.2% | 50.0% | 84.4% | 62% |
| Advanced RAG | 26.3% | 41.5% | 70.0% | 96% |
| GraphRAG | 24.2% | 37.9% | 84.4% | 98% |
| Agentic RAG | 22.8% | 39.0% | 93.3% | 100% |
| RAG | 21.7% | 36.3% | 87.8% | 98% |
Useful findings were rated on 3 of the new cases. Raters agreed 91% of the time.
The 7 final-exam cases
| Approach | Problems found | Critical and high |
|---|---|---|
| KYP | 78.4% | 83.0% |
| RAG | 43.2% | 55.6% |
| Frontier AI | 42.7% | 61.8% |
| Advanced RAG | 42.6% | 50.1% |
| GraphRAG | 42.6% | 53.9% |
| Agentic RAG | 37.3% | 56.0% |
Three files of more than 400 pages were too long for Frontier AI to open; KYP read every page of all 20 cases.
7. Is it just the instructions?
We gave Frontier AI KYP’s own instructions, word for word, but none of KYP’s system around the model: no page-by-page reading, no memory, no targeted search, no code that checks quotes.
| 13 cases | Problems found | Critical and high | Findings per document |
|---|---|---|---|
| KYP | 69.9% | 80.0% | 149 |
| Frontier AI with KYP’s instructions | 49.6% | 72.9% | 38 |
| Frontier AI | 36.5% | 56.4% | 76 |
KYP’s instructions alone account for about half of the gap. The system around the model accounts for the rest: KYP still found more in 12 of 13 cases. On critical and high problems the difference is small and not statistically clear.
Second test: seller claims against official records
A seller’s brochure can get a figure wrong: the capacity, the status, the connection date. This test asks one question: when a brochure contradicts the official record, which method catches it?
- 10 seller brochures for real public assets (US ×3, UK ×2, Germany, France, Greece, Spain, Italy)
- 31 planted wrong claims
- 29 true claims, as controls
- 3 KYP runs (Claude Opus 5.5 twice, Claude Sonnet 5 once)
Draft: we prepared the brochures and the answer key ourselves, and the key is awaiting review by an analyst. The figures may change after that review.
- Brochures. For each asset we wrote a short teaser and an information-memorandum extract (3–4 pages). Every page is marked as a synthetic test document, not issued by the owner.
- What we planted. Each brochure has 2–4 claims that contradict the official record (capacity, status, connection or commissioning date, licence holder, turbine count, protected-area distance, grid connection point and voltage, production) and 2–3 true claims. The answer key was never shown to any method.
- Methods. KYP, the same AI model prompted directly, and the same AI model with web search. Every method got the same brochure and the same short project profile. In KYP, an analyst typed the seller’s headline capacity into the profile and, for the two operating US plants, the seller’s production figures, as an analyst would.
- Scoring. An error counts as caught when the method calls the claim wrong and gives the official value or a clear contradiction. “Doubtful” without a value does not count. A false positive is a true claim called wrong. Every output was scored by hand.
| Method | Planted errors caught | True claims flagged wrong | Catches citing an official record with its date |
|---|---|---|---|
| KYP (all 3 runs gave the same result) | 90.3% (28/31) | 0% (0/29) | 96% (27/28) |
| Frontier AI + web search (Claude Opus 5.5, 2 runs) | 90.3% | 15.5% | 37% |
| Frontier AI (Claude Opus 5.5, 2 runs) | 29.0% | 6.9% | 0% |
| KYP before this round of fixes | 22.6% | 0% | 86% |
KYP caught as many errors as Frontier AI with web search, with no false positives. Web search flagged about 1 in 6 true claims as wrong, and cited a dated official record for about a third of its catches. KYP’s first run caught 22.6%; we then added the comparisons it was missing and ran the same brochures again.
What this does not show
- We wrote the brochures and planted the errors ourselves, and we fixed KYP after seeing the first run. A fresh set that we have not seen is the next test.
- Italy has no public registry that KYP can query. KYP’s only 3 misses were the Italian errors, and it said it could not check them.
- The answer key has not yet been reviewed by an independent analyst.
- The sample is small: 31 errors across 10 brochures.
- KYP used two figures an analyst typed in (headline capacity, and production for the US plants). Without the capacity figure its registry check would not run.
- Official records can lag reality. Each catch shows the date of the record it used.
Limitations
- The answer keys and the usefulness ratings were produced by AI reviewers and have not yet been independently validated by human experts.
- Each approach ran once per case. Results are from 13 cases in four countries; Germany and Greece are not yet covered.
- The final-exam cases were looked at after their run, while KYP was being developed further; the 6 new cases were not.
- Spanish cases are scored against the authority’s permit conditions, a slightly different task.
Questions about the method, or the full report with every case: jonathan.abraham@reciv.net.