AI Policy Evidence Quality Benchmark

Important limitation: each question was run once per model. The scores are rough indications of how the method works, not statistical measurements of model performance.

How to read the score

Cutoffs are deliberate: an answer must average 2.55 or higher to count as Strong, so a 2.5 sits at the top of Moderate.

The headline score averages four areas: source quality, framework coverage, jurisdiction relevance, and policy usefulness. Decimal scores appear because they are averages. For example, an answer can score between whole-number levels across the four areas.

Omission risk is coded separately. It checks whether an answer leaves out important risks, safeguards, legal issues, affected groups, or evidence gaps.

Omission risk is kept separate from the headline score so a polished answer can still be flagged for missing something important.

Filter sample cards

These 6 sample cards cover 4 of the 10 question topics. All 30 scored answers are in the downloads below.

Where the data comes from — and its limits

What the benchmark can and cannot show

Transcripts were captured manually from a single run per model. Citation URLs were not preserved in this version, so source-quality scores reflect the author's reading of each response's prose and named sources, not verification of returned links. Treat the results as evidence that the scoring method works, not as proof of which model performs best.

Benchmarking the benchmark

Applying this benchmark's own rubric to itself: one run per question, one person doing the scoring, and only 10 questions per model means these results score no better than "relevant but incomplete" as evidence. They are published as a methodology prototype. Version 1 also did not save the web links the models cited. Version 2 is planned with pinned model versions, three runs per prompt, an expanded prompt set, capture and publication of per-response citation URLs and share links or screenshots, and a second independent scorer with inter-rater reliability reporting.

Raw response transcripts

The raw response downloads preserve the 30 model outputs evaluated in the benchmark, alongside prompt id, category, model, documented model version, test date, and browsing condition.

Downloads and methodology

The benchmark preserves the raw model responses, prompts, coded results, and methodology files for review, replication, and inspection of the underlying transcripts.