Evals and verification
A repeatable evidence loop that measures whether an agent completes representative tasks correctly, safely, and efficiently.
Use it when
- A prompt, model, tool, or agent workflow will be reused in production.
- Teams need to know whether an update improved quality or only changed style.
- Failures can be turned into representative regression cases.
Choose another layer when
- A vanity benchmark that does not resemble the real workload.
- A single aggregate score that hides severe safety or task-specific failures.
Production path
Build it in this order
- 01
Define observable success
Collect representative inputs, expected properties, required evidence, forbidden behavior, and useful cost or latency measures.
Quality check: A reviewer can decide pass or fail without knowing which model produced the output.
- 02
Use layered graders
Combine deterministic checks, rubric-based model grading, trace inspection, and human review where judgment matters.
Quality check: Known good and known bad examples produce the expected grade.
- 03
Make every failure reusable
Classify failures, add important cases to the suite, and compare changes on the same set before rollout.
Quality check: A fix cannot silently regress an older capability or safety boundary.
Artifacts you may see
- evals/*
- fixtures/*
- grader schemas
- trace datasets
Connected layers
Evidence ledger
Primary documentation behind this guide
Provider-specific behavior is sourced to that provider. Open standards are sourced to their specifications and official maintainers.