# EVALS.md

> Define representative tasks, outcomes, failure taxonomies, graders, human review, operational metrics, release thresholds, and regression processes for behavior that varies across models and environments.

**Decision rule:** Evaluate the complete workflow against realistic outcomes and failure paths. Use deterministic checks where possible and calibrated human or model judgment where necessary.

## How to use this template

- Replace every angle-bracket placeholder with project-specific evidence.
- Remove sections that do not apply instead of leaving ambiguous boilerplate.
- Link decisions to their specifications, tasks, tests, traces, and owners.
- Re-review this file when the workflow, authority boundary, or primary source changes.

## Outcomes and failure taxonomy

> Measure the behavior that matters instead of a convenient proxy.

- **Success:** <Define task completion, quality, safety, latency, cost, and user-correction outcomes>
- **Failures:** <Classify instruction, context, reasoning, tool, authorization, data, and recovery failures>
- **Severity:** <Define critical, major, minor, and acceptable variance with examples>

## Evaluation set

> Represent normal use, boundaries, adversarial cases, and longitudinal behavior.

- **Cases:** <List source, owner, input, expected evidence, and protected data classification>
- **Coverage:** <Map cases to requirements, tools, users, languages, failures, and risk levels>
- **Versioning:** <Define frozen sets, hidden sets, additions from incidents, and contamination controls>

## Graders and metrics

> Make every score interpretable and periodically calibrate judgment.

- **Deterministic checks:** <List schemas, exact properties, tests, policy checks, and artifact inspection>
- **Judgment:** <Define rubric, human reviewers, model graders, blind review, and disagreement handling>
- **Operational metrics:** <Track completion, correction, retries, tool errors, latency, tokens, cost, and abandonment>

## Release and regression

> Turn evaluation into a shipping control rather than a report.

- **Thresholds:** <Define minimums, critical-case zero tolerance, confidence, and comparison baseline>
- **Release gate:** <State when a change blocks, warns, rolls out gradually, or requires sign-off>
- **Regression loop:** <Add incidents and user corrections, assign owners, and verify fixes across supported models>

## Related AgentAtlas field guides

- [Evals and verification](https://agentatlas.dev/topics/evals-verification)
- [Observability](https://agentatlas.dev/topics/observability)
- [Tasks](https://agentatlas.dev/topics/tasks)

## Primary references

- [Evaluation best practices](https://developers.openai.com/api/docs/guides/evaluation-best-practices) — Eval design, task-specific datasets, graders, continuous evaluation, and avoiding common measurement mistakes.
- [Demystifying evals for AI agents](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents) — Agent-specific evaluation design, outcome and process measures, transcript analysis, and iterative improvement.
- [OpenTelemetry semantic conventions](https://opentelemetry.io/docs/specs/semconv/) — Standard names, attributes, traces, metrics, logs, and generative-AI operations for interoperable telemetry.
- [OpenAI Agents SDK tracing](https://openai.github.io/openai-agents-python/tracing/) — Tracing model generations, tools, handoffs, guardrails, custom events, workflow grouping, and sensitive-data controls.
- [Agentic SDD workflow](https://github.github.com/spec-kit/reference/agentic-sdd.html) — Generating and analyzing implementation tasks after specification and planning quality gates.
- [A2A protocol specification](https://a2a-protocol.org/latest/specification/) — Task identity, lifecycle, messages, status, artifacts, cancellation, and communication across agent boundaries.

---

- Template version: 1.0.0
- Reviewed: 2026-07-26

Generated deterministically from reviewed AgentAtlas records.