AssuranceReviewed Markdown template

EVALS.md

Agent evaluation system

Define representative tasks, outcomes, failure taxonomies, graders, human review, operational metrics, release thresholds, and regression processes for behavior that varies across models and environments.

Download EVALS.mdReview the fields

Fillable structure

Replace prompts with project evidence.

Remove any field that does not apply. A smaller maintained file is more useful than generic documentation that agents and reviewers cannot trust.

  1. 01

    Outcomes and failure taxonomy

    Measure the behavior that matters instead of a convenient proxy.

    • Success: Define task completion, quality, safety, latency, cost, and user-correction outcomes

    • Failures: Classify instruction, context, reasoning, tool, authorization, data, and recovery failures

    • Severity: Define critical, major, minor, and acceptable variance with examples

  2. 02

    Evaluation set

    Represent normal use, boundaries, adversarial cases, and longitudinal behavior.

    • Cases: List source, owner, input, expected evidence, and protected data classification

    • Coverage: Map cases to requirements, tools, users, languages, failures, and risk levels

    • Versioning: Define frozen sets, hidden sets, additions from incidents, and contamination controls

  3. 03

    Graders and metrics

    Make every score interpretable and periodically calibrate judgment.

    • Deterministic checks: List schemas, exact properties, tests, policy checks, and artifact inspection

    • Judgment: Define rubric, human reviewers, model graders, blind review, and disagreement handling

    • Operational metrics: Track completion, correction, retries, tool errors, latency, tokens, cost, and abandonment

  4. 04

    Release and regression

    Turn evaluation into a shipping control rather than a report.

    • Thresholds: Define minimums, critical-case zero tolerance, confidence, and comparison baseline

    • Release gate: State when a change blocks, warns, rolls out gradually, or requires sign-off

    • Regression loop: Add incidents and user corrections, assign owners, and verify fixes across supported models

Related field guides

Primary references

Next template

SAFETY.md · Safety and permission system