Build evaluation cases that represent the decision, the edge conditions, and the ways the system can fail

Begin with the decision and failure consequence

State the task, affected people, allowed actions, required evidence, unacceptable outcomes, and what a release decision must determine. The dataset should reflect those consequences rather than optimizing a generic score.

Build a case taxonomy

Use multiple case families so average performance does not hide a dangerous slice.

• Representative ordinary cases across real user and data variation
• Boundary and rare cases that stress assumptions
• Missing, stale, ambiguous, and conflicting evidence
• Prohibited or out-of-scope requests
• Indirect prompt injection, poisoning, and malicious tool content
• Authorization, tenant, identity, and delegation cases
• Timeout, provider failure, retry, restart, and unknown completion
• Human-review, escalation, override, and appeal cases
• Material changes to model, prompt, retrieval, tool, or policy

Separate outcome labels from trajectory labels

Outcome labels describe whether the final result is correct, complete, safe, or useful. Trajectory labels describe whether the plan, evidence, authorization, tool calls, state transitions, delegation, retries, and stop reason were acceptable.

Trajectory-oriented research and log analysis show why process evidence is needed for credible agent evaluation. [S1] [S2]

Use human adjudication where judgment is real

Define reviewer qualifications, rubric, disagreement handling, second review, and uncertainty. Do not hide subjective or domain-specific judgments behind an automated evaluator without validation.

Protect against test leakage and overfitting

Separate development and release sets, rotate or add hidden cases, record which cases were visible during tuning, and watch for prompts or retrieval content that encode expected answers. Preserve a stable core for regression and an evolving set for new failures.

Version the complete evaluation environment

Record dataset version, model and provider, prompt and policy, retrieval corpus and index, tools, credentials and scopes, orchestration, evaluator, environment, seeds where relevant, and time. Without this context, a score cannot be reproduced or compared meaningfully.

Use slice-level decision thresholds

Define acceptance by consequence-relevant slices rather than one average. A system may perform well overall while failing one tenant boundary, user population, tool, document type, or high-consequence action.

Connect cases to monitoring

Production samples and incidents should generate new cases. Monitoring should detect when real input or behavior falls outside the evaluated envelope. NIST's monitoring work identifies context change, drift, fragmented logging, and oversight scale as persistent challenges. [S4]

Research boundary

This guide does not define universal metrics or acceptable thresholds. Domain experts, affected stakeholders, security, privacy, accessibility, legal, and operational reviewers may be required to define credible cases and judgments.