Evaluate what the agent did—not only what it said

Define the unit of evaluation

The unit may be a complete case, a tool transition, a planning decision, a human-review handoff, or an entire long-running workflow. Define it before collecting scores so results can be compared across changes.

NIST's evaluation-probe work demonstrates the value of targeted behavioral tests for agents, including manipulation and task behavior. [S1] [S2]

Build representative cases

Cases should reflect the actual users, data, environments, tools, and consequences. Include normal work, edge conditions, adversarial content, missing evidence, stale state, access failures, timeouts, tool errors, reviewer disagreement, and partial execution.

A polished demo set selected by the implementation team is not a release baseline. Cases need named owners, source provenance, expected behavior, acceptable alternatives, and severity for failure.

Measure the trajectory

Capture the plan, each observation, retrieved evidence, tool choice, arguments, authorization result, response, verification, state transition, retry, and stop reason. This permits reviewers to distinguish a correct answer from a safe and reproducible process.

• Goal preservation and task decomposition
• Source relevance, authorization, and citation correctness
• Tool selection, input validation, and verified effect
• Memory provenance and response to conflicting state
• Retry safety and recovery after partial failure
• Refusal, escalation, and human-review behavior
• Cost, latency, token use, and operational load

Use automated evaluators carefully

Model-based judges can scale review, but they are also probabilistic systems. Calibrate them against human decisions, track disagreement, version their prompts and models, and avoid using one evaluator as the sole release authority.

Deterministic checks remain important for schema validity, authorization, tool allowlists, exact citations, prohibited actions, latency limits, cost budgets, and side-effect reconciliation.

Evaluate security behavior

Agentic systems need cases for direct and indirect prompt injection, goal hijacking, malicious tool output, memory poisoning, identity confusion, data exfiltration, over-delegation, and cascading failure. OWASP's agentic guidance provides a useful threat vocabulary, but use-case-specific attack paths still need to be modeled. [S3]

Evaluate multi-agent coordination

For teams, measure role usefulness, communication overhead, disagreement handling, aggregation, minority evidence retention, and whether the team improves on a strong single-agent baseline. More messages or more participants are not evidence of better reasoning. [S4] [S5]

Translate results into release gates

A release gate ties evidence to a decision. It should name the permitted scope, blocking failures, accepted residual risk, required human review, monitoring, rollback condition, and owner. A failed gate should produce a bounded remediation decision—not an endless tuning cycle.

Research boundary

No benchmark or evaluation harness proves general safety. Results apply to the cases, versions, tools, policies, and environment tested. Material changes require re-evaluation, and domain experts must own consequence-specific judgments.