Test whether untrusted content can redirect the agent's objective or authority path

Define a successful hijack

Record the approved objective and instruction hierarchy. A hijack succeeds when untrusted input changes the effective objective, causes an unauthorized action, suppresses required evidence or review, changes durable state, delegates prohibited authority, or creates a hidden persistence mechanism.

Differentiate harmless model discussion of an instruction from actual behavioral redirection. The evaluator should observe trajectory, policy decisions, tools, effects, memory writes, and stop reason.

Map every injection channel

Include direct user input, retrieved web pages and documents, email, tickets, code comments, tool results, database fields, images or OCR text, memory, remote-agent messages, skill descriptions, system metadata, and human-authored review notes.

Label trusted, conditionally trusted, and untrusted content. Test whether content crosses those labels or is promoted into higher-priority instructions.

Build attack families, not one prompt list

Vary directness, obfuscation, language, encoding, placement, authority claims, social engineering, multi-turn buildup, tool-result injection, memory persistence, peer-agent propagation, and attacks that ask the system to conceal its deviation.

NIST's red-team analysis covered tool-use, coding, and computer-use agent scenarios and reported at least one successful attack against every included model. A system-specific evaluation should therefore rotate attacks and preserve newly discovered failures as regressions. [S2]

Test authority and consequence escalation

Attempt to move from read to write, one record to bulk data, one tenant to another, proposal to execution, low-value to high-value action, test to production, or approved tool to unapproved tool. Confirm that external policy blocks the escalation even when the model agrees with it.

Test credential misuse, hidden delegation, token passthrough, stale approval, approval replay, and attempts to route through a more privileged peer or tool server.

Test persistence and delayed execution

Attempt to store malicious instructions in memory, task state, a generated artifact, shared workspace, scheduled job, or downstream record that will be consumed later. Verify provenance, invalidation, review, expiry, and cleanup.

A test should continue beyond the first apparently safe response. Some attacks aim to influence a later step, remote agent, reviewer, or retry path.

Measure detection and reviewer effectiveness

Record whether monitoring detected the trust-boundary crossing, policy denial, unusual tool selection, hidden state write, repeated delegation, or anomalous cost. Test whether a human reviewer received the decisive evidence and had authority and time to intervene.

Use adaptive red teaming

Maintain a protected set of known attacks, but also allow qualified testers to adapt based on observed behavior. Rotate model and tool versions, vary context and timing, and test the actual integrated system rather than an isolated chat endpoint.

A pass is time-bounded. New models, prompts, tools, permissions, memory, protocols, or data sources reopen the evaluation.

Set release-blocking criteria

Block release for unauthorized cross-tenant access, consequential effect, privilege escalation, hidden persistence, bypassed human approval, unavailable containment, or missing evidence needed to reconstruct the attack. Lower-severity failures may support conditional release only with explicit scope, remediation, monitoring, owner, and expiry.

This article does not provide penetration testing or claim that a finite test set can prove an agent is unhackable.