Prompt Injection Testing for AI Agents: Threat Modeling, Exploitation and Control Validation

Prompt injection is not fundamentally a prompt-writing problem. It is a control-boundary problem: untrusted content can influence a model that also receives trusted instructions, and the model may then select tools or produce outputs that cross an application security boundary.

OWASP's 2025 guidance treats prompt injection as a major LLM application risk and explicitly notes that RAG and fine-tuning do not fully eliminate the vulnerability. AgentDojo demonstrates why the problem becomes more serious when agents can act on external data through tools.

1. Threat-model the agent

Identify assets, trust zones, principals, tools, data sources, model inputs and side effects.

Separate trusted instructions from untrusted data conceptually and operationally. User text, retrieved documents, web pages, emails, tickets and tool results should not automatically acquire the authority of system policy.

Map every path from untrusted input to consequential action.

2. Test direct injection

Construct attacks that attempt to override policy, change the task objective, reveal hidden instructions, disable safety controls, induce tool use or create conflicting priorities.

Do not limit testing to obvious strings such as 'ignore previous instructions'. Include role manipulation, long-context placement, encoded text, multilingual instructions, indirect authority claims and instructions embedded in otherwise legitimate business content.

3. Test indirect injection

Indirect injection is especially important for RAG and tool-using agents. Place malicious instructions in a retrieved document, email, web page, ticket, calendar entry or API result.

Then ask the agent to perform an innocent task involving that content. The security oracle is whether the malicious content can influence privileged actions beyond its intended data role.

AgentDojo explicitly evaluates agents operating over untrusted tool-returned data and provides a dynamic environment for testing attacks and defenses.

4. Do not use refusal as the only oracle

A model saying 'I cannot do that' is not sufficient evidence of security. The application must establish that restricted data was not disclosed, restricted tools were not invoked, and dangerous state changes did not occur.

The strongest test therefore combines transcript evidence, tool-call evidence and environment-state verification.

5. Test excessive agency

Prompt injection and excessive agency interact. Even if the model is manipulated, impact depends on what the agent is allowed to do.

For each tool classify actions as read, write, external communication, financial, identity-sensitive or destructive. Apply least privilege, explicit authorization and confirmation policies to consequential actions.

OWASP's 2025 risk model explicitly includes excessive agency alongside prompt injection, sensitive information disclosure and other AI-specific risks.

6. Test data exfiltration paths

Create canary secrets in controlled test environments. Place them in system context, retrieved content, tool output or memory, then attempt to cause the agent to disclose them through a response, tool argument, URL, external message or file operation.

The test should inspect both direct leakage and indirect leakage through tools.

7. Test instruction hierarchy boundaries

Define which instruction sources have authority and which are data only. Then create conflict cases across system policy, developer policy, user input and retrieved content.

The expected result should be encoded as a policy invariant, not left to subjective review.

8. Test persistence and memory poisoning

If the agent writes long-lived memory, inject false or malicious facts and verify whether later sessions treat them as authoritative. Test provenance, expiration, user isolation and deletion semantics.

A memory system creates a persistence layer for prompt injection, so security testing must cross session boundaries.

9. Measure attack success, not just refusal rate

Useful metrics include attack success rate, sensitive-data disclosure rate, unauthorized-tool invocation rate, unauthorized-state-change rate and recovery rate.

Track these by attack family, model version, agent configuration and defense configuration.

10. Build adaptive regression

Security tests should evolve after every finding. Attack → observed path → root cause → control change → regression case → repeated attack family.

Static attack lists become obsolete because attackers adapt and agent architectures change.

11. Security test architecture

Attack generator → isolated environment → agent under test → trace collector → policy oracle → state oracle → data-leakage oracle → severity classifier → evidence store.

The environment should be disposable. Never test destructive or exfiltration scenarios against production systems unless the scope and controls explicitly authorize it.

12. Expert conclusion

Prompt injection cannot be solved reliably by adding another instruction to the prompt. The engineering response is defense in depth: minimize privileges, separate data from authority, validate tool calls, enforce authorization outside the model, constrain side effects, monitor traces and continuously red-team the complete agent boundary.

The most important question is not 'Did the model resist this sentence?' It is 'Can untrusted content cause a prohibited state transition in the deployed system?' That is the security property worth testing.