Threat Modelling for AI Systems¶
Threat modelling is the first design activity, before you choose a single control. The goal is to understand what you are defending, where it can be attacked, and which of those attacks matter enough to spend control budget on. For AI systems the method is familiar but the attack surface is not: the model is non-deterministic, untrusted text can become instructions, and an agent can take actions you never explicitly authorised.
This page gives you a repeatable method and connects each step to the rest of the site, so the output of your threat model is a control set you can actually build.
The method in one line
Scope the system, draw the trust boundaries, enumerate the AI-specific threats, rate each against your risk tier, then select the controls that address the threats that matter.
Why AI changes threat modelling¶
Traditional threat modelling assumes the system does what its code says. AI breaks three assumptions that most models rely on:
- Inputs can become instructions. Any text the model reads, from a user, a retrieved document, a tool result, or another agent, can attempt to redirect its behaviour. Prompt injection has no equivalent in a deterministic system.
- The boundary between data and control collapses. Retrieved context, memory, and tool outputs all flow into the same prompt. Data provenance and authority become security properties, not metadata.
- Behaviour is emergent. You cannot enumerate every output. The model can be exploited through its training data, its retrieval corpus, or a cleverly worded request, and the failure may be subtle.
Step 1: Scope and assets¶
Define the system boundary and what an attacker would want. For an AI system the assets usually include:
| Asset | Why an attacker wants it |
|---|---|
| The decision or action | The system's output influences money, access, safety, or reputation |
| Training and retrieval data | Source corpora can be poisoned to change behaviour, or exfiltrated |
| Credentials and tools | An agent's tools and tokens are a path to the wider environment |
| The conversation and memory | Sensitive data accumulates in context and persistent memory |
| The model itself | Weights, system prompts, and configuration are intellectual property and attack targets |
Step 2: Draw trust boundaries¶
Map every point where data or control crosses from less-trusted to more-trusted. In an AI system the boundaries that matter most are:
- User to model (prompt injection, jailbreaks)
- Retrieved content to model (indirect injection through RAG, the largest attack surface for most systems)
- Tool result to model (poisoned tool output steering the next action)
- Agent to agent (a compromised or confused agent influencing others, the MASO (Multi-Agent Security Operations) problem)
- Model to tool (an action the model should not have been able to invoke)
Each crossing is where a control will eventually go. The infrastructure controls are organised around exactly these boundaries: identity at the entry, network segmentation between zones, and tool access controls at the model-to-tool crossing.
Step 3: Enumerate the AI-specific threats¶
Work through the attack surface systematically. The OWASP LLM Top 10 and the OWASP Agentic Top 10 are the most useful checklists; the OWASP LLM Top 10 mapping ties each risk to a control. The recurring categories:
Use MAESTRO for agentic systems
Flat checklists are good at what can go wrong but weak on where it enters in a multi-agent system. MAESTRO (Multi-Agent Environment, Security, Threat, Risk, and Outcome), the Cloud Security Alliance's agentic threat modelling framework, decomposes an agent stack into seven layers and asks you to enumerate threats at each layer and across the seams between them:
- Foundation models (the LLM itself: poisoning, jailbreaks, alignment failure)
- Data operations (ingestion, RAG corpora, memory)
- Agent frameworks (planning, tool use, delegation)
- Deployment infrastructure (hosting, sandboxing, network)
- Evaluation and observability (logging, tracing, evals)
- Security and compliance (a cross-layer function over all the others)
- Agent ecosystem (other agents, marketplaces, the wider environment)
The cross-layer and agent-to-agent seams are exactly the boundaries from Step 2 that flat checklists miss, including the MASO failures. Use MAESTRO's layers to drive enumeration, then map what you find onto the categories below.
MAESTRO structures enumeration; it does not rate or prioritise threats, and its value scales with how agentic the system is. Treat it as a lens, not a verdict: pair it with Step 4 to decide which threats actually warrant a control. The MAESTRO layer-to-control mapping ties each layer to the specific controls that address it.
| Threat category | Example | Where it enters |
|---|---|---|
| Prompt injection | User or document overrides the system prompt | User and retrieved content boundaries |
| Data and model poisoning | Corrupted training or RAG data changes behaviour | Supply chain, see Model Threat Landscape |
| Sensitive data disclosure | Model leaks PII or secrets in a response | Model to user boundary |
| Excessive agency | Agent takes an unauthorised or high-impact action | Model to tool boundary, see Agentic AI Controls |
| Supply chain | Compromised model, dependency, or tool | Build and acquisition, see Supply Chain |
| Tool and plugin abuse | Tool invoked with crafted parameters | Tool invocation, see Tool Access Controls |
| Multi-agent emergent risk | Confused-deputy and trust failures across agents | Agent to agent, hand off to MASO |
Step 4: Rate against your risk tier¶
Not every threat justifies a control. Classify the system into a risk tier based on impact, autonomy, and data sensitivity, then rate each threat by likelihood and consequence at that tier. A LOW-tier internal FAQ bot and a CRITICAL-tier credit-decision agent face the same threat catalogue but warrant very different control depth. Use Risk Assessment to quantify residual risk and decide where control budget earns the most reduction.
Step 5: Select controls¶
Map each threat that matters to a control, and consciously deselect the rest:
- Three-layer pattern (Controls): guardrails prevent known-bad, the Judge detects unknown-bad, humans decide on consequence.
- Agentic controls (Agentic AI Controls): constrain tools, scope sessions, and bound delegation when the system takes actions.
- Identity (IAM Governance): authenticate every entity, enforce least privilege across humans, services, and agents.
- Infrastructure (Infrastructure & Operations): the logging, segmentation, secrets, and incident response that make the behavioural controls enforceable.
The output of a threat model is this control set, sized to the tier. Take it into Test & Validate to confirm the controls hold, then into Infrastructure & Operations to build and run them.
Keep the model alive
A threat model is not a one-time artefact. Re-run it when the use case changes, when you add tools or agents, or when a new attack class appears. Pair it with the AI-Aware SDLC so it is revisited at each phase rather than filed and forgotten.