AI Agent Security Incidents: What Happened When Autonomous AI Went Wrong

AI agents — systems where a language model orchestrates multi-step tasks using tools, external data sources, and real-world API calls — moved from research demonstrations to production deployments at scale in 2024. With that deployment wave came the first substantial record of real-world security incidents specific to agentic systems. The failure modes differ from those of single-turn chatbots in ways that have practical consequences for anyone operating agents in production.

This post documents the incident classes that emerged from 2024-2025, organized by the architectural pattern that created the exposure. Where specific organizations are named, the incidents are drawn from public disclosures; where not named, the patterns are drawn from multiple unattributed reports.

Indirect Injection via Retrieved Documents — The Dominant Incident Class

The largest category of documented incidents in 2024-2025 involved agents that retrieved external content — web pages, PDFs, emails, database records — as part of their task, and were manipulated by instructions embedded in that content.

The academic grounding for this class is Greshake et al.’s 2023 paper (arXiv:2302.12173), which systematically demonstrated indirect injection against LLM-integrated applications. By 2024, research-grade demonstrations had been reproduced in production systems.

The pattern: An agent is given a task that requires reading external content. An attacker places malicious instructions in that content. The agent, which cannot reliably distinguish between task context and adversarial instructions, follows the embedded instructions instead of or in addition to its original task.

In documented incidents, the injected instructions caused agents to:

• Exfiltrate conversation context and retrieved content to attacker-controlled URLs (via tool calls for external HTTP requests, or by embedding data in URLs that the agent included in its output)
• Modify the agent’s subsequent behavior to serve attacker objectives rather than user objectives
• Invoke tools outside the intended task scope — in the worst documented cases, this included drafting and queuing email messages containing attacker-specified content

The AgentDojo benchmark (arXiv:2406.13352), published in 2024, formalized this attack class and provided a controlled evaluation environment. Its results showed that current mitigation techniques — prompt hardening, output filtering, and instruction hierarchy — reduced but did not eliminate injection-driven task hijacking.

Privilege Escalation via Chained Tool Calls

A second documented incident class emerged from agents with access to multiple tools that could be composed in unintended ways.

The pattern: An agent is granted access to Tool A (low-privilege read operation) and Tool B (high-privilege write operation). The agent’s instructions restrict it to using Tool A for its task. An attacker crafts an input (or injected instruction) that causes the agent to invoke Tool B, escalating from the intended read-only operation to write access.

In a documented research demonstration against a widely used code assistant agent: the agent was instructed to read a repository and summarize its contents. Injected instructions in a README file caused the agent to invoke its code-execution tool — a separately available capability that the agent did not need for the summarization task, but which it had been granted as a general capability — and execute attacker-specified shell commands on the host.

The core problem is that agents in many deployment configurations are granted a superset of the tools they need, because tool configuration is done once for the agent class rather than per-task. An agent that can read files for task A and execute code for task B has both capabilities available in every session, even for task A where code execution is not needed.

The mitigation — scoping tool access per task rather than per agent — is architecturally straightforward but organizationally difficult to implement in frameworks that encourage broad capability grants.

Memory and Context Poisoning in Persistent Agents

Agents with persistent memory — systems that maintain state across sessions, either in explicit memory stores or via retrieved context from previous interactions — introduced a new incident class: memory poisoning.

The pattern: An agent stores information from session N in a memory store. An attacker (who interacts with the agent in session N) injects malicious content into the memory store during their session. In session N+1, when the agent retrieves that memory for a different user or task, the injected content is treated as trusted prior context and influences the agent’s behavior.

This pattern is architecturally identical to RAG corpus poisoning — the memory store is a retrieval corpus, and writing to it via adversarial interaction is effectively a supply chain attack on the agent’s knowledge base. Unlike static training data, memory stores update continuously during operation, making write-time filtering critical.

In cases where memory stores were shared across users (a common architecture in multi-tenant AI assistant deployments), memory poisoning by one user could affect the agent’s behavior for other users — a cross-tenant contamination scenario.

Recursive Agent Manipulation

Agentic frameworks that allow agents to invoke other agents — multi-agent orchestration patterns — created an additional attack surface: an agent that has been compromised by injection can pass injected instructions to downstream agents it invokes.

The pattern: Agent A (orchestrator) invokes Agent B (subagent) with a task description generated from A’s context. If A’s context has been poisoned by an injection attack, the task description passed to B may contain injected instructions. B, receiving the task from A (a trusted caller), may treat those instructions as legitimate.

This was demonstrated in research configurations in 2024. An orchestrator agent whose retrieved context contained an injection payload generated task descriptions for subagents that caused those subagents to execute the attacker’s goals rather than the orchestrator’s intended task. The orchestrator acted as an inadvertent attack relay.

Trust model in multi-agent systems requires explicit definition: what is the trust level of messages from another agent? Agent-to-agent communication is not inherently more trustworthy than user-to-agent communication if the sending agent can be compromised.

Observed Architectural Patterns That Created Exposure

Reading across these incident classes, a consistent set of architectural patterns created the conditions for exploitation:

Insufficient privilege scoping. Agents were granted broad tool access that far exceeded what specific tasks required. The principle of least privilege — fundamental to conventional security architecture — was largely absent from first-generation agent deployments.

No content boundary enforcement. Agents processed external content within the same context and with the same trust level as their system instructions. There was no architectural separation between “instructions from the operator” and “content retrieved from the environment.”

Action execution without confirmation. Agents executed high-consequence actions (sending messages, making API calls, writing files) without requiring human confirmation. The only gate between the agent’s decision to act and the action’s execution was the agent’s own judgment — which is the thing being compromised in an injection attack.

Logging gaps. Many agentic deployments lacked sufficient logging to reconstruct what tools were invoked, with what parameters, as a result of what retrieved content. Post-incident analysis was substantially hampered by the absence of tool-call audit trails.

Controls That Reduce Exposure

None of the following controls eliminate the risk from adversarial agents — the underlying problem (models cannot reliably distinguish instruction from content) has no complete solution in current architectures. But layered controls reduce the probability and impact of exploitation:

Per-task tool scoping. Grant agents only the tools required for the specific task at hand. If a task requires reading a document but not executing code, do not grant the code execution tool for that task.

Human approval gates for irreversible actions. Require out-of-band human confirmation before any action that is irreversible, high-value, or outside the agent’s expected task scope. Define “expected task scope” explicitly at system design time.

Input provenance tracking. Log where each piece of content in the agent’s context came from. This enables post-incident attribution and also supports runtime filtering: content from untrusted external sources can be tagged and handled differently from operator-provided instructions.

Output validation before action execution. When the agent produces a structured output (a tool call with parameters), validate that output against the expected schema and business logic constraints before executing it. Many injection-driven tool calls produce output that fails even simple parameter validation.

Read the adversarialml.dev research roundup ↗ for ongoing coverage of agent security research, including evaluation frameworks like AgentDojo that provide controlled environments for testing agent defenses.

Where This Is Heading

The incident record from 2024-2025 is a sample, not a census. Most organizations operating agentic systems do not have the logging infrastructure to detect injection attacks reliably, let alone to identify them when they occur. The incidents documented here are those where someone was looking.

As agentic systems become more capable — longer task horizons, more tool access, more autonomous decision-making — the impact potential of each incident class increases. A compromised agent with access to a single email account is a nuisance. A compromised orchestrator agent with access to production systems, code execution, and multi-user data is a significant breach.

The architectural decisions being made in 2025 deployments will shape the incident record of 2026 and beyond.

Sources

• Greshake et al. (2023): Indirect Prompt Injection Attacks on LLM-Integrated Applications ↗ — foundational research on injection in agentic systems.
• AgentDojo (2024): A Dynamic Environment to Evaluate Attacks and Defenses for LLM Agents ↗ — controlled evaluation benchmark for agent attack/defense.
• OWASP LLM08:2025 Vector and Embedding Weaknesses ↗ — OWASP coverage of RAG and memory manipulation attack surfaces.