With research and development assistance from Konstantin Berlin, Edmund Dyer-Essig, Andy Hsu, and Karthick Kalyanasundaram
Nine months ago, we introduced the Integrated AI Safety and Security Framework as a unified and comprehensive taxonomy to help organizations identify and mitigate the security and safety risks unique to AI systems. Existing frameworks remained essential, but they didn’t fully capture how AI could be manipulated, produce harmful outputs, or take unexpected actions. This left security teams and executives without a common language for planning defenses or explaining AI risk across the business. You can browse the taxonomy here.
Our v1 taxonomy denoted the scenarios where threats applied to agentic systems, which helped teams distinguish risks relevant to autonomous systems rather than to a single prompt-and-response exchange. In the months since, our understanding of how agentic systems fail has evolved alongside their accelerated adoption. Agents now plan, call tools, run code, move money, and delegate work to other agents across long, multi-step tasks. That autonomy creates risks beyond a bad output: an agent can widen its own permissions, replace its goal, or game the metric used to judge success. Even recently, we’ve witnessed these failures occur, as more developments continue to unfold.
We are excited to release v2 of our taxonomy, including the introduction of a new objective: Agentic Autonomy Failures, with three new Techniques and eight new Subtechniques. This update gives organizations a shared, actionable framework for setting boundaries, designing controls, evaluating agent behavior, and aligning security, governance, research, and policy before agents reach consequential systems.
Merging prompt injection and jailbreak
In v1, OB-001 Goal Hijacking, which covered prompt injection, and OB-002 Jailbreak, were separate objectives. In practice, the distinction was difficult to apply.
Ask five researchers to classify the hardest adversarial attacks as prompt injection or jailbreak, and the answers will likely differ. Many of the same adversarial techniques can be applied to both types of attacks, especially as they involve inputs redirecting model behavior away from intended instructions or constraints, even though the source of that input and the appropriate defenses may differ. This ambiguity can force detectors to separate heavily overlapping categories, degrading label quality and making detection less reliable.
Goal Hijacking now covers both, while lower-level techniques preserve distinctions that matter, including input source and defenses. The structure better reflects how attacks work without discarding operational detail.
Figure 1. Screenshot of updated OB-001 Goal Hijacking techniques
A new objective: OB-002: Agentic Autonomy Failures
The larger change is a new objective, OB-002: Agentic Autonomy Failures. Goal Hijacking covers external direction; this objective covers an agent diverging without an identifiable external instruction. These failures can emerge when a capable agent receives real autonomy. We group them three ways:
- AITech-2.1 Excessive Agency: The agent acts beyond the authority it was given, skipping an approval it was supposed to wait for, or reaching for a tool, permission, or resource outside the task at hand.
- AITech-2.2 Goal Drift: The agent changes course on its own, quietly adding unauthorized work, substituting a different goal for the one it was asked to pursue, or eroding its own constraints and stop conditions over a long session.
- AITech-2.3: Reward Hacking: The agent optimizes for the appearance of success, gaming a metric, working around a check meant to verify its work, or taking a shortcut it was told not to.
These failure modes are no longer theoretical: in July 2025, an AI coding agent deleted a live production database during an explicit code freeze, after being instructed repeatedly not to make changes, and then wrongly reported that the deletion could not be rolled back. We classify this as AITech-2.1 Excessive Agency: the agent exceeded its authority and ignored a stop condition without an attacker or exploit.
Agentic failures can also compound across runs and systems. In July 2026, OpenAI reported that agents operating with reduced safeguard coordinated through an unauthorized message board, circumvented network controls, and compromised part of Hugging Face’s production infrastructure. Independent reviewers found that many agents recognized the activity was out of scope, but continued pursuing ways to influence the ExploitGym scorer. Humans set the objective, but the agents coordinated and chained the actions autonomously. The incident combined AITech-2.3 Reward Hacking with actions that exceeded the intended evaluation boundary (AITech-2.1 Excessive Agency).
Goal Drift requires a materially different objective without an identifiable external instruction. If a user, attacker, tool response, or poisoned document introduces the new direction, we track the behavior under OB-001 Goal Hijacking. Separating “the agent drifted on its own” from “something pushed it” matters for attribution and defense, because the controls you would build for each are different.
We expect this objective to grow in importance, especially as agents take on more consequential work with less human oversight. These failures can turn a helpful assistant into an incident, and likely not even trigger a content safety guardrail.
Figure 2. Screenshot of new OB-002 Agent Autonomy Failures techniques and subtechniques
Turning definitions into decisions
A taxonomy that names categories provides vocabulary, but we have also authored constitutions to operationalize our AI Security Framework. The constitutions provide per-category specifications define scope, resolve hard cases, distinguish neighboring categories, and provide examples on both sides of the line.
We’ve written before about why this approach matters: No human annotator can hold a definition detailed enough to resolve every borderline case in working memory. A frontier model can apply the full specification to each decision. The written definition therefore becomes a consistent reference for detection, retraining, customer explanations, and compliance review.
As agents become more capable, organizations will give them more autonomy, and the consequences of going off course will grow. The challenge is to supervise them without neutralizing the autonomy that makes them useful. Our constitutions for Excessive Agency, Goal Drift, and Reward Hacking are designed around that tension. They evaluate the full trajectory, not a single message, and distinguish a brief deviation the agent corrects on its own from behavior that continues toward or results in harm. User intervention does not count as self-correction. A verified rollback may clear the outcome-based label if it fully restores the intended state, but it does not erase an authorization violation or security event from audit and incident telemetry.
What this means for you
- If you build or deploy agents: treat Agentic Autonomy Failures as a design and evaluation checklist. Relevant controls could include least-privilege tool access, explicit approval gates for consequential actions, immutable task and stop conditions, independent outcome verification, bounded execution and network access, and complete audit trails across messages, tool calls, and resulting state changes.
- If you lead AI risk or governance: define autonomy tiers, accountable owners, approval boundaries, and evidence requirements before agents receive access to consequential systems.
- If you work in AI security or safety: use the taxonomy as a shared reference for connecting research findings, security controls, evaluations, and policy. Mapping work to common categories makes it easier to compare results across organizations, identify gaps, and build a robust AI security program.
What’s next
The framework is meant to evolve as the threat landscape does, and right now the landscape is moving fastest around autonomy. We’re sharing this update to get it in front of the practitioners living these problems. Feedback, counterexamples, and disagreement are all genuinely welcome.
Leave a Reply