Your procurement agent approved a vendor contract outside its authorized spend limit. The transaction cleared. The vendor has been paid. Your CFO wants to know who approved it.
You go to the logs. The action is attributed to a service account that three different agents share. The agent that executed the transaction was built by an engineer who left four months ago. The original scope document is in a Confluence page that hasn’t been updated since the agent was deployed. The agent’s permissions were elevated six weeks ago to handle a one-off edge case, and no one ever walked them back.
No one can tell you who authorized this end-to-end. No one can show you the reasoning chain. No one is sure whether this is a one-time event or whether the same agent has been doing this every time a certain condition is met.
This is the accountability vacuum. It’s not a corner case. It’s the default state of AI agent deployments in 2025. And the uncomfortable part isn’t that it happened — it’s that most engineering organizations don’t have the infrastructure to even detect it until something has already gone wrong.
The technical complexity of building AI agents has dropped sharply. The governance complexity of running them responsibly has not.
The Governance Gap
The deployment statistics tell a clear story about sequencing. 82% of organizations are already using AI agents; only 44% have formal AI governance policies in place (SailPoint, n=353 enterprise IT professionals, May 2025). That gap — 38 percentage points wide — represents the organizations running autonomous systems with live access to real data, real APIs, and real money, without documented accountability structures.
The consequences follow directly. 80% of organizations report that their AI agents have taken unintended actions (SailPoint, 2025). Gartner projects that more than 40% of agentic AI projects will be canceled by end of 2027 due to inadequate risk controls (Gartner, June 2025). These aren’t predictions about future misuse — they’re assessments based on what’s already happening in production systems today.
IBM’s 2025 analysis of enterprise AI deployments identifies four systemic failures that appear consistently across organizations:
Over-privilege without visibility: agents accumulate standing access that rarely expires, often because revoking it would break something downstream. Invisible delegation: agents reuse human user tokens, which means agent actions are logged under human identities — erasing the audit separation that compliance frameworks require. No enforcement where actions occur: governance policies exist at the policy document level but aren’t enforced at runtime, where the agent is actually calling tools and making decisions. Zero accountability after incidents: teams cannot reconstruct what happened, with whose authority, or why.
The question IBM frames as the definitive test of whether your governance infrastructure is adequate: *”Who authorized this end-to-end transaction? Can you show me an audit trail that links full accountability?”* (IBM Think, “The Accountability Gap in Autonomous AI”, 2025). In the majority of enterprise deployments, the answer is no.
Among organizations already running agents in production, only 28% say they are satisfied with their existing guardrail solutions (Cleanlab, *AI Agents in Production 2025*) — meaning nearly three-quarters are operating agents they don’t fully trust to stay within bounds. That is the only place where guardrails can intercept an agent action before it reaches an external system. Policies written in documentation cannot stop an agent from approving a vendor payment or sending a customer-facing communication. Enforcement at the point of execution is what separates governance from governance theater.
The pattern is consistent across the survey data. 34% of organizations cite governance risks as their top implementation barrier (PwC AI Agent Survey, April 2025, n=308 US business executives). Deloitte’s State of GenAI report found that regulation and risk management jumped 10 percentage points year-over-year to become the number one barrier to GenAI deployment (Deloitte, Q4 2024, n=2,773 director-to-C-suite across 14 countries). 82% of business leaders cite risk management as their top concern when scaling AI (KPMG AI Quarterly Pulse Survey, Q1 2025). The concern is pervasive. The infrastructure to address it is not.
Failure #1: The Orphaned Agent
The people who built an agent — who understand its scope, its edge cases, the reasoning behind its permission grants — rotate off projects. They’re assigned to something else, they leave the company, or the project they were on gets restructured. The agent stays in production. IBM’s analysis of enterprise deployments describes this as one of the most common and least-tracked failure modes in agentic AI: the agent outlives the institutional knowledge of the people who deployed it.
An orphaned agent is an agent in production with no designated owner — no responsible party for maintenance, scope review, access audits, or de-provisioning. It continues running. It still has access to the systems it was connected to when it was built. Those systems may have changed. The business context the agent was designed to operate in may have changed. The agent hasn’t.
The failure mode isn’t dramatic in most cases. An orphaned agent doesn’t typically go rogue on its own. What it does is accumulate risk silently. Its credentials aren’t rotated because no one is responsible for rotating them. Its permissions aren’t reviewed because no one is responsible for reviewing them. When something in the connected system changes — a new data field, a revised API contract, a modified business rule — there’s no one to update the agent’s behavior. When the agent starts making decisions based on stale logic against a changed system, the failures are often subtle enough to go undetected for weeks.
Detection requires active inventory. This means maintaining a registry of every agent in production, each one’s designated owner by name and team, the systems and data it has access to, the scope it was originally authorized for, and the date of its last access review. This is not technically complex. It is organizationally disciplined, which is why most organizations don’t do it.
The remediation process when you discover an orphaned agent is not to simply assign a new owner. The new owner needs to audit the agent against its original scope — does its current permission set match what it actually needs? Has its behavior drifted? Are its credentials still valid and appropriately scoped? In many cases, the right answer after an audit is de-provisioning. An agent that was built for a one-quarter initiative and has been running unattended for eight months with write access to your CRM is not a business asset. It’s an unmonitored attack surface.
The governance requirement is straightforward: every agent in production must have a designated owner by name, not by team. Teams change composition. Ownership assigned to “the platform engineering team” is ownership assigned to no one. When an owner leaves or changes roles, agent handoff must be an explicit, documented step in their offboarding process — not an afterthought discovered six months later during an incident review.
Failure #2: The Accountability Vacuum
When an agent takes an action, what gets logged? In most enterprise deployments: the API call, the timestamp, and the service account identifier. What doesn’t get logged: why the agent decided to take the action, what tools it considered and rejected, what context it was operating with, and which human identity or parent orchestrator authorized the delegation chain that led to the action.
Invisible delegation is IBM’s term for the specific problem of agents reusing user tokens. When an agent executes under a human user’s identity, two things break simultaneously. First, the audit log shows a human action that was actually an agent action — and any downstream review of that log will misattribute accountability. Second, the agent inherits the human user’s full permission scope, which is almost certainly broader than what the agent’s specific task requires. A customer success agent running under a sales rep’s identity has access to everything that rep has access to, which is almost certainly not what anyone intended.
Regulators are moving to require what most organizations haven’t built. ISACA’s 2025 guidance on agentic AI workflows specifies that compliance-grade audit trails must log each agent action with timestamp, capture reasoning steps and tool invocations, maintain immutable logs that cannot be retroactively altered, and link each action to the identity that authorized it — whether that’s a human or a parent orchestrator in a multi-agent system.
Here is what a minimal viable audit record looks like for a single agent action:
```python
import hashlib
import json
from datetime import datetime, timezone
def log_agent_action(
agent_id: str, # unique identity for this agent — never shared with humans or other agents
action_type: str, # e.g. "tool_call", "api_request", "data_write"
tool_name: str, # which tool was invoked
tool_input: dict, # sanitized inputs passed to the tool
tool_output_hash: str, # SHA-256 hash of the output — preserve evidence without storing sensitive data
reasoning_summary: str, # brief capture of why this action was selected
authorized_by: str, # parent orchestrator ID or human user ID that delegated to this agent
task_id: str, # the task context this action belongs to
session_id: str # the session this task belongs to
) -> dict:
record = {
"timestamp": datetime.now(timezone.utc).isoformat(),
"agent_id": agent_id,
"action_type": action_type,
"tool_name": tool_name,
"tool_input": tool_input,
"tool_output_hash": tool_output_hash,
"reasoning_summary": reasoning_summary,
"authorized_by": authorized_by,
"task_id": task_id,
"session_id": session_id,
}
# Immutability: append-only log with a chained hash
record["record_hash"] = hashlib.sha256(
json.dumps(record, sort_keys=True).encode()
).hexdigest()
append_to_immutable_log(record) # write to tamper-evident store
return record
```
The reasoning summary field is worth specific attention. It doesn’t need to be a full chain-of-thought dump — that creates storage overhead and potential data exposure. It needs to be enough for a human reviewer to understand, after an incident, why the agent made the decision it made. “Approved vendor invoice: amount within pre-authorized limit, vendor in approved registry, invoice number matches PO” is sufficient. “Completed task” is not.
Stanford’s AI Index 2025 reports that documented AI safety incidents increased 56.4% in a single year — from 149 incidents in 2023 to 233 in 2024. The majority of those incidents involve systems where post-hoc accountability reconstruction was impossible, not because the incident was sophisticated, but because the logging infrastructure was never built.
Failure #3: The Permission Accumulation Problem
Agents don’t start over-privileged. They get there gradually, through a series of individually reasonable decisions. An agent needs to read from an additional data source to handle a new edge case — access granted. A timeout issue requires elevating permissions temporarily — access elevated. A refactoring means two agents now share a service account because creating separate identities “isn’t worth the overhead right now” — access merged.
Six months later, that agent has standing access to systems it was never designed to interact with, running under a shared identity that makes its actions indistinguishable in logs from three other agents doing entirely different things. No single decision was obviously wrong. The accumulated state is indefensible.
OWASP’s Top 10 for Agentic Applications 2026 — published December 2025 with input from more than 100 industry experts — lists Tool Misuse and Exploitation (ASI02) and Identity and Privilege Abuse (ASI03) as the second and third most critical risks in agentic systems. Both trace back to the same root cause: agents holding more privilege than their task scope requires. OWASP defines Least-Agency as the core design principle: an agent should have the minimum autonomy and access necessary for its defined task, and no more.
The AWS Well-Architected Framework for Generative AI translates this into a concrete implementation recommendation: agent permissions should be calculated at runtime based on task scope and context, not pre-configured statically. Static permissions are set for the most permissive case an agent might ever encounter. Runtime-calculated permissions are set for what the agent is actually doing right now.
In practical terms, this means:
- Each agent has a unique identity — separate from human users and from other agents. No shared service accounts across agents.
- Permissions are read-only by default. Write access is explicitly granted only for the specific task requiring it, and revoked when that task is complete.
- Credentials are short-lived — issued via a credential broker at task start, with a TTL aligned to the expected task duration. Long-lived credentials that agents hold between sessions are the mechanism by which accidental over-privilege becomes persistent over-privilege.
- Access is reviewed on a schedule — quarterly minimum for agents handling sensitive data or high-risk actions. Reviews are owned by the designated agent owner, not delegated to a general platform audit.
The ISACA 2025 guidance frames this practically: the question isn’t “does this agent technically need this permission?” The question is “does this agent need this permission *right now, for this task*?” Those are different questions, and defaulting to the first rather than the second is where permission accumulation begins.
Failure #4: The Vendor Contract Trap
According to Jones Walker LLP’s analysis of AI vendor contracts, 88% cap liability at the monthly subscription fee; only 17% provide regulatory compliance warranties (Jones Walker LLP, *AI Vendor Liability Squeeze*, 2024–2025). If the vendor’s system produces a discriminatory outcome, makes a legally consequential false statement, or causes regulatory non-compliance, the contract almost certainly says this is your problem — and you’ll also be expected to indemnify the vendor against third-party claims arising from the system’s behavior.
Two court cases have clarified what this means in practice.
In Moffatt v. Air Canada (February 2024), a Canadian Civil Resolution Tribunal held Air Canada liable after its customer service chatbot incorrectly told a grieving passenger he qualified for a bereavement fare discount that didn’t exist. Air Canada’s defense — that the chatbot was a “separate legal entity” responsible for its own statements — was rejected outright. The airline was responsible for all information provided by its chatbot. The argument that software can bear its own legal accountability is not viable.
In Mobley v. Workday, a federal judge allowed a hiring bias case to proceed in July 2024, applying agency theory to hold an AI vendor directly liable for discriminatory outcomes for the first time at the federal level. In May 2025, the court granted preliminary collective certification. Workday acknowledged 1.1 billion job applications had been rejected by its tools. The potential class could number in the hundreds of millions.
The legal trajectory is clear: courts are moving toward holding deploying organizations responsible for agent behavior, on the same logic that employers are responsible for the actions of their employees and tools. Vendor contracts are simultaneously moving liability in exactly the same direction — onto the deploying organization. The organization that deploys the agent, regardless of who built the underlying model, is on the hook.
When negotiating vendor contracts for AI systems that will make consequential decisions, the minimum viable requirements are: documented audit log access that meets your regulatory obligations, explicit representations about training data sourcing and bias evaluation, indemnification language that doesn’t require you to defend the vendor against claims arising from their system’s failures, and clarity on what data the system retains and how it uses it. Most vendors will push back. Push back harder, or document internally that the risk has been consciously accepted rather than accidentally inherited.
What Regulations Actually Require
The regulation that matters is already in force. The confusion comes from conflating what applies now with what is still anticipated.
The EU AI Act entered force on August 1, 2024, with full applicability for high-risk AI systems on August 2, 2026. Article 14 is the operative provision for agentic systems: it requires that high-risk AI systems be overseen by natural persons — “commensurate with the risks, level of autonomy and context of use.” The Act does not require human-in-the-loop at every turn. It requires proportional oversight. A low-autonomy agent performing well-defined, low-stakes tasks does not need a human approving each action. A high-autonomy agent with write access to financial systems and the ability to initiate external commitments does. The distinction is risk-proportionate, not blanket. Penalties run to €35 million or 7% of global annual revenue — whichever is higher. The Act does not yet define “agentic AI” as a specific category, which means existing high-risk classifications apply based on what the agent does, not how it’s labeled.
The EU Product Liability Directive, effective December 9, 2026, extends to software and AI systems explicitly. It enables strict liability for defective AI: if a system causes harm, the developer or producer can be held liable without the plaintiff needing to prove negligence. For organizations deploying agents built on third-party foundations, this has direct implications — particularly given the vendor contract terms described above.
OWASP’s Top 10 for Agentic Applications 2026 provides the clearest engineering-level translation of what governance requires in practice. Two principles underpin the entire framework: Least-Agency (agents operate with minimum autonomy and access for their defined task) and Strong Observability (comprehensive visibility into what agents do, why, and which tools they invoke). The risk categories most relevant to governance specifically — Agent Goal Hijack (ASI01), Identity and Privilege Abuse (ASI03), Cascading Failures (ASI08), and Rogue Agents (ASI10) — are all problems that Least-Agency and Strong Observability address directly.
Singapore’s IMDA Model AI Governance Framework for Agentic AI (2025) introduces the concept of Meaningful Human Control (MHC), defined as the unity of human understanding, intervention capacity, and traceability of responsibility. The framework explicitly acknowledges that agents “operate at machine speed, beyond traditional oversight cycles” — which is precisely why MHC cannot be achieved through manual review of every action. It requires infrastructure: audit trails that make understanding and traceability possible after the fact, and intervention mechanisms that can halt or constrain agent behavior before harm is complete.
The UK’s position — “liability follows control” — is the most elegant summary of where regulatory thinking is converging, even without binding agentic AI legislation. If you control the system, you bear the liability for what it does. This is consistent with Air Canada, consistent with EU AI Act Article 14, and consistent with the direction of US case law despite the absence of federal AI legislation. The Biden Executive Order 14110 established federal accountability standards; the Trump administration revoked it in January 2025 and has moved toward deregulation. But deregulation at the federal level does not eliminate liability at the state level, the tort law level, or the contractual level — and it does not affect organizations with EU operations subject to the AI Act.
ISO/IEC 42001:2023, the first international AI management system standard, provides the operational framework for implementing governance. It uses a Plan-Do-Check-Act methodology: define your AI governance objectives, implement controls, measure their effectiveness, and improve continuously. For organizations that need to demonstrate governance maturity to customers, partners, or regulators, it provides a recognized external reference — one that carries increasing weight in EU and international procurement contexts as AI governance requirements become a standard due-diligence item.
Building Accountability Infrastructure
The gap between what most organizations have and what governance requires closes through four structural capabilities. None requires new tooling categories. All require deliberate engineering work.
Agent identity as infrastructure. Every agent gets a unique identity; cryptographically distinct from human users and from other agents. This identity is the anchor for everything else: audit logs link to it, permissions attach to it, lifecycle governance tracks it. A shared service account is not an identity. An agent that runs under a human user’s credentials has no identity for governance purposes. The cost of setting up per-agent identity is real but bounded. The cost of rebuilding audit trails for a system where identities were shared is unbounded.
Bounded autonomy by design. This means explicitly specifying the operational envelope within which an agent can act without human approval, and enforcing it at runtime rather than relying on the agent’s trained behavior. The practical implementation is a risk tier structure: high-risk actions (financial approvals above defined thresholds, legal submissions, production infrastructure changes) require explicit human approval before execution. Medium-risk actions proceed automatically but trigger human review within a defined window. Low-risk actions are fully automated with exception alerting. According to ISACA’s 2025 research on agentic AI workflows, 42% of regulated enterprises plan to introduce manager or approval features for AI agent workflows, compared to only 16% of unregulated enterprises (ISACA, *Safeguarding the Enterprise AI Evolution*, 2025). The difference reflects what those organizations are required to demonstrate to auditors. Bounded autonomy is also where the OWASP Least-Agency principle becomes operational: maximum tool calls per task, maximum retry attempts, timeouts, and confidence thresholds that route below-threshold decisions to human review rather than guessing forward.
Audit trails that support accountability reconstruction. The minimum viable standard is: timestamp, agent identity, action type, tool name, sanitized inputs, output hash, reasoning summary, and the identity of whoever authorized the delegation. Immutability matters. An audit log that can be altered after the fact is not an audit log for regulatory purposes. The reasoning summary is the element most often omitted and most often consequential in post-incident review. Regulators increasingly require end-to-end audit trails that link human authorization to agent action, not just a record that the action occurred.
Agent lifecycle governance. This is the organizational process layer that makes the technical controls sustainable. Every agent in production has a named owner. Handoff is a documented step in offboarding. Quarterly access reviews are scheduled, not ad hoc. De-provisioning has an explicit trigger: when an owner leaves without handoff, the agent’s access is suspended pending review, not left running indefinitely. An inventory of all agents, their scopes, their permissions, and their data access is maintained as a living artifact — not a stale spreadsheet from the initial deployment.
ISO/IEC 42001:2023 provides the management system wrapper for all four capabilities. Plan-Do-Check-Act applied to AI governance means: define what your agents are authorized to do (Plan), implement the technical controls that enforce those authorizations (Do), measure whether the controls are working through audit review and incident analysis (Check), and adjust as agent scope and risk profile evolve (Act).
51% of organizations using AI have experienced at least one negative consequence; inaccuracy is the most common at 30% (McKinsey, The State of AI 2025). Most of those consequences are recoverable with good audit infrastructure. Without it, they become the thing you discover in the incident call at 11pm, with no audit trail, no designated owner, and no way to determine the scope of what happened.
Where to Start
The full accountability infrastructure described above is not a weekend project. The sequence matters. These are the interventions that close the highest-risk gaps first.
- Run an agent inventory before anything else. List every agent in production, its designated owner by name, its permission scope, and its last access review date. If you can’t complete this list, you don’t know your exposure. Complete the list before building new agents.
- Assign a named owner to every agent and build handoff into offboarding. Teams change. “The platform team owns it” is not an owner. Add agent ownership transfer as an explicit line item in your HR offboarding checklist — the same way you would for a production database or a signing key.
- Implement per-agent unique identities and stop sharing service accounts across agents. This is the prerequisite for every other governance capability. Audit trails are meaningless if you can’t tell which agent took which action. Start with your highest-risk agents and work down.
- Define your risk tiers and enforce approval gates at the action level, not the deployment level. Identify which action categories require human approval before execution. Implement those gates in code — not in runbooks or team norms — so they enforce consistently regardless of who is reviewing the agent.
- Add structured audit logging to every agent action, including a reasoning summary field. The output hash and reasoning summary are the two fields most often skipped and most often needed after an incident. Build them into your logging standard now, before the incident.
- Read your AI vendor contracts before your legal team needs to. Find the liability cap, find the indemnification clause, and assess whether the regulatory compliance warranties match your obligations under the EU AI Act or applicable US state laws. Negotiate before renewal, not after an incident surfaces the gap.
The organizations that will scale AI agents successfully are not the ones that moved fastest — they’re the ones that built the accountability infrastructure before the first incident that required it.
Zartis works with engineering teams navigating exactly this build: agents in production, governance not yet caught up, and the window to close that gap before it becomes someone else’s case study getting shorter.
References
– SailPoint. (May 2025). *AI agent governance survey* (n=353 enterprise IT professionals). https://www.sailpoint.com/
– Gartner. (June 2025). *Gartner predicts agentic AI project failures due to risk controls*. https://www.gartner.com/
– Cleanlab. (2025). *AI Agents in Production 2025*. https://cleanlab.ai/ai-agents-in-production-2025/
– PwC. (April 2025). *AI Agent Survey* (n=308 US business executives). https://www.pwc.com/us/en/tech-effect/ai-analytics/ai-agent-survey.html
– Deloitte. (Q4 2024). *State of Generative AI in the Enterprise* (n=2,773 director-to-C-suite, 14 countries). https://www.deloitte.com/us/en/what-we-do/capabilities/applied-artificial-intelligence/content/state-of-generative-ai-in-enterprise.html
– McKinsey & Company. (2025). *The State of AI in 2025*. https://www.mckinsey.com/capabilities/quantumblack/our-insights/the-state-of-ai
– KPMG. (Q1 2025). *AI Quarterly Pulse Survey*. https://kpmg.com/us/en/articles/2025/ai-quarterly-pulse-survey.html
– Stanford HAI. (2025). *AI Index Report 2025*. https://aiindex.stanford.edu/report/
– IBM. (2025). *The Accountability Gap in Autonomous AI*. IBM Think. https://www.ibm.com/think/
– Jones Walker LLP. (2024–2025). *AI Vendor Liability Squeeze — market analysis of vendor contract terms*. https://www.joneswalker.com/
– ISACA. (2025). *Safeguarding the Enterprise AI Evolution: Best Practices for Agentic AI Workflows*. https://www.isaca.org/
– AWS. (2025). *Well-Architected Framework: Generative AI Lens*. https://docs.aws.amazon.com/wellarchitected/latest/generative-ai-lens/
– OWASP. (December 10, 2025). *OWASP Top 10 for Agentic Applications 2026*. https://owasp.org/www-project-top-10-for-large-language-model-applications/
– Singapore IMDA. (2025). *Model AI Governance Framework for Agentic AI*. https://www.imda.gov.sg/
– EU AI Act. (August 1, 2024). *Regulation (EU) 2024/1689 on Artificial Intelligence*. Article 14. https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX:32024R1689
– EU Product Liability Directive. (December 2024). *Directive (EU) 2024/2853*, effective December 9, 2026. https://eur-lex.europa.eu/
– ISO/IEC 42001:2023. *Artificial intelligence — Management system*. https://www.iso.org/standard/81230.html
– Squire Patton Boggs. (2025). *The Agentic AI Revolution: Managing Legal Risks* (includes analysis of Moffatt v. Air Canada). https://www.squirepattonboggs.com/
– Civil Rights Litigation Clearinghouse. *Mobley v. Workday* case tracking. https://clearinghouse.net/
– Jones Walker LLP. (2025). *AI Vendor Liability Squeeze* (Workday case analysis). https://www.joneswalker.com/
– ProPublica. (2023). *Cigna’s automated claims denial system*. https://www.propublica.org/article/cigna-pxdx-medical-health-insurance-rejection-claims
– Microsoft Azure Community Hub. (2025). *Bounded autonomy in agentic AI systems*. https://techcommunity.microsoft.com/
– Hacker News Expert Insights. (2025). *Governing AI Agents: From Enterprise Risk to Strategic Asset*. https://hackernews.expert-insights.com/