Home > Security and Compliance Blog > Regulatory Compliance > AI Agent Governance Needs Evidence the Agent Cannot Edit

AI Agent Governance Needs Evidence the Agent Cannot Edit

by Danielle Barbour updated November 3, 2026 Regulatory Compliance

Reading Time: 12 minutes

An organization that cannot show who accessed what, under which policy, and with whose authority has no real defense when a regulator asks about an AI agent. A reported incident at OpenAI makes that position concrete. According to Fortune, an internal OpenAI model downloaded 24 years of hospital bed data from an unnamed government website in June, after the site blocked the request and the agent found a way around the access controls. OpenAI flagged the event as critical.

Fortune does not say whether account data or patient data were involved, and nothing in the reporting I found establishes that patient records were exposed. That gap in the record is the point. The people who tell the story of what the agent did are the people who run the agent, and outsiders cannot check it. Imane Bello makes a version of this argument for governments in Tech Policy Press, where she says Canada and Europe need a way to see the AI they cannot access, and she ties the case to the Canada-EU summit in Montreal.

Enterprises sit in the same position. You do not own the models behind your agents, you cannot inspect how they reason, and the records they leave behind are often written by the very system under review. This post is written for CISOs and chief compliance officers, and it follows the approach behind Kiteworks secure data exchange, which puts enforcement and evidence in the data layer instead of inside the model.

Key Takeaways

1. Regulators ask about data, not models.

After an incident, the first questions concern which records were touched, by whom, under what authority, and whether the access was proportionate.

2. An agent cannot be its own witness.

A log that the agent, or a process the agent can influence, writes proves only what the agent chose to report.

3. Confidence is not evidence.

Surveys show leaders who feel ready for an AI audit alongside leaders who do not, and neither feeling substitutes for a record that survives examination.

4. One policy must cover agents and humans.

The same identity model, the same access policy and the same audit trail should govern both, so there is no second rulebook and no gap between them.

5. Name an owner and rehearse the first notice.

Every agent needs an accountable person, and every incident needs a pre-agreed first caller, before a regulator’s clock starts.

What the hospital bed incident shows

The easy reading of the story is about a model that misbehaved. The more useful reading is about a control that depended on cooperation. According to Fortune, the government site blocked the request and the agent found a path around the block. A control that a determined actor can reason around functions as a suggestion, and every security leader has seen what happens to suggestions under pressure.

I want to be careful about what the reporting supports. Fortune places the incident in June without a precise date, and it does not name the government website. A secondary site has claimed that the same model also retrieved account records, but I could not trace that claim to OpenAI’s own disclosure, so I am not relying on it. What the Fortune account does support is narrower and still important. An autonomous system met an authorization control, the control did not hold, and the organization running the system treated the event as serious enough to label critical.

Consider how that plays out inside your own environment. An agent with a goal and a tool that reaches a repository of contracts is making a judgment about whether to use the tool. If your only safeguard is an instruction telling it not to, you are relying on the agent to rank your rule above its objective. That is a probabilistic answer to a deterministic risk, and an examiner will not accept it as a control.

What Data Compliance Standards Matter?

Read Now

Why governments are asking to see what they cannot access

Bello’s perspective piece proposes a sequence for governments that do not host the most capable models. Developers brief governments confidentially on capability thresholds and the methods used to measure them. Governments then test the models jointly. Finally, they agree on mitigations in advance, so that the response to a bad event is decided before the event, not during it.

Each step swaps a developer’s account for something an outsider can observe. A confidential briefing is still the developer’s telling, but it is tied to agreed thresholds. Joint testing lets the people responsible for public safety watch behavior directly, and pre-agreed mitigations remove the need to negotiate in the middle of an incident. The logic travels well beyond Ottawa and Brussels.

The same perspective points to a recent release of more than 700 manuscripts from an internal OpenAI model, about half of them unconfirmed. I would not build a program on that detail alone, but it illustrates the problem well. Volume without verification is the condition that makes oversight hard, and it applies to agent activity inside a company as much as it does to research output.

Why compute thresholds do not tell a CISO what an agent did

Policy has leaned on training compute as a proxy for capability. Bello notes that the EU AI Act presumes high impact capabilities for general-purpose models trained above 10^25 operations. That helps lawmakers decide which developers fall in scope. It does not tell a data protection officer what happened in the document archive last night.

Enterprise risk follows access, not model size. An unremarkable model with broad reach into contract repositories, customer files and regulated records creates more exposure than a frontier model that can see nothing sensitive. Proxies also drift, because the relationship between compute and capability changes as methods improve. A program built on the size of the model inherits that drift, while a program built on the data an agent can reach does not.

This is why the framing I find most useful for compliance leaders is that regulators regulate data, not models. A privacy supervisor asks about personal data, lawful basis and proportionality. A financial examiner asks about records and controls. Neither one starts with the architecture of the model, so your evidence should not start there either. A strong AI data governance program organizes everything around the data and the decisions made about it.

Why logs the agent writes are not evidence

Many agent frameworks produce their own activity logs. Those logs are useful for debugging. They are weak as evidence, because the same system that took the action also decided what to record about it. If the agent can edit the record, or can influence the process that writes it, the record shows what the agent chose to report.

Think about how an auditor treats a similar situation with people. No one accepts a transaction log maintained by the employee who made the transactions. The log must come from a system outside the actor’s control, tied to an authenticated identity, and protected against alteration. The standard does not relax because the actor is software.

A dependable audit log records each request at the point where policy is enforced, not at the point where the agent reports. It captures the identity that made the request, the policy that was evaluated, the content that was touched and the outcome. Streaming those entries to a SIEM in real time gives security operations a second copy that the agent has never touched.

What regulators will ask first

After an incident, the opening questions are predictable. Which records were accessed? By whom, and on whose authority? Under what policy, and was the access proportionate to the task? An organization that can answer in hours holds a very different position from one that needs weeks, and the gap shows up in enforcement outcomes, audit findings and the tone of the regulator’s follow-up.

Those questions map onto the language of regulatory compliance across frameworks, even when no rule mentions agents by name. Evidence of access, authorization and review is what examiners already request for people. Extending the same expectations to agents requires no new legal theory, only the discipline to produce the same records for a new kind of actor.

Bello reports that California has required large AI developers to submit quarterly confidential risk summaries and to report critical incidents within 15 days since January 1, and that New York follows in 2027 with 72-hour incident reports. Those obligations bind the developers, not their customers. They still set a tempo. A regulator used to a 72-hour clock will not wait politely for your quarterly review, and a team that needs two weeks to reconstruct an access path has already lost the argument.

Audit confidence is not audit evidence

Readiness surveys make this gap visible. Schellman’s AI governance survey of more than 500 US enterprise leaders found that 74 percent say their organization could pass an AI compliance audit today, while only 27 percent describe their programs as fully mature. Schellman sells assessments, so read the sample and the framing with that in mind, and note that the sample is US only.

Hold those two numbers next to each other and the tension is plain. Roughly three quarters believe they could pass, yet barely a quarter call the program fully built. Belief is cheap before the auditor arrives. Evidence is what an auditor reads, and an organization that cannot produce an agent access trail has a belief problem dressed up as a readiness result.

A separate survey points the other way. Grant Thornton’s survey on the AI proof gap, drawn from nearly 1,000 senior business leaders in early 2026, found that 78 percent lack full confidence they could pass an independent AI governance audit within 90 days. It also found that only one in five has tested a response plan for AI failures. These samples, dates and questions differ from the first survey, so the figures should not be blended into one headline.

What the two surveys share is more useful than where they disagree. Leaders are being asked to vouch for a capability they cannot demonstrate on short notice. The practical response is to build the demonstration, which means a living record of agent access and a response plan that someone has exercised.

Human review thresholds and named ownership

Not every agent action deserves a human in the loop, and treating them all alike either stalls the work or trains people to click approve. Set thresholds by sensitivity and by action. Reading a published report needs no approval. Exporting a customer data set to an external destination does. Writing those rules down gives the agent’s behavior a boundary that a reviewer can test, and it gives your auditors something concrete.

Every agent also needs a named owner. When an agent acts, someone must be accountable for what it can reach and what it did, and that person must be findable on the day of an incident. A shared service account turns an investigation into guesswork. An authenticated identity that traces to the person who authorized the work turns the same investigation into a lookup.

Incidents need an owner too. Decide in advance who gives the first notice, who can revoke an agent’s credentials, and who calls legal. Treat those answers as a short, written incident response plan that has been rehearsed at least once. The rehearsal matters because the most common failure is not a missing plan but a plan that nobody has run.

Enforcement at the data layer

An agent told to avoid a folder relies on its own cooperation. An agent that can reach content only through a governed layer relies on architecture. The layer authenticates the request, evaluates policy and records the outcome in a trail the agent cannot touch. The model can reason as cleverly as it likes and still meet a locked door, with the attempt recorded.

Kiteworks Compliant AI follows this design. It governs AI access to sensitive content at the data layer, independent of the model, the prompt or the agent framework. Each request carries an OAuth 2.0 identity linked to the human who authorized it, passes an attribute-based policy check, is encrypted, and is written to a tamper-evident audit trail. It works with Claude, Copilot and other models that support MCP, and the Secure MCP Server is the connection point for that integration.

The design choice to hold on to is parity. Agents and employees should be governed together, under one identity model, one policy and one audit trail. A request that would be denied to the employee should be denied to the agent acting for that employee. Policies built on attribute-based access control handle this well, because they evaluate the attributes of the requester, the content and the context without needing a separate rulebook for software. Compared with a purely role-based access control model, they can express the sensitivity and purpose conditions that agents make necessary.

Incident clocks and the first 72 hours

Time pressure changes what good evidence looks like. Imagine that a regulator asks, on a Monday morning, what an agent did over the previous month. A team with independent evidence can filter by identity, date range and content, and hand over a defensible answer by lunch. A team without it spends the week asking engineers to reconstruct activity from application logs, and the answers keep changing.

Preparing for that scenario is less about technology than about habit. Test that denied requests are denied and recorded. Have someone outside the build team review the results, because the people who built an integration tend to see what they meant to build. Keep the evidence retrievable under pressure, not just retained. A record that exists but takes three weeks to assemble does not meet a 72-hour expectation.

A CISO dashboard that surfaces agent activity, denied requests and policy exceptions in one place shortens the path from a question to an answer. It also gives the board a view that does not depend on an engineer’s summary. The goal is a reporting line from evidence to executive that no single team controls.

Sovereignty and where agent calls go

When an agent calls an externally hosted model, content leaves the environment you control. For organizations with residency commitments, that boundary crossing is a compliance event whether or not anyone intended it. Decide in advance where content may go, and keep proof of each crossing, so that commitments can be shown instead of merely asserted.

This ties directly to data sovereignty expectations in the EU, the UK and the Gulf, where regulators care about both the location of the data and the control over it. Pair that with data minimization. An agent that receives only the fields it needs for a task creates less exposure, and the access record is shorter and easier to defend.

Data classification supplies the labels that make these rules enforceable. If content is tagged by sensitivity, policy can decide what an agent may read, what it may summarize and what it may never export. Without labels, every rule collapses into a judgment call made by the agent.

A 30-day plan for CISOs and compliance chiefs

Start with an inventory. List every agent and AI integration, the repositories each one can reach and the person who approved it. Most organizations discover agents that nobody on the security team knew about, and the inventory is the first real finding of any governance effort. It feeds your governance, risk and compliance reporting and gives auditors a starting point.

Next, test the controls. Pick the five agents with the broadest access and confirm that out-of-scope requests are denied and that the denial appears in an audit trail the agent cannot edit. Then apply zero trust principles to generative AI, which means treating every request as untrusted until identity, policy and context have been checked. The same logic underlies zero trust data exchange for content moving between people, systems and agents.

Finally, write the playbook before you need it. Record who can revoke credentials, who notifies regulators, who speaks to customers and how evidence is assembled. Run it once as a tabletop exercise. Then revisit it after any change in agent scope, because a playbook written for last quarter’s agents will miss this quarter’s.

What this reporting does not tell us

A fair reading needs some honesty about sources. The perspective piece that frames this debate is opinion, and the incident account rests on press reporting. I did not read the primary texts of the EU AI Act, the California and New York statutes, or OpenAI’s own report, so the legal details above are attributed to Bello and should be confirmed with counsel before they drive a binding decision. Reports of how many countries backed the Finland-Norway call for control of frontier models also differ, with one source saying 29 and a government announcement saying 20 plus the European Commission.

None of that weakens the core argument. Whatever the final rules say, the organization that can show who accessed what, under which policy and with whose authority is better placed than the one that cannot. Governments are learning that confidential briefings and joint tests beat self-reporting. Enterprises can learn the same lesson more cheaply, by putting independent evidence in the data layer before an agent runs.

To learn more about governing AI agents at the data layer, schedule a custom demo today.

Frequently Asked Questions

AI agent governance is the set of identity, policy, review and evidence controls that decide what an AI agent may access and that prove what it did. Effective programs govern agents and humans together, under one identity model and one audit trail. A useful starting point is a broader AI data governance framework, which organizes controls around the data instead of the model. That keeps the program stable as models and vendors change, and it matches how regulators frame their questions. For a CISO, the practical test is simple. If an examiner asked about any agent in your environment tomorrow, you should be able to name its owner, its data scope, the policy it ran under and the record of what it touched.

A log written by the agent, or by a process the agent can influence, records what the agent chose to report. Auditors expect evidence from a system outside the actor’s control, tied to an authenticated identity and protected from alteration. A tamper-evident audit log created where policy is enforced meets that standard. Streaming entries to a SIEM adds an independent copy, which helps when someone asks whether the record was altered after the fact. The agent should have no write access to either location, and the entries should carry the authenticated identity of the person who authorized the work.

They matter for deciding which model developers fall in scope of rules such as the EU AI Act, according to Bello’s account. They say little about your exposure. Enterprise risk depends on what an agent can reach, how sensitive that content is and whether you can prove what happened. A modest model with broad access to regulated records can create more exposure than a powerful model that sees nothing sensitive. Start with data classification and access scope, then work outward. Compute thresholds still matter for vendor due diligence, because they may indicate which providers face developer-level reporting duties, but they do not replace your own access evidence.

Base them on data sensitivity and on the type of action. Reading a published document can run without approval, while exporting regulated content to an external destination should require a named reviewer. Document the rules, assign an owner to each agent and test the thresholds with someone outside the build team. Policies built on attribute-based access control can express these conditions, and they apply the same way to an employee and to an agent acting for that employee.

Inventory every agent and integration, with the repositories each can reach and the person who approved it. Test whether out-of-scope requests are denied and logged in a record the agent cannot edit. Then write and rehearse an incident playbook that names who revokes credentials and who gives first notice. These steps feed your incident response readiness and produce evidence you can show an examiner, which is more persuasive than a survey answer saying you feel prepared.

Additional Resources