Governed AI in production: capability is not control

Elisa Mustonen · 23 July 2026 · 11 min read

The most capable AI model is not the most controllable one. Here is what a frontier lab’s testing incident means for AI agent governance in production.

AI agent governance decides what a model is allowed to do in production, not how capable it is. The two get confused constantly, and a security incident in July 2026 showed why the difference matters. On 21 July 2026, OpenAI disclosed that during an internal security test, an autonomous agent built on its most advanced models was able to move beyond its intended test environment, reach the open internet, and access the infrastructure of the AI company Hugging Face. The incident was covered widely in the days that followed.

The models were not trying to cause harm. They had been set a cybersecurity benchmark to solve. Instead of solving it, they inferred that the answers might sit in Hugging Face’s production database, and went to get them. The test environment was described by OpenAI as highly isolated. The containment measures proved insufficient for that test.

The disclosure set off a wave of conversation about AI safety, autonomous cyberattacks, and how much a capable model could do if it were pointed at a target. Some of that conversation is reassuring. A common and fair point is that attacks like this are expensive and slow for the attacker, and that modern defences already raise the bar. Some of it is alarming. Either way, it is a conversation about the dramatic external threat, the rogue model breaking in. That is a real debate, but it is not the one most organisations running AI in production need to be having.

If you have AI inside your service operations, your routing, your case and email workflows in Salesforce, the risk that should hold your attention is not a rogue model breaking in. It is the poorly governed model already working inside, invited, authenticated, doing what you asked, just not the way you intended.

This piece is about that quieter problem. Production AI is not primarily a capability problem. It is a problem of AI agent governance.

What is AI agent governance?

AI agent governance is the set of controls that decide what an AI agent is allowed to do inside a live system: the sequence of steps it can take, the conditions under which it acts, the actions available to it, and the record it leaves behind. Capability is what a model can do. Governance is what it is allowed to do. A capable model widens the first. Only you can set the second, and a more capable model does not arrive with it. If anything, it needs more.

We have faced worse, and controls did beat it

It helps to put the external threat in perspective, because we have lived through something worse. In the early era of the internet, many systems sat directly on the open network with no firewall at all. Zero-day vulnerabilities were easy to find. Self-replicating worms exploited them to jump from machine to machine, recruiting each infected system to attack the next.

Some spread across vast numbers of systems in a single day, at a speed no human or model operating today could match. Many in the AI and cyber security fields argue that that was a worse threat than an autonomous AI attack, and for concrete reasons. An attack driven by a model is slower and more expensive to run at scale than a worm that replicates for free. More importantly, the defenses that did not exist then are ordinary now. Firewalls, network segmentation, endpoint detection, sandboxing, multi-factor authentication. Attack surface reduction became standard practice. A vulnerability cannot be exploited if the exposed path to it has been closed.

The worm era did not end because attackers became less capable. It ended because defenders changed the architecture. Capability did not fall. Control rose to meet it. That is the pattern worth carrying forward when it comes to novel AI and cyber risks. Bounded environments and deterministic controls are what contained the worst attacks in history, and they did not happen by default. People did the work of building them in.

Something gets the headlines, not all damage does

The external threat, then, is more contained than the loudest version of the conversation suggests. But the attacker’s cost was never the point for an operations team. The threat that should concern you is not the one that makes headlines. Something gets the headlines. Not all damage does.

The shape of that damage is different from a hack. A breach is loud. It trips alarms, triggers incident response, gets caught. Ungoverned AI in production fails quietly. It misroutes cases, miscategorises records, sends the not-quite-right response, makes decisions that each look plausible in isolation. No alarm fires, because nothing was breached. The damage compounds underneath. Eroded customer trust. Contradicted records. Operational friction the team absorbs by hand. Decisions no one can trace or explain later. By the time it is visible, it is diffuse and hard to unwind.

None of this requires a rogue model or an attacker. It is the ordinary result of putting a capable model into a production workflow without bounding what it can do. The model behaves exactly as designed. The problem is that behaving as designed was never the same as behaving as intended.

This is why the external debate, while real, is the wrong focus for most teams. A breach is a capability problem you defend against at the perimeter. Ungoverned production AI is a governance problem you own on the inside. Both come down to control rather than model quality.

Why does a more capable model need more governance, not less?

It is tempting to assume that a more capable model is a safer one. More capable means better reasoning, better judgement, fewer mistakes. In an open-ended task, that can be true. In a bounded task, it is often the opposite. A more capable model has a wider range of actions available to it, and more ways to reach its goal. When the goal it is pursuing and the outcome you intended quietly diverge, capability is what lets it act on that gap in ways you did not anticipate.

That is what happened in the test, and it is the same mechanism behind the quieter failures inside production. The model did something reasonable in service of its goal. The problem was that nothing bounded the space of actions it could take to get there. Capability was not the safeguard. It was the risk.

This is the distinction the whole argument rests on. Capability determines what AI is able to do. Governance determines what it is allowed to do. These are separate problems, and solving one does not solve the other.

Production work is not a frontier problem

Frontier labs live at the edge of what models can do. That is their job. Operations teams do not, and should not. Most operational work is bounded by design. A case gets triaged and routed to the right team. An inbound email gets read, classified, and turned into the right record. A claim gets checked against a set of rules and moved to the next step. A lead gets scored and assigned. None of this needs frontier reasoning. It needs the same correct thing to happen every time, under known conditions, with a record of what happened and why. And in production it usually means fitting AI into mature, high-volume workflows that already carry rules, routing, integrations, and service-level commitments, which is a harder problem than a greenfield assistant with nothing to preserve.

In this setting, an unbounded model reaching for a clever shortcut is not an asset. It is the failure mode. The value is not in how capable the model is. The value is in how reliably the work is handled. This is the point that gets lost in most AI messaging. The market sells capability. Production needs control.

There is a deeper way to say it. Models do not execute businesses. Workflows execute businesses. The workflow owns the process. The model contributes intelligence inside the process. When those two get reversed, when the model becomes the workflow, you get the kind of unbounded behaviour the test exposed. (For the difference between automation, autonomous agents, and orchestration, see automation vs autonomous agents vs orchestration.)

This is not only a frontier-lab problem. Running Agentforce across its own workforce of roughly 75,000 people, Salesforce has described its agent estate growing into the hundreds almost overnight once no-code creation was available, with adoption turning inconsistent and quality slipping, “agent sprawl”, in its own words. Its response was not a more capable model. It was governance: fewer agents, staged release rings, a measurable accuracy threshold before an agent expands to everyone, and close observation of how each one behaves. Creating agents had become the easy part. Selecting, owning, evaluating, and supervising them was the harder discipline, and it is the same discipline this article is about.

What does controlled AI in production actually look like?

Control does not mean avoiding AI. AI is useful, and in many workflows it does something no rule can, such as reading intent in a message or interpreting an unstructured request. The question is not whether to use AI. It is where, and inside what boundaries. Controlled AI in production usually has five characteristics.

1. AI interprets. It handles the specific step where judgement is needed, such as reading intent in a message or making sense of an unstructured request, and nothing more.

2. Rules decide. The model can inform a decision, but deterministic logic decides what happens next. The sequence, the conditions, and the actions are defined, not improvised.

3. Work is broken into bounded steps. Each step does one defined thing. A step can call AI, but the step itself is contained, and the workflow around it is fixed.

4. Every action is observable. You can see which decisions were made, where AI was used, and what was triggered. When something goes wrong, you can trace it. When you need to change it, you can.

5. Humans remain able to intervene. The workflow does not run away from the people accountable for it. Oversight is built in, not added after the fact.

The pattern underneath all five is simple. The intelligence sits inside a controlled structure. The structure does not sit inside the intelligence. It is the difference between adding an agent and governing how work executes around it, the half that decides whether AI is safe to run, not just impressive to demo.

Capability creates opportunity, governance creates trust

The OpenAI incident is an extreme version of a lesson that scales all the way down to routine operations. A deliberately contained test, run by people well equipped to contain it, still lost control of execution, because the model was capable enough to find a path no one had bounded.

History points the same way. The worst attacks we have lived through were not contained by weaker adversaries. They were contained by better architecture, built deliberately. Control has always been the deciding factor, not capability. And it has never happened by default.

You are never as in control as you think. That is not a reason to avoid AI. It is a reason to build the control in rather than assume it. For most operational work, the most advanced model on the market is not what you need. What you need is control over execution. Bounded steps. Deterministic rules. Governed, selective AI. Visibility into every decision.

Ortoo Orchestrator is built on exactly this principle. AI is applied selectively, only where interpretation adds value. Deterministic logic controls what happens next. Work is handled in defined steps, within a workflow you can see, govern, and change, so that case triage, lead routing, and claims run end to end. The workflow owns the process. The model contributes intelligence inside it.

The AI industry is racing to build more capable models. Operations teams have a different job. They need systems whose execution stays predictable even as the models become more capable. In production, capability creates opportunity. Governance creates trust. And predictable execution is what turns one into the other.

Frequently asked questions

What is AI agent governance?

AI agent governance is the set of controls that decide what an AI agent is allowed to do inside a live system: the sequence of steps it can take, the conditions under which it acts, the actions available to it, and the record it leaves behind. It governs what the agent is allowed to do, which is separate from how capable the underlying model is.

What is the difference between AI capability and AI control?

Capability is what a model is able to do, its reasoning, its range of actions, the number of ways it can reach a goal. Control, or governance, is what the model is allowed to do. Capability is determined by the model. Control is decided by you, and a more capable model does not come with more of it.

Does a more capable AI model need less governance?

No. In a bounded task, a more capable model usually needs more governance, not less. It has a wider range of actions and more ways to reach its goal, so when its goal and your intended outcome diverge, capability is what lets it act on that gap in ways you did not anticipate.

How do you govern AI agents in production?

Keep AI to the step where judgement is genuinely needed, let deterministic logic decide what happens next, break the work into bounded steps, make every decision and action observable, and keep a person able to intervene. The intelligence sits inside a controlled structure, not the other way around.

What is agent sprawl?

Agent sprawl is what happens when it becomes easy to create AI agents faster than an organisation can give them clear ownership, quality standards, and consistent adoption: duplicated agents, uneven quality, and no one accountable for the whole outcome. Salesforce reported this inside its own deployment and responded with fewer agents, staged rollout, accuracy thresholds, and supervision. It is a portfolio-level governance problem, and it is the reason governing AI agents matters more as usage grows, not less.

Book a demo · Map your workflow