Who approves what the AI writes?

The oldest control in the book is quietly failing

Separation of duties is one of the oldest controls we have. Long before software, it was the rule that the person who writes the check does not also sign it, and the person who records a transaction does not also approve it. The logic is blunt and durable: concentrate authorship and authorization in one pair of hands and you have built a single point at which error and fraud become invisible. Split them, and someone with the standing to say no has to look.

Software inherited that rule wholesale. In any environment that answers to auditors, the engineer who writes a change is not supposed to be the one who approves and releases it. That control is now under a quiet, structural strain — not because anyone repealed it, but because of who, or what, is now doing the writing. As AI agents open a growing share of pull requests, the human on the other end is often the only human in the loop, and the approval is degrading into a reflex. The form of the control is intact. The substance is leaking out of it.

The claim of this post is narrow: a human clicking approve is not the same as human oversight, and the gap between the two is exactly where the risk now lives. Every serious governance instrument being written for AI says the same thing in its own dialect — oversight has to be real to count. The click was always meant to be the visible trace of a judgment. When the judgment stops happening, the trace keeps getting recorded anyway.

What separation of duties was actually for

It helps to remember what the control was defending against, because the defense only works if the mechanism is intact. Under the Sarbanes-Oxley Act, management has to assess internal control over financial reporting, and those assessments are almost always organized against the COSO Internal Control framework, where segregation of duties sits among the core control activities. In the systems that produce financial numbers, that principle lands as an IT general control over change management: the person who develops a change is not the person who approves its move to production.

The point of that split was never the paperwork. It was independent judgment — a second party who understood the change, had the authority to reject it, and would bear some accountability if it went wrong. The approval step is a preventive control precisely because it interposes that judgment before the change can do damage. Strip the judgment out and leave the step behind, and you have not weakened the control by a little. You have removed it and kept its shadow, which is worse, because the shadow still reports green.

Why agent-authored pull requests break the assumption underneath it

Segregation of duties made a quiet assumption that held for decades: the author was a person. That mattered in ways the control never had to spell out. A human author worked at human speed, so the volume of change arriving at the approver was bounded by how fast someone could write it. A human author was accountable, so the approver was checking the work of a named colleague who could be asked what they were thinking. And the effort of writing was itself a filter — nobody produced a thousand-line change casually.

An agent removes all three at once. Volume is no longer throttled by typing, so a single approver can face more change, faster, than attention can absorb. Accountability detaches from authorship, because the agent cannot be accountable for anything. And the effort filter is gone — a large, plausible, well-formatted change is now cheap to generate, and plausibility is exactly the property that makes a thin review feel sufficient. The approver is doing the same job under conditions the job was never designed for.

What fills the gap is automation bias — the well-documented human tendency to over-trust a confident automated output, especially when it is usually right and checking it is expensive. When the diff is clean, the tests are green, and the model is correct most of the time, approving is the path of least resistance and rejecting is friction no one is rewarded for absorbing. The result is a rubber stamp that everyone can feel happening and the audit trail cannot see. From the record's point of view, a careful approval and a reflexive one look identical.

A click is not oversight — which is precisely what the rules now say

This is where the emerging governance instruments turn out to be talking about the same problem as the engineering one. The EU AI Act, for high-risk AI systems, requires human oversight, and it is unusually specific about what that means. Article 14 says oversight measures must let a real person understand the system's capabilities and limitations, correctly interpret its output, and decide in any given case not to use it or to override it — and it names automation bias directly, requiring that overseers remain aware of the tendency to over-rely on automated output. The law anticipated the rubber stamp and wrote against it.

The other frameworks converge from different directions. The NIST AI Risk Management Framework puts accountability and human oversight at the center of its Govern function: someone must be responsible, and the responsibility cannot dissolve into the tooling. ISO/IEC 42001, the AI management-system standard, requires defined roles and an accountable owner for AI in the organization. None of these treat oversight as a checkbox. All of them treat it as a capability a named human has to actually exercise — which is another way of saying that approve has to mean something.

Read against SOX, the message is consistent rather than novel. Segregation of duties already required that the approver be independent of the author and capable of refusing. The AI instruments are not inventing a new control; they are restating an old one for a world where the author might be a machine, and insisting that the independence and the judgment be genuine. The through-line from a 2002 financial-controls statute to a 2024 AI regulation is a single sentence: the person who signs off has to be a person who could have said no, and did the work to know whether to.

Rubber stamp versus real review

The difference between the two is not a matter of attitude; it is observable in what the approval leaves behind. Held against the capabilities the EU AI Act's Article 14 expects of a human overseer, the contrast is concrete.

The reviewer is now the bottleneck — and the last real control

There is a bright side buried in this, and it is the same insight from the metrics side of the house. When generation gets cheap, review becomes the scarce resource and the approver becomes the load-bearing control — the single point where human judgment still enters the system. That is not a problem to route around; it is the control working as designed, finally under enough pressure to matter. The failure mode is not that review is the bottleneck. It is pretending it is not while quietly starving it.

You can even read whether the approvals were real in the delivery data. Change failure rate — one of the four DORA metrics from the DevOps Research and Assessment program, not to be confused with the European Union's similarly named financial-sector regulation — is close to a direct measurement of whether change is actually being trusted well. When approval thins out under a rising tide of generated diffs, failure rate is where it eventually shows up, after the fact and in production. The metric stays honest even when the process turns performative.

Two things make the control real again rather than nominal. First, independence: the reviewer has to be able to fail differently from the author, which means the approver's judgment cannot come from the same agent, or the same model family, that wrote the change. A reviewer that shares the author's blind spots is a mirror, not a check. Second, provenance: the audit trail has to record what actually produced the change and who actually assessed it — that an agent drafted it, which human reviewed it, and that a separate human approved it. Capturing model involvement is not about blaming a tool; it is what makes the trail complete enough to stand behind.

What to watch

A few things are worth watching over the next year, because this is moving from principle to practice. Watch whether approve stops being a bare button and starts asking the approver to attest to what they actually checked — the small design change that turns a reflex back into a judgment. Watch whether agent authorship gets captured in the provenance record by default, rather than being reconstructed after an incident. And watch the review-to-volume ratio inside your own organization: if the number of changes an agent opens is climbing while the human review capacity behind them is flat, the control is already eroding, whatever the dashboard says.

Watch the outside pressure, too. As the EU AI Act's high-risk obligations phase in through 2026 and 2027, effective human oversight moves from good practice to a documented duty, and auditors, buyers, and investors doing diligence will start asking not whether a human approved a change but whether that approval meant anything. The honest answer to who approves what the AI writes cannot be a name and a timestamp. It has to be a person who understood the change, was independent of the thing that wrote it, and could have said no. Everything else is a signature on an empty page.

Frequently asked questions