Separation of duties when AI writes — and reviews — the code

Separation of duties: the oldest rule in internal control

Separation of duties — segregation of duties in the accounting literature — is one of the oldest ideas in internal control, and its premise fits in a sentence: no single person should control every stage of a transaction. Classical practice splits four incompatible functions — authorizing an action, recording it, holding custody of the asset it touches, and reconciling the result — so that a mistake or a manipulation by one hand must pass through another before it becomes real. The control does not assume anyone is dishonest. It assumes everyone is fallible, and arranges the work so that a single failure is not enough.

In financial reporting the idea is codified rather than optional. The Sarbanes-Oxley Act of 2002 makes management responsible for the effectiveness of internal control over financial reporting (Section 404), and the COSO Internal Control — Integrated Framework, the reference model most issuers use, treats the segregation of incompatible duties as a core control activity. Where those controls meet software, they live in IT general controls (ITGC): the change-management and access rules that decide who may move code and configuration into the systems that produce the numbers.

The software form of the rule is concrete and familiar. The person who writes a change is not the person who approves it, and neither is the person who presses deploy. Author, reviewer, and release are meant to be distinct roles, enforced by access rights rather than good intentions. A developer who can write a change, approve their own pull request, and ship it to production has collapsed three control points into one — and it is exactly that collapse, not any individual's character, that an auditor flags.

Why it exists: independence, not distrust

The reason the maker cannot be the checker is subtle, and it is worth stating plainly because AI is about to test it. A review has value only when the reviewer can fail differently from the author. If the two parties share the same assumptions, the same blind spots, and the same incentives, the second look adds ceremony without adding safety — it will wave through precisely the errors the first pass produced. Independence is the whole point, and separation of duties is the organizational machinery for manufacturing it.

That is why the control targets structure rather than sentiment. It does not ask whether the developer is careful or the reviewer attentive; it arranges roles so that a single error or a single bad actor cannot carry a change all the way to production unchecked. The four-way split in accounting and the author, reviewer, and release split in software are the same design: force a decision to survive contact with a party who did not originate it.

Hold on to that definition — a reviewer who can fail differently from the author — because it is the exact property automation puts under pressure. Everything that follows is a variation on one question: when the author is a model, what does a genuinely independent checker look like, and how do you keep from quietly making the maker the checker again?

How AI-authored change strains the control

AI does not repeal separation of duties; it erodes it in ways that are easy to miss because the org chart still looks compliant. The signatures are all present, the roles are all named — and the independence they were meant to guarantee has quietly leaked out. Four pressures matter most.

• Self-approval by another name: an agent granted write and merge rights is a developer who approves their own pull requests. The control is not weakened; it is bypassed.
• Volume that outruns review: separation of duties assumes the reviewer actually reviews. When changes arrive faster than they can be understood, the second signature is real on paper and empty in fact.
• Correlated reviewers: two systems drawn from the same model lineage can be confidently, identically wrong. Cross-checking them surfaces nothing, because they agree by construction, not by verification.
• Diffused accountability: when a human requested it, a model wrote it, a pipeline merged it, and a second model summarized the diff, the simple question of who the maker is no longer has a simple answer — and a control you cannot attribute is a control you cannot separate.

Who is the maker when a model wrote it?

Separation of duties presupposes something automation quietly removes: a clear answer to who did what. You cannot separate roles you cannot identify. When a change is co-authored by a person and a model, the maker is no longer a single name on a commit; it is a small chain — the human who set the intent, the model and version that generated the code, the prompt and context that shaped it, and the human or system that let it through.

This is why provenance moves from hygiene to prerequisite. Recording that a change was AI-authored, by which model at which version, under whose request, is not documentation for its own sake — it is what makes the rest of the control possible. If you cannot tell AI-written code from human-written code, you cannot enforce that the model stays out of its own approval path, you cannot size how much of your change volume is machine-generated, and you cannot honestly tell an auditor who the maker was. Attribution is the foundation the separation is built on.

Provenance also has to be tamper-evident to mean anything. A label a committer can set by hand and remove at will is a suggestion, not evidence. Signed commits, build-provenance approaches such as the open SLSA specification, and append-only history move attribution from claim to record — the same shift, from assertion to artifact, that separates a screenshot from proof.

Patterns that preserve separation of duties

No single practice restores the control; a small set together does. Each one targets a specific way automation collapses the author, reviewer, and release split, and each is deliberately boring — the durable ones usually are.

• Attribute AI-authored change at the source. Mark it, capture the model and version and the requesting human, and make the mark tamper-evident (signed commits, build provenance) so the record cannot be quietly edited after the fact.
• Name a human attester who is not the requester. The person who prompted or requested a change should not be its sole approver. A meaningful signature comes from someone with the authority and the context to reject it — a distinct, accountable human, recorded by name.
• Enforce review gates in the machine, not the culture. Branch protection, required reviews, and policy-as-code make the separation something the author cannot bypass. A rule that depends on the author choosing to honor it is not a control.
• Keep the model out of its own approval path. A model may propose; it must not merge its own work. If a model assists review, it must not be the same instance or lineage that authored the change, and a named human must own the final decision — automation informs, a person approves.
• Right-size volume to real review capacity. If changes arrive faster than a human can understand them, either the throughput or the review model has to change. Speed that outruns comprehension does not preserve the control; it hollows it out.

What 'meaningful human review' has to mean

The escape hatch every organization reaches for is human in the loop — a person signs off, so separation of duties is intact. It is a real answer only if the review is real. A human who approves a change they did not read, or read but could not understand, has provided a signature, not a second judgment. That is the rubber stamp the control exists to prevent, now wearing a compliance badge.

The problem sharpens when the reviewer is also AI-assisted, which is increasingly the default. A reviewer leaning on the same class of model that wrote the code inherits its blind spots at exactly the moment independence matters most. Meaningful review, in that setting, has a few non-negotiable properties: the reviewer can actually understand the change and could reconstruct why it is correct; the reviewer has genuine authority to reject it, not merely to acknowledge it; the reviewer is accountable by name for the decision; and the reviewer is independent of the author in the way that counts — not sharing the model, prompt, or training set that would make them fail together.

Both the standards and the law point the same direction. The EU AI Act requires that high-risk AI systems be designed so that natural persons can effectively oversee them (Article 14), with explicit attention to automation bias — the human tendency to over-rely on a machine's output. The NIST AI Risk Management Framework places human accountability and oversight in its Govern function. Neither treats a nominal human as sufficient. Oversight has to be effective, informed, and owned, or it is oversight in name only.

Where the standards land

None of these instruments was written specifically for the case of an AI agent opening a pull request, but each speaks to it directly. Read together, they say the same thing from four directions: automation may do the work, but a separate, accountable human must be able to check it — and prove they did.

A checklist for teams shipping AI-authored code

Whatever your stack, the questions that test whether separation of duties survives AI authorship are the same. Ask them of your own pipeline before an auditor does — and treat any question you cannot answer with evidence as a gap, not a pass.

• Can any agent or account both write and merge the same change? If yes, the maker is the checker.
• Is AI-authored code attributed at the source — model, version, requesting human — in a record that cannot be silently edited?
• Is the approver always a different, named human from the requester of the change?
• Are review gates enforced by the system (branch protection, required reviews, policy-as-code), or only by convention?
• If a model helps review, is it barred from reviewing its own output, and does a named human own the final call?
• Does your change volume still fit within real human review capacity, or has approval quietly become a rubber stamp?
• Could you show an auditor, for any given change, who made it, who approved it, and that the two were independent?

Frequently asked questions