AI governing AI: how law and measurement are co-evolving

The shift: from point-in-time audits to continuous assurance

Traditional assurance is a snapshot. An auditor scopes a period, samples a subset of controls, tests them, and issues an opinion that begins to age the moment it is signed. A SOC 2 Type II report describes how controls operated across an observation window that has already closed; SOX certification runs on a quarterly and annual cadence; ISO management-system certificates rest on surveillance audits spaced across a multi-year cycle. Each is a considered judgment about the past.

That model held while software changed on a human timescale. It strains badly now. A system under continuous deployment can be materially different by the afternoon — new dependencies, new access grants, a refactor an assistant proposed and a pipeline merged. The gap between what was tested and what is true today is no longer a rounding error; it is the space where incidents live. This mismatch is what pushes teams toward continuous control monitoring: evidence gathered on the system's own clock rather than the auditor's calendar.

The endpoint of that trajectory is machine-verifiable assurance — controls whose operation emits signed, timestamped, reproducible evidence that a reviewer can check without re-interviewing the people who ran them. The audit stops being a periodic excavation and becomes a standing, queryable state. That is a genuine improvement in fidelity. It also quietly moves the reviewer's seat closer to the machine, which is where the interesting problems begin.

Regulation is converging on management systems, not checklists

The instruments emerging around AI share a family resemblance, and it is not the prescriptive checklist of earlier compliance regimes. They ask for a management system: documented governance, risk assessment tied to how a system is actually used, lifecycle controls, logging, transparency, and a named human who is accountable. Each of those requirements implies evidence that is produced continuously, not a certificate framed once a year.

• Common thread one: risk is judged by context of use, not by the technology in the abstract — the same model can be low-risk in one deployment and high-risk in another.
• Common thread two: documentation, logging, and traceability are load-bearing — regulators want to see how a system behaves over time, not a one-time snapshot.
• Common thread three: a human stays accountable — oversight and ownership cannot be delegated to the system itself.

AI now builds — and increasingly reviews — the software it is governed by

For most of computing history, the tools that produced software and the controls that governed it were clearly separate layers. That separation is dissolving. Assistants now write, refactor, and review code; they also draft policies, map controls to frameworks, summarize logs, triage alerts, and flag the anomaly buried in a week of activity. The governance layer itself is becoming a place where models do the work.

This is the recursion at the heart of the matter. Increasingly, the artifact being assessed and the thing doing the assessing are drawn from the same well — auto-generated code reviewed by an auto-generated reviewer, often descended from overlapping training data and shaped by similar prompts. When the author and the checker share ancestry, they can share blind spots. A reviewer that fails in precisely the places the author fails is not a control; it is a mirror.

None of this makes automation the enemy. Machines are consistent, tireless, and fast enough to keep pace with change that humans cannot track by hand — exactly the property continuous assurance demands. The point is narrower and more uncomfortable: automating both sides of a check collapses an independence that older assurance simply took for granted, and independence does not reconstitute itself. If we want it back, we have to engineer it in on purpose.

Who verifies the verifier? The independence problem

Control theory has one rule that predates computing: the maker cannot be the checker. Segregation of duties, tester independence, and the auditor's arm's-length stance all encode the same intuition — a judgment is only worth something if the party making it can fail differently from the party being judged. When both parties are models, that difference has to be constructed rather than assumed.

The failure to watch for is correlated error. Two systems that share a base model, a prompt scaffold, or a training corpus can be confidently, identically wrong, and no amount of automated cross-checking between them will surface it — they agree because they are the same, not because they are right. Genuine independence means the reviewer differs where it counts: a different model lineage, held-out or differently sourced data, or a human placed at the decision point rather than downstream of it.

This is why provenance becomes the center of gravity. Which model, at which version, under which prompt, over which inputs, produced this artifact — and which model, at which version, produced the judgment about it? Without those answers, an approval is an assertion dressed as evidence. And it is why every serious instrument keeps a named human on the hook: the EU AI Act's human-oversight requirement for high-risk systems and the accountable-owner idea in management-system standards both insist that automation may inform a decision but a person must own it. Accountability is the one thing that does not automate.

What counts as evidence when both the artifact and its reviewer are machines

If a reviewer can be a model, then evidence has to mean more than a reviewer's say-so. The word has to shift from an assertion to an artifact: a record with provenance, a timestamp, and enough fixity that an independent party can re-derive the same conclusion. A screenshot proves someone once saw a screen. A signed, reproducible record proves what happened, when, on which system, and by which decision-maker — human or model.

• Describe the model, not just the output: model cards, datasheets, and system cards capture intended use, known limits, and evaluation results.
• Fix the evaluation so it can be repeated: pinned versions, retained inputs, and fixed seeds turn a one-off result into a reproducible one.
• Sign the chain of custody: build-provenance and supply-chain attestation approaches (for example, the open SLSA specification) let a downstream party verify what produced an artifact.
• Keep the raw signal reachable: a reviewer must be able to inspect the underlying record, not only the summary a model wrote about it.

You cannot govern what you cannot measure — and a gamed metric governs nothing

Continuous governance runs on measurement, which makes measurement integrity a control in its own right. The hazard is old and well named: Goodhart's law, in the paraphrase popularized by Marilyn Strathern, holds that when a measure becomes a target, it ceases to be a good measure. Point a single number at an optimizer — human or machine — and it will find the number, not the thing the number was standing in for.

The durable frameworks resist single-metric tyranny by spanning dimensions no one lever can satisfy at once. The SPACE framework (Forsgren, Storey, and colleagues, 2021) deliberately covers Satisfaction and well-being, Performance, Activity, Communication and collaboration, and Efficiency and flow, precisely so that gaming one dimension shows up as damage in another. The DORA program's delivery metrics — deployment frequency, lead time for changes, change failure rate, and time to restore service — balance speed against stability so that shipping faster cannot quietly masquerade as shipping better.

When AI produces both the work and the measurements about the work, that balance is not automatic — the same system that optimizes the metric may also be reporting it. The defensive posture is boring and effective: instrument at the source rather than trusting a computed summary, keep humans able to see the raw signal, span more than one dimension, and treat any number you cannot trace back to its origin as unmeasured rather than as a low score. An honest gap is a better input to a decision than a confident fabrication.

What good looks like: continuous, explainable, human-attributable controls

Pulling the threads together, the controls that survive an era of AI-on-AI governance share five properties. They are less a product category than a design stance, and they map cleanly onto what the emerging instruments already ask for.

• Continuous: evidence is produced on the system's own clock — read-only at the source, append-only in the record — so assurance tracks the live system instead of a sampling window.
• Explainable: every automated judgment carries its provenance — model, version, inputs, and rationale — so a human or an independent system can re-derive it rather than take it on faith.
• Human-attributable: a named owner is accountable for each control and each material decision, consistent with the EU AI Act's human-oversight duty and the accountable-owner concept in management-system standards.
• Independent: the reviewer does not share the author's failure modes, achieved through a different model lineage, differently sourced data, or a human placed at the decision point.
• Reproducible: pinned versions, retained inputs, and fixed seeds mean an evaluation or review can be re-run and return the same answer, by someone who was not there the first time.

How to evaluate any AI governance approach or tool

Whatever combination of framework, platform, or process a team adopts, the questions worth asking are the same, because they target the failure modes automation introduces rather than any one vendor's feature list:

• Does every automated judgment come with its provenance — model, version, inputs, and rationale — or only a verdict?
• Is the reviewer genuinely independent of the author, or do they share a model, prompt, or training set that would make them fail together?
• Is there a named human accountable for each control and each material decision, or does accountability dissolve into the system?
• Is evidence signed, timestamped, and append-only, or can it be edited after the fact without a trace?
• Can a result be reproduced — same inputs, same version, same answer — by an independent party?
• Does the system tell you what it could not verify, or does every dashboard read green regardless of what the evidence supports?

Frequently asked questions