Measuring what AI writes: trust is the new bottleneck

The bottleneck moved from writing code to trusting it

For most of software's history, the expensive step was producing the code. Design was cheap talk; typing was the tax. Almost every habit the industry built — story points, velocity charts, lines-of-code counts, the reflex to celebrate a big merge — assumes that output is the scarce thing and therefore the thing worth counting.

That assumption is dissolving. When a capable model can draft a plausible thousand-line change in seconds, generation is no longer the constraint. The constraint has moved downstream, to review, verification, and the plain question of whether a change can be trusted in production. This is not a tooling footnote. It reorders what is worth measuring: when output is abundant and trust is scarce, counting output measures the wrong thing — and often measures it in exactly the direction that hides the risk.

Why output metrics break when a model writes the draft

Output metrics were never good, but they were survivable when a human had to type every line — the effort of writing was itself a crude throttle on volume and a crude signal of thought. Remove the throttle and the signal inverts. A velocity chart can climb while the thing you actually care about — small, reviewable, reversible change — gets worse: larger diffs, more of them, and fewer human eyes per line.

The dangerous dynamic is simple arithmetic. Draft volume can now scale almost without limit; human review capacity cannot. Hold review capacity flat while output rises and review depth per change falls, which means risk concentrates precisely where the dashboard looks healthiest. A team shipping more, faster, with thinner scrutiny will post beautiful activity numbers right up until the incident.

• Lines of code and diff size now measure the model's fluency, not the team's judgment.
• Commit and pull-request counts reward fragmentation and volume, both of which AI makes trivially cheap.
• Per-developer output charts, always fragile, become noise once authorship is shared with a tool.
• None of these say anything about whether a change was understood, reviewed, or is safe to reverse.

DORA and SPACE still work — because they never measured typing

The measurement frameworks that survive this shift are the ones that never counted individual output in the first place. DORA (the DevOps Research and Assessment program — not to be confused with the European Union's Digital Operational Resilience Act, an unrelated financial-sector regulation that shares the acronym) measures the delivery system's ability to ship change safely. Its four widely cited keys describe outcomes, so they are indifferent to who or what wrote the code.

Notice what each key actually rewards: delivering working change, keeping changes small enough to move quickly, and recovering fast when one goes wrong. Change failure rate is the sharpest of the four in an AI-authored world — it is close to a direct read on trust in change, and it rises the moment review thins out beneath a rising tide of generated diffs. Later DORA research added a fifth dimension, reliability, in the same spirit: operational outcome, not authorship.

SPACE — the developer-productivity framework introduced by Forsgren and colleagues in 2021 — makes the underlying point explicit. Productivity, it argues, is multi-dimensional (Satisfaction and well-being, Performance, Activity, Communication and collaboration, Efficiency and flow) and cannot be captured by any single metric; activity, the output dimension, is the most misleading in isolation. That was sound advice when humans wrote everything. When a model writes the draft, it stops being advice and becomes a warning.

The harder question: who is accountable for a change no human wrote?

Metrics tell you whether the system is healthy. They do not answer the question that keeps auditors and incident reviewers up at night: when a change breaks something, who is accountable for it? A generated change does not arrive with accountability attached. Someone still has to have understood it, reviewed it, and authorized it — and in any controlled environment, that someone must be a human with a name.

This maps cleanly onto controls that already exist. IT general controls and segregation of duties hold that the party who authors a change should not be the same party who approves it. A model in the author's seat does not relax that rule; it makes the human reviewer and approver the load-bearing controls, because they are now the only points where judgment enters. The provenance question — what produced this change, who checked it, who approved it — becomes the spine of the audit trail rather than a footnote to it.

The practical work is capturing model involvement as part of that record: not to assign blame to a tool, which is meaningless, but so the trail is complete and reproducible. An auditor who asks how a change reached production should be able to see that a model drafted it, a named engineer reviewed it, and a separate approver released it. That is not bureaucracy for its own sake — it is the difference between a change you can explain and a change you merely tolerate.

Regulation is arriving faster than the metrics

While teams argue about dashboards, the governance layer is being written into standards and law. Three instruments matter most for anyone building or buying software that uses AI, and it is worth being precise about what each one is, because they are frequently confused.

• The EU AI Act is risk-based: obligations scale with how a system is used, and general-purpose model providers carry their own transparency and documentation duties.
• ISO/IEC 42001 gives an organization a certifiable structure to prove its AI governance exists and operates — the AI analogue to an ISO/IEC 27001 information-security management system.
• The NIST AI RMF is voluntary and practice-oriented; many teams use it to shape day-to-day risk work and reach for ISO/IEC 42001 when they need something certifiable to show a customer.

What good looks like: metrics that survive AI-authored code

Read the three instruments together and a common thread appears: none of them prescribe a velocity number, and none reward output. Every one asks whether you can document, justify, and evidence how a system was built and changed. That is a trust question wearing a compliance costume — the same question the metrics section arrived at from the engineering side. A measurement program built for a world where AI writes the code should follow the same instinct.

• Measure the delivery system, not the person: favor the DORA keys — especially change failure rate and lead time — over any per-author output count.
• Promote review to a first-class metric now that drafts are nearly free: track review coverage and depth, not just that a review happened.
• Record provenance end to end: what generated a change, what a human reviewed, and who approved it, kept as a reproducible trail.
• Treat unmeasured risk as a real, reportable state — never let a dashboard read green over something you have not actually verified.
• Attach AI governance to the control frameworks you already run rather than standing up a disconnected parallel program.

What to watch next

Three things are worth watching over the next year. First, whether general-purpose model transparency and documentation obligations mature into something buyers actively request in due diligence, the way SOC 2 reports became table stakes. Second, whether ISO/IEC 42001 certification moves from novelty to procurement checkbox, which would pull AI governance out of the lab and into contracts. Third, whether the measurement community adds an explicit provenance dimension to DORA- and SPACE-style thinking — a standard way to record that a change was model-drafted and human-approved.

The throughline under all of it is old and slightly unfashionable: shipping fast only matters if you can still trust what you shipped. AI has made the shipping part cheap. The interesting work — and the scarce resource — is now everything that turns a generated change into one you can stand behind. That is what these guides will keep measuring.

Frequently asked questions