AI development metrics: a catalog for AI-assisted SDLC and AI systems
Updated
This is the comprehensive, vendor-neutral catalog of AI development metrics, organized by lens. When AI makes code abundant, raw throughput stops being the constraint and trust in each change becomes the scarce resource, so the metrics that matter shift from how much you ship to whether it can be trusted.
It spans two lenses: metrics for the humans-plus-AI process that ships software — contribution, acceptance, review burden, provenance, and authorship-split change failure — and metrics for the AI systems you ship, covering evals, groundedness, hallucination, drift, jailbreak resistance, agent-run reliability, cost, and governance. The taxonomy is still emerging, so treat every entry as a widely-discussed signal, not a settled benchmark, and read each one alongside its blind spot.
How to use this catalog
For most of software history, engineering metrics optimized for throughput because writing code was the constraint. When a capable model drafts code on demand and agents open pull requests, production stops being scarce. The bottleneck moves downstream, to the people and systems that must decide whether a change is correct, safe, and worth keeping — so justified trust in each change, not raw output, becomes the resource worth measuring. This is still an emerging, unstandardized area: the taxonomy below adapts settled delivery and governance frameworks to the human-plus-AI split rather than reading off a finished standard.
Use the catalog the way you would a menu, not a scorecard. Pick a few metrics per lens that match what you actually need to know, and always read a metric next to its blind-spot column — every entry here is useful and every one is gameable. Above all, no single metric is a target: Goodhart's law says that once a measure becomes the goal it stops measuring what you cared about, so pair each number with a counter-metric and watch trends and distributions rather than a lone absolute.
- Pick a few metrics per lens that map to a real decision — not the whole list.
- Read every metric next to its blind spot; each one is useful and each one is gameable.
- No single metric is a target (Goodhart's law): pair it with a counter-metric and watch the trend.
- Weight authorship-split outcomes (change failure, rework) over activity counts (acceptance, contribution rate).
AI-assisted SDLC metrics: measuring humans + AI shipping software
This is an emerging, still-unstandardized measurement area — the metrics below adapt established delivery frameworks (DORA, SPACE, DevEx, Flow, NIST/OWASP, FinOps) to the human+AI authoring split rather than drawing on a settled AI-specific standard, so treat them as widely-discussed signals, not benchmarks. Weight authorship-split outcome metrics (CFR, rework) over activity metrics (acceptance, contribution rate), which are the easiest to game.
| Metric | What it measures | Blind spot / how it is gamed | Source |
|---|---|---|---|
| AI Code Contribution Rate | Share of merged code (lines, commits, or PRs) generated or materially assisted by AI. | Lines-of-code is a poor value proxy — inflated by verbose AI output and silent on correctness. | DORA (2024 State of DevOps, AI adoption measures); GitHub Copilot telemetry |
| AI Suggestion Acceptance Rate | Proportion of inline AI suggestions a developer accepts into the editor. | Acceptance is not retention or quality — devs accept then delete; it rewards convenient autocomplete, not outcomes. | GitHub Copilot productivity research, framed on the SPACE framework |
| Rework / Code Churn Rate on AI Code | Fraction of AI-authored code reverted or rewritten within a short window (e.g., two weeks). | Churn can be legitimate iteration; AI attribution is fuzzy and the short-window cutoff is easy to game. | DevEx framework (Forsgren, Storey, Noda et al.); GitClear code-quality research |
| Change Failure Rate by AI-vs-Human Authorship | DORA change failure rate split by whether a change was AI-authored/assisted or human-authored. | Co-authorship makes attribution ambiguous; failures can be relabeled to protect the AI-adoption narrative. | DORA (Four Keys / core DORA metrics) |
| AI Change Review Coverage (Review Burden) | Percent of AI-generated changes that receive human code review, and the review load they create. | High coverage can be rubber-stamping — it counts reviews, not review depth or reviewer understanding. | SPACE framework (Collaboration; Efficiency & Flow); Google Engineering Practices code review |
| Provenance / Attestation Coverage | Percent of AI-authored changes carrying provenance metadata: model, prompt, and human approver. | Presence of metadata is not accuracy — self-reported provenance can be incomplete or falsified. | NIST SSDF (SP 800-218); SLSA supply-chain attestation; emerging AI-BOM |
| Time-to-Review AI PRs (Review Latency) | Median time from an AI-authored PR opening to first review and to merge. | Fast turnaround can mean shallow review; latency alone hides queue depth and oversized batches. | SPACE (Efficiency & Flow); DORA lead-time decomposition |
| AI Adoption & Trust Rate | Share of developers using/relying on AI tooling and their reported trust in its output. | Adoption is a leading input, not an outcome; self-reported trust is subjective and novelty-biased. | DORA (2024 State of DevOps, AI chapter) |
| Flow Efficiency of AI-Assisted Work Items | Active work time versus total flow time for work items touched by AI. | Efficiency can rise while downstream rework worsens; wait-state definitions are easily gamed. | Flow Framework / Flow Metrics (Mik Kersten, Project to Product); Kanban |
| Vulnerability Introduction Rate in AI Code | Security defects per unit of AI-generated code, surfaced by scanning and review. | Scanners miss logic and business-logic flaws — a low number may just mean weak detection. | OWASP (Top 10; Top 10 for LLM Applications); NIST SSDF secure-coding practices |
| Developer Experience / Perceived AI Productivity | Perceptual DevEx signals — satisfaction, flow, cognitive load — attributed to AI use. | Self-reported and hype-sensitive; perceived speed routinely diverges from measured delivery. | DevEx framework (Forsgren, Storey, Noda et al.); SPACE (Satisfaction & Well-being) |
| AI Tooling Cost per Change (Unit Economics) | AI token/subscription spend per merged PR or per developer — the cost side of AI-assisted delivery. | Optimizing cost can degrade quality; ignores value delivered and downstream rework cost. | FinOps Foundation (FinOps Framework / FinOps for AI, FOCUS) |
AI System, Eval & Governance Metrics
Metrics for the AI you actually ship — how well it's evaluated, whether it drifts or gets jailbroken, what a run costs, and who is on the hook when it fails. Each maps to the NIST AI RMF Measure function and an ISO/IEC 42001 AIMS control.
| Metric | What it measures | Blind spot / how it is gamed | Source |
|---|---|---|---|
| Faithfulness / Groundedness | Share of generated claims that are actually supported by the retrieved source context. | A faithful answer can still be irrelevant or answer the wrong question; usually scored by an LLM judge that shares the generator's blind spots. | RAGAS / TruLens RAG Triad; NIST AI RMF MEASURE 2.9 (valid & reliable); ISO/IEC 42001 |
| Context Precision / Recall (retrieval quality) | Whether the retriever surfaced the right supporting chunks and how much of the needed evidence it recalled. | Isolates retriever from generator, so a perfect retrieval score says nothing about the final answer; sensitive to how ground-truth relevance is labeled. | RAGAS; NIST AI RMF MEASURE 2.9; ISO/IEC 42001 |
| Hallucination Rate | Fraction of outputs containing fabricated or unsupported statements. | Entirely dependent on the reference set; a model games it by abstaining or hedging, trading hallucination for uselessness. | Vectara HHEM / TruthfulQA; NIST AI RMF MEASURE 2.3 & 2.9 (trustworthiness) |
| Task Success Rate / pass@k | Share of tasks solved correctly within k sampled attempts (code, agent, or QA). | Benchmark contamination and narrow task distributions inflate scores; pass@k rewards lucky retries over first-shot reliability. | OpenAI HumanEval (pass@k) / SWE-bench; NIST AI RMF MEASURE 2.5 & 2.9 |
| Drift (data / concept / embedding) | Distributional shift in inputs, outputs, or embeddings versus the evaluation baseline over time. | Detectors flag that something changed, not that harm occurred; thresholds are arbitrary and slow degradation slips under them. | MLOps monitoring practice; NIST AI RMF MEASURE 2.4 (deployed-system monitoring); ISO/IEC 42001 |
| Guardrail Block Rate / Over-Refusal | Fraction of requests stopped by safety filters, split against legitimate requests wrongly refused. | A high block rate can mask over-refusal and lost helpfulness; the headline number rarely separates true positives from false positives. | OWASP Top 10 for LLM Apps; NIST AI RMF MEASURE 2.6 (safe); ISO/IEC 42001 |
| Jailbreak / Prompt-Injection Resistance (attack success rate) | Share of adversarial or injected prompts that bypass safety and policy controls. | Only ever measured against a known attack corpus; a low ASR on a static red-team set gives false assurance against novel attacks. | OWASP LLM01 (Prompt Injection) / MITRE ATLAS; NIST AI RMF MEASURE 2.7 (secure & resilient) |
| Agent Run Success & Step-Completion Rate | Whether multi-step agent runs reach the goal and complete each intermediate step correctly. | Goal-level success hides unsafe or wasteful intermediate actions and reward-hacking; a run can 'succeed' via a forbidden shortcut. | Emerging agent evals (τ-bench, SWE-bench, WebArena); NIST AI RMF MEASURE 2.5 |
| Cost per Successful Request (AI unit economics) | Token/inference spend normalized to a request, or better, to a successful outcome. | Optimizing cost alone rewards cheap-but-wrong answers and ignores retry amplification that hides in the 'success' denominator. | FinOps Foundation (Cloud + AI FinOps, unit economics) |
| Latency (p95 / time-to-first-token) | Tail and streaming response times against a latency SLO. | Averages hide the slow tail; a fast TTFT can mask a slow full completion, and speed says nothing about correctness. | Google SRE — Four Golden Signals / SLIs & SLOs |
| Harmful-Bias / Toxicity Rate | Rate of biased, toxic, or disparate outputs across protected subgroups. | Aggregate fairness numbers average away subgroup harms, and the choice of fairness metric can flip the verdict. | NIST AI RMF MEASURE 2.11 (harmful bias); ISO/IEC TR 24027; ISO/IEC 42001 |
| % AI Artifacts with Named Human Attestation | Share of models, prompts, and datasets with a documented, accountable owner who signed off. | A signature is a checkbox — it proves someone is named, not that the artifact was actually reviewed or is safe. | ISO/IEC 42001 (roles & accountability); NIST AI RMF GOVERN 2/3; Model Cards (Mitchell et al.) |
| Model / Config Provenance & Lineage Coverage | Fraction of deployed models with traceable version, training-data lineage, and config history (AI-BOM). | Recorded provenance is not verified provenance; complete lineage still leaves the model itself opaque and unexplained. | ISO/IEC 42001; NIST AI RMF MAP 2.2; CycloneDX ML-BOM / SLSA supply-chain provenance |
| AI Incident Count & MTTD/MTTR | Number of AI-specific incidents (harmful output, safety or security breach) and time to detect and resolve them. | Only counts detected and reported incidents; under-reporting makes an unsafe system look clean with a low count. | NIST AI RMF MANAGE 4 (incident response); OECD AI Incidents Monitor; ISO/IEC 42001 |
Frequently asked questions
Why do classic throughput metrics break when AI writes the code?
Because they were proxies for a constraint AI removed. Velocity, commits, and lines all assumed writing code was the bottleneck; when a model drafts code cheaply those numbers inflate almost for free while saying nothing about whether the change is correct or safe. The scarce resource becomes justified trust in each change, so measurement has to move to stability, oversight, and provenance.
Is there a standard set of AI development metrics?
Not yet — this is an emerging, still-unstandardized area. The metrics in this catalog adapt established frameworks (DORA, SPACE, DevEx, Flow, NIST AI RMF, ISO/IEC 42001, OWASP, FinOps) to the human-plus-AI split, so treat them as widely-discussed signals rather than settled benchmarks.
Which AI development metrics are easiest to game?
The activity metrics: suggestion acceptance rate, AI code contribution rate, and lines generated. They measure model eagerness and habit, not delivered value, and a verbose model can move them while creating more code to review. Weight authorship-split outcomes — change failure rate and rework on AI code — over those activity counts.
What is the difference between the two lenses?
The first lens measures the process — humans plus AI shipping software — with signals like contribution rate, review burden, provenance, and change failure rate split by authorship. The second measures the AI systems you ship: eval scores, groundedness, hallucination, drift, jailbreak resistance, agent-run reliability, cost, and governance. Most teams building with AI need both.
How should we pick metrics from this catalog?
Choose a few per lens that map to a decision you actually face, read each one next to its blind spot, and never turn a single metric into a target — Goodhart's law guarantees it will then stop measuring what you cared about. Pair every number with a counter-metric and watch the trend, not the absolute.
Published by ShipReady Metrics, an evidence-based technology and compliance intelligence platform. This guide is educational and vendor-neutral.