Test data management in the AI era: realistic data without the risk

What test data management is

Test data management (TDM) is the practice of getting the right data into the right environment to test software — data that looks and behaves like production, is safe to handle, and is available when a test needs it. Every non-production environment (development, QA, staging, a CI pipeline, a demo) needs data to run against. Where that data comes from, and what it contains, is the whole subject.

The tension at the core of TDM is simple to state and hard to resolve. Tests are most trustworthy when they run against data that mirrors reality: the same shapes, the same edge cases, the same volume and messiness as production. The most faithful mirror of production is production itself — which is exactly the data you are least allowed to copy, because it is full of real people. TDM is the set of techniques for resolving that tension: realistic enough to be useful, safe enough to be allowed.

Done well, it is invisible plumbing. Done badly, it shows up in two places: as flaky, low-coverage tests that pass while production breaks, and as a compliance finding when someone notices that a copy of the customer database is sitting in a developer sandbox.

Why production data in lower environments is a finding

The oldest shortcut in testing is to clone production into a lower environment and test against that. It is fast, it is faithful, and it is a liability. Lower environments typically have more users, broader access, weaker monitoring, and shorter-lived infrastructure than production. Copying live personal data into them takes information that was collected and protected for one purpose and scatters it across systems that were never designed to protect it.

This is why data-protection regimes converge on the same instinct: minimize. The GDPR's data-minimisation principle (Article 5(1)(c)) requires that personal data be limited to what is necessary for the purpose; its purpose-limitation principle (Article 5(1)(b)) means data collected to run a service is not automatically fair game to test with. The California Consumer Privacy Act, as amended by the CPRA, carries a comparable expectation and a carve-out only for genuinely de-identified data. HIPAA treats protected health information copied into a test system as still protected, unless it has been de-identified under the Safe Harbor or Expert Determination methods in 45 CFR 164.514. Anonymised data can fall outside the GDPR entirely (Recital 26) — but only if it is genuinely not re-identifiable, which is a higher bar than it sounds.

For an auditor, the mechanics matter less than the exposure. Copying production data — especially regulated data — into an environment with weaker controls is a classic IT general controls (ITGC) weakness: it widens the population of people and systems that can touch sensitive data, and it does so outside the access controls that protect the source. That is what makes production data in lower environments a finding, not a style preference. The remedy is not to test against worse data; it is to test against data that is realistic without being real.

The mechanics: masking, subsetting, and referential integrity

If you cannot copy production wholesale, you have to manufacture a safe stand-in. A handful of long-standing techniques do most of the work. Masking transforms sensitive fields into safe substitutes — a real name becomes a plausible fake one, an account number becomes a different well-formed number — in a copy that no longer contains the originals. Subsetting takes a smaller but internally consistent slice of production, so environments are cheaper to run and faster to refresh. Tokenization swaps sensitive values for format-preserving tokens that map back to the originals only inside a secured vault, which is useful when a value has to be re-linked later. Synthetic generation, covered below, manufactures records from scratch.

The discipline that separates working test data from a broken pile of it is referential integrity. Real data is relational: an order points to a customer, a claim points to a patient, a payment points to an account. Mask a customer id in one table but not the order that references it, or subset the orders without the customers they depend on, and tests fail for reasons that have nothing to do with the code. Preserving those relationships consistently — the same transformation applied everywhere a key appears — is the part teams most often underestimate.

The other two disciplines are refresh and coverage. Test data ages: schemas change, new features introduce new shapes, and a dataset captured six months ago no longer resembles what production looks like today, so tests drift out of touch with reality. And a dataset can be safe, fresh, and still useless if it does not contain the cases that matter — the null field, the unusual character set, the account in an unexpected state. Managing refresh cadence and measuring coverage are what keep test data honest over time.

Techniques compared

The four techniques are not competitors so much as a toolkit; most real programs combine them — subset, then mask, and generate synthetic data for the cases the subset does not contain. Each makes a different trade between fidelity, safety, and effort.

The AI angle, part one: synthetic data generated for testing

The AI era changes TDM from two directions. The first is generation: using models to produce synthetic test data that preserves the statistical shape of production — the distributions, correlations, and edge cases — without copying any real record. Instead of masking real people, you learn what real data looks like and manufacture new, fictional data with the same behavior. At its best this is the cleanest answer to the core tension: realistic and, by construction, containing no real person.

The catch is that synthetic does not automatically mean private, and realistic and private pull in opposite directions. A generative model trained on production data can memorize specific records and reproduce them, and the closer its output tracks the real distribution, the more it can leak. This is not hypothetical: membership-inference attacks (Shokri and colleagues, 2017) can determine whether a particular individual was in the training data by probing a model's behavior. A synthetic dataset can carry a real person's fingerprint even when no row is copied verbatim.

Two more failure modes travel with generation. Bias: a model learns from history, so synthetic data reproduces — and can amplify — whatever skew the source carried, handing the same blind spots to anything tested only against it. And coverage: a generator reproduces the patterns it learned, which means the genuinely novel, malformed, or adversarial inputs where defects tend to hide may simply be absent unless someone deliberately injects them.

• Leakage and memorization: a model trained on production can regurgitate real records; a synthetic label is not a privacy guarantee, and re-identification risk has to be tested rather than assumed.
• Fidelity versus privacy: the closer synthetic data tracks the real distribution, the more useful and the more leaky it is; formal guarantees such as differential privacy (Dwork and colleagues, 2006) bound that leakage at a measurable cost to fidelity.
• Bias inheritance: synthetic data learned from skewed history reproduces that skew, so a system tested only on it inherits the same blind spots.
• Coverage gaps: generated data reflects patterns the generator learned; malformed, out-of-distribution, and adversarial cases must be injected deliberately, not expected to appear on their own.

The AI angle, part two: test data for AI systems

The second shift is subtler, and it hides in the same phrase. Test data for an AI system is not the synthetic data you use to test ordinary software — it is the data you use to test the AI itself. When the thing under test is a model or an AI-powered feature, the classic notion of a test case becomes an evaluation: an input, an expected behavior, and a way to score how close the system came.

This is where a distinct set of datasets earns its keep — evals, edge-case and adversarial inputs, and red-team collections — and where a distinct hazard appears. The integrity of an evaluation depends on its data being independent of the training data. If examples used to measure a model also appeared in what trained it, the score measures memorization, not capability. Keeping evaluation data held out and provenance-tracked is therefore a control, not a chore.

• Evaluation sets (evals): curated, labeled inputs with expected behavior, used to measure an AI system before and after each change — the AI analogue of a regression suite.
• Edge-case and adversarial data: malformed inputs, prompt-injection attempts, and out-of-distribution examples that probe where a system fails, not only where it succeeds.
• Red-team datasets: adversarial cases assembled to elicit unsafe, biased, or non-compliant behavior — a practice the NIST AI RMF Generative AI Profile (NIST AI 600-1, 2024) and the EU AI Act (adversarial testing for general-purpose models with systemic risk) both point to.
• Held out and uncontaminated: eval data must stay separate from training data; overlap silently inflates scores, so the provenance of the eval set is itself part of the control.

What good looks like

Put the two eras together and a defensible TDM practice has a recognizable shape. It is less a product than a set of properties, and they line up with what data-protection law and AI-governance frameworks already ask for: minimize the real data, preserve the relationships, keep the privacy claim testable, and record where every dataset came from.

• Data stays inside its blast radius by default: production PII stays in production; lower environments get masked, subsetted, or synthetic data, so a breach of a test system does not become a breach of customer data.
• Referential integrity is preserved end to end: masking and subsetting keep keys consistent across tables, so tests exercise real relationships rather than broken ones.
• Refresh and coverage are managed together: data is refreshed on a known cadence and measured for coverage, so tests neither run against stale shapes nor silently lose the cases that matter.
• Provenance is recorded: which technique produced a dataset, from which source, under which policy — so an auditor or an automated reviewer can trace a test result back to compliant data.
• The privacy claim is tested, not asserted: re-identification and leakage risk are evaluated, not assumed away because a dataset is labeled synthetic or masked.

How to evaluate any TDM approach

Whatever mix of masking, subsetting, tokenization, and generation a team lands on, the questions that separate a real TDM practice from a hopeful one are the same, because they target the exposures the shortcuts create:

• Does any real production PII reach a lower environment, and if so, under what legal basis and what controls?
• Is referential integrity preserved across masked or subsetted data, or do tests run against broken relationships?
• For synthetic data, is the privacy claim measured — re-identification and membership-inference risk — rather than assumed from the label?
• Is test coverage tracked, so subsetting or generation has not quietly dropped the edge cases where defects live?
• For AI systems, are eval sets held out from training data and provenance-tracked, so scores are not inflated by contamination?
• Can each dataset be traced to its source, technique, and policy — the evidence an auditor or reviewer needs?

Frequently asked questions