Public-sector GenAI procurement: how California buys AI safely

How California buys GenAI, in brief

Generative AI procurement in California runs through the Government Operations Agency (GovOps), which coordinates the state's GenAI work across the administration. GovOps oversees the component departments that actually do the buying and the standard-setting: the Department of General Services (DGS), the state's primary procurement arm; the California Department of Technology (CDT); the Office of Data and Innovation (ODI); and the California Department of Human Resources (CalHR). The exact roster count varies by source, but that membership is consistent.

Two things distinguish public-sector GenAI buying from an ordinary software purchase. First, a state entity is expected to justify the need and assess the risk before it acquires a tool, not after. Second, the vendor is increasingly expected to demonstrate — not merely assert — how its model handles bias, illegal content, and civil-liberties risk. Both expectations flow from a pair of executive orders and the guidelines issued under them.

Two executive orders frame the rules

The current regime rests on two Newsom executive orders. Executive Order N-12-23 (2023) set the foundation and produced the 2024 guidelines; Executive Order N-5-26 (2026), titled Trusted AI Procurement, builds on it and turns toward what vendors must certify. The second order is forward-looking — per public analyses it does not direct agencies to re-open existing contracts, and it is not itself a statute; it leverages the state's purchasing power to shape vendor behavior, and it applies to vendors nationwide seeking to contract with California.

Before you buy: the 2024 GenAI Guidelines

The operational rulebook for state entities is the State of California GenAI Guidelines for Public Sector Procurement, Uses and Training, released March 21, 2024 to implement EO N-12-23 and jointly authored by GovOps, CDT, DGS, ODI, and CalHR. Per public summaries of the guidelines, they were issued as interim guidance, with a more permanent procurement and training policy anticipated after piloting. The guidelines put the burden of diligence on the acquiring agency before a contract is signed.

• Establish the business need first — identify the problem before procuring a GenAI tool to solve it.
• Assess risks and impacts, including to the security and privacy of California residents, and complete a Generative AI Risk Assessment (SIMM 5305-F) to gauge the risk exposure of a planned deployment.
• Test models before deployment to reduce bias and errors, rather than relying on a vendor's claims alone.
• Stand up a GenAI team for continuous evaluation, so oversight persists after go-live instead of ending at purchase.
• Complete mandatory GenAI training: from March 29, 2024, state purchasing officials access Procurement GenAI Training through the California Procurement and Contracting Academy (CalPCA).

What vendors will be asked to attest to under N-5-26

Executive Order N-5-26 is where the diligence turns outward, toward the vendor. It does not, on its own, enact a finished certification; per law-firm analyses of the order, it directs DGS and CDT to develop new vendor certification criteria within 120 days of the March 30, 2026 signing. Once those criteria are finalized, vendors seeking state contracts are to attest to and explain their policies and safeguards across three areas the order names. Treat the table below as the shape of what is coming, not a checklist already in force.

Beyond certification: what else N-5-26 sets in motion

The certification standards are the headline, but the order tasks GovOps and its component agencies with a broader set of deliverables on the same 120-day clock. For vendors and agency staff, these signal where procurement expectations are heading — toward provenance, data minimization, and safer employee use of vetted tools.

• Watermarking guidance: CDT and GovOps are to issue guidance on labeling AI-generated or significantly manipulated images and video (referencing Cal. Bus. & Prof. Code sections 22757.2 and 22757.3).
• Data-minimization toolkit: multiple agencies are to develop templates, special contract provisions, and review checklists that limit the data a deployment collects and retains.
• Contractor-responsibility reforms: GovOps, with DGS and CDT, is to propose reforms targeting entities judicially determined to have unlawfully undermined privacy or civil liberties.
• Vetted employee tools and training: GovOps component agencies are to facilitate employee access to vetted GenAI tools with security safeguards, and to expand employee AI training.
• A life-events public portal: the order directs a pilot GenAI-powered portal organized around life events such as disaster relief, starting a business, and finding a job — a directive to pilot, not an already-live tool.
• Federal-procurement review: the CDT State Chief Information Security Officer is to independently review federal supply-chain risk determinations and other federal procurement changes, and facilitate continued state procurement where appropriate.

A practical checklist for selling AI to California government

If you are a vendor, the through-line across both orders is that the state increasingly wants demonstrable evidence, not assurances. The following maps the requirements above onto what a well-prepared bid can show today, before the N-5-26 criteria are finalized:

• Be ready to support a SIMM 5305-F risk assessment: know your data flows, model provenance, and residency well enough for an agency to complete one.
• Document the model, not just the output: intended use, known limitations, and evaluation results, in the spirit of a model or system card.
• Show your bias work: how you test for harmful bias, what governance reduces it, and how those results are reproducible rather than one-off.
• Have illegal-content safeguards written down: your policies against generating or distributing CSAM and non-consensual intimate imagery, and how they are enforced.
• Support provenance and watermarking: be able to label AI-generated or manipulated media, anticipating the forthcoming CDT and GovOps guidance.
• Design for data minimization: collect and retain the minimum needed, and be ready to accept special contract provisions that require it.
• Expect testing before deployment: assume the agency will validate the model itself, and make that easy rather than opaque.

How this fits the wider AI-governance landscape

California's approach is a procurement regime layered on the same posture emerging across AI governance more broadly. Where the EU AI Act (Regulation (EU) 2024/1689) sets binding, risk-tiered duties, ISO/IEC 42001:2023 offers a certifiable management system, the NIST AI Risk Management Framework provides voluntary US guidance, and the Colorado AI Act (SB 24-205) imposes a duty of reasonable care on high-risk systems, California uses the state's buying power as its lever. A vendor that has done the work for those frameworks — documented intended use, tested for bias, kept provenance and logs — is largely doing the work California procurement asks for too.

The common thread is that assertion is no longer enough. Both the 2024 guidelines and N-5-26 push toward evidence an agency can inspect: a completed risk assessment, tested-before-deployment results, attestations a vendor must explain rather than merely sign, and provenance for AI-generated media. For anyone selling into or working within government, the durable move is to treat that evidence as a standing capability, not a one-time bid artifact.

Frequently asked questions