What AI features are measured, not generative?

Audience

Audience: a security reviewer, finance owner, or engineering lead asking whether ShipReady Metrics uses an LLM to compute scores or store agent prompts. This page covers surfaces 3–8 from the E31 inventory — not compliance chat or trust buyer chat.

Deterministic score graders — not LLM

AI ROI (src/lib/scoring/scores/ai-roi.ts), AI Readiness (src/lib/scoring/scores/ai-readiness.ts), and Agent Health (src/lib/scoring/scores/agent-health.ts) are pure deterministic graders. They combine declared inputs and connector signals; null dimensions are omitted — never baseline-filled. They are not LLM and do not call Anthropic.

Agent Health reads agent_telemetry_* data ingested via src/lib/agent-telemetry/ingest.ts and src/app/api/ingest/agent-telemetry/route.ts. Run volume is context, not a graded dimension (see agent-health.ts header).

AI-generated badges

src/components/compliance/ai-generated-badge.tsx is a unified UI label for AI-generated content across auditor-facing views. It shows artifact class, attestation state (attested / rejected / pending_attestation / legacy), and auditor acceptability. It is not a generative feature — it labels artifacts that already exist.

Agent telemetry — no prompts in telemetry

src/lib/agent-telemetry/ingest.ts header: NEVER store raw prompts, secrets, or transcripts — only redacted metadata/refs. The POST handler at src/app/api/ingest/agent-telemetry/route.ts authenticates a bearer token, normalizes batches, and upserts into agent_telemetry_* tables to feed Agent Health. Write-only — the endpoint never returns tenant data.

Cost and usage metering

src/lib/ai/anthropic-admin.server.ts fetches real USD cost and token usage from the Anthropic Admin API when ANTHROPIC_ADMIN_KEY is set. Without the admin key, functions return null and the UI shows estimates only. Failure is always null, never zero — returning zero would fabricate a financial figure.

src/lib/ai/usage.server.ts records best-effort token/cost per feature into ai_usage_events. Feature keys include compliance_chat, trust_buyer_chat, policy_ai, questionnaire_ai, remediation_ai, vendor_review_ai, commitment_ai, security_audit, and attestation. Recording never breaks the user's AI request.

Feature model registry / AI provenance

resolveFeatureModel in src/lib/compliance/sox/ai-provenance.server.ts pins the model per (org, feature) when an approved+active sox_ai_config_version row exists; any read error or missing config returns the fail-safe default ANTHROPIC_MODEL / claude-opus-4-8. Binding a call site to the registry cannot break the AI call — it only takes effect once an admin approves and activates a version.

What you need to do now

Connect sources and optional agent telemetry before expecting AI Readiness, AI ROI, or Agent Health to grade. Unmeasured dimensions stay visible gaps — the product does not invent scores.

If you need real Anthropic spend (not estimates), provision ANTHROPIC_ADMIN_KEY — a different credential from the model API key.

Checklist

Verify these items when describing measured AI surfaces in a security review.

• Confirm reviewers distinguish deterministic scores from generative compliance chat.
• Confirm agent telemetry policy: no prompts in telemetry — only redacted metadata/refs.
• Confirm AI-generated badges are labels, not generated content.
• Confirm Admin API cost shows null/estimates when ANTHROPIC_ADMIN_KEY is absent — never a fabricated zero.
• Review which AiFeature keys your org's usage dashboard records.