Operational guidance, not legal advice. This page distills named public sources (regulator guidance and industry practice). It is not a legal determination, not a notification decision, and not a substitute for your counsel, insurer, or a retained DFIR firm. Verify applicability and current deadlines for your facts and jurisdiction.
What is continuous evidence collection?
Updated
Collecting control evidence on a schedule from connected systems rather than screenshotting configurations before an audit. It gives you a period record instead of a snapshot, and it makes drift visible while you can still fix it.
Explainer, last verified 10 September 2026 against the AICPA Trust Services Criteria monitoring criteria, ISO/IEC 27001:2022 Clause 9, and NIST SP 800-137 on information security continuous monitoring. Continuous collection is industry best practice and, in some framings, a monitoring control — it is not a legal mandate anywhere on this page. This page is not legal advice, does not determine YOUR obligations, does not decide which frameworks apply to you, does not start a clock, and does not file with an auditor.
What this page is, and what it is not
Audience: a security or compliance leader deciding whether to invest in automated evidence collection, or being sold it, and wanting to know what it genuinely changes.
This page is not legal advice. It does not determine YOUR obligations, does not decide which frameworks apply to your organisation, does not start a clock, and does not file with an auditor. Last verified 10 September 2026.
This is also not marketing. Continuous collection is genuinely useful and it is oversold. The useful part is narrow and real: for controls whose state lives in a system with an interface, a scheduled capture produces a record across the period rather than a snapshot at the end, and it surfaces drift within days rather than at fieldwork. The oversold part is the implication that it produces audit readiness. It does not. It produces fresher inputs to a judgement that a human still has to make.
- Continuous collection changes the freshness and completeness of evidence. It does not change what a control requires or whether yours works.
- Nothing on this page is a legal requirement. Continuous monitoring appears in guidance and as a control objective, not as a statutory duty.
- The share of your evidence that is genuinely automatable is smaller than platform demonstrations suggest — policies, reviews, judgements, and third-party reports are manual everywhere.
- Collection is not the same as monitoring. Capturing a configuration daily and nobody looking at it produces a large archive and no control.
Point-in-time compared with continuous collection
First column is the dimension, then how each approach behaves, then the honest note. Neither approach is required; the choice is about cost and about how early you find problems. Last verified 10 September 2026. Not legal advice.
| Dimension | Point-in-time capture | Continuous collection | Honest note |
|---|---|---|---|
| What it produces | A snapshot showing the state on the day it was captured | A series of captures showing the state across the period | For a period-of-time report the series is what the criteria actually contemplate. A single snapshot supports a design assessment, not an operating one. |
| When gaps surface | During preparation or fieldwork, when remediation time has run out | Within the collection interval, while the period is still open | This is the largest real benefit, and it is about time to remediate rather than about evidence quality. |
| Effort profile | Concentrated: weeks of scramble per cycle | Front-loaded into integration, then low per cycle for connected sources | The scramble does not vanish. It shrinks to the manual residue, which is bigger than expected. |
| Drift detection | None between captures. A setting relaxed in March and restored in November looks compliant | Visible, if the captures are compared and someone reviews the difference | Automated capture alone does not detect drift. Comparison plus a human review does. This distinction is where most value is lost. |
| Population completeness | Usually derived from a one-off export at the end, which is fine if the source retained the whole period | Assembled progressively, so short source retention does not destroy the population | Continuous collection genuinely rescues populations that platform retention would otherwise expire. That is an underrated benefit. |
| Coverage | Whatever you remember to capture | Whatever is connected — comprehensively for those sources, not at all for the rest | Coverage becomes a property of your integrations, which creates a new failure mode: confident dashboards over partial scope. |
| Failure mode | Missing or stale evidence, and screenshots nobody dated | A green dashboard covering a fraction of the control set, and an archive nobody reviews | Both failures are recoverable. The continuous one is more dangerous because it feels like assurance. |
| Cost | Low tooling cost, high recurring people cost | Higher tooling cost, lower recurring people cost for connected sources | For a small control set and one framework, point-in-time can be the rational choice. That is not a compliance judgement, it is a budget one. |
| What an auditor does with it | Tests the evidence provided against the criteria | Tests the evidence provided against the criteria | Identical. No auditor accepts a control because collection was automated. The collection method is not itself evidence of effectiveness. |
What automation can and cannot evidence
The honest boundary. Left column is what a scheduled integration can genuinely capture; right column is what it cannot, and where a human artifact remains the only evidence. Last verified 10 September 2026. Not legal advice.
| Control area | What automation can capture | What it cannot capture | What still has to be human |
|---|---|---|---|
| Logical access | Current user, group and role membership; multi-factor enforcement settings; privileged-role lists, on a schedule | Whether the access each person holds is appropriate to their job | The access review: a named reviewer's decision per account, with revocations evidenced. |
| Change management | Merged-change and deployment populations, branch-protection configuration, required-check results | Whether reviews were meaningful, or whether the tests that gate merges test anything | Review quality judgement, and the control narrative that explains where authorisation happens. |
| Vulnerability management | Findings with severity, first-seen and closed dates, remediation timing against stated service levels | Whether a risk acceptance was reasonable, or whether an unscanned system exists | Risk-acceptance decisions with named approvers, and confirmation of scan coverage against a real asset inventory. |
| Monitoring and alerting | Logging and alert-rule configuration, and alert volumes | Whether alerts were investigated competently | Triage records showing what was concluded and by whom. |
| Incident management | Ticket and incident metadata, timestamps, and status transitions | The severity judgement, the impact assessment, and the reportability decision | The written decisions with named decision-makers, and the post-incident review. |
| Policies and governance | That a document exists and when it changed, where the repository exposes it | Approval, understanding, and whether anyone follows it | The approval record, the management review, and the training evidence. |
| Vendor management | Little to nothing, unless the vendor exposes an interface you can query | Vendor report contents, subservice-organisation assessment, and criticality judgement | Collecting and reviewing the reports, and recording the assessment. |
| People controls | Some onboarding and offboarding timestamps, where the human-resources system is connected | Background-check outcomes, training comprehension, and disciplinary process | Records held by people operations, provided as manual evidence. |
| Physical and environmental | Nothing, for most modern companies using cloud providers | Everything | The provider's own attestation report, reviewed and retained by you. |
| Risk management | That the register changed | Whether the risk assessment is sound | The assessment itself, the treatment decisions, and the management review. |
Which kind of authority continuous collection carries
Continuous is frequently sold as though a framework demanded it. None cited here does. Last verified 10 September 2026.
| Statement | Which kind of authority | What it does not mean |
|---|---|---|
| No statute on this page requires continuous evidence collection. | Statement about the absence of a legal requirement. | Does not mean monitoring is optional as a control. It means the collection method is your choice, not a legal obligation. |
| AICPA TSC CC4.1 and CC4.2: the entity evaluates and communicates internal control deficiencies, and monitors its controls. | Attestation criteria, applying because you sought a report. | Requires monitoring, not automated collection. A quarterly manual review that is genuinely performed and documented satisfies the criteria. |
| ISO/IEC 27001:2022 Clause 9: monitoring, measurement, analysis, evaluation, internal audit, and management review; plus surveillance audits during a certification cycle. | Certification requirement where you seek or hold certification. | Requires that you monitor and evaluate on your stated cadence. It does not name a tool, a frequency, or an automation level. |
| NIST SP 800-137 information security continuous monitoring. | Regulatory and technical guidance; binding only where imposed, for example under a federal contract or FedRAMP. | Even where it binds, it describes an approach to maintaining awareness, not a requirement to buy a collection platform. |
| PCI DSS v4.0.1 introduced targeted risk analyses and expectations that certain controls operate on a defined frequency with evidence. | Legal requirement in the contractual sense, for entities in scope. | Requires defined frequency and evidence of operation for named requirements. It does not require continuous automated collection. |
| Collect continuously for the controls whose state lives in a connected system, and keep the rest on a recorded manual cadence. | ShipReady Metrics recommendation, and common industry best practice. | Not a requirement, and not audit readiness. It improves freshness, completeness, and time to remediate. |
| Compare successive captures and have a human review the differences. | ShipReady Metrics recommendation. | Not named in any framework here. It is the step that converts collection into monitoring, and it is the step most often skipped. |
Checklist
A question list for a continuous-collection programme. Not a determination that any framework applies to you. Last verified 10 September 2026. Not legal advice.
- Do we know what share of our in-scope control set is genuinely covered by automated collection, expressed as a fraction of controls rather than as a dashboard colour?
- Is the uncovered remainder written down, with an owner and a manual cadence per item?
- Are successive captures compared, and does a named person review the differences?
- Does anything happen when a captured setting changes — an alert, a ticket, a decision record — or does it only accumulate?
- Do captures cover the whole period we intend to be assessed over, including the interval before we turned collection on?
- Is the collection interval short enough that a relaxation and its restoration between captures could not hide?
- Do we retain the captures themselves long enough to serve as the period record?
- When collection breaks — expired credential, revoked token, changed interface — do we find out, and is the gap recorded?
- Does the scope of each connected source match reality, so a newly created account, project, or repository is included automatically?
- Do we avoid describing automated collection as evidence of effectiveness in our control narratives?
What to do now
Ordered so the honest scoping happens before the tooling. None of these steps determines YOUR obligations or files anything with an auditor.
- List your in-scope controls and mark each as automatable, partly automatable, or manual. That fraction is the realistic ceiling on any automation investment.
- Connect the highest-volume, highest-drift sources first: identity, cloud control plane, and source control. They cover the most controls per integration.
- Turn collection into monitoring by comparing captures and routing differences to a named reviewer, with the review recorded.
- Put the manual remainder on a calendar with owners, because it will otherwise be discovered during fieldwork.
- Add failure detection for the collectors themselves, and record collection outages as evidence gaps rather than letting them pass silently.
- Make source scope dynamic where you can, so new accounts, projects, and repositories are covered without a manual step.
- Set retention on the captures to cover your audit period, since the series is the period record.
- Write the control narrative honestly: say that collection is automated for these controls and manual for those, and never imply automation demonstrates effectiveness.
- Re-verify the cited criteria annually against the primary sources below and record the date. Ours says 10 September 2026.
Where this shows up in ShipReady Metrics
Only shipped behaviour is described here, including its limits.
Connectors ingest on a schedule. The GitHub App is read-only across Contents, Metadata, Pull requests, Actions, Administration, the Dependabot, code-scanning and secret-scanning alert feeds, and organisation Members; it ingests those alerts as findings. GitLab ingest covers security findings. Vulnerability data is ranked using CISA KEV, EPSS and CVSS, deduplicated across sources, and searchable for blast radius over captured dependencies, including transitive npm dependencies where a lockfile was fetched. Delivery Health computes DORA posture from connected sources including GitHub Actions runs.
If you already have a session: signed-in app → Compliance holds evidence collection, which assembles control-mapped artifacts for starter control subsets from those connected sources. Coverage is a starter subset, not your whole control set, and everything outside it arrives as a manual row. Evidence review is where a named human accepting a manual row renders it met and rejecting it renders it a gap, with a timestamp — that human accept is deliberately part of the design, because ingestion establishes that an artifact exists, not that a control works. The result is a timestamped compliance artifact, not a downloadable evidence binder, not a regulator filing pack, and not an auditor's opinion.
Readiness figures are internal indicators. They are not certification, not CE marking, and not an assurance opinion, and they should never be read as a statement that a control is effective.
Primary sources (last verified 10 September 2026)
Each source is labelled by the kind of authority it carries.
AICPA Trust Services Criteria CC4.1 and CC4.2 are professional attestation criteria requiring monitoring, not automation. ISO/IEC 27001:2022 Clause 9 is a certification requirement setting monitoring, internal audit and management review on your stated cadence. NIST SP 800-137 is guidance, binding only where a control baseline is imposed. PCI DSS v4.0.1 binds entities in scope contractually and sets defined frequencies for named requirements without requiring automated collection. Not a complete list, and not legal advice.
The repository page in this cluster covers where collected evidence lives, the prove-a-control page covers what makes it sufficient, and the retention pages cover how long the series has to survive.
Frequently asked questions
Is this legal advice?
No. It is an operational explainer, with each source labelled by the kind of authority it carries. No statute cited here requires continuous evidence collection. Whether SOC 2, ISO/IEC 27001, PCI DSS, or any other framework reaches your organisation is a question for counsel and your compliance owner. This page does not determine YOUR obligations and does not file anything with an auditor.
Does any framework require continuous evidence collection?
None cited here. The AICPA criteria require monitoring of controls; ISO/IEC 27001 Clause 9 requires monitoring, internal audit and management review on your stated cadence; NIST SP 800-137 describes an approach and binds only where imposed. All of those can be satisfied by a manual review that genuinely happens and is documented. Automation is a means, not a requirement.
What share of our evidence can actually be automated?
Only the part whose state lives in a system with an interface — access configuration, change populations, findings, pipeline results, cloud settings. Policies, approvals, access-review judgements, risk acceptances, vendor report reviews, training comprehension, and physical controls remain manual everywhere. Score your own control list before investing; the honest fraction is usually well below what a demonstration implies.
Does automated collection make an auditor accept a control?
No. An auditor tests the evidence against the criteria regardless of how it was collected, and the collection method is not itself evidence of effectiveness. What automation buys is fresher, more complete evidence and more time to fix problems found while the period is still open. A green dashboard over partial coverage is a risk, not an assurance.
What breaks most often in a continuous-collection setup?
Silent collector failure — an expired credential, a revoked token, or a changed interface — leaving a gap in the series that nobody notices until fieldwork. The second is scope drift, where new accounts, projects, or repositories fall outside the connected scope. Detect collector failure explicitly, record the outage as an evidence gap, and make source scope dynamic where the platform allows.
Published by ShipReady Metrics, an evidence-based technology and compliance intelligence platform. This guide is educational and vendor-neutral.