Operational guidance, not legal advice. This page distills named public sources (regulator guidance and industry practice). It is not a legal determination, not a notification decision, and not a substitute for your counsel, insurer, or a retained DFIR firm. Verify applicability and current deadlines for your facts and jurisdiction.
What penetration testing evidence do auditors accept?
Updated
An audit-acceptable test has a documented scope covering the in-scope environment, a stated methodology, an independent tester, dated findings with severity, remediation records, and retest evidence. A vulnerability scan is not a penetration test and does not substitute for one.
Penetration-testing evidence map, last verified 10 September 2026 against PCI DSS v4.0.1 Requirement 11.4 and the PCI Security Standards Council's penetration-testing guidance, NIST SP 800-115 (Technical Guide to Information Security Testing and Assessment), the Penetration Testing Execution Standard, the OWASP Web Security Testing Guide, and the AICPA Trust Services Criteria. PCI DSS is the only source here that sets a testing frequency, and only for entities in scope. This page does not rank or recommend providers, does not determine YOUR obligations, does not start a clock, and does not file with an auditor. Not legal advice.
What this page is, and what it is not
Audience: a security lead, engineering leader, or compliance owner who needs penetration testing to satisfy an auditor and an enterprise customer, and who has been burned by a report that turned out to be a scan with a cover page.
This page is not legal advice. It does not determine YOUR obligations, does not decide which frameworks apply to your organisation, does not start a clock, and does not file anything with an auditor. Last verified 10 September 2026.
It also does not rank, recommend, or name testing providers. The provider landscape changes constantly, accreditations differ by scheme and by jurisdiction, and a ranking without stated criteria is marketing. What this page gives instead is the set of criteria to apply yourself, and the instruction to verify what a given provider is actually accredited to perform before you rely on it.
- Independence is the property auditors care about most and teams think about least. A test performed by the team that built the system evidences diligence, not independent assurance.
- Scope is what makes a report reusable. A report whose scope does not cover the environment in your audit boundary answers a question nobody asked.
- The report is the beginning of the evidence, not the end. Remediation records and retest evidence are what show the finding was closed.
- An attestation letter is for sharing; the full report is for your auditor. Prepare both, and know which one goes where.
What makes a penetration test audit-acceptable
This is the checklist to apply to a report before you rely on it, and to a statement of work before you sign it. Read the second column as the property, the third as the artifact that shows it, and the fourth as what makes an auditor reject it. Last verified 10 September 2026. Not legal advice.
| Property | What it means | Artifact that shows it | What makes it rejected |
|---|---|---|---|
| Independence of the tester | The test was performed by someone who did not build or operate the system, with organisational separation where the framework or contract expects it. | The engagement contract or statement of work, naming the performing organisation and its relationship to you. | A test run by the engineering team that owns the code, presented as independent assurance. |
| Documented scope | The systems, applications, network ranges, environments, and user roles tested — and, explicitly, what was excluded and why. | The scope section of the report and the statement of work, with named targets and exclusions. | A scope that omits the in-scope environment, or an unstated exclusion discovered when the auditor compares the report to your system boundary. |
| Rules of engagement | Test window, permitted techniques, escalation contacts, handling of production data, and stop conditions — agreed before testing. | A signed rules-of-engagement document, referenced by the report. | No rules of engagement at all, which raises questions about authorisation and about what was not attempted. |
| Stated methodology | The methodology followed, named rather than implied — for example the OWASP Web Security Testing Guide for web applications, or the phases described in NIST SP 800-115 and the Penetration Testing Execution Standard. | The methodology section of the report, naming the framework and the coverage per phase. | A report with no methodology, so nobody can tell what was in scope conceptually rather than just by hostname. |
| Manual testing, not only automated scanning | A human tested authorisation logic, business-process abuse, chained weaknesses, and context-specific flows that a scanner does not model. | Findings that describe reasoning and reproduction steps rather than tool output; a methodology naming manual phases. | Scanner output with a cover page. See the comparison table below. |
| Tester qualification, verified | The performing individuals or organisation hold relevant qualifications, and the organisation is accredited for the kind of testing performed where a scheme applies. | Named certifications or accreditation references in the report or contract — verified with the issuing scheme, not taken on trust. | Accreditation asserted but not verifiable, or an accreditation for a different kind of assessment than the one performed. |
| Dated findings with severity and evidence | Each finding has an identifier, a severity with a stated rating basis, affected assets, reproduction detail, and impact. | The findings section, with a stated severity method rather than adjectives. | Severity assigned with no stated basis, so it cannot be reconciled with your own risk ranking. |
| Remediation records | Each finding has an owner, a decision, and a fix or documented acceptance, traceable to the change that carried it. | Your own tracking records, joined to the report's finding identifiers. | A report with no follow-through, which evidences that you were told and stopped there. |
| Retest or verification evidence | The tester or an equivalent verification confirmed closure of the significant findings. | A retest letter or a retest section referencing the original finding identifiers and their new status. | Self-asserted closure with no verification, particularly for findings the auditor sampled. |
| Attestation letter for external sharing | A short letter stating that the test was performed, its scope, its date, and the current remediation position, suitable to share with customers. | The letter itself, issued by the testing organisation. | Sharing the full report externally, which discloses exploitable detail and is usually a mistake. |
| Frequency and trigger conditions | A stated cadence, plus the conditions that trigger an additional test — significant change, new environment, or a material architectural shift. | Your testing policy plus the test history showing the cadence was met. | A cadence in policy that the test history contradicts, or a significant change with no test after it. |
| Retention of the report and its evidence | Reports, letters, and retest evidence retained long enough to show a trend, with access limited appropriately. | The evidence repository entry, with retention and access recorded. | Last year's report lost with the departed employee's mailbox. |
Scan versus manual penetration test
These are complementary, and one is routinely sold as the other. Both belong in a programme; neither replaces the other. Where PCI DSS applies, Requirement 11.3 addresses vulnerability scanning and Requirement 11.4 addresses penetration testing as separate obligations, which is the clearest statement of the distinction. Last verified 10 September 2026. Not legal advice.
| Dimension | Automated scanning | Manual penetration test |
|---|---|---|
| What it finds | Known vulnerable versions, missing patches, common misconfigurations, and pattern-matched weaknesses across a wide surface. | Authorisation flaws, business-logic abuse, chained weaknesses, tenant-isolation failures, and context-specific issues that require understanding what the application is for. |
| Coverage and repeatability | Broad and repeatable. Runs on every build or on a schedule, and produces comparable results over time. | Deeper and less repeatable. Coverage depends on scope, time, and tester skill, which is why methodology and scope documentation matter. |
| Cost profile | Low marginal cost per run once configured, so it can be continuous. | Engagement cost per test, which is why frequency is a budget and risk decision rather than a technical one. |
| Evidence value to an auditor | Evidences ongoing vulnerability management: coverage, cadence, and remediation over the period. | Evidences independent assurance at a point in time, with an outside perspective on the control set. |
| Where PCI DSS treats it | Requirement 11.3 addresses internal and external vulnerability scanning, with its own frequency and rescan expectations for entities in scope. | Requirement 11.4 addresses penetration testing, with its own methodology, frequency, and post-significant-change expectations for entities in scope. |
| What it cannot do | Cannot reason about intent. It will not notice that a valid API call from one tenant returns another tenant's data. | Cannot provide continuous coverage. A test in March says nothing about a vulnerability introduced in July. |
Legal requirement, guidance, best practice, or our recommendation
Annual penetration testing is widely assumed to be mandatory. For most organisations it is a customer contract term or a self-set standard, not a statutory duty. Label the reason yours exists. Last verified 10 September 2026. Not legal advice.
| Statement | Which kind of authority | What it does not mean |
|---|---|---|
| PCI DSS v4.0.1 Requirement 11.4: perform penetration testing according to a defined methodology, externally and internally, at a stated frequency and after significant changes, with findings corrected and testing repeated to verify corrections. | Legal requirement in the contractual sense, for entities in scope through card-brand and acquirer agreements. | Does not set a testing duty for an organisation with no cardholder data in scope, and does not name a provider or an accreditation scheme. |
| Customer contract terms requiring an annual test, an attestation letter, or a right to test. | Legal requirement in the contractual sense — often the real reason a test happens, and frequently stricter than any framework. | Does not appear in your framework mapping unless someone puts it there. Read the contracts. |
| AICPA Trust Services Criteria CC4.1 monitoring and CC7.1 detection of vulnerabilities. | Attestation criteria. They require monitoring and vulnerability detection; they do not name penetration testing or a frequency. | Does not mean SOC 2 requires an annual penetration test. Auditors commonly expect one as a practical way to evidence the criteria, which is a different statement. |
| ISO/IEC 27001:2022 Annex A 8.8 and 8.29, and Clause 9.2 internal audit. | Certification requirement where you seek or hold certification. It requires vulnerability management and security testing, not a third-party penetration test specifically. | Does not mandate an external tester, though your own risk assessment and stated policy may. |
| NIST SP 800-115, the Penetration Testing Execution Standard, and the OWASP Web Security Testing Guide. | Regulatory and standards guidance, and community best practice. These are the methodologies a report should name. | None of them binds a commercial company by default, and none certifies a provider. |
| Test annually, and additionally after any significant architectural change, using an independent tester with a named methodology. | Industry best practice, and the pattern most enterprise buyers expect. | Not a legal duty outside the regimes and contracts above. Once written in your policy it becomes auditable, so state a cadence you will meet. |
| Keep the report, the attestation letter, the remediation records, and the retest evidence together in one evidence-repository entry, retained for at least three years. | ShipReady Metrics recommendation. | Not required by any framework named here. It exists because the report is usually findable and the retest evidence usually is not. |
Choosing a provider without ranking anyone
We do not rank testing providers and do not name any. The landscape changes, accreditations differ by scheme and jurisdiction, and a ranking without defensible stated criteria is advertising. Apply your own criteria, verify claims with the issuing body, and record the reasoning — the record of why you chose is itself useful evidence of a considered decision.
- Verify accreditation with the scheme, not with the provider's website, and check that the accreditation covers the kind of testing you are buying rather than a neighbouring activity.
- Ask for a redacted sample report before signing. The quality of the findings write-up is the single best predictor of the engagement's value, and it tells you immediately whether manual testing is real.
- Confirm who performs the work and their qualifications, not only who sells it. Sub-contracting is common and legitimate; undisclosed sub-contracting is a scope and confidentiality problem.
- Agree the retest terms in the contract, before findings exist. Retest priced later is retest that quietly does not happen.
- Confirm scope against your audit boundary explicitly, target by target, rather than assuming the provider inferred it.
- Check insurance, confidentiality terms, and data-handling commitments where testing touches production or personal data — this is where counsel should look.
- Record the criteria you applied and the decision. Where you rotate providers to get a fresh perspective, record that reasoning too.
Checklist
A question list for the testing side of your evidence set. Not a determination that any framework applies to you. Last verified 10 September 2026. Not legal advice.
- Does the report's scope cover the environment inside our audit boundary, target by target?
- Was the tester independent of the team that built and operates the system?
- Does the report name its methodology, and does it show manual phases rather than only tool output?
- Have we verified the provider's accreditation with the issuing scheme, and that it covers this kind of testing?
- Do findings carry identifiers, a stated severity basis, affected assets, and reproduction detail?
- Can we join every significant finding to a remediation record and the change that carried the fix?
- Do we hold retest or verification evidence for the significant findings, referencing the original identifiers?
- Do we have an attestation letter suitable for customers, distinct from the full report?
- Does our testing history match the cadence in our policy, including tests after significant changes?
- Are reports and retest evidence retained together, with access limited appropriately?
- Do we know whether any contract or regime — PCI DSS scope, a customer term — makes testing an obligation rather than a choice?
What to do now
Ordered so the gaps that invalidate an existing report close first. None of these steps is a legal determination, and none files anything with an auditor.
- Compare your most recent report's scope against your audit boundary today. A scope mismatch is the most expensive thing to discover late, because the remedy is another test.
- Locate the retest evidence for the last round of significant findings. If it does not exist, that is the fastest available improvement to your evidence position.
- Get an attestation letter you can share with customers, so the full report stops being the thing you send out.
- Check the contracts for testing obligations — customer terms are the most common real driver, and they often specify frequency and scope you were not tracking.
- Write the testing policy with a cadence and a trigger for significant change, then reconcile it against the actual history and fix whichever side is wrong.
- Negotiate retest into the next engagement contract rather than after findings arrive.
- Keep the report, letter, remediation records, and retest evidence in one evidence-repository entry, so next year's auditor gets one link.
- Re-verify the PCI SSC guidance, NIST SP 800-115, and the OWASP testing guide annually, and record the date. Ours says 10 September 2026.
Where this shows up in ShipReady Metrics
Only shipped behaviour is described here. This product does not perform third-party penetration tests, does not issue attestation letters, does not accredit providers, and does not file anything with an auditor.
First-party SAST and DAST exist in this product and complement, rather than replace, third-party penetration testing. They give breadth and repeatability; an independent manual test gives an outside perspective on logic and context. Presenting first-party automated results as a penetration test would be wrong, and an auditor would treat it as such.
Vulnerability management ranks findings using CISA KEV membership, EPSS, and CVSS, deduplicates across sources, and supports blast-radius search over captured dependencies including transitive npm dependencies where a lockfile was fetched. Penetration-test findings from an external provider are tracked as evidence rows rather than ingested automatically.
If you already have a session: signed-in app → Compliance holds evidence collection for control-mapped artifacts across starter control subsets, and evidence review, where a named human accepting a manual row renders it met and rejecting it renders it a gap, with a timestamp. That is a timestamped compliance artifact — not a forensic chain of custody, not a downloadable evidence binder, and not an auditor's opinion.
Primary sources (last verified 10 September 2026)
Each source is labelled by the kind of authority it carries.
PCI DSS v4.0.1 Requirements 11.3 and 11.4, and the PCI Security Standards Council's penetration-testing guidance, bind entities in scope through card-brand and acquirer contracts. NIST SP 800-115 (Technical Guide to Information Security Testing and Assessment) is guidance. The Penetration Testing Execution Standard and the OWASP Web Security Testing Guide are community best practice and methodology references. AICPA Trust Services Criteria CC4.1 and CC7.1 are professional attestation criteria that do not name penetration testing. ISO/IEC 27001:2022 Annex A 8.8 and 8.29 and Clause 9.2 are certification requirements. Not a complete list, and not legal advice. No provider is ranked or recommended anywhere on this page.
The vulnerability-management page in this cluster covers the scanning and remediation side, and the audit evidence repository page covers where reports and retest letters should live.
Frequently asked questions
Is this legal advice?
No. It is an operational guide to penetration-testing evidence, with every source labelled by the kind of authority it carries. Whether PCI DSS, a customer contract, or any statute requires testing in your case is a legal and commercial question for counsel and your contracts. This page does not determine YOUR obligations and does not file anything with an auditor.
Does SOC 2 require an annual penetration test?
The AICPA Trust Services Criteria do not name penetration testing or a frequency; they require monitoring and vulnerability detection. In practice many auditors expect a test as a straightforward way to evidence those criteria, and many customer contracts require one outright. So the honest answer is that it is usually expected rather than required — and where PCI DSS Requirement 11.4 applies, testing obligations are explicit.
Is a vulnerability scan a penetration test?
No. A scan finds known vulnerable versions and common misconfigurations across a broad surface, repeatably and cheaply. A manual test reasons about authorisation logic, business-process abuse, and chained weaknesses that a scanner cannot model. PCI DSS treats them as separate requirements — 11.3 for scanning and 11.4 for penetration testing — which is the clearest available statement that they are not interchangeable.
Which penetration testing provider should we use?
We do not rank or recommend providers, because the landscape changes and a ranking without defensible stated criteria is advertising. Apply your own criteria: verified accreditation for the kind of testing you are buying, a redacted sample report you have read, named performers with disclosed sub-contracting, retest terms agreed in the contract, and scope confirmed target by target against your audit boundary. Record the reasoning.
Can we use ShipReady Metrics DAST instead of a penetration test?
No. First-party SAST and DAST exist here and complement third-party testing — they give breadth and repeatability across every run. They do not replace an independent manual test, and presenting automated first-party results as a penetration test would fail with any auditor or enterprise reviewer who read the methodology.
Published by ShipReady Metrics, an evidence-based technology and compliance intelligence platform. This guide is educational and vendor-neutral.