AI Pentest Reports

Purple Firefish pentest reports turn authorized campaign results into local, redacted evidence packages that operators can review or share internally.

The current report generator lives in app/pentest/reporting.py and produces:

• JSON report for APIs, automation, and internal review.
• Markdown report for human-readable assessment notes.
• HTML report for local preview.
• SARIF-like JSON summary for CI-friendly finding ingestion.

Safety Model

Reports are safe by default:

• Raw evidence is not exported unless include_raw_evidence=True is requested

while PENTEST_ALLOW_RAW_EVIDENCE_EXPORT=true and SECURE_DEBUG_MODE=true.

• Raw evidence previews are still redacted when explicitly requested.
• Evidence rows include hashes so reviewers can correlate proof without exposing raw content.
• Secrets, API keys, bearer tokens, passwords, and authorization values are redacted.
• Target metadata is filtered before it appears in JSON, Markdown, HTML, or SARIF-like output.

Raw evidence export is intended only for authorized review workflows. Normal UI/API surfaces should use redacted previews and hashes.

Example Usage

from app.pentest.reporting import generate_pentest_report

report = generate_pentest_report(
    engagement=engagement,
    session=session,
    targets=targets,
    campaigns=campaigns,
    results=module_results,
    module_specs=module_specs,
    output_dir="data/pentest_reports",
)

print(report.json_path)
print(report.markdown_path)

JSON Snippet

{
  "schema_version": "firefish.pentest_report.v1",
  "executive_summary": {
    "finding_count": 2,
    "max_severity": "high",
    "top_firefish_protection_outcome": "missed"
  },
  "privacy": {
    "raw_evidence_included": false,
    "secrets_redacted": true
  }
}

Markdown Snippet

## Firefish Protection Effectiveness

| Outcome | Count |
| --- | --- |
| prevented | 3 |
| detected_only | 1 |
| missed | 0 |

Report Sections

Generated reports include:

• Executive summary.
• Scope and authorization.
• Target inventory.
• Campaigns run.
• Findings by severity.
• OWASP LLM Top 10 coverage.
• MITRE ATLAS coverage.
• Coverage matrix with tested, vulnerable, protected, inconclusive, and not-tested counts.
• Firefish protection effectiveness.
• Evidence timeline.
• Reproduction steps.
• Remediation plan.
• Appendix with module versions and payload IDs.

Coverage Matrix

The coverage matrix summarizes what the campaign actually exercised by OWASP and MITRE mapping. It helps reviewers distinguish tested coverage from not-tested gaps without exposing raw prompts, secrets, or unsafe payloads.

The matrix is an evidence aid. It should not be read as a guarantee that an AI system is secure.

API Export Status

Pentest report export is available behind FIREFISH_PENTEST_ENABLED. Keep exports local, redacted, and scoped to authorized engagements.