AI Pentest CI/CD Campaigns

Purple Firefish can run a headless AI pentest campaign from pentest_campaign.yaml and return CI-friendly exit codes:

• 0: no findings at or above the configured threshold
• 1: one or more findings at or above the configured threshold
• 2: configuration or execution error

The runner is designed for authorized defensive testing. Keep campaigns dry-run by default, use synthetic targets/canaries, and do not point CI jobs at public or third-party systems unless the engagement scope explicitly authorizes them.

CI campaigns are best used as regression checks for AI apps already protected by Firefish. They help detect changes in prompt handling, RAG source behavior, tool governance, output handling, and protection outcomes before release.

Run Locally

python -m app.pentest.ci_runner --config pentest_campaign.yaml --output-dir data/pentest_ci

The script wrapper is equivalent:

python scripts/run_pentest_campaign.py --config pentest_campaign.yaml --output-dir data/pentest_ci

A starter config is available at config/pentest_campaign.yaml.example.

Config Format

name: Firefish CI AI Pentest
engagement_id: ci-engagement-local
session_id: ci-session-local
campaign_id: ci-campaign-local

target:
  target_id: ci-target
  name: Local Firefish-protected AI app
  target_type: chat_gateway
  base_url: http://localhost:8000
  capabilities:
    - target_metadata
    - safe_benchmark
  metadata:
    allowed_run_modes:
      - dry_run

modules:
  - recon.echo_target
  - benchmark.smoke_test

run_mode: dry_run

authorization:
  approved_by: security-owner
  approval_reference: CI-AUTH-001
  owner: ci
  allowed_hosts:
    - localhost
    - 127.0.0.1

thresholds:
  fail_at_severity: high
  max_findings_at_or_above: 0

report_formats:
  - json
  - sarif
  - junit

limits:
  max_prompts: 10
  max_tokens: 2048
  max_seconds: 60
  max_rps: 2

Reports

Requested reports are written to the configured output directory:

• json: full redacted Firefish pentest report
• sarif: SARIF-like JSON for CI/security tooling
• junit: JUnit XML suitable for build summaries
• markdown / md: human-readable report
• html: local HTML report

Raw evidence is not exported by the CI runner. Reports use redacted evidence and hashes.

GitHub Actions Example

This workflow is an example only. Add it to .github/workflows/ai-pentest.yml when your team is ready to gate PRs.

name: ai-pentest

on:
  pull_request:
    paths:
      - "app/**"
      - "firefish/**"
      - "pentest_campaign.yaml"
  workflow_dispatch:

jobs:
  firefish-ai-pentest:
    runs-on: ubuntu-latest
    env:
      APP_ENV: dev
      LOCAL_ONLY_MODE: "true"
      FIREFISH_PENTEST_ENABLED: "true"
      DATABASE_URL: sqlite:///./data/ci-pentest.db
      OPENAI_API_KEY: ""
      LOG_RAW_INPUT: "false"
    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
          cache: "pip"

      - name: Install Firefish
        run: |
          python -m pip install --upgrade pip
          python -m pip install -e ".[server]"

      - name: Run AI pentest campaign
        run: |
          python -m app.pentest.ci_runner \
            --config pentest_campaign.yaml \
            --database-url sqlite:///./data/ci-pentest.db \
            --output-dir data/pentest_ci

      - name: Upload AI pentest reports
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: firefish-ai-pentest-reports
          path: data/pentest_ci/

PR Gate Pattern

Start with passive and dry-run modules:

• recon.echo_target
• benchmark.smoke_test
• passive RAG/tool-governance modules against fake or staging targets

Use thresholds.fail_at_severity to decide when the PR should fail. A common first gate is:

thresholds:
  fail_at_severity: high
  max_findings_at_or_above: 0

That lets informational, low, and medium findings show up in reports while failing the build only for high and critical regressions.

Safety Notes

• CI campaigns should remain dry_run unless a security owner approves safe_active against a staging target.
• Use synthetic canaries only.
• Never include real secrets in YAML.
• Keep LOCAL_ONLY_MODE=true.
• Do not enable hosted LLM providers for CI unless your organization has explicitly approved that data flow.