AI Pentest Quickstart

Purple Firefish AI pentesting is available behind the FIREFISH_PENTEST_ENABLED feature flag. In the local Docker review stack, the flag is enabled so you can open Security Workbench and try the workflow now.

1. Open Security Workbench

Go to:

http://localhost:8000/app/

Select Security Workbench.

Security Workbench shows sample engagements, local targets, modules, campaign runs, findings, coverage, and a report preview using safe synthetic data.

2. Use The Built-In Local Lab

Start with the built-in fake targets:

• protected_firefish_gateway
• vulnerable_chatbot
• protected_rag_corpus
• vulnerable_rag_corpus
• protected_tool_agent
• vulnerable_tool_agent

These are deterministic fixtures. They do not call the network, do not send email, do not run real tools, and do not contain real secrets. Canaries are clearly synthetic, such as FIREFISH_CANARY_SECRET_001.

3. Choose A Target

For a first review, use one of these:

• Local Firefish Gateway for gateway and policy checks.
• Synthetic RAG Corpus for source-aware RAG checks.
• Mock Tool Broker for tool-governance checks.

Each target declares the run modes and capabilities it supports.

4. Pick Modules

Good first modules:

• recon.echo_target
• benchmark.smoke_test
• recon.capability_probe
• rag.citation_integrity_check
• agent.unapproved_external_send
• prompt_injection.direct_instruction_override

The module library also includes RAG, jailbreak smoke, data exposure, output handling, resource-consumption, and agent/tool-governance checks.

5. Build A Dry-Run Campaign

Use the campaign builder to choose:

• Target.
• Module categories or individual modules.
• Run mode.
• Max prompts.
• Max turns.
• Max tokens.
• Max duration.
• Authorization scope metadata.

Dry-run mode is the safest way to inspect the plan before any active test behavior.

6. Review The Plan

The plan preview shows:

• Modules to run.
• Payload counts.
• Expected target capabilities.
• Safety warnings.
• Required approvals.
• OWASP and MITRE coverage.

Campaigns cannot start without authorization metadata.

7. Review Findings And Evidence

After a run or sample run, Firefish shows:

• Findings by severity.
• Firefish protection outcome.
• OWASP LLM Top 10 coverage.
• MITRE ATLAS coverage.
• Evidence timeline.
• Redacted report preview.

Evidence is safe by default: raw unsafe payloads and secrets are not displayed in normal UI or reports.

Local Docker Review Mode

To relaunch Docker with Security Workbench visible:

$env:FIREFISH_PENTEST_ENABLED='true'
docker compose -f docker/docker-compose.yml up -d --build

To hide it again:

$env:FIREFISH_PENTEST_ENABLED='false'
docker compose -f docker/docker-compose.yml up -d --build

Safety Defaults

• Feature flag off unless enabled.
• Local-only mode on by default.
• Dry-run first.
• Synthetic canaries only.
• Request budgets and rate limits.
• No destructive actions by default.
• No raw evidence export by default.
• No hosted model provider required.