CI Red-Team Regression

Use this when you want pull requests or releases to prove that known AI security controls still behave as expected.

The Firefish CI red-team runner executes safe synthetic cases against a local Firefish instance or a controlled test target. It compares current behavior with a baseline and can fail the build when configured regression thresholds are exceeded.

What it checks

• Prompt injection and jailbreak regression cases.
• RAG/document injection cases.
• Tool-call governance cases.
• DLP and output redaction cases.
• Benign hard-negative false positives.
• Stable reason codes and risk levels.

Local-first defaults

• Test payloads are synthetic and de-weaponized.
• Reports use stable case IDs, redacted previews, hashes, and summary metadata.
• Public internet targets are not used unless explicitly configured and allowed.
• No hosted model provider is required.

Initialize a config

firefish-redteam init --output firefish.redteam.yaml

Run a local regression suite

firefish-redteam run \
  --config firefish.redteam.yaml \
  --output-dir .firefish-redteam

Compare against a baseline

firefish-redteam compare \
  --baseline .firefish-redteam/baseline.json \
  --current .firefish-redteam/results.json

GitHub-friendly output

firefish-redteam annotate \
  --results .firefish-redteam/results.json \
  --summary .firefish-redteam/summary.md \
  --sarif .firefish-redteam/firefish.sarif

Interpreting failures

A regression means a previously blocked, quarantined, redacted, or approval-gated synthetic risk now looks allowed, or benign hard-negative false positives increased beyond your configured threshold. Improvements are reported but should not fail the build.

Firefish CI checks do not prove your application is secure. They are regression gates for known classes of AI security behavior.