Quickstart Benchmarking Agent Firewall RAG Firewall DLP Security Ops CI Red-Team Threat Model Deployment Limits

Benchmarking

Use this when you want evidence-backed local proof that Firefish is exercising important AI security controls.

Firefish benchmarks compare protected behavior with a simple allow-all baseline using safe synthetic cases from the repository. The results are useful for local proof and regression tracking, not for claiming complete protection.

What the benchmark covers

Direct prompt injection.
Indirect RAG/document injection.
Unsafe tool calls.
Sensitive-data leakage.
Output redaction.
Benign hard-negative false positives.
CI red-team regression behavior.
Latency overhead.

Regenerate local results

python benchmarks/firefish_benchmark.py \
  --output data/benchmark_results/latest.json \
  --markdown data/benchmark_results/latest.md

Then open:

http://localhost:8000/benchmark

The public benchmark page renders real artifact data when available. If the artifact is missing, it shows a placeholder instead of fake numbers.

Metrics

malicious_block_or_mitigate_rate: malicious synthetic cases that Firefish blocked, quarantined, redacted, or approval-gated.
benign_allow_rate: benign synthetic cases allowed without unnecessary blocking.
prompt_injection_mitigation_rate: prompt-injection cases mitigated by scanner, RAG firewall, or policy.
rag_injection_quarantine_rate: poisoned document or chunk cases quarantined.
unsafe_tool_action_block_or_approval_rate: unsafe tool actions blocked or approval-gated.
dlp_secret_detection_rate: synthetic secret cases detected by DLP.
output_redaction_rate: output leakage cases redacted.
false_positive_rate_on_benign_cases: benign cases incorrectly blocked or over-restricted.
p50_latency_ms and p95_latency_ms: measured local harness latency.

False positives and false negatives

Review false positives first when Firefish blocks benign developer, support, or security-education prompts. Review false negatives by stable case ID and hash, not by raw payload text. When you add a new case, keep the example synthetic and de-weaponized.

What benchmarks do not prove

Benchmarks do not guarantee immunity to prompt injection, data leakage, or unsafe agency. They do not certify compliance. They do not replace human review. Run the suite against your own applications and add local cases that match your threat model.