Local benchmark proof
Firefish Benchmarks
Proof, not vibes.
See how Purple Firefish performs against safe synthetic AI security cases for prompt injection, poisoned RAG, unsafe tool calls, secrets, output leakage, and benign false positives.
Threat path
Poisoned document
Retrieved context
Agent tool call
Firefish decision
Audit evidence
Evidence preview: [REDACTED_SYNTHETIC_CASE] + stable hashes only.
Top metrics
Current local artifact
Benchmark artifact not generated yet.
Run the local benchmark harness to populate this page.
python benchmarks/firefish_benchmark.py --output data/benchmark_results/latest.json --markdown data/benchmark_results/latest.md
Attack-path coverage
What the synthetic suite exercises
Direct prompt injection
Awaiting local benchmark artifact.
Indirect RAG/document injection
Awaiting local benchmark artifact.
Agent tool misuse
Awaiting local benchmark artifact.
Sensitive-data leakage
Awaiting local benchmark artifact.
Output redaction
Awaiting local benchmark artifact.
Benign hard negatives
Awaiting local benchmark artifact.
Methodology
ExecutionLocal and synthetic fixture cases only.
TargetsNo public targets and no live third-party systems.
SecretsNo raw secrets in public artifacts; evidence uses hashes and redacted previews.
Source dataMetrics are generated from repo fixtures by the local benchmark harness.
ValidationRun the suite against your own apps before relying on the result.
Latest run
Benchmark artifact not generated yet. Run the local benchmark harness to populate this page.
- Expected JSON
data/benchmark_results\latest.json- Expected Markdown
data/benchmark_results\latest.md
What this does not prove
- Does not guarantee immunity to prompt injection.
- Does not replace human review.
- Does not certify compliance.
- Does not prove all future attacks are blocked.
CI proof
The same benchmark style can run in CI/CD as a regression gate so teams can catch changes in AI security behavior before release.