feat: AI security benchmark adapters (JBB, DoNotAnswer, OPI, AICGSecEval, SecCodeBench, AgentDojo)

[AnandShivansh]

Summary

Adds AI security benchmark adapters for issue #24, built on a shared SecurityBenchmark base with safety-dimension scoring (1.0 = safe/refusal, never averaged into correctness).

Phase 1 — Prompt/refusal benchmarks

• JailbreakBench — HF JailbreakBench/JBB-Behaviors
• DoNotAnswer — HF LibrAI/do-not-answer
• OpenPromptInjection — indirect prompt injection scenarios
• JailBreakV28K — HF SaFoLab/JailBreakV-28K with JBB fallback

Phase 2 — Secure code generation

• AICGSecEval — CWE tasks with HumanEval-style sandbox checks
• SecCodeBench — thin adapter with optional [benchmarks-seccode] extra

Phase 3 — Agent security

• AgentDojo — utility vs attack-success metrics; supports run(model=...) or run(target=BaseTarget)

Also adds _security_utils.py, fetch_github_json() in dataset_cache.py, bundled smoke fixtures under benchmarks/data/, and unit tests for all adapters.

Running full datasets

See README AI Security Benchmarks → Running full datasets for details. Quick reference:

Type	Benchmarks	Full run
HF-backed	JailbreakBench, DoNotAnswer, JailBreakV28K	Omit limit — full dataset fetched from HuggingFace and cached under ~/.cache/harness_evals/benchmarks/
Fixture-backed	OpenPromptInjection, AICGSecEval	Bundled JSON (2–3 items) loads first today; rename bundled file to fall through to GitHub fetch, or prefetch with fetch_github_json()
Path override	SecCodeBench	SecCodeBench(manifest_path="sec-code-bench/datasets/benchmark/python/python.json")
Path override	AgentDojo	AgentDojo(tasks_path="/path/to/full_tasks.json")

Use offline=True after the first download. Check size with len(await benchmark.load_dataset()) before running.

Follow-up: remote-first loading so bundled fixtures are test/offline-only (no file rename workaround).

Test plan

• pytest tests/benchmarks/ -v -m unit — 160 tests pass
• ruff check src/ tests/ — clean
• Manual smoke: JailbreakBench().run(model=...) with HF cache populated
• Manual smoke: AgentDojo().run(target=...) with bundled tasks

Notes

• Bundled JSON under benchmarks/data/ is intentionally minimal (2–3 items) for offline CI; HF-backed benchmarks load full datasets.
• Version bumped 0.13.0 → 0.15.0 across three commits.

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}