Skip to content

feat(red-team): govern generated attacks behind curate, review, and a new grant - #55

Open
worldofhacks wants to merge 1 commit into
mainfrom
redteam/generative-two-stage-loop
Open

feat(red-team): govern generated attacks behind curate, review, and a new grant#55
worldofhacks wants to merge 1 commit into
mainfrom
redteam/generative-two-stage-loop

Conversation

@worldofhacks

Copy link
Copy Markdown
Owner

Tier 1 of generative red-teaming as a governed two-stage loop — stages 1–3.

Base, please read first

Branched from b0e641c (codex/integration-b34-reconcile-802), per the task's "reusing 0022's
governed path (PR #50, landing on b0e641c)". That branch is not published, so this PR shows
7 commits — 6 are your unpublished integration base and only 0678a7c is mine. Retarget
once b0e641c is pushed, or review 0678a7c alone. Nothing outside my 7 new files is touched.

What this closes

The 2026-07-25 coverage review recorded that hosted generation was implemented but structurally
undispatchable
: the coordinator requires every proposal to equal the reviewed corpus
byte-for-byte, so a generated attempt falls outside the grant that authorized the run.

That check is correct and is not loosened here. An authorization is content-addressed over the
exact corpus hash, so a generated case is by construction outside every prior grant. The missing
piece was a governed way to mint a new corpus identity and take a new grant.

The invariant

Every generated attack passes curate → human review → new corpus hash → new authorization
before it touches the target.

Stage Module State
1 curate agents/red_team/curation.py implemented, 16 tests
2 human review gate agents/red_team/review_gate.py implemented, 16 tests
3 new hash + fresh authorization campaign/generated_profile.py implemented, 16 tests
4 dispatch not wired — stacks on 0022
5 evaluate (oracle/canary Judge) existing unchanged, not reached

The generator (generate_traced, qwen, target_scope:none) is unmodified and still never
touches a target.

Two things worth your attention

The generator authors input_sequence and nothing else. oracle_expectation is forced to
kind: "none", and severity / exploitability / ground_truth_refs are neutralized rather
than inherited from the governed template. This is not cosmetic: a generated case inheriting the
template's synthetic_canary_match oracle would assert a deterministic canary hit its own turns may
never trip, and would still pass schema validation. ground_truth_refs is deliberately
non-resolving, so writing a generated case into the authored corpus fails the corpus validator's
bidirectional case↔label check until a human authors its label.

Minimization is structural only, and skipped for denial_of_service — collapsing repeats in a
token-exhaustion attack would destroy it while reporting success. Semantic (delta-debugging)
minimization needs target feedback, so it belongs to Tier 2.

Honest limits

  • Stage 4 is not wired. prepare_generated_dispatch() runs every gate and stops, rather than
    no-op'ing — a silent no-op would look exactly like a successful dispatch.
  • Tier 2 (multi-round feedback loop) is NEXT, not done. Verdicts do not yet steer the next round.
  • No live run, no verdict, no confirmed exploit is claimed from any generated case.

Coordination

  • No migration. Stages 1–3 are pure and content-addressed, following the existing
    tool_profile.py / corpus.py precedent — no revision number to reconcile with A.
  • No re-export from agents/red_team/__init__.pycampaign already depends on
    agents.red_team; re-exporting would invert the layering.
  • Review records use the exact provenance shape load_live_100_corpus already validates
    (source_kind: hosted_red_team), rather than a second notion of "approved".

Gates

ruff check .            All checks passed          (ruff 0.16.0 — CI's resolved version, not local 0.15.x)
ruff format --check .   714 files already formatted
pytest tests/           1901 passed, 3 skipped
new suites              48 passed
scripts/secret_scan.sh  clean (926 files)

Pushed identically to GitHub and GitLab at 0678a7c.

🤖 Generated with Claude Code

… new grant

Hosted generation was implemented but structurally undispatchable: the coordinator
requires every proposal to equal the reviewed corpus byte-for-byte, so a generated
attempt falls outside the grant that authorized the run. That check is correct. The
missing piece was a governed way to mint a NEW corpus identity and take a NEW grant.

Tier 1, stages 1-3 of the two-stage loop. The generator itself is unmodified and
still never touches a target.

* curate (agents/red_team/curation.py) - total, deterministic, network-free:
  normalize -> materialize + validate against attack-case v1 -> structurally
  minimize -> dedupe within batch and against the base corpus -> novelty-score ->
  content-address. Every candidate becomes a curated case or a typed rejection, so
  nothing is silently dropped and the reviewer sees what was refused.

  The untrusted generator authors input_sequence and nothing else. oracle_expectation
  is forced to kind "none", and severity/exploitability/ground_truth_refs are
  neutralized rather than inherited: a generated case carrying the template's
  synthetic_canary_match oracle would assert a deterministic canary hit its own turns
  may never trip, and would still pass schema validation. ground_truth_refs is
  deliberately non-resolving so a generated case cannot enter the authored corpus
  until a human authors its label.

  Minimization is structural only; it is skipped for denial_of_service, where
  collapsing repeats would destroy the attack while reporting success. Semantic
  minimization needs target feedback and belongs to Tier 2.

* review gate (agents/red_team/review_gate.py) - the human authorization decision,
  bound to exact bytes. Refuses a bundle mutated after presentation, self-approval by
  the generating principal, approval by omission, and a decision transplanted onto
  different content. Emits records in the provenance shape load_live_100_corpus
  already validates (source_kind hosted_red_team) rather than a second notion of
  "approved".

* fresh authorization (campaign/generated_profile.py) - mirrors tool_profile.py.
  Re-derives each approved case hash rather than trusting the field, produces a
  corpus identity provably distinct from the base, and refuses a scope carrying the
  base corpus hash, a foreign corpus id, or a spent run nonce.

Stage 4 (dispatch) stacks on the 0022 governed four-role path and is NOT wired;
prepare_generated_dispatch runs every gate and stops there rather than no-op'ing,
because a silent no-op would look like a successful dispatch. Tier 2 (multi-round
feedback) is next, not done. No live run, verdict, or confirmed exploit is claimed.

No migration: stages 1-3 are pure and content-addressed, following the existing
tool_profile.py precedent, so there is no revision number to reconcile.

48 new tests; full suite 1901 passed / 3 skipped; ruff 0.16.0 check + format clean.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@worldofhacks
worldofhacks changed the base branch from codex/integration-docs-after-pr48 to main July 26, 2026 06:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants