governed 0022: real four-role composition over the reviewed corpus (v1 blocker) - #50
Conversation
…osition Blueprint for run_governed_acceptance: reuse HostedFourRoleRuntime.run_attempt with a reviewed corpus seed + a governed policy_gateway_dispatch (corpus-hash invariant -> real PolicyGateway -> ExecutionRecorder -> real evidence) + real oracle Judge + terminalization-safe lifecycle; migration 0022 (governed target-bound four-role authority, down_revision 0021, two-person auth); green e2e test with a controlled target (proves wiring, NOT the evidence); the real four-role evidence is a separate post-deploy authorized live-target campaign. Confirmed: 0018 canonical, b176 is an ancestor (no deconflict), terminalization = fix/provider-evidence-terminalization. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Add run_kind 'governed_acceptance' (down_revision 0021): a real four-role attack over the reviewed corpus dispatched through the Policy Gateway to the target. Combines campaign-style two-person authorization (a live dispatch is never system-provenanced) with the four-role hosted config + exactly one bounded target dispatch (v3 governed limits: network_scope=policy_gateway_target, target_call_limit=1). Campaign + the target-free v1/v2 acceptance envelopes stay unchanged. Single head on 0018->0019->0020->0021->0022; applies + downgrades (refuses while governed rows exist) + round-trips. 2 governed migration tests + updated single-head assertion; full acceptance migration suite green (35). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…alid + downgrade coverage Pre-merge tightening (m-required): the governed authority shape now requires acceptance_context_ sha256 AND acceptance_attempt_id NOT NULL — a governed_acceptance row can't carry a null identity. Add DB coverage: a well-formed governed row inserts (two-person auth + config + reviewed-corpus attempt via the deferred FK); a governed row with null context or null attempt is rejected by the authority-shape check; downgrade refuses while a populated governed row exists. 6 governed migration tests + full acceptance suite green (39). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Fold the authority-shape gap the composition surfaced: the 0020/0021 agent_acceptance guards are hardwired to run_kind='agent_acceptance' and enforce an ABSOLUTE zero-target-traffic invariant, so a governed_acceptance run (four hosted roles + one bounded target dispatch) was unguarded for four-role lineage and Judge binding at the DB level. Per the reserved-migration owner's call, ISOLATE rather than generalize: add a dedicated m1d_validate_governed_acceptance_execution + provider-invocation guard, leaving the agent_acceptance guards and their absolute zero-target-traffic guarantee untouched. Each guard body is gated on its own run kind, so on a governed row the acceptance guard is a no-op and vice versa. The governed Judge is the real calibrated, human-enabled independent Judge (calibration-bound + explicit decision authority on success), not the failed-advisory acceptance Judge; the single bounded dispatch is anchored by the existing attempt_result UNIQUE(campaign_run_id, attempt_id). 12 governed migration tests green; agent_acceptance + core migration suites unchanged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Add the governed run authority the composition needs, isolated from the target-free acceptance store methods (which forbid target traffic): - create_governed_acceptance_run: human-launched under a live two-person authorization; binds acceptance_context_sha256 to the reviewed case's content hash (the seed-replay corpus-hash anchor); actor/provenance stay NULL. - start_governed_agent_execution: four-role advisory execution that PERMITS the one bounded dispatch (no zero-target-traffic assertion); the Judge is the real calibrated, human-enabled independent Judge (starts 'enabled', not failed). - complete_governed_acceptance_run: requires four measured successful calls, an adjudicated calibration-bound Judge, AND exactly one recorded target dispatch. - abort_governed_acceptance_run: kill switch with no new authority. Two-person control's first line stays upstream (a self-approved decision can never be recorded); the store's launcher!=approver check is defense in depth. 9 governed store tests green; agent_acceptance store suite unchanged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…nce) Build the v1-blocker composition on the tested schema + store foundation, reusing HostedFourRoleRuntime.run_attempt so a real reviewed-corpus attack flows end to end: Orchestrator selects the reviewed case -> Red Team replays THAT exact reviewed case (seed-replay) -> Policy Gateway dispatches it to the bound target -> the Execution Recorder persists the REAL response -> the independent calibrated Judge evaluates that real response (deterministic oracle keeps precedence) -> Documentation drafts (blocked pending human approval). - Seed-replay authorization invariant: the DISPATCHED CONTENT (the P10 projection) must equal the reviewed case's seed-replay bytes; classification metadata may ride on the Red Team object but only the reviewed attempt reaches the target. Any drift aborts BEFORE the target is touched. - The gateway is the sole cap-enforcing target exit; the Red Team never holds a credential. A native terminalization-safe governed lifecycle records every logical + physical fact so no execution dangles (a Judge-finish failure still terminalizes). - Deterministic precedence: the calibrated model Judge cannot launder a confirmed exploit safe; the oracle over the REAL transcript decides. Controlled-target e2e on real Postgres (test proves WIRING, never the evidence): asserts the Judge saw the REAL controlled response (canary present, not a synthetic canary), four governed executions in the four-role chain all succeeded, verdict EXPLOIT_CONFIRMED by oracle precedence, run completes; and a drifted Red Team proposal aborts before the target with zero recorded dispatch. 33 governed tests + the base runtime suite green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The terminalization reconcile (PR #47) deleted the fix-3 test, leaving uncovered the path where a dispatched call 500s unbilled: a 500 is non-retryable, so exactly one physical attempt is made and the transport records it not_observed. The terminal record must keep physical_attempts == 1 (a dispatched call is NEVER recorded as never-made) while cost stays honestly not_observed — distinct from the no-dispatch not_observed, where physical_attempts is 0. Drives the real transport over httpx.MockTransport + the real lifecycle + store + Postgres, like the rest of the suite. 7 preservation tests green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The oracle-precedence CONFIRM was already guarded (a deterministic oracle/canary wins). The REVERSE was only unreachable-by-construction, not enforced: a model Judge emitting state=EXPLOIT_CONFIRMED with a non-confirming oracle fell straight through _deterministic_precedence as a model-sourced confirmed exploit. Isolate and reject it: reaching that branch means the deterministic path did NOT confirm, so an EXPLOIT_CONFIRMED there originates from the calibrated MODEL Judge. Only the deterministic oracle/canary/human may CONFIRM (the verdict contract already restricts a CONFIRMED confirmation_source to exactly those), so strip the unauthorized confirmation to EXPLOIT_LIKELY — the strongest verdict a model may assert — rather than emit a model-originated confirmation. Tested at both layers: a base-runtime unit test (model CONFIRMED + silent oracle -> EXPLOIT_LIKELY, reason recorded, still documented as a draft) and a governed e2e on real Postgres (oracle genuinely silent — no canary in the real response — model over-claims CONFIRMED -> LIKELY, run completes, Judge adjudicated by the calibrated model). 97 tests green across the blast radius. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Integration review: NO-SHIP at The reserved migration is clean: The runtime composition bar is not yet met:
After PR #34 lands, rebase/replay this stack onto the new integration tip and publish the exact Please supersede or update #50 only after these points are closed; do not merge the current head. |
|
Integration addendum for the final Please add a Direct-DB migration tests must bypass the application store and prove: self-approval rejected, distinct approver accepted, unlinked/missing lineage rejected, trigger absent at 0021 and restored at 0022, with The application-layer companion is isolated at local commit |
|
Correction — this supersedes the aggregate design in my prior addendum. Adversarial review reproduced a real multi-evidence failure and found a race; do not implement the earlier The existing API contract treats any finding-evidence link count other than exactly one as ambiguous/unavailable. Keep parity:
Required tests now also include same-launcher multi-link denial, cross-organization isolation, campaign and governed-acceptance success, rejected/resolved unaffected, populated invalid-upgrade refusal, link-insert-after-approval refusal, and full 0021↔0022 round trip. The final revision must still be the sole head The app companion |
Repair-bar item 3: the governed store must accept the confirmed production authority (Judge max_calls=56, roles 9/19/19/9, retries=1), which the closed global-4/per-role-1 envelope rejected. Per the reviewed decision, DERIVE the per-role/global call+spend budget from the staged, content-hashed config (the 4-call harness config OR the 56-call production config alike), with three guardrails held explicit: - One-dispatch invariant is STRUCTURAL, never derived: target_call_limit=1 + network_scope=policy_gateway_target are pinned by construction in both 0022's relaxed _governed_limits_sql() and the store's canonical_governed_acceptance_limits(). A relaxed budget cannot relax the dispatch ceiling. - Derivation binds ONLY to the staged reviewed config: the store matches the run's limits against canonical_governed_acceptance_limits(loaded_config), so derived caps are always the authorized ones (no unreviewed dispatch). - Retries live at the agent-reasoning level only (config retries=1 accepted); target_call_limit stays 1 regardless. 0022's governed constraint becomes structural (valid four-role shape + absolute one-dispatch + budget positivity, global calls <= the 56 platform ceiling); row-level store check is structural, role-level is exact config-match. Folded into 0022 to keep it the single head. Tests: production 56-call config derives + creates a governed run with target_call_limit=1 pinned; a second dispatch, over-ceiling budget, or target-free scope are rejected. Backward-compatible (the 4-call config derives the prior closed envelope). Acceptance suites unchanged. RE-SERIALIZATION FLAG FOR m: 0022's body grew again (relaxed _governed_limits_sql + the store derivation) — single head 0022 on 0021 unchanged, but please re-run the single-head/serialization check. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Release-route carry-forward for the 0022 composition: OpenRouter's live catalog currently exposes qwen/qwen3.5-397b-a17b through provider slug alibaba and does not list together for this model. Any production configuration or seam assertion in this stack must accept the exact staged alibaba identity; do not hard-code or attest together. The shared OpenRouter secretref remains fail-closed and will be proven by Runner preflight after deployment. Calibration will re-attest the exact staged production identity before campaign enablement. |
|
Re-audit at 127099a: NO-SHIP. PR #52 is now merged; the new published integration base is 0ce980c and mirrored exactly to GitLab. Rebase/replay onto that SHA. Verified passing: sole Alembic head 0022 -> 0021; focused PR50 suite 80/80; config-derived global/Judge 56 behavior works in a probe. Still blocking:
Mechanical replay of PR50 and merged PR52 was clean in either order and combined focused tests passed. Publish the repaired replay on codex/** so exact-head GitHub CI runs; preserve redteam/** in ci.yml. |
…54 cal independently found the AD-04 hole in HostedFourRoleRuntime._deterministic_ precedence was REAL (a model EXPLOIT_CONFIRMED with no oracle hit reached a Documentation draft), plus a second hole: the NO_EXPLOIT_OBSERVED->INDETERMINATE guard trusts a model-self-emitted 'calibrated' boolean. cal fixed BOTH + pinned canonical tests on PR #54. My earlier 4ad125b re-enforced only the first hole (leaving the calibrated-boolean hole open) on the SAME function — a partial, divergent duplicate. Per coordination with m, there must be ONE canonical guard (cal's). Revert my _deterministic_ precedence edit + its two tests so PR #50 does not re-enforce AD-04 separately; the composition will sit on cal's fixed class once m rebases PR #50 onto the tip that carries PR #54. Reverts byte-identical to the pre-4ad125b function. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Coordination — AD-04 canonical guard + PR #50 status (for m) cal's finding is confirmed and correct. My earlier
Sequencing request: integrate PR #54 first, then rebase PR #50 onto that tip. I'm holding the 1/2/4/5 composition rebuild until the base carries PR #54, so it sits on cal's fixed class rather than the unfixed base. One coordination question before I resume 1/2/4/5 — where should the composition's canonical Judge-decision guard live? The repair-bar item 2 (invoke
Please confirm which path is canonical (or that |
…el-authority holes Wires the re-attest harness so a calibration can only ever attest the Judge the deployment is actually running, and repairs two places where the model's own output was granting it authority. The re-attest itself has NOT been run — it is blocked on inputs I do not own (below). No number in this commit is a new measurement. WHY THE 8ce852b NUMBER IS SUPERSEDED, PRECISELY Its provider calls were real: 54 unique OpenRouter request ids, 54 distinct token triples, costs read from the provider's own usage.cost field that match Gemini 2.5 Pro list price (1.25/10 per M) rather than this repo's 5/30 ceilings. The arithmetic reproduces byte-for-byte. What disqualifies it is what it was bound to: capture_judge_calibration.py built its OWN HostedConfigurationSet. Since judge_model_version == judge_role.configuration_sha256, which hashes credential_reference, prices and limits along with model and prompt, and since those limits were sized to the label count, the attested identity drifted with corpus size and could never equal the deployed one. load_judge_calibration_status compares judge_identity against the running identity and returns invalidated/identity_drift — so the identity RESULT_2026-07-24.md offered under "hand this to integration" would have been rejected by the runtime it was handed to. Pinned by test: the same evaluator over 54 vs 56 labels yields different judge_model_version values. Two further properties, disclosed before but now measured rather than described: the evidence judged is authored, never executed (campaign_run_id "ground-truth-unexecuted"), and the pooled headline includes 12 oracle-backed samples the model cannot influence. Restratified: pooled agreement 0.9259 over 54, but 0.9048 over the 42 samples the model actually decides, with 4 ambiguous over-calls the false-positive rate structurally cannot see. CONFIRMATION AUTHORITY — two real holes, both reproduced by execution HostedFourRoleRuntime is the four-role composition an enabled model Judge runs inside, and the class PR #50 builds run_attempt on. It took two authorities from the model's own response: 1. its verdict enum included EXPLOIT_CONFIRMED and _deterministic_precedence returned it unchanged when no oracle or canary had fired. run_attempt drafts a vulnerability report for any EXPLOIT_CONFIRMED/EXPLOIT_LIKELY verdict, so a model claiming a confirmation produced Documentation output for a finding with no evidence behind it. 2. the guard refusing an uncalibrated NO_EXPLOIT_OBSERVED was keyed on a "calibrated" boolean the MODEL emitted about itself — so a model asserting its own calibration could clear a case. That is the dangerous direction. Repaired: EXPLOIT_CONFIRMED removed from the model's output schema, a model-claimed confirmation downgraded to EXPLOIT_LIKELY with reason model_confirmation_refused, calibration status passed in from the caller's enablement gate. The class had no production composition root before now, which is why this had not yet produced a bad finding; PR #50 gives it one. WHAT THIS ADDS - capture_judge_calibration.py: --hosted-configuration-set and --expected-configuration-sha256 are now required. Identity is derived from the staged set, never synthesized. Reconstruction is itself a check — a prompt_sha256 that is not this release's served prompt is a refusal. Capacity preflight refuses a corpus past the staged budget or the 56-call platform ceiling rather than relaxing limits, which would change the identity being attested. - analyze_judge_calibration.py: restates a result over the NON-ORACLE stratum, the only cases an enabled model decides, and counts the two failure modes the contract's counters cannot express — a positive scored safe, and an over-call on an ambiguous sample. - enable_model_judge.py: the human gate. Refuses on identity drift, on a non-oracle stratum breach even when the pooled headline passes, on ground truth without a two-person blind attestation bound to the exact slice_set_sha256, and without a named approver. Re-checks the written artifact through require_model_judge_enablement. - REATTEST_RUNBOOK.md: the procedure, what the identity does and does not cover, and the fact that the replay path validates bundle SHAPE only — a hand-written bundle with 0 tokens and $0 cost yields a contract-valid passing artifact under the strict policy with no API key. Any re-attest must be reported as "the harness measured this bundle" unless an out-of-band OpenRouter usage export is attached. BLOCKED ON (neither is mine to resolve) - m: no staged production hosted configuration set exists on disk; production role configs live in the Postgres hosted_configuration_sets table. - g: the 100-case ground-truth set is GT-CAND-M11-LIVE100, 200 candidate labels, calibration_status AUTHORED_PENDING_HUMAN_ATTESTATION, human_labeler and distinct_reviewer both null, and its labels carry no evidence_envelope — so it is neither human-labeled nor runnable through CalibrationGate. The committed 54 labels are all rule- or agent-authored. - Scope note: 200 labels cannot be captured in one run at all. max_calls is capped at HOSTED_MAX_PHYSICAL_CALLS=56, and raising it would change the identity. 1547 passed, 3 skipped. tests/test_packaging.py::test_wheel_installed_outside_repo_validates_corpus fails identically on the untouched base (fresh-venv wheel install), unrelated. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
Closing as superseded by the governed four-role and corpus integrations now on main through 938326d. The branch is retained for history; do not merge its stale migration variants. |
Governed 0022 — the real four-role composition (v1 hard blocker)
Builds the real four-role composition so a real attack flows end to end over the existing reviewed/authorized corpus — closing the gap PR #43 left (four providers callable, not composed). Reuses
HostedFourRoleRuntime.run_attempt(m's Option 1) on the reserved base6badfbd(PR #47). Single migration head0022on0018→0021.No unreviewed generation (Horizon 2): the dispatched content must equal the reviewed case's seed-replay bytes, else the run aborts before the target is touched.
What's in the stack (5 commits on
6badfbd)9df945c, tightened9634d84): addsrun_kind='governed_acceptance'(v3 target-bound limits, one dispatch); a governed row cannot carry nullacceptance_context_sha256/acceptance_attempt_id; downgrade refuses while governed rows exist.bb044c1): the fold for the authority-shape gap (below).2055350):create_governed_acceptance_run(two-person launch;acceptance_context_sha256= reviewed case content hash),start_governed_agent_execution(permits the one bounded dispatch; calibrated enabled Judge),complete/abort_governed_acceptance_run.run_governed_acceptancecomposition (9cf6778): seed-replay invariant + real gateway/recorder/envelope dispatch + oracle over the real transcript + terminalization-safe governed lifecycle. Controlled-target e2e on real Postgres.14cb508): the fix-3 case the terminalization reconcile deleted (dispatched, unbilled 500 →not_observedwithphysical_attempts=1).The composition surfaced a real authority-shape gap: the 0020/0021
agent_acceptanceguards are hardwired torun_kind='agent_acceptance'and enforce an absolute zero-target-traffic invariant, so a governed run (four roles + one bounded dispatch) was unguarded for four-role lineage/Judge binding at the DB level. Per your Option 1 (isolate) call, I folded a dedicatedm1d_validate_governed_acceptance_execution+ provider guard into 0022 and left theagent_acceptanceguards (and their absolute guarantee) untouched — each kind's invariant stays absolute and separately verifiable.The revision graph is unchanged (still single head
0022on0021), but the 0022 body grew beyond thecampaign_runsconstraints it originally touched. Please re-run the single-head serialization check on the updated0022before merge.Tests vs. evidence (kept separate, per your instruction)
EXPLOIT_CONFIRMEDby oracle precedence; run completes with no dangling execution; and a drifted Red Team proposal aborts before the target with zero recorded dispatch.Validation: 139 tests green across the blast radius (governed migration/store/e2e, base runtime, provider-evidence preservation, hosted-agent lineage, core migrations); ruff clean; single head
0022.Please do not merge — reserved for your review + the serialization re-check.
🤖 Generated with Claude Code