corpus: land the reviewed 100-case canary corpus + wired consumption oracle (rebased, single head 0022) - #59
Merged
worldofhacks merged 5 commits intoJul 25, 2026
Conversation
… 6 categories Author 100 attack-case v1 cases (PI20/DX18/TM18/SC15/DoS14/IR15; 79 single + 21 two-turn = 121 physical) with content-addressed workload manifest, per-case generation+review sidecars, and the embedded 14-case reviewed baseline. Each case is canary-anchored to a genuinely seeded synthetic canary (system-prompt or cross-patient) or bound to the consumption oracle (DoS); non-echoable cases are LLM-assessed (kind=none), never bound to unwired oracles. Reconcile corpus.py LIVE_100 constants to 6 categories; MVP/FULL_SCAN identities intact. Loader + validate_attack_case (100/100) + workload registry tests pass. Manifest sha256: 07d649d482dd1f59a70e2b7238506e59eacddb8f39b56c419ccc6aab52ca252d Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…measured dims) _adjudicate now builds a ResourceObservation from the gateway's MEASURED trio (elapsed_ms, request_count, response_size) re-read from Postgres and runs ResourceLimitOracle alongside CanaryOracle, so DoS/unbounded-consumption cases fire deterministically on a real measured breach. Target-internal tokens/tool calls/cost are NOT observable from black-box /chat and are left None (skipped, never fabricated, never a silent pass). recorder-resource-limits-v1 runtime_wired=True is justified by an end-to-end measured-breach test, not a bare flag. Deterministic oracle precedence (canary first) + Judge transcript containment preserved. Migration 0018 (expand-only nullable JSONB) single head <-0017. Full suite 1524 passed; independent code + security review APPROVED. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… for m) Raise safety_caps to the operator-authorized envelope so the 100-case / 121-physical campaign is admissible: logical_case_limit=100, physical_request_limit=121, target_retries_per_turn=0 (exact-match to the corpus per runner live-100 gate), budget_usd=50 hard cap, run_timeout_seconds=3600, rps=0.5, max_attempts_per_run=130. Validated: corpus exact-match 100/121/0; RunCaps within hard maxima; test_runner_campaign 29 passed. REVIEW NOTE for m: envelope was written to production.json blocks as well as staging + the dedicated target file; if production should retain tighter caps, scope this to only the catalog the live-100 run binds. Not pushed. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… lint gate Two things blocked this lane from integrating. **Parallel Alembic head.** The branch numbered its consumption migration `0018` with `down_revision 0017`, but upstream `0018` is already `provider_call_lineage` on the same parent. Rebasing put two distinct revisions with the identical id `0018` in `migrations/versions/`, which Alembic rejects and which violates the single-head invariant. Renumber to `0022` on `down_revision 0021` (the integration head's revision), fix the docstring header to match, and update the readiness assertion. `ScriptDirectory.get_heads()` now returns exactly `['0022']`. Note for the 0022 lane: PR #50 also claims revision `0022`. Whichever lands second must renumber to `0023`; they cannot both keep the id. **Lint gate.** `ruff check` failed with 4 errors, and because that is step 1 of the `test` job, the entire pytest suite — including this lane's own 566-line `tests/test_consumption_oracle_wiring.py` — had never executed in CI. Rewrap three over-long comments and drop an unused `sqlalchemy.text` import; `ruff format` then settles three files. Corpus content is untouched. Independently re-verified after the rebase: `resolve_workload` loads all 100 cases against the pinned manifest digest `07d649…252d`; categories are PI20/DX18/TM18/SC15/DoS14/IR15; 79 single-turn + 21 two-turn = 121 physical; oracle bindings are 33 synthetic-canary + 26 system-prompt-canary + 14 resource-limit + 23 none + 4 unwired baseline. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two lanes asserted mutually exclusive safety_caps in the same four files: this branch raised ALL four targets to $50/130/121/0, while the capacity lane (769242e) narrowed the raise to week2 and restored week1 to legacy. Capacity's verifier rejects the broad blobs outright, so the two could never both land. Owner decision: capacity's narrowed envelope is authoritative. Take 769242e's four blobs wholesale. copilot-week2 / clinical-copilot-week2 budget $50 · attempts 100 · logical 100 · physical 121 · retries 0 · 3600s · 0.5 rps copilot-week1 / clinical-copilot-week1 legacy: budget $1 · attempts 40 · logical 40 · physical 60 · retries 1 · 1800s It matches the corpus exactly (100 logical / 121 physical), keeps the $50 hard cap, and applies least privilege — only the target actually being scanned is raised. `retries 0` is the outer campaign safety envelope and is independent of the governed run's agent-level retries; the two are not in tension. Also fix a second hardcoded head assertion missed by the 0022 renumber (test_agent_acceptance_migration.py:281). The point of that test is that there is exactly ONE head, not that the head is any particular revision, so it now asserts 0022 and the 0022->0021 link. Verified: 395 catalog/caps/target/workload tests pass against the narrowed envelope; validate_target_catalog clean; ruff lint + format clean. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
worldofhacks
merged commit Jul 25, 2026
9ccf0de
into
codex/platform-observability-followup
10 checks passed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Replays g's PR #51 onto the integration head. PR #51 is CONFLICTING and red on
test+secret-scan; none of that was corpus substance.Two real defects fixed
1. Parallel Alembic head. The branch numbered its consumption migration
0018withdown_revision 0017— but upstream0018is alreadyprovider_call_lineageon the same parent. After rebase,migrations/versions/held two distinct revisions with the identical id0018. Alembic rejects that outright. Renumbered to0022ondown_revision 0021;ScriptDirectory.get_heads()now returns exactly['0022'].2. The lint gate hid the whole suite.
ruff checkfailed with 4 errors, and that is step 1 of 11 in thetestjob. So this lane's own 566-linetests/test_consumption_oracle_wiring.py— the test carrying its central claim — had never executed in CI. Fixed (3 over-long comments, 1 unused import). CI will now run it for the first time.secret-scanwas the.gitleaksignorestaleness affecting every pre-ec85a45branch; the rebase clears it. The507d032doc conflict is a duplicate of what landed via #48, so the integration head's authoritative 004medium/ 005low/ 006lowwin.Independent verification — I re-derived every claim rather than trusting it
07d649d4…52ca252d✅ matchesresolve_workload(..., expected_content_hash=…)Local gates:
ruff checkclean,ruff format --check727 files clean, 36 targeted tests pass.(A first pass hashing raw file bytes reported 100 mismatches — that was my error, not corruption. The loader hashes canonical JSON:
sort_keys=True,separators=(",",":"). Recorded here so the next reviewer doesn't repeat it.)Honest limits — these bound what a 100-case run can conclude
runtime_wiredoracle (the consumption/resource-limit family). Of its three measured dimensions, only some are reachable against a black-boxPOST /chat.patient_scope_violation×2,forbidden_tool_call×2) and can therefore only ever return INDETERMINATE. That is disclosed, not hidden.07d649…is pinned nowhere in the repo — it must be supplied at authorization time, so an accidental edit to any case file is caught only by an operator-typed value. Worth a follow-up CI assertion.$50 / 130 / 121 / 0. The capacity lane (769242e, unpushed) narrows the raise to week2 only and its verifier rejects these blobs. A human must pick one authority before either lands. No committed authorization artifact exists for the $50 envelope —authorization-request-week2.jsonstill carries the old$1 / 40 / 60envelope and corpus idm11-seed-corpus-v1.