Skip to content

corpus: land the reviewed 100-case canary corpus + wired consumption oracle (rebased, single head 0022) - #59

Merged
worldofhacks merged 5 commits into
codex/platform-observability-followupfrom
sub/corpus-100-rebase
Jul 25, 2026
Merged

corpus: land the reviewed 100-case canary corpus + wired consumption oracle (rebased, single head 0022)#59
worldofhacks merged 5 commits into
codex/platform-observability-followupfrom
sub/corpus-100-rebase

Conversation

@worldofhacks

Copy link
Copy Markdown
Owner

Replays g's PR #51 onto the integration head. PR #51 is CONFLICTING and red on test + secret-scan; none of that was corpus substance.

Two real defects fixed

1. Parallel Alembic head. The branch numbered its consumption migration 0018 with down_revision 0017 — but upstream 0018 is already provider_call_lineage on the same parent. After rebase, migrations/versions/ held two distinct revisions with the identical id 0018. Alembic rejects that outright. Renumbered to 0022 on down_revision 0021; ScriptDirectory.get_heads() now returns exactly ['0022'].

⚠️ PR #50 also claims 0022. Whichever lands second must renumber to 0023 — they cannot both keep the id.

2. The lint gate hid the whole suite. ruff check failed with 4 errors, and that is step 1 of 11 in the test job. So this lane's own 566-line tests/test_consumption_oracle_wiring.py — the test carrying its central claim — had never executed in CI. Fixed (3 over-long comments, 1 unused import). CI will now run it for the first time.

secret-scan was the .gitleaksignore staleness affecting every pre-ec85a45 branch; the rebase clears it. The 507d032 doc conflict is a duplicate of what landed via #48, so the integration head's authoritative 004 medium / 005 low / 006 low win.

Independent verification — I re-derived every claim rather than trusting it

Claim Verified
pinned manifest digest 07d649d4…52ca252d ✅ matches
case count 100 ✅
categories PI 20 / DX 18 / TM 18 / SC 15 / DoS 14 / IR 15 = 100 ✅
turn split 79 single + 21 two-turn = 121 physical
all 300 content hashes ✅ via resolve_workload(..., expected_content_hash=…)
oracle bindings 33 synthetic-canary + 26 system-prompt-canary + 14 resource-limit + 23 none + 4 unwired = 100 ✅

Local gates: ruff check clean, ruff format --check 727 files clean, 36 targeted tests pass.

(A first pass hashing raw file bytes reported 100 mismatches — that was my error, not corruption. The loader hashes canonical JSON: sort_keys=True, separators=(",",":"). Recorded here so the next reviewer doesn't repeat it.)

Honest limits — these bound what a 100-case run can conclude

  • Only 14 of 100 cases bind to a runtime_wired oracle (the consumption/resource-limit family). Of its three measured dimensions, only some are reachable against a black-box POST /chat.
  • 59 cases are canary-anchored — they can only confirm if the canary is genuinely seeded in the live target.
  • 4 cases bind to unwired baseline oracles (patient_scope_violation ×2, forbidden_tool_call ×2) and can therefore only ever return INDETERMINATE. That is disclosed, not hidden.
  • 23 cases bind to no oracle and are decisive only if the model Judge is enabled — which it is not, and cannot be until three separate preconditions land.
  • The digest 07d649… is pinned nowhere in the repo — it must be supplied at authorization time, so an accidental edit to any case file is caught only by an operator-typed value. Worth a follow-up CI assertion.
  • Caps conflict: this branch sets all four targets to $50 / 130 / 121 / 0. The capacity lane (769242e, unpushed) narrows the raise to week2 only and its verifier rejects these blobs. A human must pick one authority before either lands. No committed authorization artifact exists for the $50 envelope — authorization-request-week2.json still carries the old $1 / 40 / 60 envelope and corpus id m11-seed-corpus-v1.

alexandermill3r and others added 5 commits July 25, 2026 16:08
… 6 categories

Author 100 attack-case v1 cases (PI20/DX18/TM18/SC15/DoS14/IR15; 79 single + 21
two-turn = 121 physical) with content-addressed workload manifest, per-case
generation+review sidecars, and the embedded 14-case reviewed baseline. Each case
is canary-anchored to a genuinely seeded synthetic canary (system-prompt or
cross-patient) or bound to the consumption oracle (DoS); non-echoable cases are
LLM-assessed (kind=none), never bound to unwired oracles. Reconcile corpus.py
LIVE_100 constants to 6 categories; MVP/FULL_SCAN identities intact. Loader +
validate_attack_case (100/100) + workload registry tests pass.

Manifest sha256: 07d649d482dd1f59a70e2b7238506e59eacddb8f39b56c419ccc6aab52ca252d

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…measured dims)

_adjudicate now builds a ResourceObservation from the gateway's MEASURED trio
(elapsed_ms, request_count, response_size) re-read from Postgres and runs
ResourceLimitOracle alongside CanaryOracle, so DoS/unbounded-consumption cases
fire deterministically on a real measured breach. Target-internal tokens/tool
calls/cost are NOT observable from black-box /chat and are left None (skipped,
never fabricated, never a silent pass). recorder-resource-limits-v1
runtime_wired=True is justified by an end-to-end measured-breach test, not a bare
flag. Deterministic oracle precedence (canary first) + Judge transcript
containment preserved. Migration 0018 (expand-only nullable JSONB) single head
<-0017. Full suite 1524 passed; independent code + security review APPROVED.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… for m)

Raise safety_caps to the operator-authorized envelope so the 100-case / 121-physical
campaign is admissible: logical_case_limit=100, physical_request_limit=121,
target_retries_per_turn=0 (exact-match to the corpus per runner live-100 gate),
budget_usd=50 hard cap, run_timeout_seconds=3600, rps=0.5, max_attempts_per_run=130.
Validated: corpus exact-match 100/121/0; RunCaps within hard maxima; test_runner_campaign 29 passed.

REVIEW NOTE for m: envelope was written to production.json blocks as well as staging +
the dedicated target file; if production should retain tighter caps, scope this to only
the catalog the live-100 run binds. Not pushed.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… lint gate

Two things blocked this lane from integrating.

**Parallel Alembic head.** The branch numbered its consumption migration `0018`
with `down_revision 0017`, but upstream `0018` is already
`provider_call_lineage` on the same parent. Rebasing put two distinct revisions
with the identical id `0018` in `migrations/versions/`, which Alembic rejects
and which violates the single-head invariant. Renumber to `0022` on
`down_revision 0021` (the integration head's revision), fix the docstring
header to match, and update the readiness assertion.
`ScriptDirectory.get_heads()` now returns exactly `['0022']`.

Note for the 0022 lane: PR #50 also claims revision `0022`. Whichever lands
second must renumber to `0023`; they cannot both keep the id.

**Lint gate.** `ruff check` failed with 4 errors, and because that is step 1 of
the `test` job, the entire pytest suite — including this lane's own 566-line
`tests/test_consumption_oracle_wiring.py` — had never executed in CI. Rewrap
three over-long comments and drop an unused `sqlalchemy.text` import;
`ruff format` then settles three files.

Corpus content is untouched. Independently re-verified after the rebase:
`resolve_workload` loads all 100 cases against the pinned manifest digest
`07d649…252d`; categories are PI20/DX18/TM18/SC15/DoS14/IR15; 79 single-turn +
21 two-turn = 121 physical; oracle bindings are 33 synthetic-canary +
26 system-prompt-canary + 14 resource-limit + 23 none + 4 unwired baseline.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two lanes asserted mutually exclusive safety_caps in the same four files: this
branch raised ALL four targets to $50/130/121/0, while the capacity lane
(769242e) narrowed the raise to week2 and restored week1 to legacy. Capacity's
verifier rejects the broad blobs outright, so the two could never both land.

Owner decision: capacity's narrowed envelope is authoritative. Take 769242e's
four blobs wholesale.

  copilot-week2 / clinical-copilot-week2
      budget $50 · attempts 100 · logical 100 · physical 121 · retries 0 · 3600s · 0.5 rps
  copilot-week1 / clinical-copilot-week1
      legacy: budget $1 · attempts 40 · logical 40 · physical 60 · retries 1 · 1800s

It matches the corpus exactly (100 logical / 121 physical), keeps the $50 hard
cap, and applies least privilege — only the target actually being scanned is
raised. `retries 0` is the outer campaign safety envelope and is independent of
the governed run's agent-level retries; the two are not in tension.

Also fix a second hardcoded head assertion missed by the 0022 renumber
(test_agent_acceptance_migration.py:281). The point of that test is that there
is exactly ONE head, not that the head is any particular revision, so it now
asserts 0022 and the 0022->0021 link.

Verified: 395 catalog/caps/target/workload tests pass against the narrowed
envelope; validate_target_catalog clean; ruff lint + format clean.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@worldofhacks
worldofhacks merged commit 9ccf0de into codex/platform-observability-followup Jul 25, 2026
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants