judge: bind calibration to the deployed identity; close two model-authority holes (blocked on staged prod config + human ground truth)

[worldofhacks]

Judge calibration → deployed identity, and two model-authority holes closed

Status: harness wired. The re-attest has NOT been run — there is no new number in this PR, and I am not asking you to enable anything. It is blocked on two inputs neither of which is mine (§3). What is here is the machinery that makes the re-attest bind to something real, plus two defects found on the way that matter to you before #50 lands.

Stacks conceptually behind #51 (the model-eval lane intake); this branch is off 8ce852b, so it will need the same replay treatment onto the integration head.

---

1. Why the old number is superseded — and what part of it was actually sound

The headline at 8ce852b was agreement 0.9259 / FN 0.0000 / FP 0.0000. Three separate statements, kept apart because they are not equally serious:

The provider calls were real. 54 unique OpenRouter request ids, 54 distinct token triples, ledger total $0.75823375. Per-sample costs are read from the provider's own usage.cost (providers/openrouter.py:647) and match Gemini 2.5 Pro's real list price (1.25 / 10 per M) — not this repo's own 5 / 30 ceilings, which is what a fabricator working from repo constants would have produced. The arithmetic reproduces byte-for-byte. I want that on the record, because "fabricated" would be the wrong word for it.

The evidence judged was authored, never executed. Every ground-truth envelope carries campaign_run_id: "ground-truth-unexecuted", and the trusted canary detail says so outright: "Authored deterministic confirmation for offline calibration; it is not evidence that a live target was executed." That is the right design for a labelled corpus — you cannot harvest 54 labelled exploits from a live run — but it means the measurement is about the evaluator, never the target.

The identity it attested does not exist in production. This is the disqualifying part. capture_judge_calibration.py built its own HostedConfigurationSet. Since

judge_model_version == judge_role.configuration_sha256
                    == sha256(role, provider, model_id, upstream_provider,
                              credential_reference, prompt_sha256, policy_sha256,
                              prices, limits)

and those limits were sized to the label count, the attested identity drifted with corpus size and could never equal the deployed one. load_judge_calibration_status compares judge_identity against the running identity and returns invalidated / identity_drift — so the identity RESULT_2026-07-24.md handed you under "Frozen identity — hand this to integration" would have been rejected by the runtime it was handed to. Pinned by test: the same evaluator over 54 vs 56 labels yields different judge_model_version values.

RESULT_2026-07-24.md now carries a superseded banner; the identity block is marked do-not-use.

---

2. ⚠️ Two model-authority holes — please read before landing #50

HostedFourRoleRuntime is the class #50 builds run_attempt on. It was taking two authorities out of the model's own response. Both reproduced by execution, both fixed here.

(a) The model could confirm an exploit. Its verdict enum included EXPLOIT_CONFIRMED, and _deterministic_precedence returned it unchanged when no oracle or canary had fired:

deterministic ground truth : {'state': 'INDETERMINATE', 'canary_triggered': False}
model assessment           : {'state': 'EXPLOIT_CONFIRMED', 'calibrated': True, ...}
RESULT state               : EXPLOIT_CONFIRMED   <-- model-sourced confirmation

run_attempt drafts a vulnerability report for any EXPLOIT_CONFIRMED/EXPLOIT_LIKELY verdict, so this produced Documentation output for a finding with no evidence behind it.

(b) The model could vouch for its own calibration. The guard refusing an uncalibrated NO_EXPLOIT_OBSERVED was keyed on a calibrated boolean the model emitted about itself (hosted_runtime.py:708, required at :710). A model asserting calibrated: true got its safe verdict accepted. That is the direction that clears a real exploit.

Fixed: EXPLOIT_CONFIRMED removed from the model's output schema; a model-claimed confirmation is downgraded to EXPLOIT_LIKELY with reason model_confirmation_refused; calibration status is passed in from the caller's enablement gate. Pinned by tests/test_hosted_runtime_confirmation_authority.py.

The class had no production composition root before now — only tests constructed it, and __init__ is itself gated by require_model_judge_enablement — which is why this had not yet produced a bad finding. #50 gives it one.

---

3. Blocked on — I need these from you and from g

	Input	Owner
B1	Staged production hosted configuration set (HostedConfigurationSet.canonical_payload() shape) + the configuration_sha256 the deploy attested	you
B2	Two-person human ground-truth attestation bound to the exact slice_set_sha256	g + a distinct approver

On B1: there is no staged config anywhere on disk — production role configs live in the Postgres hosted_configuration_sets table. I need the payload and its hash, not the secret; only the credential_reference string is identity-bearing.

Two deployment gaps you will hit even after a clean re-attest, both verified:

• AGENTFORGE_JUDGE_CALIBRATION_PATH is set nowhere. It is read exactly once, at src/agentforge/runner.py:820, and appears in no .env.example, no .env.local, and no Railway config. Without it the runner has no calibration path, load_judge_calibration_status returns unavailable, and a correctly enabled artifact is simply never loaded — the model Judge stays off no matter what we approve. Please wire it as part of the deploy.
• HEADSHOT_JUDGE_* is dead and misleading. .env.example:111 / .env.local:61 set HEADSHOT_JUDGE_PROVIDER=anthropic and HEADSHOT_JUDGE_MODEL=claude-sonnet-5, with zero consumers anywhere in src/ or scripts/ — and it names a different provider and model than the actual judge (google/gemini-2.5-pro via openrouter:google-vertex). Anyone reading the env file will get the judge identity wrong. Worth deleting.

On B2: g's set is GT-CAND-M11-LIVE100 — 200 candidate labels, calibration_status: AUTHORED_PENDING_HUMAN_ATTESTATION, human_labeler and distinct_reviewer both null, and the labels carry no evidence_envelope, so they cannot be fed through CalibrationGate at all. g was scrupulous about labelling it a candidate. It is not a human-labeled ground-truth set yet. The committed 54 labels are all policy_rule / deterministic_oracle / deterministic_canary — none human.

Scope correction while you plan: 200 labels cannot be captured in one run under any valid configuration. HostedLimits.max_calls is capped at HOSTED_MAX_PHYSICAL_CALLS = 56, and raising it is self-defeating because limits sit inside judge_model_version — a widened cap produces a different identity than the deployed one. It has to be several identity-bound batches. #51's note that "the 56-call hosted cap requires batching" is exactly right. Cost scales with it: 54 labels ≈ $0.76, so ~200 ≈ $2.80, not the ~$0.76 budgeted.

---

4. What's in the diff

• scripts/capture_judge_calibration.py — --hosted-configuration-set and --expected-configuration-sha256 now required; the synthesis path is gone. Reconstruction is itself a check: HostedRoleConfiguration rejects a prompt_sha256 that is not this release's served prompt, so prompt drift between deploy and calibration is a refusal. Capacity preflight refuses a corpus past the staged budget or the platform ceiling rather than relaxing limits. Manifest gains identity_binding and evidence_provenance blocks.
• scripts/analyze_judge_calibration.py (new) — restates a result over the non-oracle stratum, the only cases an enabled model decides, and counts what the contract's own counters cannot express: a positive scored safe, and an over-call on an ambiguous sample. On the committed artifact: pooled 0.9259 over 54, 0.9048 over the 42 the model actually decides, all 12 oracle-backed samples at 1.0000, 4 over-calls, 0 safe-misses.
• scripts/enable_model_judge.py (new) — the human gate. Refuses on identity drift, on a non-oracle breach even when the pooled headline passes, on ground truth without a two-person blind attestation bound to the exact slice set, and without a named approver. Re-checks the written artifact through require_model_judge_enablement.
• docs/evidence/judge-calibration/REATTEST_RUNBOOK.md (new) — procedure, and the honest limits below.
• RESULT_2026-07-24.md — superseded banner.

---

5. Honest limits I am not able to close from here

The replay path cannot prove a provider call happened. run_judge_calibration.py --captured-results replays an operator-supplied JSON file, and calibration_results.py validates its shape only — no request-id format check, no cost re-derivation, no network. A hand-written bundle naming a nonexistent model with 0 tokens and $0.00 cost, with no OPENROUTER_API_KEY set, yields a contract-valid state: passed artifact with agreement 1.0 under the strict policy. Reproduced independently twice during review. So: any re-attest must be reported as "the harness measured this bundle", not "the model was measured", unless an out-of-band OpenRouter usage export for the capture window is attached. That export is the only thing that closes it.

The identity covers model, prompt, policy and budget limits — but not everything. HostedCallBounds (per-call max tokens, reasoning budget, timeout), the Judge assessment schema, and the 24k evidence-truncation bound are outside JudgeIdentity; temperature/top_p are never sent on the Judge path. So a per-call budget change will not invalidate a calibration even though it can move behaviour. Pin HostedCallBounds alongside the config when you attest. Extending JudgeIdentity is a judge_calibration contract change and belongs with contract-steward.

policy_sha256 is an unvalidated free hex label. In the superseded capture it was sha256("judge-calibration-capture:judge:v1"), which resolves to no registered policy. Please stage the real one.

Calibration does not prove the Judge cannot be fooled. It measures one identity against one authored corpus at one time. The containment is structural — oracle precedence, no confirmation state in the model's vocabulary, INDETERMINATE never counting as safe, human approval on publication — not statistical.

---

6. Unchanged, and verified

Confirmation stays oracle / canary / human. On the HostedEvaluator path this is enforced twice: the assessment schema has no EXPLOIT_CONFIRMED member, and verdict.json restricts a confirmed verdict's confirmation_source to oracle | canary | human, so calibrated_model cannot carry one. On the HostedFourRoleRuntime path it is enforced as of this PR (§2). INDETERMINATE stays visible and is never safe — is_safe() is true only for NO_EXPLOIT_OBSERVED.

What enabling buys, and the whole of it: cases with no trusted hit stop being forced to INDETERMINATE and receive an EXPLOIT_LIKELY / NO_EXPLOIT_OBSERVED assessment.

---

Tests

1547 passed, 3 skipped, ruff clean. tests/test_packaging.py::test_wheel_installed_outside_repo_validates_corpus fails identically on the untouched base (fresh-venv wheel install) — pre-existing, unrelated.

🤖 Generated with Claude Code

[worldofhacks]

Integration intake: keep this split into two deliverables.

Pre-deploy deliverable: replay only the owned calibration foundation + 8d96f11 harness/authority changes onto the current published integration base 0ce980c. Exclude old reason_code, findings 004/005/006, capability-doc history, and contaminated ancestry already present elsewhere. Preserve redteam/** in ci.yml. It must be an exact-head GitHub-green branch and must use/accept the verified Qwen upstream identity alibaba, not together. This harness may land while its result remains explicitly NOT RUN.

Post-deploy evidence deliverable: after m stages the exact production four-role payload and proves resource_id == canonical configuration hash plus Runner secretref/Langfuse resolution, m will hand cal the canonical payload and attested SHA. Cal then captures/re-attests that exact identity, attaches the OpenRouter usage receipt, receives the separate human ground-truth/enablement approvals required by the harness, and returns the immutable enabled artifact before any campaign.

The raw PR is NO-SHIP because it conflicts and replays already-merged findings/history; do not merge it as-is.

[worldofhacks]

Update — batching at a fixed identity, and provenance that makes the artifact measured

Both review follow-ups are in (5504d54). Still no re-attest and no enablement — those stay blocked on the staged config and on ground-truth labels carrying both principals + evidence envelopes.

Batching instead of a wider cap

Taken exactly as directed: sub-runs against one unchanged staged configuration, so judge_model_version is constant and the batches are aggregatable.

• capture_judge_calibration.py gains --batch-size / --batch-index / --plan-only. Batch size defaults to the staged Judge role's max_calls and may never exceed it — the refusal tells you to shrink the batch, not raise the limits.
• generation_policy_sha256 now binds the campaign (corpus size + batch size) rather than the per-batch sample count, so it's constant across sub-runs too and the merged bundle has one coherent value. A per-batch count would make the trailing short batch disagree for no meaningful reason.
• merge_calibration_batches.py refuses unless every batch carries a byte-identical judge_identity and pinned provenance, the batches are disjoint, and their union covers the corpus exactly. Coverage is checked against the slice directory rather than the batch manifests — a silently dropped batch would measure a smaller, easier corpus and report a better agreement rate.

Verified lossless. I split the committed 54-sample bundle into 2 batches of 27, merged, and re-measured:

batched == unbatched metrics : True
calibration_id identical     : True
merged 2 batches -> 54 samples, measured $0.75823375

Plan output over the real corpus at --batch-size 20:

corpus            54 labels
batch size        20 (one physical Judge call per label)
batches           3
judge_model_ver   001a82e7…  (constant across every batch)
generation_policy 5c958cde…  (bound to the campaign, not the batch)

Provenance — the artifact now has to be measured, not just shape-valid

verify_calibration_provenance.py reconciles the bundle against OpenRouter's own usage export: every provider_request_id must appear in it, with matching model, cost and tokens, and the summed cost must reconcile with the ledger. A fabricated id has nothing to match. The resulting attestation is a required input to enable_model_judge.py — a shape-valid bundle can no longer license runtime authority.

CSV or JSON, case-insensitive column aliases. An export whose columns can't be resolved prints the headers it actually saw and refuses rather than guessing, because a column guessed wrong would "verify" nothing.

The attestation states its own limit in its body: it does not prove the evidence judged came from a live target. The corpus is authored synthetic ground truth. It establishes only that the Judge model calls really happened and cost what the bundle claims.

Per-batch visibility

analyze_judge_calibration.py --batch-manifest reports per-sub-run metrics beside the aggregate. The aggregate governs; the batches are shown so a degraded sub-run stays visible. On the rehearsal split that's not hypothetical:

stratum           n    agree      FN      FP  safe-miss  over-call
pooled           54   0.9259  0.0000  0.0000          0          4
oracle-backed    12   1.0000  0.0000  0.0000          0          0
NON-ORACLE       42   0.9048  0.0000  0.0000          0          4

per batch         n    agree  non-oracle n  non-oracle agree
batch-0          27   0.9630            21            0.9524
batch-1          27   0.8889            21            0.8571

Batch-1's non-oracle agreement is materially weaker than batch-0's, and that is invisible in the 0.9048 aggregate.

Cost

One physical Judge call per label. The 54-label set measured $0.75823375 → ~$0.014/label, so ~$2.80 for 200 labels across 4 batches of 56/56/56/32. Matches your budget.

Housekeeping

Three capacity tests in test_judge_calibration_capture_binding.py are retired — their concern (a corpus larger than one batch) is no longer a refusal but a split, and is covered in test_judge_calibration_batching.py.

1568 passed, 3 skipped, ruff clean. The one pre-existing packaging failure is unchanged and unrelated.

[worldofhacks]

Answer for q: reconcile_judge_assessment was already safe — but the guard is now genuinely shared

Direct answer: #54 previously covered only _deterministic_precedence. reconcile_judge_assessment did not have either hole — it was never the vulnerable path. Verified by execution against the manual-skeleton path before changing anything:

1. model claims EXPLOIT_CONFIRMED, no oracle hit  -> REFUSED (raise): "Evaluator response state is invalid"
2. model smuggles calibrated:true on a safe verdict -> REFUSED (raise): "Evaluator response has an invalid shape"
3. caller says calibration unavailable              -> INDETERMINATE | model_decisive: False
4. caller says enabled                              -> NO_EXPLOIT_OBSERVED | authority: calibrated_model

Why it was safe: _validate_assessment rejects any state outside the assessment vocabulary, _model_verdict's reason map would KeyError on EXPLOIT_CONFIRMED, the assessment shape is an exact key-set so a smuggled calibrated field is rejected outright, and calibration_state was always a caller parameter. _deterministic_precedence was the outlier on all three counts.

But q is right that this needed doing anyway — done (4058312)

The two paths were enforcing the same rule from two private copies, which had already diverged once. Same-rule-two-places is how the next chokepoint gets a third copy. So the rule now lives once, in src/agentforge/agents/judge/model_authority.py, and both chokepoints call it:

	manual skeleton	four-role
vocabulary	MODEL_ASSESSMENT_STATES	same object (is-identical, pinned by test)
model claims confirmation	sanitize_model_state(..., on_confirmation="refuse")	sanitize_model_state(..., on_confirmation="downgrade")
may the model decide?	model_may_decide(...)	model_may_decide(...)

The two dispositions are an explicit parameter, not a silent divergence. refuse fails closed and suits a single evaluation, where a schema-violating response makes the whole invocation suspect. downgrade keeps the unsafe signal as EXPLOIT_LIKELY and suits a long campaign, where aborting every remaining case over one malformed response costs more than it protects. Neither ever yields a confirmation — that part is not parameterised.

model_may_decide reads only caller-held values (calibration_state, model_authority_allowed). Nothing the model reports about itself is an input, by construction.

The module imports nothing from agentforge, so any chokepoint can depend on it without a cycle. Callers translate ModelAuthorityViolation into their own typed error, so existing error contracts are unchanged (HostedEvaluatorError / HostedCompositionError messages are identical to before).

q: route through this, don't build a second

from agentforge.agents.judge.model_authority import (
    MODEL_ASSESSMENT_STATES,   # the enum you hand the model — no EXPLOIT_CONFIRMED
    ModelAuthorityViolation,
    model_may_decide,          # authority from YOUR gate, never the model's output
    sanitize_model_state,      # on_confirmation="refuse" | "downgrade"
)

tests/test_model_authority_guard.py (13 tests) pins the shared behaviour and asserts both chokepoints draw from the same object, so a future divergence fails CI rather than shipping.

Per-batch weakness — carried into the runbook

Flagged as directed. REATTEST_RUNBOOK.md now instructs the real run to report the per-batch table and to call out explicitly if any batch sits near or under min_agreement_rate while the aggregate passes, rather than letting the aggregate speak for it. Recorded rationale: batches are cut from the corpus sorted by label_id, so a batch can concentrate one category's hard cases — the rehearsal spread (0.9524 vs 0.8571 against a 0.9048 aggregate) is exactly that shape. A batch failing alone is not automatically disqualifying, since the aggregate is the defined gate, but it is a finding.

Unchanged

Still correctly blocked on the staged prod config, the two attestors, and the OpenRouter usage export. No re-attest, no enablement.

(Aside, not acted on: I see a note that integration may have moved from m to sub. This PR still targets codex/platform-observability-followup — tell me if it should be retargeted and I'll redo it rather than assume.)

[worldofhacks]

Rebased onto sub's integration head — and the base was already correct

Acting on "retarget to sub's integration base", I checked before moving anything, and the base ref did not need to change:

• judge: bind calibration to the deployed identity; close two model-authority holes (blocked on staged prod config + human ground truth) #54 already targets codex/platform-observability-followup;
• sub's own PRs target the same ref — judge: land cal's canonical AD-04 confirmation-authority guard (rebased) #57 (merged) and corpus: land the reviewed 100-case canary corpus + wired consumption oracle (rebased, single head 0022) #59 (open).

So that is sub's integration base. What actually needed fixing was the head, not the base.

What I found

Sub had already rebased and merged my first two commits via #57:

935e740  Merge pull request #57 from worldofhacks/sub/cal-ad04-rebase   <- integration head
df0ed6e  test(judge): retarget calibration tests at the package-owned prompt authority
b7747f3  feat(judge): batch the calibration at a fixed identity, and require provider-reconciled provenance
b7ebe07  feat(judge): bind calibration to the deployed identity; close two model-authority holes

scripts/merge_calibration_batches.py, verify_calibration_provenance.py and enable_model_judge.py are all present on the head; src/agentforge/agents/judge/model_authority.py is not. So the canonical guard was the only undelivered piece, and #54 was showing two already-merged commits alongside it.

Rebased

#54 now contains exactly one commit (731839b) against the integration head:

docs/evidence/judge-calibration/REATTEST_RUNBOOK.md |  10 +
src/agentforge/agents/hosted_runtime.py             |  63 +++---
src/agentforge/agents/judge/hosted.py               |  33 ++-
src/agentforge/agents/judge/model_authority.py      | 133 ++++++++++++
tests/test_model_authority_guard.py                 | 225 +++++++++++++++++++++

1937 passed, 3 skipped, 0 failed on the integration base, ruff clean.

My fault, noted — thank you for the fix

df0ed6e exists because I cut this branch from 8ce852b, which predates agents/hosted_prompts.py becoming the agentforge.agents.prompts package. Three of my test modules imported a symbol that no longer exists — green locally, broken at collection on the head. Sub had to author the repair to land my work.

The rebase picks that fix up, and I checked the new file specifically: tests/test_model_authority_guard.py imports neither hosted_prompt nor load_prompt_registry, so it does not reintroduce the problem. I'll cut from the integration head rather than from my lane's own tip going forward.

Unchanged

Still blocked on the staged prod config (sub), the two attestors, and the OpenRouter usage export. No re-attest, no enablement.

[worldofhacks]

Graded provenance — the deadline relaxation, implemented so the downgrade can't be lost

Both relaxations are in. Two corrections to the framing first, because they change what the disclosure has to say.

1. There is no separate automated-labeled set, and the labels are not model-labeled

I searched every lane worktree. No labeling lane has produced a new set. What exists is the same 54 committed labels, and they are rule-derived in code — _LABEL_TABLE[slug] in scripts/build_calibration_corpus.py, resolved from the sample slug with no model involved. label_source ∈ {policy_rule, deterministic_oracle, deterministic_canary}.

So disclosing them as "automated-labeled baseline (model X)" would name a model that does not exist. I've implemented a model_labeled tier for when a genuine one lands, and disclosed what we actually have as rule_derived. (g's 200-label candidate set is still AUTHORED_PENDING_HUMAN_ATTESTATION with no evidence envelopes — not runnable regardless.)

2. I did not verify the 54 calls — I found strong circumstantial evidence

Worth stating precisely since the whole point is the label being accurate. What I established was unique provider-shaped request ids, provider-reported costs matching Gemini list price rather than repo constants, and distinct token counts. That is not reconciliation against provider records.

That distinction is now code, not prose. lineage_consistent is earned by five checks that actually run:

unique_request_ids  provider_shaped_request_ids  distinct_costs  distinct_token_counts  nonzero_costs

The committed bundle passes all five → lineage_consistent. A hand-written bundle fails at least one → unverified. Tests cover each check individually.

What's implemented

src/agentforge/agents/judge/provenance.py. Provenance is computed from the evidence, never declared — naming a strong tier does not grant it.

Ground truth		Provider calls
human_two_person	two distinct blind principals	usage_export_reconciled	every sample matched to provider records
model_labeled	every label names its model	lineage_consistent	what the bundle earns — not proof
rule_derived	what is on disk	unverified	shape-valid only
unattested	nothing

The approver names the weakest tier they accept; enablement refuses if the real tier is weaker. And because judge_calibration.json is additionalProperties: false, I encoded the accepted tiers into approver_ref, which is inside the frozen contract:

approver_ref = "gt=rule_derived;prov=lineage_consistent;by=headshot:morgan"

A sidecar file can be separated from the artifact in a copy. This cannot. The downgrade travels wherever the artifact travels, and the same field names who accepted it.

Enablement is still a human act. Relaxing label provenance does not relax --approver-ref / --confirm.

The disclosure at current tiers, verbatim

Ground truth: automated-labeled baseline — labels derived in code from a static design table (scripts/build_calibration_corpus.py), NOT model-labeled and NOT human ground truth. The labels encode what the corpus author intended each sample to be, so the measurement shows agreement with that intent, not with an independent judgement of the evidence. Provider calls: consistent with a real provider run, NOT reconciled against the provider's records — unique provider-shaped request ids, provider-reported per-sample costs and distinct token counts. Strong circumstantial evidence; it is not proof the calls occurred.

The second clause of the ground-truth line is the one that matters for reading the number: against rule-derived labels, agreement measures the Judge against the corpus author's stated intent, not against an independent reading of the evidence.

⚠️ Still blocked — and this one was not lifted

The staged production hosted configuration set. I re-checked the integration head: no staged payload exists anywhere in the tree; production role configs live in the Postgres hosted_configuration_sets table. Without it there is no prod identity to derive, so no identity-bound calibration can be produced — the two relaxations don't reach this.

I deliberately did not fall back to re-measuring the existing bundle: that is bound to the capture-local identity 30f743a3… and would reproduce 0.9259, the superseded number tied to a configuration production does not run.

To unblock, I need exactly two things from the integrator:

1. the staged set as HostedConfigurationSet.canonical_payload() JSON (no secret — only the credential_reference string is identity-bearing), and
2. its configuration_sha256.

Given those, the batched capture, merge, measurement, stratified report and tier-disclosed enablement all run end-to-end — rehearsed already on the committed bundle.

[worldofhacks]

Adopted GT-AUTO-M11-LIVE100 as the model_labeled baseline

Verified the artifact directly from 7bb472f before adopting anything. Everything checks out:

artifact_id	GT-AUTO-M11-LIVE100
content_sha256	0ffc85fa065080f1a83f56a887b642507b0ac67adc4e830ab28b1e3fd5432c81 ✓
labels	200 = 100 automated_model_agent + 100 rule_derived ✓
labeller	claude-opus-5[1m] — different family from the Judge (gemini-2.5-pro) ✓
honesty	two_person_human_gate_satisfied: false, calibration_status: AUTOMATED_NOT_HUMAN_ATTESTED ✓

The labelling lane was right to leave the enum to this lane, and right about why.

The enum extension, and the privilege it does not grant

ground-truth-slice.v1.json now admits automated_model_agent, with labeler_model_id required — a model label that doesn't name its model can't be graded at all.

What it deliberately does not inherit is human_expert's privilege. validation.py:1326 permits an EXPLOIT_CONFIRMED with no oracle and no canary hit only for human_expert + confirmation_source: human + human_confirmed in the reason codes. I did not extend that. Only an oracle, a canary, or a human may confirm, and a model labeller is none of those.

That turns out to matter more than a schema nicety: it makes the 8-case exclusion structurally enforced rather than a hand-kept list. A model-labelled CONFIRM with no bound oracle is now rejected by validation outright — it cannot enter a ground-truth slice even if someone tried. Pinned by test.

The eight exclusions

DX-003 · DX-118 · PI-003 · TM-009 · TM-010 · TM-013 · TOOL-PROMPTFOO-68E9DFE5599D · TOOL-PYRIT-1D2BE5077815

Your reasoning holds, and it's worse than "phantom false negatives" — I checked the scoring path. CalibrationGate._reasons raises the hard confirmed_exploit_missed invariant on any expected EXPLOIT_CONFIRMED with a non-positive actual, and that fails the entire calibration regardless of every rate threshold. So these 8 wouldn't just distort the FN rate; they could hard-fail a calibration against a Judge behaving exactly as designed.

Implementation detail worth flagging: the exclusion is derived from the artifact's own disagreements records, not hard-coded, and cross-checked against its declared over_claimed_provability_n. A drift between the list and the headline is an error, not a quietly shorter exclusion. Excluded labels are partitioned out and counted: 16 labels excluded, 184 scored, of 200.

The honest headline, carried

Independent automated labelling of the 100-case corpus by claude-opus-5[1m], blind to the candidate labels, to each case's own oracle_expectation, and to Judge output: 92/100 agreement (Cohen's κ 0.743), 0 false negatives, confusion one-directional — all 8 disagreements run LIKELY → CONFIRM; there is no CONFIRM → LIKELY cell at all. Per category: DoS / identity-role / state-corruption 1.00; data-exfiltration 0.889; prompt-injection 0.85; tool-misuse 0.833. Inter-pass 0.88 (κ 0.596), 12 adjudicated.

This is not human ground truth. No human attested any label. The labeller over-claimed provability on 8 cases and under-claimed on none — its bias runs toward asserting deterministic provability, which is why those 8 are excluded rather than scored.

The one-directional confusion is the part I'd keep in front of a reader: 0 false negatives isn't "we got lucky on a small sample", it's that the independent labeller never once down-graded a candidate CONFIRM.

One thing the adoption does not change

These labels carry no evidence_envelope, and the artifact states corpus_execution_status: NOT_EXECUTED. CalibrationGate.evaluate feeds the evaluator label["evidence_envelope"] per label — so this set can grade provenance (which is what it's adopted for) but cannot by itself drive a Judge calibration run. Envelopes come from executing the corpus, which is the live-campaign path.

So the standing blocker is unchanged and now joined by a second: the staged prod configuration set (no identity to bind to), and evidence envelopes for the 100 cases. Neither is in this lane.

1975+ passed, ruff clean. Fixture vendored at tests/fixtures/independent-automated-labels.v1.json so the numbers above are asserted against the real artifact in CI, not a mock.