fix: always dump case ledgers, even when a phase dies

[sei-ahouseholder]

• Closes An escaping assertion destroys the scenario's ledger artifacts, so the invariant harness fails on artifact download #2239

Summary

The case-ledger dump ran as the last statement of every run_*_demo(), so any assertion escaping a demo_check block skipped it while main()'s finally: assert_demo_success() still failed the job — the demo failed with an empty devlogs/ and the invariant-harness job died on artifact download instead of reporting an invariant result. A shared harness now owns the ordering: run phases, always dump, then assert.

Changes

• vultron/demo/helpers/harness.py (new): scenario_harness(demo_name) resets the failure accumulator, always dumps the ledgers on the way out, and calls assert_demo_success() last. On the failing path the original exception propagates unchanged with the accumulated demo_check failures attached via add_note() — a dump error can never replace the real cause (DEMOMA-23-001/-004, DEMOCI-10-004). _run_dump() is total: the no-dump-registered manifest write runs inside demo_step, _backstop_manifest() logs and swallows its own failure rather than displacing the dump error already in flight, and every path — including a BaseException that demo_step does not catch — is contained.
• vultron/demo/helpers/ledger_dump.py (new): the one implementation of the dump. Writes devlogs/<demo>/dump-manifest.json from a finally, recording the case ID, captured/total targets, and each missing actor with its route key and reason — so the artifact exists even when there were no ledgers to capture (DEMOCI-10-002, DEMOMA-17-001). default_devlogs_root() is the single resolution of where devlogs/ lives: DEVLOGS_DIR when set, else a repo-root-relative default. vultron/demo/report.py and test/ci/invariants/common.py both delegate to it, so the writer, the reader, and the invariant harness cannot disagree.
• All nine vultron/demo/scenario/*_demo.py: body wrapped in with scenario_harness("<demo>") as harness:; the dump is registered via harness.dump_with(...) as soon as the case exists — before any subsequent network call — so every later phase can fail without costing the ledgers. Each _phase_dump_case_ledgers is now a thin wrapper over dump_case_ledgers() and takes demo_name=harness.demo_name, making the harness's name authoritative; main() no longer owns the accumulator (DEMOMA-23-002/-003).
• test/ci/invariants/common.py: load_devlogs() no longer skips when the demo ran, dumped, and produced nothing. Manifests are read unconditionally, so a manifest with no ledger files — or one that parses as something other than an object — pytest.fails whatever else the dump produced, and reproduces the manifest's own per-actor account of what was missing and why. It still skips when there is genuinely no artifact (DEMOCI-10-003).
• test/demo/test_fv_demo.py: the harness's honest assert_demo_success() surfaced a masked, pre-existing in-process SYNC-2 check failure that earlier runs accumulated and dropped. The test now asserts on that one known failure exactly, so any other phase failure still fails it (tracked in In-process scenario tests must assert on accumulated demo failures (and drop the FV stopgap) #2267), and pins DEVLOGS_DIR to tmp_path so the always-dump change does not write into the repo root (In-process FV demo test writes to the repo-root devlogs/, so a local unit run after the integration suite fails the FV invariant harness #2274).
• vultron/metadata/specs/registry.py: select yaml.CSafeLoader when available. load_registry() goes 3.44s → 0.35s, which stops spec-metadata tests from tripping the global 5s thread timeout that this PR's spec additions were pushing over the edge. Partial mitigation for Integration suite aborts on the global 5s thread timeout, masking the merge signal #2270 (measurements in this comment); the timeout_method = "thread" fragility itself is untouched.
• specs/demo-ci.yaml (1.9.0) and specs/multi-actor-demo.yaml (1.4.0): new groups DEMOCI-10 "Case-Ledger Artifact Availability on Failure" and DEMOMA-23 "Shared Scenario Harness".
• vultron/demo/AGENTS.md, notes/demo-ci-invariants.md, vultron/demo/report.py: new pitfall section, the fail-vs-skip design note, and a stale docstring pointing at a module deleted long ago.

Verification

Run at bdc5442a, after merging origin/main (2f1a9354).

• Unit suite (uv run pytest): 6611 passed, 335 skipped, 2 xfailed, 5552 subtests passed, 1 failed in 89.7s. The one failure is test_fv_invariants.py::test_invariant_5_expected_event_types_present[validate_report] — it reads the gitignored local devlogs/fv/ left by an earlier demo run and is fv demo emits no validate_report eventType, so invariant 5 fails on a clean main #2273/In-process FV demo test writes to the repo-root devlogs/, so a local unit run after the integration suite fails the FV invariant harness #2274, not this branch. With a fresh devlogs root (DEVLOGS_DIR=$(mktemp -d)) the whole file skips, and the path test/ci/invariants/common.py resolves is byte-identical to the pre-branch one, so no behaviour changed here. CI runs on a fresh checkout, and the Python application workflow is green at this commit.
• Integration suite (uv run pytest -m integration): 1129 tests, exit 0.
• Every XFAIL in both suites references a live open issue (Remove alias_generator=to_camel from core-layer classes (ARCH-12-004) #1991, Remove core-layer actor classes from wire VOCABULARY registry (ARCH-12-003) #1992, Concern: SE-07 phrase tests using defaultdict cannot detect unfillable slot bugs #1898, CM-22 derives-from-non-accepted-adr signal not firing in decision-audit inventory #1993); the one XPASS references open test_demo in test_initialize_participant_demo.py references closed #464 — needs new tracking issue #1994.
• New tests: 21 in test/demo/test_issue_2239_ledger_dump_in_finally.py + test/demo/test_scenario_harness.py; the first parametrizes all nine scenarios, patching a phase to raise and asserting the manifest lands. TestUnwritableDevlogsRoot adds four covering both _run_dump branches against a devlogs root that genuinely cannot be written — the exact shape that would otherwise turn the pytest job red. 8 new tests in test/ci/invariants/test_common.py cover every fail/skip branch (49 there in total).
• black, flake8, mypy (667 files, no issues), pyright (0 errors), spec-lint all clean.

CI state

Python application, Spec Check, and Lint Markdown are green. Demo Integration is red, and every red job on it is red on main at the same base commit (2f1a9354) with the same root cause: 422 Unprocessable Content on trigger/engage-case, tracked in #2233. Nothing on this branch regressed a demo job.

That red is also the evidence this PR works. fvcv-handoff Invariant Harness used to die on artifact download without asserting anything; it now runs and reports 39 passed, 2 failed against the dumped ledgers, failing on a real protocol claim (Expected eventType 'engage_case' not found in case-actor log). fcvcv Invariant Harness is green even though fcvcv Demo Integration failed — the ledgers survived the failure, which is exactly what #2239 asked for.

Follow-ups already tracked

• Case-ledger dump is still skipped when a scenario dies before scenario_harness() is entered #2281 (filed from this PR's triage): the dump is still skipped when a scenario dies before scenario_harness() is entered — a health-check sys.exit(1), an import error, or a docker compose startup failure. The residual hole in DEMOCI-10-002; needs its own design pass because a tolerant download step alone would reintroduce the false-green skip DEMOCI-10-003 forbids.
• Pre-existing and independent: engage-case 422: participant RM state is read from the inline case copy but written to the standalone record #2233 (engage-case 422), Add demo_gate context manager and make gate tests exercise real control flow #2201/Migrate all 9 demo scenarios to causal gates and fix the latent fccv-handoff defects #2203 (demo_gate is documented in AGENTS.md, DEMOCI-01-007 and ADR-0058 but does not exist in code, leaving ~20 unguarded wait_for_case_participants calls), In-process scenario tests must assert on accumulated demo failures (and drop the FV stopgap) #2267 (in-process tests must assert on accumulated failures), Integration suite aborts on the global 5s thread timeout, masking the merge signal #2270 (global 5s thread timeout, now parented to Test & code infrastructure #2089), fv demo emits no validate_report eventType, so invariant 5 fails on a clean main #2273 and In-process FV demo test writes to the repo-root devlogs/, so a local unit run after the integration suite fails the FV invariant harness #2274 (devlogs pollution and the missing validate_report event). Integration suite aborts on the global 5s thread timeout, masking the merge signal #2270 and fv demo emits no validate_report eventType, so invariant 5 fails on a clean main #2273 had no parent epic and now do.

[sei-ahouseholder]

PR Triage: #2279 — fix: always dump case ledgers, even when a phase dies

Linked issues: #2239 (An escaping assertion destroys the scenario's ledger artifacts…)
Changed files: 25 files — demo/helpers, demo/scenario (all nine), test/ci/invariants, specs, metadata/specs
CI status: ❌ failing — 4 red jobs, of which 1 is PR-introduced and 3 are pre-existing on main
Merge state: ✅ mergeable (UNSTABLE — red non-required checks)
Base branch: main (up to date, merge-base 110da916)
Needs integration tests: yes

---

Findings

#	Phase	Severity	Description	Outcome
phase8-default-output-root-container-path-0	code-review	❌ FAIL	DEFAULT_OUTPUT_ROOT = "/app/devlogs" hardcodes the container path as the only non-container fallback — root cause of the red pytest job	fix-now
phase10-fv-test-missing-devlogs-dir-1	test-coverage	❌ FAIL	test_full_workflow_succeeds is the only touched test not pinning DEVLOGS_DIR to tmp_path, making its new exact-match assertion environment-dependent	fix-now
phase8-harness-manifest-replaces-cause-2	code-review	❌ FAIL	Bare write_dump_manifest() in the no-dump-registered branch lets a manifest error replace the scenario's exception — violates DEMOCI-10-004 / DEMOMA-23-004, the PR's headline guarantee	fix-now
phase8-fvcv-extension-late-dump-registration-3	code-review	❌ FAIL	fvcv_extension_demo registers the dump after an extra get_actor_by_id call (DEMOMA-23-003); the argument forcing the delay is unused	fix-now
phase11-ci-pytest-regression-4	linter-ci	❌ FAIL	Tests (pytest) red here, green on the main baseline → PR-introduced regression	fix-now
phase8-ensure-dump-manifest-masks-dump-error-5	code-review	⚠️ IMPROVE	ensure_dump_manifest backstop can overwrite the dump's real error	fix-now
phase8-three-devlogs-root-resolutions-6	code-review	⚠️ IMPROVE	Three independent resolutions of the devlogs root now coexist; one path yields a false-green skip	fix-now
phase8-nondict-manifest-silently-dropped-7	code-review	⚠️ IMPROVE	A null/non-dict manifest is dropped → skip instead of fail (DEMOCI-10-003)	fix-now
phase3-pr-body-verification-stale-8	pr-body-format	⚠️ IMPROVE	Verification claims "6949 tests, exit 0"; CI reports 1 failed / 7706 passed	fix-now
phase8-demo-name-duplicated-per-scenario-9	code-review	⚠️ IMPROVE	Demo name spelled twice per scenario; an overridden demo_name splits ledgers from the fallback manifest	fix-now
phase8-dump-skipped-before-harness-entry-10	code-review	⚠️ IMPROVE	Dump still skipped when a scenario dies before harness entry (health-check sys.exit(1), import error); download step has no if: always()	new-issue-no-ask

Total: 5 FAIL · 6 IMPROVE · 0 NEW-ISSUE (1 IMPROVE routed to a new issue)

---

The one PR-introduced failure

vultron/demo/helpers/ledger_dump.py:62 sets DEFAULT_OUTPUT_ROOT = "/app/devlogs" — a container-absolute path used whenever DEVLOGS_DIR is unset. The Tests (pytest) job runs on bare ubuntu-latest (no container), so mkdir(parents=True) cannot create /app and raises PermissionError: [Errno 13] Permission denied: '/app'.

Two things make this land as a red job rather than a silent no-op:

1. test_full_workflow_succeeds is the only test in the diff that does not monkeypatch.setenv("DEVLOGS_DIR", str(tmp_path)) — its three siblings at :1237, :1283, :1323 all do, and the PR's own new vultron/demo/AGENTS.md guidance says to.
2. The harness now honestly asserts the accumulated failures, so the four STEP FAILED: Dumping case ledger … entries join the list and the new exact-match assertion fails.

The container default is redundant regardless: docker/docker-compose-multi-actor.yml:299 already sets DEVLOGS_DIR=/app/devlogs explicitly, so the fallback only ever fires where it is wrong. vultron/demo/report.py:1103 already does this correctly (DEVLOGS_DIR, else _REPO_ROOT / "devlogs").

Baseline comparison — what is not this PR's fault

Against main run 31632495576 @ 65fe33f1:

Job	This PR	main	Verdict
Tests (pytest)	❌	✅	PR-introduced
fcvcv Demo Integration	❌	❌	pre-existing — 422 on /trigger/engage-case (#2229 / #2266)
fvcv-handoff Demo Integration	❌	❌	pre-existing — same 422
fvcv-handoff Invariant Harness	❌	❌	pre-existing — engage_case emitted by no code (#2243 decided, #2266 implements)

Worth stating plainly: fvcv-handoff Invariant Harness now fails on a real invariant assertion (2 failed, 39 passed) instead of dying on artifact download. That is exactly the outcome #2239 asked for — the red is expected until #2266 lands.

Verified clean

Black, flake8, mypy, pyright, spec-lint, CodeQL all pass. DEMOCI-10-001..004 and DEMOMA-23-001..004 are defined, cross-referenced and traceable into code, tests and AGENTS.md. Notes frontmatter valid; no ADR warranted (no new layer, protocol or persistence change). The nine scenario conversions are mechanically consistent apart from finding -3, both handoff scenarios' closures correctly pick up the reassigned case, and the failure accumulator is reset on harness entry so nothing leaks across scenarios in run_all.

---

Triage artifact: .claude/pr-2279-triage.json
Next step: /pr-execute applies the fixes.

[sei-ahouseholder]

PR Execute: #2279 — fix: always dump case ledgers, even when a phase dies

Fixes applied: 3 commits (817f1945, 67bb9d0d, bdc5442a)
Issues filed: 1 (#2281)
Deferred (awaiting your input): 0
Tests run: unit + integration
CI status after push: ❌ failing — Demo Integration only, and every red job on it is red on main at the same base commit
Base sync: ✅ merged main @ 2f1a9354 — 0 conflicts resolved

---

Fixed

Finding	Commit
phase8-default-output-root-container-path-0: DEFAULT_OUTPUT_ROOT = "/app/devlogs" hardcoded the container path	817f1945
phase10-fv-test-missing-devlogs-dir-1: test_full_workflow_succeeds did not pin DEVLOGS_DIR	817f1945
phase8-harness-manifest-replaces-cause-2: bare write_dump_manifest() could replace the scenario's exception	817f1945
phase8-fvcv-extension-late-dump-registration-3: dump registered after a network call	817f1945
phase11-ci-pytest-regression-4: Tests (pytest) red here, green on main	817f1945
phase8-ensure-dump-manifest-masks-dump-error-5: backstop manifest could displace the dump error	817f1945
phase8-three-devlogs-root-resolutions-6: three divergent devlogs-root resolutions	817f1945
phase8-nondict-manifest-silently-dropped-7: non-object manifest silently dropped	817f1945
phase8-demo-name-duplicated-per-scenario-9: demo name spelled twice per scenario	817f1945
phase3-pr-body-verification-stale-8: stale "6949 tests, exit 0" claim	PR body (untracked in git)

The first two are the same bug seen from both ends, and together they are the whole of the red pytest job. DEFAULT_OUTPUT_ROOT hardcoded the demo container's absolute path as the only fallback, so on a GitHub runner — runs-on: ubuntu-latest, no container: — every dump raised PermissionError. There is now one resolution, default_devlogs_root(): DEVLOGS_DIR when set, else a repo-root-relative default. vultron/demo/report.py and test/ci/invariants/common.py both delegate to it, so the writer, the reader and the invariant harness cannot disagree about where the ledgers live (DEMOMA-17-001).

_run_dump() is now total. Triage flagged the never-propagate invariant as untested — the existing tests only covered the registered-dump path against a writable tmp_path, and the early-death test accepted pytest.raises((RuntimeError, DemoFailureError)), which would have passed even if the harness swapped the cause. TestUnwritableDevlogsRoot adds four tests against a devlogs root that genuinely cannot be written (DEVLOGS_DIR under a regular file, so every mkdir raises), covering both _run_dump branches and asserting the body's own exception survives.

Filed as Issues

Finding	Issue
phase8-dump-skipped-before-harness-entry-10: dump still skipped when a scenario dies before scenario_harness() is entered	#2281

new-issue-no-ask — added to Project #24 (Schedule=Someday). It needs its own design pass: a tolerant download step alone would reintroduce the false-green skip DEMOCI-10-003 forbids.

Skipped

None.

---

Base sync

sync-with-main.sh main exited 0 — origin/main (2f1a9354, PR #2272) auto-merged cleanly. specs/demo-ci.yaml and notes/demo-ci-invariants.md were touched by both sides and merged without markers. One judgment call: both sides had independently bumped demo-ci.yaml from 1.7.0 to 1.8.0, so the merged file carried two changesets under one version. bdc5442a bumps it to 1.9.0 so a version identifies one registry state. spec-lint clean, DEMOCI-10 intact.

Tests

• Unit (uv run pytest): 6611 passed, 335 skipped, 2 xfailed, 5552 subtests passed, 1 failed, 89.7s.
• Integration (uv run pytest -m integration): 1129 tests, exit 0.
• xfail ratchet: clean — every XFAIL references an open issue (#1991, #1992, #1898, #1993), and the one XPASS references open #1994.
• black, flake8 (vultron/ test/), mypy (667 files), pyright (0 errors, full tree), spec-lint: all clean.

The single unit failure is test_fv_invariants.py::test_invariant_5_expected_event_types_present[validate_report], and it is not branch-owned. It reads the gitignored local devlogs/fv/ left behind by an earlier demo run — #2273/#2274. Control: with DEVLOGS_DIR=$(mktemp -d) the entire file skips. The path test/ci/invariants/common.py resolves is identical before and after this branch (_REPO_ROOT / "devlogs" → default_devlogs_root()), so no branch behaviour is involved, and Python application is green at bdc5442a on a fresh checkout.

CI state

Green: Python application, Spec Check, Lint Markdown.

Red: Demo Integration — fcvcv Demo Integration, fvcv-handoff Demo Integration, fvcv-handoff Invariant Harness. All three fail identically on main at 2f1a9354 (run 31653820183 vs this branch's 31654890058), all with 422 Unprocessable Content on /trigger/engage-case followed by RM.ACCEPTED / VFD replica timeouts. That root cause is tracked by open #2233. Note for the record: triage recorded #2243 and #2266 as the trackers; both are closed, so #2233 is the live one. Zero branch-owned CI failures.

That red is also the evidence this PR does what it claims. fvcv-handoff Invariant Harness used to die on artifact download without asserting anything; it now runs and reports 39 passed, 2 failed against the dumped ledgers, failing on a real protocol claim (Expected eventType 'engage_case' not found in case-actor log). And fcvcv Invariant Harness is green while fcvcv Demo Integration failed — the ledgers survived the failure, which is exactly what #2239 asked for.

Review threads

No inline review comments on this PR (gh api .../pulls/2279/comments → 0).

[sei-ahouseholder]

PR Verify: #2279 — fix: always dump case ledgers, even when a phase dies

Overall verdict: ❌ GAPS-FOUND — flag UNVERIFIED-CI-FAILING
CI status: ❌ failing — Demo Integration (3 jobs); all 17 other checks pass
Merge state: ✅ MERGEABLE (UNSTABLE) — base main
Base sync in execute: ✅ merged @ 67bb9d0d (0 conflicts resolved)
Integrity check: ✅ all 11 findings accounted for

---

Finding Verdicts

Finding	Severity	Outcome	Verdict
phase8-default-output-root-container-path-0	❌ FAIL	fixed @ 817f1945	✅ CONFIRMED
phase10-fv-test-missing-devlogs-dir-1	❌ FAIL	fixed @ 817f1945	✅ CONFIRMED
phase8-harness-manifest-replaces-cause-2	❌ FAIL	fixed @ 817f1945	✅ CONFIRMED
phase8-fvcv-extension-late-dump-registration-3	❌ FAIL	fixed @ 817f1945	✅ CONFIRMED
phase11-ci-pytest-regression-4	❌ FAIL	fixed @ 817f1945	✅ CONFIRMED
phase8-ensure-dump-manifest-masks-dump-error-5	⚠️ IMPROVE	fixed @ 817f1945	✅ CONFIRMED
phase8-three-devlogs-root-resolutions-6	⚠️ IMPROVE	fixed @ 817f1945	✅ CONFIRMED
phase8-nondict-manifest-silently-dropped-7	⚠️ IMPROVE	fixed @ 817f1945	✅ CONFIRMED
phase3-pr-body-verification-stale-8	⚠️ IMPROVE	fixed (PR body)	✅ CONFIRMED
phase8-demo-name-duplicated-per-scenario-9	⚠️ IMPROVE	fixed @ 817f1945	✅ CONFIRMED
phase8-dump-skipped-before-harness-entry-10	🎫 NEW-ISSUE	filed as #2281	📋 NOTED

Verified at HEAD (bdc5442a), not just in the commit diff:

• vultron/demo/helpers/ledger_dump.py — no /app/devlogs outside the docstring that explains why it must not be the default; default_devlogs_root() at :161, _REPO_ROOT from parents[3] at :65.
• test/demo/test_fv_demo.py — 4 monkeypatch.setenv("DEVLOGS_DIR", str(tmp_path)) sites, including test_full_workflow_succeeds.
• vultron/demo/helpers/harness.py — _backstop_manifest() at :109 with its own except BaseException, _run_dump() at :131 with the manifest write inside demo_step and an outer except BaseException.
• vultron/demo/scenario/fvcv_extension_demo.py — harness.dump_with( at :926 precedes get_actor_by_id(vendor2_client, …) at :941.
• vultron/demo/report.py:1106 and test/ci/invariants/common.py:51 both call default_devlogs_root().
• test/ci/invariants/common.py:153 pytest.fails on a non-object manifest.
• All 9 vultron/demo/scenario/*_demo.py pass demo_name=harness.demo_name.
• 817f1945 is on bug/2239-always-dump-ledgers; #2281 is OPEN.
• git grep for surviving conflict markers: clean.

---

❌ What blocks the verdict

UNVERIFIED-CI-FAILING. Demo Integration is red, and execute recorded integration_tests_run: true, so the gate requires that job green. It is not.

For the record, the analysis behind that red — which is why this is a gate to clear rather than a defect to fix on this branch:

• Failing jobs: fcvcv Demo Integration, fvcv-handoff Demo Integration, fvcv-handoff Invariant Harness.
• All three fail identically on main at the same base commit 2f1a9354 (run 31653820183 vs this branch's 31654890058), with the same root cause: 422 Unprocessable Content on /trigger/engage-case, then RM.ACCEPTED / VFD replica timeouts. Tracked by open #2233.
• Zero branch-owned CI failures. Tests (pytest) — the one job this PR did regress — is green at bdc5442a.

fvcv-handoff Invariant Harness red is also the deliverable working: it now reports 39 passed / 2 failed against dumped ledgers instead of dying on artifact download without asserting anything. fcvcv Invariant Harness is green while fcvcv Demo Integration failed — the ledgers survived the failure, which is precisely what #2239 asked for.

So the gate is honest but the PR is not the thing failing it. Clearing it needs a decision that is yours, not the pipeline's: land #2233 first, or accept a Demo Integration red that predates this branch.

---

Filed, not folded

Finding	Issue	Recommendation
phase8-dump-skipped-before-harness-entry-10	#2281	Leave for the issue — a tolerant download step alone would reintroduce the false-green skip DEMOCI-10-003 forbids, so it needs its own design pass

---

Artifacts preserved — verdict is GAPS-FOUND.