diff --git a/AGENTS.md b/AGENTS.md index b5aeadc2e..8203b83fd 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -143,7 +143,14 @@ six-step checklist (enum → pattern → use-case → map → tests). - **Inbox**: `vultron/adapters/driving/fastapi/routers/actors/` (package; `_routes.py` defines endpoints) - **Errors**: `vultron/errors.py` - **Demo**: `vultron/demo/cli.py` (entry point) -- **Case States**: `vultron/case_states/` — enums are authoritative +- **Case States**: `vultron/core/states/cs.py` — CS/VFD/PXA enums are + authoritative; `vultron/core/states/cs_invariants.py` holds the CS validity, + transition and history invariants (CSB-17). `vultron/core/case_states/` is the + legacy string-pattern reference model, retained as an independent oracle and + still imported by `states/cs.py` and `use_cases/query/action_rules.py`. Reach + for `cs_invariants.py` for new protocol-path work; the legacy module's only + remaining new-code use is as the oracle in the CSB-17 equivalence tests + (ADR-0060) Full core-layer map → [`vultron/core/AGENTS.md`](vultron/core/AGENTS.md). Full wire-layer map → [`vultron/wire/as2/AGENTS.md`](vultron/wire/as2/AGENTS.md). @@ -486,8 +493,7 @@ See [notes/agents-md-structure.md](notes/agents-md-structure.md) for routing pol and semantic-registry layers, and confirm against `graphify explain ""` call edges, before asserting absence. CONCERN-2243 filed a Concern on this basis for an event emitted by all nine - scenarios. - See also ISSUE-1784 (tracking the script fix). + scenarios. *Source: CONCERN-2243* - **`git rebase` "local changes would be overwritten" With a Clean Working Tree** — this error can be a false positive when the rebased branch diverges far from main and both sides touched the same files. Fix: cherry-pick onto a fresh branch @@ -495,7 +501,9 @@ See [notes/agents-md-structure.md](notes/agents-md-structure.md) for routing pol instead of rebasing. The error message is misleading — it is NOT evidence of uncommitted work. See also: single large-commit branches with 70+ files trigger a sequencer duplicate-pick bug; the cherry-pick workaround resolves both variants. - *Sources: ISSUE-1518, ISSUE-1504* + If `freshen-branch.sh` took this path and then hit a conflict, it can leave the + temp branch behind — delete it by hand (ISSUE-1784). + *Sources: ISSUE-1518, ISSUE-1504, ISSUE-1784* - **Verify Issue ACs Against Current Code Before Starting** — an issue may already be fully implemented by a prior PR that did not include a `Closes #N` footer. Check current `main` against all ACs before writing any code; if satisfied, close diff --git a/docs/adr/0060-re-express-legacy-cs-invariants.md b/docs/adr/0060-re-express-legacy-cs-invariants.md new file mode 100644 index 000000000..1a9c6e097 --- /dev/null +++ b/docs/adr/0060-re-express-legacy-cs-invariants.md @@ -0,0 +1,257 @@ +--- +status: accepted +date: 2026-08-12 +deciders: Allen D. Householder +consulted: [] +informed: [] +--- + +# Re-express the Legacy Case-State Invariants and Keep the Hypercube as Reference + +## Context and Problem Statement + +`vultron/core/case_states/` is the original implementation of the MPCVD +state-based model (CMU/SEI-2021-SR-021). It encodes the case-state (CS) rules as +**six-character strings and regular expressions** — `"vfdpxa"`, `"v..P.."` — +validated by `validations.py` and analysed by `hypercube.py` (a networkx / +numpy / pandas model that enumerates states, transitions, histories, and scores +them). The protocol implementation has since moved to typed enums (`CS`, +`CS_vfd`, `CS_pxa`), `pytransitions` machines, and per-machine dimension objects +(ADR-0036). The two worlds do not talk to each other. + +The legacy module holds rules the current models do not enforce anywhere. The +most consequential is `is_valid_history`, which makes validity a **causal** +property of an entire event sequence rather than a point-in-time state check or +a wall-clock timestamp comparison — exactly what CONCERN #2181 asked for. +`VFDPXA` and `VFDXAP` visit only valid states, yet only the first is a case +history any real case could have produced; nothing in the current code can tell +them apart. + +Two questions had to be settled together (issue #2237, which blocks #2236): + +1. Which legacy rules are still valid, which have been superseded, and which are + wrong — and how should the survivors be expressed against the current models? +2. What is the legacy module's status: keep, retire, or archive? + +Answering (2) required first checking the issue's stated premise that the module +is "completely orphaned". **It is not.** There are exactly two live importers +outside its own tree: + +- `vultron/core/use_cases/query/action_rules.py` imports + `vultron.core.case_states.patterns.potential_actions.action`, reached from the + live `actors_get_action_rules` FastAPI endpoint. +- `vultron/core/states/cs.py` imports `ensure_valid_state` from + `validations.py` and decorates four string helpers with it. + +The second is the load-bearing one: the *current* CS enum module depends on the +*legacy* validator, so "retire the legacy module" is not a delete — it is a +migration. + +## Decision Drivers + +- The rules, not the code, are the asset. The issue's own framing: treat + `case_states/` as the **specification of record**, and prefer a deliberate + line-by-line rewrite over `import and use`. +- Re-expression must be provably faithful. A rewrite that silently admits a + different rule set is worse than no rewrite, because it looks authoritative. +- Real case histories are usually **incomplete**; a rule family that only + validates all six events is not usable on live cases. +- Cross-machine rules (RM/EM × CS emit guards) belong to #2236, not here. +- `hypercube.py` pulls in networkx, numpy and pandas. Nothing on the protocol + path should acquire that dependency weight. +- The 500-line module guideline (CS-18-001): `cs.py` is already over the + guideline, so new code goes in a sibling module. `cs_invariants.py` itself + lands over it too; it is kept whole deliberately, because the rule family is a + single closed set of invariants whose only natural split — predicates apart + from the tables they read — would put a rule and its enforcement in different + files. Splitting is revisited if a second rule family lands here. + +## Considered Options + +- **A. Import and delegate** — have the current models call + `validations.is_valid_transition` / `is_valid_history`, converting enums to + strings at the boundary. +- **B. Re-express the surviving rules against the current enums; keep the legacy + module as the analytical model and as the test oracle.** +- **C. Re-express and retire** — rewrite, then delete `validations.py` and + `hypercube.py` in this change. +- **D. Archive the whole tree** — move `case_states/` out of `vultron/` to + `docs/` or a research repo, and rebuild any needed rule from the SEI report. + +## Decision Outcome + +Chosen option: **B — re-express the surviving rules in current idiom; keep the +legacy module, demoted to reference model and test oracle.** + +### The rule inventory + +Every rule in `validations.py` and the graph construction in `hypercube.py` was +assessed. No rule was found to be **wrong**; the verdicts split between *still +valid* and *superseded by structure*. + +| Legacy rule | Verdict | Where it now lives | +|---|---|---| +| `is_valid_state`: `vF` and `fD` are impossible → 32 valid states | Still valid, but **structural** | `CS_vfd` has only 4 members, so the impossible combinations are unrepresentable. Expressed as a ratchet test, not a runtime predicate. CSB-17-001 | +| `is_valid_transition`: Hamming distance 1 | Still valid, unenforced | `cs_transition_event()`, CSB-17-002 | +| `is_valid_transition`: monotone (no UC→lc), same dimension | Still valid, partly enforced per-dimension | Delegated to `is_valid_vfd_transition` / `is_valid_pxa_transition`; compound-level check in `is_valid_cs_transition()`, CSB-17-002 | +| `TRANSITION_RULES[1]`: `...pX. → ...PX.` | Still valid | `required_next_cs_events()`. Partly covered by CSB-13-001 (entry cascade); CSB-17-003 generalises it to every successor of a `pX` state | +| `TRANSITION_RULES[0]`: `v..P.. → V..P..` | Still valid, asserted by SM-09-001 as a persistence-time normalization but with no CS-behavior rule of its own | `required_next_cs_events()`, CSB-17-003, which `refines` SM-09-001 by restating it as a trajectory rule | +| `is_valid_history`: causal event ordering (`V≺F≺D`, `P≺X`/`XP`, `V≺P`/`PV`) | Still valid — **the most valuable rule in the module** | `is_valid_cs_history()` / `is_valid_cs_history_prefix()` / `replay_cs_history()`, CSB-17-004 | +| `is_valid_pattern` and the whole `.`-wildcard regex pattern language | **Superseded** | Enum membership tuples (`VFD_VENDOR_AWARE`, `PXA_EXPLOIT_PUBLIC`, …) and the `is_*` predicates already do this, type-safely | +| `hypercube.py` scoring, tf-idf, pagerank, `DESIDERATA`, adjacency matrices | Out of scope — analytical, not normative | Stays in `hypercube.py`; no protocol path needs it | + +The survivors are implemented in `vultron/core/states/cs_invariants.py` as +CSB-17, raising the current-idiom errors (`VultronInvalidStateTransitionError`, +`VultronValidationError`) rather than the legacy `CvdStateModelError` tree. + +### Two re-expression choices worth recording + +**Transitions delegate to the dimension machines instead of re-deriving +monotonicity.** `is_valid_cs_transition` identifies the single changed dimension +and hands the check to `is_valid_vfd_transition` / `is_valid_pxa_transition` — +the same tables `VfdDimension` / `PxaDimension` use. Monotonicity and the VFD +prerequisite chain then come for free and, more importantly, **cannot drift** +from what the dimension objects enforce. Only the two ephemeral rules are +genuinely new logic at the compound level. + +**History validity is expressed as causal replay, not as ordering predicates.** +The legacy formulation compares permutation indices; the re-expression replays +the sequence through the transition rule from `CS.vfdpxa`. The two are provably +equivalent — for a permutation of `VFDPXA`, replay succeeds iff `V≺F`, `F≺D`, +`index(P)−index(X) ≤ 1`, and `index(V)−index(P) ≤ 1`, and the two ephemeral +rules can never be active simultaneously (one needs `P` set, the other needs it +unset). Replay was chosen because it **generalises to prefixes**: real cases are +in progress, and `is_valid_cs_history_prefix` validates what has happened so far +without demanding all six events. The index-comparison formulation cannot do +that. The equivalence is asserted directly in the tests, so the two formulations +cannot silently diverge. + +### The legacy module's status: keep, demoted + +`validations.py` and `hypercube.py` are **retained**, with a changed role: + +- **Not** on the protocol path. `cs_invariants.py` is the runtime source of + truth for CS validity. New code MUST NOT import `case_states.validations`. +- **Reference model.** `hypercube.py` remains the derivation of the 32/58/70 + figures and the home of the analytical tooling (scoring, desiderata, pagerank) + that has no protocol counterpart and no reason to acquire one. +- **Test oracle.** `test/core/states/test_cs_invariants.py` compares the new + implementation against the legacy string implementation over the whole space — + 64 candidate states, 32×32 candidate transitions, all 720 permutations. This + is the strongest available evidence that the rewrite is faithful, and it only + works while both implementations exist. + +Retirement is therefore **deliberately deferred, not forgotten**, and is gated on +two migrations that are out of scope here: + +1. `vultron/core/states/cs.py` must stop importing `ensure_valid_state`. Its four + decorated string helpers (`vfd`, `pxa`, `state_string_to_enums`, + `state_string_to_enum2`) need an enum-native validator. +2. `vultron/core/use_cases/query/action_rules.py` must stop importing + `case_states.patterns.potential_actions` — which raises the separate question + of whether the `actors_get_action_rules` endpoint should be served from the + pattern language at all. + +Option A was rejected because it would make the enum models depend on +string-and-regex validation permanently, and would let the legacy error tree leak +into protocol paths — the opposite of the issue's explicit instruction to prefer +rewriting. Option C was rejected because deleting the legacy module in the same +change that rewrites it destroys the only independent oracle proving the rewrite +correct, and because two live importers make it a migration rather than a +deletion. Option D was rejected for the same reason plus the loss of the +analytical model, which is genuinely useful and genuinely non-normative. + +### Consequences + +- Good: the `vP → VP` rule is now enforceable against the current enums for the + first time. SM-09-001 already required it, but only at the persistence + boundary and only with the legacy string-pattern module as its implementation; + CSB-17-003 gives it a CS-behavior expression, and CSB-17-005 settles what it + means for an in-progress history (a prefix may end in `vP`; a persisted state + may not be `vP`). +- Good: history validity is available as a causal check on complete *and* + partial histories, which is what CONCERN #2181 needs. +- Good: compound-transition validity cannot drift from the dimension objects, + because it delegates to their tables rather than re-deriving them. +- Good: the equivalence tests fail loudly if either implementation changes, + making the legacy module useful precisely as long as it is still present. +- Neutral: two implementations of the same rules coexist. Acceptable because one + is explicitly non-normative and the tests pin them together. +- Neutral: `cs_invariants.py` is a library, not an enforcement point. Wiring it + into emit/BT paths is #2236's job; nothing calls it on the protocol path yet. +- Bad: the retirement of `case_states/` is now a recorded intention with two + named prerequisites rather than a completed act, so it can still rot if those + prerequisites are not tracked. + +## Validation + +- `test/core/states/test_cs_invariants.py` — exhaustive equivalence + against the legacy implementation: the `CS` enum equals the legacy 32-state + set; the new transition rule admits exactly the legacy 58 edges; the causal + replay admits exactly the legacy 70 histories; and `is_valid_cs_history` + agrees with the legacy result on all 720 permutations. +- Ephemeral-state coverage: the 12 `vP`/`pX` states are identified + independently of the predicate under test, and each is asserted to have + exactly one valid successor. +- Prefix coverage: every prefix of every valid history validates, and the + causally impossible prefixes (`F` first, `XA`, `PA`) are rejected. The + converse is pinned too — the accepted set is exactly the prefix-closure of + the 70 valid histories, so no accepted prefix is a dead end (CSB-17-005). +- Input coercion: because `CSEvent` is a `StrEnum`, the single-letter strings of + the legacy API are accepted on every entry point and produce identical + verdicts; the bool-returning predicates answer `False` for non-events rather + than raising. + +## Pros and Cons of the Options + +### A. Import and delegate to the legacy validators + +- Good, because it is the smallest change and cannot diverge from the legacy + rules by construction. +- Bad, because it makes the enum models permanently depend on string/regex + validation and on the legacy `CvdStateModelError` tree. +- Bad, because it cannot express prefix validity — `is_valid_history` requires + all six events, so live in-progress cases remain unvalidatable. +- Bad, because it contradicts the issue's explicit preference for a deliberate + rewrite over `import and use`. + +### B. Re-express; keep the legacy module as reference and oracle + +- Good, because the current models gain the rules in their own idiom, with their + own errors and their own types. +- Good, because the legacy implementation becomes an independent oracle that + proves the rewrite faithful over the entire state space. +- Good, because replay-based history validity generalises to prefixes. +- Neutral, because two implementations coexist until the migration completes. +- Bad, because retirement becomes a tracked intention rather than a done deed. + +### C. Re-express and retire the legacy module now + +- Good, because it leaves exactly one implementation of the rules. +- Bad, because it destroys the only independent evidence that the rewrite is + correct, at the moment that evidence is most needed. +- Bad, because two live importers make this a migration; `cs.py` itself depends + on `validations.ensure_valid_state`. +- Bad, because it would discard the analytical model (scoring, desiderata, + pagerank) that has no current-idiom replacement and needs none. + +### D. Archive the whole `case_states/` tree out of `vultron/` + +- Good, because it makes the non-normative status unmistakable. +- Bad, because it has all of C's problems plus the loss of the existing + `test/core/case_states/` suite. +- Bad, because the `actors_get_action_rules` endpoint would break immediately. + +## More Information + +The "completely orphaned" premise in issue #2237 is incorrect and the correction +matters for the decision — see `plan/incoming/learnings/`. Cross-machine +(RM/EM × CS) emit-guard enforcement is #2236, which this issue blocks. + +Source model: Householder, A. D., and Spring, J. *A State-Based Model for +Multi-Party Coordinated Vulnerability Disclosure (MPCVD)*, CMU/SEI-2021-SR-021, +. + +Generated spec requirements: `cs-behavior.yaml` CSB-17-001 through CSB-17-004. +Related: ADR-0036 (dimension objects), CSB-13-001 (pX→PX entry cascade), +CSB-16-001/002 (write-boundary transition validation). diff --git a/docs/adr/index.md b/docs/adr/index.md index a6d214d4b..cad572663 100644 --- a/docs/adr/index.md +++ b/docs/adr/index.md @@ -129,6 +129,7 @@ General information about architectural decision records is available at Note: `test/demo/test_pcr_late_joiner.py::test_late_joiner_receives_case_replica` +> and `test/metadata/test_decision_audit_inventory.py` were **not flaky tests** — +> they were honest 3.5-4.3s tests colliding with a 5s ceiling sized for the unit +> suite. Because `timeout_method = "thread"` kills the whole pytest process, +> `uv run pytest -m integration` aborted with **no summary line**, so a red +> integration run carried no information about the branch. Reliably red in random +> order (2/2 on clean `origin/main` 65fe33f1b); passed under `-p no:randomly`, +> which is what made it look like nondeterminism. Only *which* test tripped +> followed the `pytest-randomly` seed. +> +> **Fixed by #2270** — `test/conftest.py` now gives `integration`-marked tests a +> 60s tier while the unit suite runs at 30s (raised from 5s in the same issue, +> because AST-walking ratchets at ~3.4s were tripping the old ceiling under +> full-suite load). Verified 2/2 random-order runs at +> exit 0, 0 timeout aborts, 1101 passed. Never catalogued as flaky; rows added +> and removed in the same change (2026-08-12). +> +> **Lesson**: before adding a row here, ask whether the test is nondeterministic +> or whether the *ceiling* is wrong. A timeout tuned for one tier of tests will +> masquerade as flakiness in another. See also #2249 for the opposite error — +> cataloguing a deterministic protocol bug as noise. + +--- + ## CI / Demo Integration Jobs (job name granularity) | Job name | Issue | Last blocked | @@ -87,23 +115,51 @@ and fall through to Level 2 (GitHub label search). | `fcvcv Invariant Harness` | #2233 | 2026-08-13 | | `fcv-reject Invariant Harness` | #2233 (was #2121, closed) | 2026-08-13 | | `fv Invariant Harness` | #2233 | 2026-08-13 | - -> Some of these jobs fail intermittently due to inter-container HTTP delivery -> timeouts (async race windows). Root cause documented in +| `fv Demo Integration` | #2241 | 2026-08-13 | + +> `fv Demo Integration` is a different animal from both the #2233 rows and the +> async-race rows above it. It passed at `dc31b6c6` and failed at +> `0b607c11` — a docs-only diff — while base `fe951d00` does not fail it at all, +> so the trigger is genuinely intermittent. But the *failure* is deterministic +> once triggered: `add-note-to-case` returns an intermittent 422, and +> `vultron/demo/helpers/notes.py:92` then reads `result` outside the +> `with demo_step(...)` block that assigned it. `demo_step` suppresses the +> exception, so control falls through and raises +> `UnboundLocalError: cannot access local variable 'result'`, which buries the +> real 422 under a traceback pointing at the wrong line. Pre-existing base code +> (last touched by #1387, #543). +> +> **#2241 already owns this pattern** — "assignment inside a swallowing +> `demo_check` block then used after it" — so this row cites it rather than a new +> issue. The concrete callsite and run evidence are recorded there; note the +> pattern reaches `demo_step` too, not just `demo_check`. The related reporting +> failure is #2240. The `ValueError: No case ledger entries` later in the same run +> is *not* a second bug: `ledger_dump.py:434` raises it deliberately because the +> run died before any ledger was written. See also #2281. +> +> The rows with no issue number fail intermittently due to inter-container HTTP +> delivery timeouts (async race windows). Root cause documented in > `plan/incoming/learnings/` entry `20260731-async-race-windows-in-fv-demo.md`. > When a new occurrence is confirmed, `pr-execute` will open or comment on a > `flaky-test` + `bug` issue and record it here. > -> **The six rows pointing at #2233 are not flaky** — they fail on *every* run -> until the engage-case 422 lands, and they are listed here only because -> `pr-execute`'s dedup procedure looks here first. The Demo Integration pair -> fails on `SvcEngageCaseUseCase failed: TransitionParticipantRMtoAccepted`; the -> four Invariant Harness rows fail downstream of it on +> **The six rows pointing at #2233 are not flaky** — they are deterministic and +> branch-independent, failing on *every* run until the engage-case 422 lands. The +> Demo Integration pair fails on +> `SvcEngageCaseUseCase failed: TransitionParticipantRMtoAccepted`; the four +> Invariant Harness rows fail downstream of it on > `test_invariant_5_expected_event_types_present[engage_case]`, because the 422 -> aborts the trigger before `GuardedCommitCaseLedgerEntryBT` can record the -> entry that #2266 made universally required. Keep them in one row set: routing -> them to separate issues is what left #2216 and #2121 as stale pointers here -> after they were closed. +> aborts the trigger before `GuardedCommitCaseLedgerEntryBT` can record the entry +> that #2266 made universally required across all nine scenarios — turning one +> silent gap into a red `Invariant Harness` job per scenario. `origin/main` +> `06bf60c2` fails **15** of these jobs on its own, so a PR that fails a subset +> of them has not caused them. +> +> Keep them in one row set: routing them to separate issues is what left #2216 +> and #2121 as stale pointers here after they were closed. They are listed here +> at all because `pr-execute`'s dedup procedure looks here first and records +> blocked jobs regardless of cause. Do not re-diagnose them as nondeterminism, +> and do not "fix" them on a feature branch. --- diff --git a/plan/incoming/learnings/20260812-cs-hypercube-premise-was-wrong.md b/plan/incoming/learnings/20260812-cs-hypercube-premise-was-wrong.md new file mode 100644 index 000000000..44cdc0d3a --- /dev/null +++ b/plan/incoming/learnings/20260812-cs-hypercube-premise-was-wrong.md @@ -0,0 +1,62 @@ +--- +title: The "completely orphaned" premise in #2237 was wrong, and docs carried stale 64-state claims +timestamp: 2026-08-12T00:00:00Z +source: ISSUE-2237 +type: learning +signal: process-issue +--- + +# The "completely orphaned" premise in #2237 was wrong + +*Areas: case states, legacy code, documentation drift, specs.* + +## What happened + +Issue #2237 described `vultron/core/case_states/` as completely orphaned, +implying retirement was a deletion. Verification found **two live importers** +outside its own tree: + +- `vultron/core/use_cases/query/action_rules.py` imports + `case_states.patterns.potential_actions`, reached from the live + `actors_get_action_rules` FastAPI endpoint. +- `vultron/core/states/cs.py:26` imports `validations.ensure_valid_state` — the + authoritative module depends on the legacy one. + +That made retirement a **migration, not a deletion**, and drove ADR-0060's +"keep, demoted" decision with two named prerequisites instead of the archive the +issue anticipated. + +## Documentation drift found alongside it + +All four of these asserted a 64-state hypercube or a nonexistent path, and were +corrected in the same PR: + +- `notes/case-state-model.md` — "2^6 = 64-node hypercube"; the truth is 32 valid + states, 58 valid transitions, 70 valid complete histories of 720 permutations. +- `notes/codebase-structure.md` — listed a `vultron/case_states/enums/` directory + that does not exist. +- `notes/documentation-strategy.md` — three stale `vultron/case_states/` paths. +- `AGENTS.md` — Key Files Map had no entry for the authoritative + `vultron/core/states/cs.py`. + +## Also found: `references:` in specs YAML is silently dropped + +`specs/*.yaml` files accept a `references:` key that is **not** a field on +`StatementSpec` (`vultron/metadata/specs/schema.py`). It is silently discarded by +`spec-dump` with no lint error. Two pre-existing occurrences in +`specs/inbox-orchestration.yaml` are equally dead. The real field is `adr:`, +which `spec-lint` does validate against ADR filenames. + +## How to apply + +- Treat an issue's characterisation of code as a hypothesis. "Orphaned", + "unused", "dead" are claims to verify with an importer search before choosing + between deletion and migration — the answer changes the shape of the work. +- When touching a model documented in `notes/`, grep for the model's headline + numbers, not just its symbol names. "64" was wrong in prose that named no + symbol at all, so a symbol grep would have missed it. +- Adding a key to a spec YAML file proves nothing. Round-trip it through + `PYTHONPATH= uv run spec-dump` and confirm it appears, because unknown keys + vanish without complaint. + +Related: [[20260812-integration-timeout-tier]] diff --git a/plan/incoming/learnings/20260812-integration-timeout-tier.md b/plan/incoming/learnings/20260812-integration-timeout-tier.md new file mode 100644 index 000000000..2e6f61ab0 --- /dev/null +++ b/plan/incoming/learnings/20260812-integration-timeout-tier.md @@ -0,0 +1,87 @@ +--- +title: A timeout tuned for one test tier masquerades as flakiness in another +timestamp: 2026-08-12T00:00:00Z +source: ISSUE-2270 +type: learning +signal: tooling-issue +--- + +# A timeout tuned for one test tier masquerades as flakiness in another + +*Areas: testing, pytest, timeouts, flaky-test triage.* + +## What happened + +While validating #2237, `uv run pytest -m integration` exited 1 with two +`+++ Timeout +++` dumps and **no summary line**. Which test tripped it moved +with the `pytest-randomly` seed, and `-p no:randomly` made the suite pass — the +classic signature of a flaky test. + +It was not flaky. `timeout = 5` in `pyproject.toml` is sized for the unit suite, +but several integration tests do 3.5-4.3s of honest work. Because +`timeout_method = "thread"` kills the *whole pytest process* rather than the one +slow test, a single spurious trip aborted the session. At the suite level the +failure was **reliably red** (2/2 on clean `origin/main` 65fe33f1b); only the +location was nondeterministic. + +Fixed by #2270: a two-tier ceiling — 60s for `integration`-marked tests via +`test/conftest.py::apply_integration_timeout`, and the unit default raised from +5s to 30s in `pyproject.toml`. + +## This was diagnosed four times before it was fixed + +The same root cause was written up in three earlier learning files, each time as +a workaround rather than a fix: + +| File | Source | What it concluded | +|---|---|---| +| `20260803-pytest-full-suite-timeout.md` | ISSUE-1925 | full suite "never finishes"; worked around with scoped runs | +| `20260805-per-test-timeout-marginal-under-load.md` | ISSUE-1988 | AST ratchets at ~3.4s sit near the 5s ceiling; mitigated one ratchet with a prefilter | +| `20260808-pytest-thread-timeout-fakes-nondeterminism.md` | ISSUE-2086 | thread-method aborts fake nondeterminism; cost real time chasing phantoms | + +Three sessions correctly identified that `timeout = 5` plus +`timeout_method = "thread"` was the problem, and all three treated the ceiling as +fixed background. The unit tier was raised to 20s in this session only because +those three files were read together and the pattern became visible. + +Note the second file's finding is why widening only the integration tier would +have been an incomplete fix: the *unit* suite has AST-walking ratchets at ~3.4s, +so the 5s ceiling had a load-dependent margin there too. + +## Why it matters + +A red integration run carried **no information about the branch**. That is worse +than a failing test — it trains everyone to re-run rather than read, and it lets +a genuinely broken branch be waved off as "the usual timeout". + +The near-miss: the ready-made move was to add two rows to +`notes/flaky-tests.md` and proceed. That would have permanently catalogued a +config defect as noise, which is exactly the error #2249 was filed to correct in +the opposite direction. + +## How to apply + +- Before cataloguing a test as flaky, ask whether the test is nondeterministic + or whether the **ceiling** is wrong. Distinguish *suite reliably red* from + *test intermittently red*; if only the failure's location varies, suspect a + global resource limit, not the tests. +- `-p no:randomly` flipping a suite green is not evidence of flakiness. It is + evidence of order- or timing-sensitivity, which a too-tight global timeout + produces. +- Check for a summary line before concluding anything. A session aborted by + `timeout_method = "thread"` looks identical to a clean pass if you only count + `FAILED` lines. `notes/flaky-tests.md` already recorded this trap; it caught a + second victim anyway, so check for the `+++ Timeout +++` marker explicitly. +- Timeouts are diagnostics, not correctness invariants. When a tier fires on + honest work rather than catching hangs, change the tier — do not contort the + tests or paper over it with scattered `@pytest.mark.timeout(N)`. +- When widening a timeout, prefer widening the ceiling over switching + `timeout_method` off `thread`. The signal method cannot interrupt code blocked + in a C extension, so it converts a noisy abort into an invisible hang. +- **A recurring workaround is a signal to read the earlier write-ups, not to add + another.** Three learning files independently named this root cause and each + worked around it. Before writing a learning file about tooling friction, grep + `plan/incoming/learnings/` for the same symptom — if it is already there, the + finding is not the symptom, it is that nobody fixed the cause. + +Related: [[20260812-cs-hypercube-premise-was-wrong]] diff --git a/pyproject.toml b/pyproject.toml index aceb1f15c..10a597518 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -120,7 +120,29 @@ pythonpath = ["."] # # See AGENTS.md "Testing" section for full guidance. addopts = "-ra -q -m 'not integration'" -timeout = 5 +# Unit-tier per-test timeout. Raised 5 -> 30 in #2270. +# +# `timeout_method = "thread"` cannot cancel one test — it kills the whole pytest +# process, so a trip yields no summary line at all. That makes a too-tight +# ceiling actively harmful: it converts a slow test into an uninformative +# aborted session. At 5s the margin was thin enough that AST-walking +# architecture ratchets (~3.4s in isolation) tripped it nondeterministically +# under full-suite load, and four separate sessions re-diagnosed it as flakiness +# (ISSUE-1925, ISSUE-1988, ISSUE-2086, ISSUE-2237). +# +# Sized from measurement, not taste: the slowest unit test on an idle machine is +# ~3.1s (`test_real_specs_load` setup), so this is ~10x headroom. An +# intermediate value of 20s was tried and still tripped once while a background +# graphify rebuild competed for CPU — contention on a loaded dev box or CI +# runner inflates these well beyond their idle cost. +# +# A generous ceiling costs nothing on a genuine hang: that test was never going +# to finish, and 30s is still far below the CI job timeout. The suite stays fast +# because total runtime (~120s) is bounded by the tests, not by this ceiling. +# +# Integration tests get a wider tier; see `INTEGRATION_TIMEOUT_SECONDS` in +# `test/conftest.py`. +timeout = 30 timeout_method = "thread" testpaths = [ "test", diff --git a/specs/cs-behavior.yaml b/specs/cs-behavior.yaml index 48c9fe49e..7f4c946db 100644 --- a/specs/cs-behavior.yaml +++ b/specs/cs-behavior.yaml @@ -1453,3 +1453,218 @@ groups: spec_id: SDO-02-004 - rel_type: refines spec_id: BTND-10-001 + +- id: CSB-17 + title: CS Compound State and History Validity + description: >- + CS-internal validity invariants over the compound (VFD x PXA) case state + and over whole case histories. Where CSB-16 validates a single write at + the persistence boundary, CSB-17 defines which compound states exist, + which single-event transitions between them are causally possible, and + which orderings of the six CS events a real case could have produced. + Mined from the legacy `vultron.core.case_states` hypercube model and + re-expressed against the current CS enums; see ADR-0060. + specs: + - id: CSB-17-001 + priority: MUST + kind: protocol + statement: >- + An implementation MUST treat exactly 32 compound CS states as valid: the + cross product of the 4 valid VFD states (vfd, Vfd, VFd, VFD) and the 8 + PXA states. A fix MUST NOT be ready while the vendor is unaware (vF is + impossible) and a fix MUST NOT be deployed that was never ready (fD is + impossible). + rationale: >- + The VFD dimension is a linear chain, not a free product of three bits: + readiness presupposes awareness and deployment presupposes readiness. + Enumerating only the reachable combinations makes the two impossible + combinations unrepresentable rather than merely rejected at runtime, + which is why `CS_vfd` has four members and not eight. + testable: true + postconditions: + - description: The CS enum has exactly 32 members, one per valid compound state + relationships: + - rel_type: refines + spec_id: SM-09-002 + note: >- + SM-09-002 states the vF/fD prohibition as a rejection rule; this + refines it to a structural guarantee by enumerating only the 32 + reachable combinations, so the impossible ones are unrepresentable. + adr: + - ADR-0060 + + - id: CSB-17-002 + priority: MUST + kind: protocol + statement: >- + A CS transition MUST change exactly one of the six CS dimensions, and + MUST change it from the not-yet-happened value to the happened value. + Transitions that change two or more dimensions at once, that reverse a + dimension, or that skip a VFD prerequisite MUST be rejected. These three + conditions alone admit 72 transitions; applying the ephemeral-state rule + of CSB-17-003 as well leaves exactly 58 valid CS transitions. + rationale: >- + CS events are irreversible and are observed one at a time; a compound + state change of Hamming distance greater than one conflates two distinct + events and loses their ordering, which is exactly the information history + validity depends on. + testable: true + preconditions: + - description: A (source, destination) pair of compound CS states + steps: + - order: 1 + actor: validator + action: >- + Identify the single dimension that differs between source and + destination; reject if more than one differs. If none differs, the + write is a same-state re-assertion, which CSB-16-001 and CSB-16-002 + permit as a status confirmation: accept it only when the caller has + opted in (the allow_null parameter), and reject it otherwise, because a + transition validator's default answer for "no change" must be "that is + not a transition". + expected: A single CS event accounts for the change, or the caller has + explicitly asked for same-state writes to pass + - order: 2 + actor: validator + action: >- + Delegate to the changed dimension's own transition table + (is_valid_vfd_transition or is_valid_pxa_transition). + expected: Monotonicity and VFD prerequisite ordering are enforced + - order: 3 + actor: validator + action: >- + Apply the ephemeral-state rule of CSB-17-003 to the source state. + expected: A transition out of a vP or pX state fires only the event that + state requires + postconditions: + - description: Only Hamming-distance-1 monotone transitions are accepted + relationships: + - rel_type: constrains + spec_id: CSB-16-001 + note: >- + Adds the one-dimension-at-a-time and monotonicity constraints that a + per-dimension write validator cannot see. The same-state write CSB-16 + permits is preserved via the opt-in described in step 1. + - rel_type: constrains + spec_id: CSB-16-002 + - rel_type: depends_on + spec_id: CSB-17-003 + note: The 58-transition count holds only with the ephemeral rule applied. + adr: + - ADR-0060 + + - id: CSB-17-003 + priority: MUST + kind: protocol + statement: >- + An implementation MUST treat the compound states vP (public aware while + the vendor is unaware) and pX (exploit public while the public is + unaware) as ephemeral. From a vP state the next CS event MUST be V; from + a pX state the next CS event MUST be P. Any other next event MUST be + rejected. + rationale: >- + Neither condition can persist in reality: public awareness reaches the + vendor, and a published exploit makes the public aware. Modelling these + as ephemeral rather than invalid preserves the causal ordering of the + events while forbidding the case from progressing in any other direction + first. The pX ephemeral rule is the pX to PX invariant of CSB-13-001 + generalised from the entry cascade to every successor of a pX state. + testable: true + preconditions: + - description: The case is in one of the 12 ephemeral compound CS states + postconditions: + - description: Each ephemeral state has exactly one valid successor state + relationships: + - rel_type: extends + spec_id: CSB-13-001 + - rel_type: refines + spec_id: SM-09-001 + note: >- + SM-09-001 governs the persistence boundary: an incoming status write + carrying vP or pX MUST be promoted to the forced form before it is + stored. CSB-17-003 governs trajectory validation, where the same two + conditions are modelled as ephemeral states with exactly one legal + successor. The two are consistent because they apply at different + boundaries: no vP or pX state is ever persisted, but such a state is + still a legal intermediate point in a replayed history, and CSB-17-004 + therefore permits a history prefix to end in one. + adr: + - ADR-0060 + + - id: CSB-17-004 + priority: SHOULD + kind: protocol + statement: >- + A consumer validating a reported CS history SHOULD validate it causally + by replaying the event sequence through CSB-17-002 and CSB-17-003 from + the initial state vfdpxa, rather than checking states point-in-time or + comparing wall-clock timestamps. A complete history contains all six CS + events exactly once; there are exactly 70 valid complete histories out of + 720 orderings. + rationale: >- + Validity of a case trajectory is a property of the whole ordered + sequence, not of any single state: VFDPXA and VFDXAP visit only valid + states, but only the first is a possible case history. Wall-clock + timestamps cannot substitute, because they may be missing, skewed, or + recorded by different participants. SHOULD rather than MUST because a + consumer holding only a current state, with no event log to replay, + cannot do better. + testable: true + preconditions: + - description: An ordered sequence of CS events, possibly incomplete + steps: + - order: 1 + actor: validator + action: Reject any event that occurs more than once in the sequence + expected: Each CS event happens at most once + - order: 2 + actor: validator + action: >- + Replay the sequence from vfdpxa, requiring each step to satisfy + CSB-17-002 and CSB-17-003 given every event that preceded it. + expected: Causally impossible orderings are rejected + postconditions: + - description: A complete valid history terminates in VFDPXA + adr: + - ADR-0060 + + - id: CSB-17-005 + priority: MUST + kind: protocol + statement: >- + An incomplete CS history MUST be validated as a prefix: validation MUST + NOT require all six CS events, and a prefix MAY end in an ephemeral + state. The accepted prefixes are exactly the prefixes of the 70 valid + complete histories of CSB-17-004, so every accepted prefix MUST have at + least one completion. + rationale: >- + Real cases are usually still in progress, so requiring six events would + reject every live case. Ending in an ephemeral state is unavoidable at the + prefix level: CSB-17-003's forced successor has not happened yet when the + prefix is cut, which is different from persisting an ephemeral state — + SM-09-001 still forbids that. Pinning acceptance to CSB-17-004's prefix + closure keeps both directions honest: a validator that merely never + rejects a genuine prefix could still accept dead ends. + testable: true + preconditions: + - description: An ordered sequence of CS events with fewer than six entries + steps: + - order: 1 + actor: validator + action: >- + Apply CSB-17-004's replay to the sequence without requiring that it + terminate in VFDPXA. + expected: >- + The prefix is accepted iff it is a prefix of at least one valid + complete history + postconditions: + - description: >- + A prefix ending in an ephemeral state is accepted, and its forced + successor per CSB-17-003 is the only legal continuation + relationships: + - rel_type: refines + spec_id: CSB-17-004 + - rel_type: depends_on + spec_id: CSB-17-003 + adr: + - ADR-0060 diff --git a/specs/state-machine.yaml b/specs/state-machine.yaml index 897227150..2aa485a36 100644 --- a/specs/state-machine.yaml +++ b/specs/state-machine.yaml @@ -157,9 +157,11 @@ groups: statement: 'The CS state machine MUST enforce two forced (ephemeral) transitions at validation time: (1) `pX → PX` — exploit-public forces public-aware; (2) `vP → VP` — public-aware forces vendor-aware. Any incoming CS state that contains `pX` or `vP` MUST be promoted to the forced form before being persisted.' - rationale: These transitions are protocol-correctness invariants, not optional normalizations. A CS state of `pXa` (exploit + rationale: 'These transitions are protocol-correctness invariants, not optional normalizations. A CS state of `pXa` (exploit public, public unaware) is semantically impossible — the exploit being public implies the vulnerability is public. Enforced - by `vultron/core/case_states/validations.py` `TRANSITION_RULES`. + by `vultron/core/states/cs_invariants.py` (`is_ephemeral_cs_state`, `required_next_cs_events`). This requirement governs + the *persistence* boundary only; CSB-17-003 and CSB-17-005 cover trajectory validation, where an ephemeral state is a + legal prefix terminus.' relationships: - rel_type: implements spec_id: VP-14-001 diff --git a/test/AGENTS.md b/test/AGENTS.md index 1b424f1cf..99a803d0f 100644 --- a/test/AGENTS.md +++ b/test/AGENTS.md @@ -43,11 +43,48 @@ uv run pytest test/test_semantic_activity_patterns.py -v ### Per-Test Timeout Guardrail -Default 5-second timeout (`pytest-timeout`, `pyproject.toml`). When a test -trips it: mock slow deps, avoid `time.sleep()`, restructure integration tests. +Timeouts are **two-tier** (`pytest-timeout`): + +| Tier | Ceiling | Set in | +|---|---|---| +| Unit (default) | 30s | `timeout = 30`, `pyproject.toml` | +| `@pytest.mark.integration` | 60s | `INTEGRATION_TIMEOUT_SECONDS`, `test/conftest.py` | + +`test/conftest.py::apply_integration_timeout` widens the ceiling for +integration-marked tests at collection time. An explicit +`@pytest.mark.timeout(N)` on a test always wins over the tier default. + +**Why these numbers** (#2270): `timeout_method = "thread"` kills the *whole +pytest process*, not the one slow test, so a trip produces **no summary line**. +A too-tight ceiling therefore does not surface a slow test — it converts the run +into an uninformative abort. Both tiers used to be 5s, which was thin enough +that honest work tripped it under load: + +- integration tests doing 3.5-4.3s of real HTTP work, and +- AST-walking architecture ratchets at ~3.4s in isolation. + +Four separate sessions re-diagnosed the result as flakiness (ISSUE-1925, +ISSUE-1988, ISSUE-2086, ISSUE-2237) before the ceiling itself was fixed. Raising +it costs nothing on a genuine hang — that test was never going to finish — and +the suite stays fast because total runtime is bounded by the tests, not by this +ceiling. + +Both tiers are sized from measurement: the slowest unit test is ~3.1s idle and +the slowest integration test ~4.3s. The headroom is deliberately large because +contention (a CI runner, or a background graphify rebuild) inflates these well +beyond their idle cost — an intermediate unit value of 20s was tried and still +tripped once under exactly that. + +When a test trips its tier: mock slow deps, avoid `time.sleep()`, or move it +behind the `integration` marker if it really does exercise the full stack. `@pytest.mark.timeout(N)` is a last resort and MUST have a comment explaining why. Do not use it to paper over slow tests. +A timeout ceiling is a diagnostic tool, not a correctness invariant — if a tier +is firing on honest work rather than catching hangs, change the tier rather +than contorting the tests around it. Do not add a row to +`notes/flaky-tests.md` for a test that is merely near its ceiling. + --- ### `monkeypatch.undo()` MUST Precede `reload_config()` in Fixture Teardown diff --git a/test/conftest.py b/test/conftest.py index 5c5a7925a..f06c4ca03 100644 --- a/test/conftest.py +++ b/test/conftest.py @@ -20,6 +20,12 @@ Also registers the ``spec`` pytest marker and validates spec IDs referenced by ``@pytest.mark.spec`` against the loaded SpecRegistry (SR-05-001, SR-05-002). + +Finally, it applies the integration-tier per-test timeout. The 30-second +default in ``pyproject.toml`` is sized for the unit suite; integration tests +exercise the full HTTP stack and legitimately need longer. See +``INTEGRATION_TIMEOUT_SECONDS`` and ``test/AGENTS.md`` § "Per-Test Timeout +Guardrail". """ import os @@ -39,6 +45,21 @@ warn_unknown_spec_id, ) +#: Per-test timeout for ``@pytest.mark.integration`` tests, in seconds. +#: +#: The global ``timeout = 30`` in ``pyproject.toml`` is sized for the unit +#: suite, where the slowest honest test runs at ~3.1s. It is still too tight +#: for integration tests: several run at 3.5-4.3s of honest work against a +#: much wider load-dependent spread, and because ``timeout_method = "thread"`` +#: kills the *whole pytest process* rather than the one slow test, a single +#: spurious trip aborted the session with no summary line — turning a red +#: integration run into no signal at all. See issue #2270. +#: +#: 60s is still a bounded hang detector (2x the unit ceiling) while leaving +#: ample headroom over the slowest honest integration test. Tests needing more +#: keep their own explicit ``@pytest.mark.timeout(N)``, which wins over this. +INTEGRATION_TIMEOUT_SECONDS = 60 + def pytest_configure(config): """Register the ``spec`` marker (SR-05-001).""" @@ -48,13 +69,36 @@ def pytest_configure(config): ) +def apply_integration_timeout(items): + """Give ``integration``-marked tests the integration-tier timeout. + + Applied to every item marked ``integration`` that does not already carry + an explicit ``timeout`` marker. An explicit marker always wins, so the + deliberate per-test values in the demo suite are left untouched. + + Returns the number of items modified (for tests and diagnostics). + """ + modified = 0 + for item in items: + if item.get_closest_marker("integration") is None: + continue + if item.get_closest_marker("timeout") is not None: + continue + item.add_marker(pytest.mark.timeout(INTEGRATION_TIMEOUT_SECONDS)) + modified += 1 + return modified + + def pytest_collection_modifyitems(session, config, items): - """Warn for unknown spec IDs in ``@pytest.mark.spec`` markers (SR-05-002). + """Apply the integration timeout, then warn for unknown spec IDs. - Emits :class:`~vultron.metadata.specs.UnknownSpecIdWarning` (non-blocking) - for any marker that references a spec ID not found in the registry. - Skips silently when no YAML files exist in ``specs/`` (e.g., before SR.6). + Spec-ID warnings (SR-05-002) emit + :class:`~vultron.metadata.specs.UnknownSpecIdWarning` (non-blocking) for + any ``@pytest.mark.spec`` marker referencing an ID not found in the + registry. Skips silently when no YAML files exist in ``specs/``. """ + apply_integration_timeout(items) + spec_dir = Path(__file__).parent.parent / "specs" if not spec_dir.is_dir(): return diff --git a/test/core/states/test_cs_invariants.py b/test/core/states/test_cs_invariants.py new file mode 100644 index 000000000..0e238c395 --- /dev/null +++ b/test/core/states/test_cs_invariants.py @@ -0,0 +1,743 @@ +#!/usr/bin/env python +# Copyright (c) 2026 Carnegie Mellon University and Contributors. +# - see Contributors.md for a full list of Contributors +# - see ContributionInstructions.md for information on how you can Contribute to this project +# Vultron Multiparty Coordinated Vulnerability Disclosure Protocol Prototype is +# licensed under a MIT (SEI)-style license, please see LICENSE.md distributed +# with this Software or contact permission@sei.cmu.edu for full terms. +# Created, in part, with funding and support from the United States Government +# (see Acknowledgments file). This program may include and/or can make use of +# certain third party source code, object code, documentation and other files +# ("Third Party Software"). See LICENSE.md for more details. +# Carnegie Mellon®, CERT® and CERT Coordination Center® are registered in the +# U.S. Patent and Trademark Office by Carnegie Mellon University + +"""Tests for `vultron.core.states.cs_invariants`. + +The exhaustive tests here are the regression anchor for the re-expression of +the legacy `vultron.core.case_states` rules against the current enum models +(issue #2237, ADR-0060). They compare the new implementation against the +legacy string-pattern implementation over the *whole* state space — 64 +candidate states, 32x32 candidate transitions, and all 720 event +permutations — so any drift between the two is caught immediately, and the +legacy module can be retired only once these comparisons are deliberately +removed. +""" + +from itertools import permutations + +import pytest + +from vultron.core.case_states.validations import ( + is_valid_history as legacy_is_valid_history, + is_valid_state as legacy_is_valid_state, + is_valid_transition as legacy_is_valid_transition, +) +from vultron.core.states.cs import ( + CS, + CS_pxa, + CS_vfd, + PXA_Trigger, + VFD_Trigger, + is_valid_pxa_transition, + is_valid_vfd_transition, + is_vfd_vendor_aware, +) +from vultron.core.states.cs_invariants import ( + CS_EVENT_TO_PXA_TRIGGER, + CS_EVENT_TO_VFD_TRIGGER, + CS_EVENTS, + CSEvent, + PXA_EVENTS, + VFD_EVENTS, + apply_cs_event, + cs_dimensions, + cs_from_dimensions, + cs_transition_event, + ensure_valid_cs_history, + ensure_valid_cs_transition, + is_ephemeral_cs_state, + is_valid_cs_history, + is_valid_cs_history_prefix, + is_valid_cs_transition, + next_cs_states, + replay_cs_history, + required_next_cs_events, + valid_cs_histories, +) +from vultron.errors import ( + ValidationError as LegacyValidationError, + VultronInvalidStateTransitionError, + VultronValidationError, +) + +# --- helpers --------------------------------------------------------------- + + +def _all_candidate_state_strings() -> list[str]: + """Every one of the 2**6 vfdpxa letter combinations, valid or not.""" + combos = [] + for i in range(2**6): + bits = format(i, "06b") + combos.append( + "".join( + letter.upper() if bit == "1" else letter + for letter, bit in zip("vfdpxa", bits) + ) + ) + return combos + + +def _legacy_valid_states() -> set[str]: + valid = set() + for candidate in _all_candidate_state_strings(): + try: + legacy_is_valid_state(candidate) + except LegacyValidationError: + continue + valid.add(candidate) + return valid + + +def _legacy_valid_transitions() -> set[tuple[str, str]]: + states = sorted(_legacy_valid_states()) + edges = set() + for src in states: + for dst in states: + try: + legacy_is_valid_transition(src, dst) + except LegacyValidationError: + continue + edges.add((src, dst)) + return edges + + +def _legacy_valid_histories() -> set[str]: + valid = set() + for perm in permutations("VFDPXA"): + history = "".join(perm) + try: + legacy_is_valid_history(history) + except LegacyValidationError: + continue + valid.add(history) + return valid + + +def _as_string(history) -> str: + return "".join(event.value for event in history) + + +# --- CSEvent --------------------------------------------------------------- + + +def test_cs_events_are_canonical_order(): + assert _as_string(CS_EVENTS) == "VFDPXA" + + +def test_event_dimension_partition(): + assert VFD_EVENTS | PXA_EVENTS == set(CS_EVENTS) + assert not VFD_EVENTS & PXA_EVENTS + + +def test_event_trigger_maps_cover_their_dimensions(): + assert set(CS_EVENT_TO_VFD_TRIGGER) == VFD_EVENTS + assert set(CS_EVENT_TO_PXA_TRIGGER) == PXA_EVENTS + assert set(CS_EVENT_TO_VFD_TRIGGER.values()) == set(VFD_Trigger) + assert set(CS_EVENT_TO_PXA_TRIGGER.values()) == set(PXA_Trigger) + + +@pytest.mark.parametrize( + "event,trigger", + [ + (CSEvent.V, VFD_Trigger.V), + (CSEvent.F, VFD_Trigger.F), + (CSEvent.D, VFD_Trigger.D), + ], +) +def test_vfd_trigger_map_pairs_by_letter(event, trigger): + assert CS_EVENT_TO_VFD_TRIGGER[event] is trigger + + +@pytest.mark.parametrize( + "event,trigger", + [ + (CSEvent.P, PXA_Trigger.P), + (CSEvent.X, PXA_Trigger.X), + (CSEvent.A, PXA_Trigger.A), + ], +) +def test_pxa_trigger_map_pairs_by_letter(event, trigger): + assert CS_EVENT_TO_PXA_TRIGGER[event] is trigger + + +# --- compound state validity ---------------------------------------------- + + +@pytest.mark.spec("CSB-17-001") +def test_cs_enum_is_exactly_the_legacy_valid_state_set(): + """The 32 CS members are the legacy model's 32 valid states. + + This is the state-validity rule (`vF` and `fD` are impossible) expressed + structurally: `CS_vfd` has only four members, so the impossible + combinations cannot be constructed at all. + """ + assert {state.name for state in CS} == _legacy_valid_states() + + +def test_cs_has_32_states(): + assert len(list(CS)) == 32 + + +def test_impossible_vfd_combinations_are_not_constructible(): + """No CS_vfd member has F without V, or D without F.""" + for state in CS_vfd: + vendor_aware, fix_ready, fix_deployed = ( + char.isupper() for char in state.name + ) + assert not (fix_ready and not vendor_aware) + assert not (fix_deployed and not fix_ready) + + +def test_dimension_round_trip(): + for state in CS: + assert cs_from_dimensions(*cs_dimensions(state)) is state + + +def test_cs_from_dimensions_covers_the_full_cross_product(): + pairs = { + cs_from_dimensions(vfd_state, pxa_state) + for vfd_state in CS_vfd + for pxa_state in CS_pxa + } + assert pairs == set(CS) + + +# --- ephemeral states ----------------------------------------------------- + + +def test_ephemeral_states_are_the_twelve_vp_and_px_states(): + ephemeral = {state.name for state in CS if is_ephemeral_cs_state(state)} + expected = { + state.name + for state in CS + # vP: public aware, vendor unaware + if (state.name[3] == "P" and state.name[0] == "v") + # pX: exploit public, public unaware + or (state.name[4] == "X" and state.name[3] == "p") + } + assert ephemeral == expected + assert len(ephemeral) == 12 + + +@pytest.mark.parametrize( + "state,expected", + [ + (CS.vfdPxa, {CSEvent.V}), + (CS.vfdPXA, {CSEvent.V}), + (CS.vfdpXa, {CSEvent.P}), + (CS.VFDpXA, {CSEvent.P}), + (CS.vfdpxa, set()), + (CS.VfdPxa, set()), + (CS.VFDPXA, set()), + ], +) +@pytest.mark.spec("CSB-17-003") +def test_required_next_cs_events(state, expected): + assert required_next_cs_events(state) == frozenset(expected) + + +def test_the_two_ephemeral_rules_are_mutually_exclusive(): + """vP requires P set; pX requires P unset — they cannot both apply.""" + for state in CS: + required = required_next_cs_events(state) + assert len(required) <= 1 + + +def test_ephemeral_states_have_exactly_one_successor(): + for state in CS: + if is_ephemeral_cs_state(state): + assert len(next_cs_states(state)) == 1 + + +# --- transition validity -------------------------------------------------- + + +@pytest.mark.spec("CSB-17-002") +def test_transition_set_matches_legacy_exactly(): + """The re-expressed transition rule admits the legacy model's 58 edges.""" + new_edges = { + (src.name, dst.name) + for src in CS + for dst in CS + if is_valid_cs_transition(src, dst) + } + assert new_edges == _legacy_valid_transitions() + + +def test_there_are_58_valid_transitions(): + count = sum( + 1 for src in CS for dst in CS if is_valid_cs_transition(src, dst) + ) + assert count == 58 + + +def _satisfies_structural_conditions(src: CS, dst: CS) -> bool: + """Conditions 1-3 of CSB-17-002, *without* the ephemeral rule (condition 4). + + Distance-1, monotone, and permitted by the changed dimension's own table — + but indifferent to whether *src* is ephemeral. + """ + if src is dst: + return False + event = cs_transition_event(src, dst) + if event is None: + return False + + src_vfd, src_pxa = cs_dimensions(src) + dst_vfd, dst_pxa = cs_dimensions(dst) + if event in VFD_EVENTS: + return src_pxa is dst_pxa and is_valid_vfd_transition(src_vfd, dst_vfd) + return src_vfd is dst_vfd and is_valid_pxa_transition(src_pxa, dst_pxa) + + +@pytest.mark.spec("CSB-17-002") +def test_structural_conditions_alone_admit_72_transitions(): + """Pins CSB-17-002's attribution of the 72 -> 58 reduction. + + The 58 figure needs all four conditions. Conditions 1-3 alone admit 72; + the ephemeral rule of CSB-17-003 removes the other 14. CSB-17-002's + statement restates both counts, and a restated count next to a + requirement reference drifts silently unless something asserts it + (see `20260812-restated-counts-in-spec-cross-references-drift.md`). + """ + structural = { + (src, dst) + for src in CS + for dst in CS + if _satisfies_structural_conditions(src, dst) + } + valid = { + (src, dst) + for src in CS + for dst in CS + if is_valid_cs_transition(src, dst) + } + + assert len(structural) == 72 + assert len(valid) == 58 + + # The 14 removed edges all leave an ephemeral state by the wrong event. + assert valid < structural + assert all(is_ephemeral_cs_state(src) for src, _ in structural - valid) + + +def test_every_transition_changes_exactly_one_dimension(): + for src in CS: + for dst in next_cs_states(src): + assert cs_transition_event(src, dst) is not None + + +def test_transitions_are_monotone(): + """No transition ever un-sets a bit; CS events are irreversible.""" + for src in CS: + for dst in next_cs_states(src): + for before, after in zip(src.name, dst.name): + assert not (before.isupper() and after.islower()) + + +def test_null_transition_rejected_by_default_and_allowed_when_asked(): + assert not is_valid_cs_transition(CS.Vfdpxa, CS.Vfdpxa) + assert is_valid_cs_transition(CS.Vfdpxa, CS.Vfdpxa, allow_null=True) + + +def test_ensure_valid_cs_transition_allows_null_when_asked(): + ensure_valid_cs_transition(CS.Vfdpxa, CS.Vfdpxa, allow_null=True) + + +@pytest.mark.parametrize( + "src,dst", + [ + (CS.vfdpxa, CS.Vfdpxa), # V + (CS.Vfdpxa, CS.VFdpxa), # F + (CS.VFdpxa, CS.VFDpxa), # D + (CS.Vfdpxa, CS.VfdPxa), # P + (CS.VfdPxa, CS.VfdPXa), # X + (CS.VfdPxa, CS.VfdPxA), # A + (CS.vfdPxa, CS.VfdPxa), # ephemeral vP resolved by V + (CS.vfdpXa, CS.vfdPXa), # ephemeral pX resolved by P + ], +) +def test_valid_transitions(src, dst): + assert is_valid_cs_transition(src, dst) + ensure_valid_cs_transition(src, dst) + + +@pytest.mark.parametrize( + "src,dst,reason", + [ + (CS.Vfdpxa, CS.vfdpxa, "not monotone"), + (CS.vfdpxa, CS.VfdPxa, "two dimensions change"), + (CS.vfdpxa, CS.VFdpxa, "F requires V first"), + (CS.Vfdpxa, CS.VFDpxa, "F and D cannot both fire at once"), + (CS.vfdpxa, CS.VFDPXA, "everything changes at once"), + (CS.vfdPxa, CS.vfdPxA, "vP requires V next"), + (CS.vfdPxa, CS.vfdPXa, "vP requires V next"), + (CS.vfdpXa, CS.vfdpXA, "pX requires P next"), + (CS.VFDpXa, CS.VFDpXA, "pX requires P next"), + ], +) +def test_invalid_transitions(src, dst, reason): + assert not is_valid_cs_transition( + src, dst + ), f"{src.name} -> {dst.name} should be rejected: {reason}" + with pytest.raises(VultronInvalidStateTransitionError): + ensure_valid_cs_transition(src, dst) + + +def test_ephemeral_rejection_message_names_the_required_event(): + with pytest.raises(VultronInvalidStateTransitionError) as exc_info: + ensure_valid_cs_transition(CS.vfdpXa, CS.vfdpXA) + message = str(exc_info.value) + assert "ephemeral" in message + assert "'P'" in message + + +def test_terminal_state_has_no_successors(): + assert next_cs_states(CS.VFDPXA) == () + + +def test_initial_state_successors(): + """From vfdpxa only V, P, X and A can fire — F and D need prerequisites.""" + events = { + cs_transition_event(CS.vfdpxa, dst) + for dst in next_cs_states(CS.vfdpxa) + } + assert events == {CSEvent.V, CSEvent.P, CSEvent.X, CSEvent.A} + + +def test_cs_transition_event_returns_none_for_non_unit_diffs(): + assert cs_transition_event(CS.vfdpxa, CS.vfdpxa) is None + assert cs_transition_event(CS.vfdpxa, CS.VfdPxa) is None + + +def test_cs_transition_event_identifies_the_changed_dimension(): + assert cs_transition_event(CS.vfdpxa, CS.Vfdpxa) is CSEvent.V + assert cs_transition_event(CS.vfdpxa, CS.vfdpxA) is CSEvent.A + + +# --- apply_cs_event ------------------------------------------------------- + + +def test_apply_cs_event_advances_the_state(): + assert apply_cs_event(CS.vfdpxa, CSEvent.V) is CS.Vfdpxa + assert apply_cs_event(CS.Vfdpxa, CSEvent.F) is CS.VFdpxa + assert apply_cs_event(CS.vfdpXa, CSEvent.P) is CS.vfdPXa + + +def test_apply_cs_event_agrees_with_the_transition_predicate(): + for state in CS: + for event in CS_EVENTS: + try: + result = apply_cs_event(state, event) + except VultronInvalidStateTransitionError: + assert not any( + cs_transition_event(state, dst) is event + for dst in next_cs_states(state) + ) + continue + assert is_valid_cs_transition(state, result) + assert cs_transition_event(state, result) is event + + +def test_apply_cs_event_rejects_a_repeated_event(): + with pytest.raises(VultronInvalidStateTransitionError, match="already"): + apply_cs_event(CS.Vfdpxa, CSEvent.V) + + +def test_apply_cs_event_rejects_an_unmet_prerequisite(): + with pytest.raises( + VultronInvalidStateTransitionError, match="prerequisite" + ): + apply_cs_event(CS.vfdpxa, CSEvent.F) + + +def test_apply_cs_event_rejects_a_violated_ephemeral_rule(): + with pytest.raises(VultronInvalidStateTransitionError, match="ephemeral"): + apply_cs_event(CS.vfdpXa, CSEvent.A) + + +def test_apply_cs_event_reports_the_nearest_blocker_when_several_apply(): + """F from vfdpXa is blocked twice over: pX is ephemeral *and* V is unmet. + + The message names the ephemeral rule, because that is the constraint the + caller has to satisfy first. + """ + with pytest.raises(VultronInvalidStateTransitionError) as excinfo: + apply_cs_event(CS.vfdpXa, CSEvent.F) + + message = str(excinfo.value) + assert "ephemeral" in message + assert "prerequisite" not in message + # Both blockers really are live, so this is a precedence choice, not luck. + assert required_next_cs_events(CS.vfdpXa) == frozenset({CSEvent.P}) + assert not is_vfd_vendor_aware(cs_dimensions(CS.vfdpXa)[0]) + + +# --- history validity ----------------------------------------------------- + + +@pytest.mark.spec("CSB-17-004") +def test_valid_histories_match_legacy_exactly(): + """The causal replay admits exactly the legacy model's 70 histories. + + This is the equivalence proof for `is_valid_history`: the legacy + permutation-ordering formulation and the replay-through-transitions + formulation accept the same set. + """ + new = {_as_string(history) for history in valid_cs_histories()} + assert new == _legacy_valid_histories() + + +def test_there_are_70_valid_histories(): + assert len(valid_cs_histories()) == 70 + + +def test_valid_histories_are_distinct(): + histories = valid_cs_histories() + assert len(set(histories)) == len(histories) + + +def test_is_valid_cs_history_agrees_with_legacy_on_every_permutation(): + legacy = _legacy_valid_histories() + for perm in permutations(CS_EVENTS): + assert is_valid_cs_history(list(perm)) == (_as_string(perm) in legacy) + + +def test_is_valid_cs_history_agrees_with_valid_cs_histories(): + enumerated = set(valid_cs_histories()) + for perm in permutations(CS_EVENTS): + assert is_valid_cs_history(list(perm)) == (perm in enumerated) + + +@pytest.mark.parametrize( + "history", + [ + "VFDPXA", + "VPFXDA", + "PVFDXA", # V immediately follows P + "VFXPDA", # P immediately follows X + "AVFDPX", + ], +) +def test_accepted_histories(history): + events = [CSEvent(char) for char in history] + assert is_valid_cs_history(events) + ensure_valid_cs_history(events) + + +@pytest.mark.parametrize( + "history,reason", + [ + ("FVDPXA", "V must precede F"), + ("VDFPXA", "F must precede D"), + ("VFDXAP", "P must precede X or immediately follow it"), + ("PAVFDX", "V must precede P or immediately follow it"), + ("XAPVFD", "P must immediately follow X"), + ], +) +def test_rejected_histories(history, reason): + events = [CSEvent(char) for char in history] + assert not is_valid_cs_history( + events + ), f"{history} should be rejected: {reason}" + with pytest.raises( + (VultronValidationError, VultronInvalidStateTransitionError) + ): + ensure_valid_cs_history(events) + + +def test_every_valid_history_reaches_the_terminal_state(): + for history in valid_cs_histories(): + assert replay_cs_history(history) is CS.VFDPXA + + +def test_is_valid_cs_history_rejects_incomplete_and_repeated_sequences(): + assert not is_valid_cs_history([CSEvent.V, CSEvent.F]) + assert not is_valid_cs_history([CSEvent.V] * 6) + + +def test_ensure_valid_cs_history_reports_missing_events(): + with pytest.raises(VultronValidationError, match="missing"): + ensure_valid_cs_history([CSEvent.V, CSEvent.F, CSEvent.D]) + + +def test_ensure_valid_cs_history_reports_repeated_events(): + with pytest.raises(VultronValidationError, match="more than once"): + ensure_valid_cs_history([CSEvent.V, CSEvent.V]) + + +# --- history prefixes ----------------------------------------------------- + + +def test_empty_prefix_is_valid(): + assert is_valid_cs_history_prefix([]) + + +@pytest.mark.spec("CSB-17-005") +def test_every_prefix_of_every_valid_history_is_valid(): + for history in valid_cs_histories(): + for length in range(len(history) + 1): + assert is_valid_cs_history_prefix(history[:length]) + + +@pytest.mark.spec("CSB-17-005") +def test_accepted_prefixes_are_exactly_the_prefixes_of_valid_histories(): + """The converse of the test above — the half that gives the API meaning. + + One direction alone would also be satisfied by a predicate that accepts + everything. This pins the accepted set to *exactly* the prefix-closure of + the 70 valid histories, so every accepted prefix is a trajectory some real + case could actually be on, and no legal in-progress case is rejected. + """ + closure = { + history[:length] + for history in valid_cs_histories() + for length in range(len(history) + 1) + } + + accepted = { + candidate + for length in range(len(CS_EVENTS) + 1) + for candidate in permutations(CS_EVENTS, length) + if is_valid_cs_history_prefix(list(candidate)) + } + + assert accepted == closure + + +@pytest.mark.spec("CSB-17-005") +def test_every_accepted_prefix_extends_to_a_complete_valid_history(): + """No accepted prefix is a dead end. + + Stated the way a caller cares about it: a case sitting on any prefix the + predicate accepts always has some legal route left to `VFDPXA`. Note that + the accepted set is derived from the predicate, not from + `valid_cs_histories()`, so this is not a restatement of the closure test. + """ + completions = valid_cs_histories() + + for length in range(len(CS_EVENTS) + 1): + for candidate in permutations(CS_EVENTS, length): + if not is_valid_cs_history_prefix(list(candidate)): + continue + assert any( + history[:length] == candidate for history in completions + ), f"accepted prefix {_as_string(candidate)} has no completion" + + +@pytest.mark.parametrize( + "prefix", + [ + [CSEvent.V], + [CSEvent.V, CSEvent.F], + [CSEvent.X], # ends in ephemeral pX — nothing has come next yet + [CSEvent.P], # ends in ephemeral vP + [CSEvent.A, CSEvent.V], + ], +) +@pytest.mark.spec("CSB-17-005") +def test_accepted_prefixes(prefix): + assert is_valid_cs_history_prefix(prefix) + + +@pytest.mark.parametrize( + "prefix", + [ + [CSEvent.F], # F before V + [CSEvent.D], # D before F + [CSEvent.X, CSEvent.A], # pX requires P next + [CSEvent.P, CSEvent.A], # vP requires V next + [CSEvent.V, CSEvent.V], # repeated event + ], +) +def test_rejected_prefixes(prefix): + assert not is_valid_cs_history_prefix(prefix) + + +def test_prefix_can_start_from_a_non_initial_state(): + assert is_valid_cs_history_prefix([CSEvent.F], start=CS.Vfdpxa) + assert not is_valid_cs_history_prefix([CSEvent.V], start=CS.Vfdpxa) + + +def test_replay_from_a_non_initial_state(): + assert replay_cs_history([CSEvent.F], start=CS.Vfdpxa) is CS.VFdpxa + + +def test_replay_rejects_repeated_events(): + with pytest.raises(VultronValidationError, match="more than once"): + replay_cs_history([CSEvent.V, CSEvent.P, CSEvent.V]) + + +def test_replay_of_empty_sequence_returns_the_start_state(): + assert replay_cs_history([]) is CS.vfdpxa + assert replay_cs_history([], start=CS.VFdPxa) is CS.VFdPxa + + +# --- input coercion ------------------------------------------------------- +# +# `CSEvent` is a StrEnum, and the legacy `case_states.validations` API this +# module replaces took plain strings ("VFDPXA"), so a migrating caller will +# naturally pass them. These pin that the string form works and that garbage +# makes the predicates answer False rather than raise. + + +@pytest.mark.parametrize( + "history,expected", + [ + ("VFDPXA", True), + ("FVDPXA", False), # F before V + ("VDFPXA", False), # D before F + ("VFDXPA", True), # X then P — pX satisfied by the adjacent P + ("PVFDXA", True), # P then V — vP satisfied by the adjacent V + ], +) +def test_string_histories_match_the_enum_form(history, expected): + assert is_valid_cs_history(history) is expected + assert is_valid_cs_history([CSEvent(c) for c in history]) is expected + + +def test_string_prefixes_are_accepted(): + assert is_valid_cs_history_prefix("VF") + assert is_valid_cs_history_prefix(["V", "F"]) + assert not is_valid_cs_history_prefix("F") + + +def test_apply_cs_event_accepts_the_string_form(): + assert apply_cs_event(CS.vfdpxa, "V") is CS.Vfdpxa + + +@pytest.mark.parametrize( + "events", + [ + ["Z"], # not a CS event letter + ["v"], # lowercase is a state letter, not an event + [None], + [1], + [["V"]], # unhashable + ], +) +def test_predicates_reject_non_events_without_raising(events): + """A predicate must answer, not explode, on bad input.""" + assert is_valid_cs_history_prefix(events) is False + assert is_valid_cs_history(events) is False + + +def test_ensure_valid_cs_history_names_the_offending_value(): + with pytest.raises(VultronValidationError, match="is not a CS event"): + ensure_valid_cs_history(["Z", "F", "D", "P", "X", "A"]) + + +def test_replay_rejects_non_events(): + with pytest.raises(VultronValidationError, match="is not a CS event"): + replay_cs_history(["V", "Z"]) diff --git a/test/test_integration_timeout_tier.py b/test/test_integration_timeout_tier.py new file mode 100644 index 000000000..bd0370ba0 --- /dev/null +++ b/test/test_integration_timeout_tier.py @@ -0,0 +1,260 @@ +# Copyright (c) 2026 Carnegie Mellon University and Contributors. +# - see Contributors.md for a full list of Contributors +# - see ContributionInstructions.md for information on how you can Contribute to this project +# Vultron Multiparty Coordinated Vulnerability Disclosure Protocol Prototype is +# licensed under a MIT (SEI)-style license, please see LICENSE.md distributed +# with this Software or contact permission@sei.cmu.edu for full terms. +# Created, in part, with funding and support from the United States Government +# (see Acknowledgments file). This program may include and/or can make use of +# certain third party source code, object code, documentation and other files +# ("Third Party Software"). See LICENSE.md for more details. +# Carnegie Mellon®, CERT® and CERT Coordination Center® are registered in the +# U.S. Patent and Trademark Office by Carnegie Mellon University + +""" +Tests for the integration-tier per-test timeout (issue #2270). + +The global ``timeout = 30`` in ``pyproject.toml`` is sized for the unit suite. +``test/conftest.py`` widens it for ``integration``-marked tests only. These +tests pin both tiers so neither ceiling can silently creep back down and abort +the session again. + +``TestApplyIntegrationTimeout`` exercises the hook against a stub item, which +pins its own contract but cannot show that pytest-timeout honours a marker +added at collection time. ``TestResolvedTimeoutsUnderRealPytest`` closes that +gap by running a real pytest session and reading the timeout each item +actually resolved to. +""" + +import json + +import pytest + +from test.conftest import ( + INTEGRATION_TIMEOUT_SECONDS, + apply_integration_timeout, +) + +pytest_plugins = ["pytester"] + + +class FakeItem: + """Minimal stand-in for a pytest ``Item``. + + Only the two marker operations the hook uses are implemented: + ``get_closest_marker`` and ``add_marker``. + """ + + def __init__(self, *marker_names, timeout=None): + self.names = set(marker_names) + self.added = [] + self.explicit_timeout = timeout + + def get_closest_marker(self, name): + if name == "timeout": + if self.explicit_timeout is not None: + return pytest.mark.timeout(self.explicit_timeout).mark + for mark in self.added: + if mark.name == "timeout": + return mark + return None + return ( + pytest.mark.__getattr__(name).mark if name in self.names else None + ) + + def add_marker(self, marker): + self.added.append(marker.mark) + + @property + def applied_timeout(self): + for mark in self.added: + if mark.name == "timeout": + return mark.args[0] + return None + + +class TestApplyIntegrationTimeout: + def test_integration_item_gets_the_integration_tier_timeout(self): + item = FakeItem("integration") + + assert apply_integration_timeout([item]) == 1 + assert item.applied_timeout == INTEGRATION_TIMEOUT_SECONDS + + def test_unit_item_is_left_at_the_global_default(self): + item = FakeItem() + + assert apply_integration_timeout([item]) == 0 + assert item.applied_timeout is None + + def test_explicit_timeout_marker_wins(self): + """A deliberate per-test value must not be overwritten.""" + item = FakeItem("integration", timeout=180) + + assert apply_integration_timeout([item]) == 0 + assert item.applied_timeout is None + + def test_explicit_shorter_timeout_also_wins(self): + """Tests that assert on a timeout firing must keep their short value.""" + item = FakeItem("integration", timeout=1) + + assert apply_integration_timeout([item]) == 0 + assert item.applied_timeout is None + + def test_mixed_collection_only_touches_integration_items(self): + integration = FakeItem("integration") + unit = FakeItem() + explicit = FakeItem("integration", timeout=10) + + assert apply_integration_timeout([integration, unit, explicit]) == 1 + assert integration.applied_timeout == INTEGRATION_TIMEOUT_SECONDS + assert unit.applied_timeout is None + assert explicit.applied_timeout is None + + def test_applying_twice_is_idempotent(self): + """Re-running the hook must not stack duplicate timeout markers.""" + item = FakeItem("integration") + + apply_integration_timeout([item]) + assert apply_integration_timeout([item]) == 0 + assert [m.name for m in item.added].count("timeout") == 1 + + def test_empty_collection_is_a_no_op(self): + assert apply_integration_timeout([]) == 0 + + +_UNIT_TIER_FOR_PROBE = 30 + +_PROBE_CONFTEST = """ +import json +import pathlib + +import pytest_timeout + +from test.conftest import apply_integration_timeout + +_resolved = {} + + +def pytest_configure(config): + config.addinivalue_line("markers", "integration: integration test") + + +def pytest_collection_modifyitems(items): + apply_integration_timeout(items) + + +def pytest_runtest_setup(item): + _resolved[item.name] = pytest_timeout._get_item_settings(item).timeout + + +def pytest_sessionfinish(session, exitstatus): + pathlib.Path(session.config.rootpath / "resolved.json").write_text( + json.dumps(_resolved) + ) +""" + +_PROBE_TESTS = """ +import pytest + + +@pytest.mark.integration +def test_integration_tier(): + pass + + +def test_unit_tier(): + pass + + +@pytest.mark.integration +@pytest.mark.timeout(7) +def test_explicit_marker_wins(): + pass +""" + + +class TestResolvedTimeoutsUnderRealPytest: + """Assert the timeout each item *actually* resolves to (issue #2270). + + The stub-based tests above verify the hook's own logic. This one runs a + real pytest session and asks pytest-timeout what it resolved for each + item, which is the only way to catch a hook-ordering regression: the + marker is added at collection time, but pytest-timeout reads it much + later, in ``pytest_runtest_protocol``. + """ + + @pytest.fixture + def resolved(self, pytester): + pytester.makeconftest(_PROBE_CONFTEST) + pytester.makepyfile(test_probe=_PROBE_TESTS) + + result = pytester.runpytest( + "-p", + "no:randomly", + "-o", + f"timeout={_UNIT_TIER_FOR_PROBE}", + "-o", + "timeout_method=thread", + ) + result.assert_outcomes(passed=3) + + return json.loads((pytester.path / "resolved.json").read_text()) + + def test_integration_item_resolves_to_the_integration_tier(self, resolved): + assert resolved["test_integration_tier"] == INTEGRATION_TIMEOUT_SECONDS + + def test_unit_item_resolves_to_the_unit_tier(self, resolved): + assert resolved["test_unit_tier"] == _UNIT_TIER_FOR_PROBE + + def test_explicit_marker_beats_the_tier_default(self, resolved): + """The demo suite's deliberate values (10, 180) depend on this.""" + assert resolved["test_explicit_marker_wins"] == 7 + + +class TestIntegrationTimeoutValue: + def test_is_comfortably_above_the_slowest_honest_integration_test(self): + """The slowest honest integration test measured ~4.3s (issue #2270). + + A ceiling that is merely a little above that is what caused the + original spurious aborts, so require real headroom. + """ + assert INTEGRATION_TIMEOUT_SECONDS >= 30 + + def test_is_still_a_bounded_hang_detector(self): + assert INTEGRATION_TIMEOUT_SECONDS <= 300 + + +class TestUnitTierTimeout: + """Pin the unit-tier ceiling in ``pyproject.toml`` (issue #2270). + + Four sessions re-diagnosed a too-tight unit ceiling as flakiness before it + was raised. These assertions make a silent revert fail loudly. + """ + + @staticmethod + def _configured_timeout(pytestconfig): + return int(pytestconfig.getini("timeout")) + + def test_unit_tier_clears_the_slowest_ast_ratchet(self, pytestconfig): + """AST-walking architecture ratchets run ~3.4s in isolation. + + Under full-suite load they were tripping a 5s ceiling. Require enough + headroom that load variance cannot reach it. + """ + assert self._configured_timeout(pytestconfig) >= 15 + + def test_unit_tier_is_still_a_bounded_hang_detector(self, pytestconfig): + assert self._configured_timeout(pytestconfig) <= 60 + + def test_integration_tier_is_wider_than_the_unit_tier(self, pytestconfig): + assert INTEGRATION_TIMEOUT_SECONDS > self._configured_timeout( + pytestconfig + ) + + def test_thread_method_is_still_in_use(self, pytestconfig): + """Documents the coupling the tier values depend on. + + If this ever changes to "signal", a timeout fails one test instead of + aborting the session, and the generous ceilings above can be revisited. + """ + assert pytestconfig.getini("timeout_method") == "thread" diff --git a/vultron/core/states/__init__.py b/vultron/core/states/__init__.py index 17af1d85c..f2f8fa942 100644 --- a/vultron/core/states/__init__.py +++ b/vultron/core/states/__init__.py @@ -34,6 +34,28 @@ state_string_to_enums, vfd, ) +from vultron.core.states.cs_invariants import ( + CSEvent, + CS_EVENTS, + CS_EVENT_TO_PXA_TRIGGER, + CS_EVENT_TO_VFD_TRIGGER, + PXA_EVENTS, + VFD_EVENTS, + apply_cs_event, + cs_dimensions, + cs_from_dimensions, + cs_transition_event, + ensure_valid_cs_history, + ensure_valid_cs_transition, + is_ephemeral_cs_state, + is_valid_cs_history, + is_valid_cs_history_prefix, + is_valid_cs_transition, + next_cs_states, + replay_cs_history, + required_next_cs_events, + valid_cs_histories, +) from vultron.core.states.em import ( EM, EM_EMBARGO_ACTIVE, diff --git a/vultron/core/states/cs_invariants.py b/vultron/core/states/cs_invariants.py new file mode 100644 index 000000000..1c4f18f7f --- /dev/null +++ b/vultron/core/states/cs_invariants.py @@ -0,0 +1,562 @@ +#!/usr/bin/env python +# Copyright (c) 2026 Carnegie Mellon University and Contributors. +# - see Contributors.md for a full list of Contributors +# - see ContributionInstructions.md for information on how you can Contribute to this project +# Vultron Multiparty Coordinated Vulnerability Disclosure Protocol Prototype is +# licensed under a MIT (SEI)-style license, please see LICENSE.md distributed +# with this Software or contact permission@sei.cmu.edu for full terms. +# Created, in part, with funding and support from the United States Government +# (see Acknowledgments file). This program may include and/or can make use of +# certain third party source code, object code, documentation and other files +# ("Third Party Software"). See LICENSE.md for more details. +# Carnegie Mellon®, CERT® and CERT Coordination Center® are registered in the +# U.S. Patent and Trademark Office by Carnegie Mellon University + +"""CS-internal validity, transition, and history invariants. + +This module expresses the Case State (CS) validity rules of the MPCVD +state-based model against the current enum models in +`vultron.core.states.cs` — `CS`, `CS_vfd`, `CS_pxa`, and the per-dimension +transition tables that back `VfdDimension` / `PxaDimension`. + +Three rule families live here: + +Compound state validity + The 32 members of `CS` *are* the valid compound states. The two + impossible-combination rules of the formal model (`vF` — a fix ready + while the vendor is unaware; `fD` — a fix deployed that was never ready) + are enforced structurally by `CS_vfd` having only four members, so no + runtime predicate is needed. `test_cs_invariants.py` ratchets the enum + against an independent derivation of the rule. + +Transition validity (`is_valid_cs_transition`) + A CS transition changes exactly one of the six dimensions, always from + the not-yet-happened to the happened value, and must respect the two + *ephemeral state* rules below. + +Ephemeral states (`is_ephemeral_cs_state`, `required_next_cs_events`) + Two compound states are transient in the formal model and constrain the + very next event rather than being forbidden outright: + + - ``vP`` (public aware, vendor unaware) — the next event MUST be ``V``. + - ``pX`` (exploit public, public unaware) — the next event MUST be ``P``. + +History validity (`is_valid_cs_history`, `is_valid_cs_history_prefix`) + A *history* is the order in which the six events occurred. History + validity is a **causal** property of a whole sequence, not a + point-in-time or wall-clock one: it is what lets a consumer reject a + reported case trajectory that no real case could have produced. + +The complete-history ordering rules + +- ``V ≺ F ≺ D`` +- ``P ≺ X`` or ``P`` immediately follows ``X`` +- ``V ≺ P`` or ``V`` immediately follows ``P`` + +are provably equivalent to replaying the sequence through +`is_valid_cs_transition` from `CS.vfdpxa`; both admit exactly the same 70 +histories. `test_cs_invariants.py` asserts that equivalence directly, so +the two formulations cannot drift apart. + +Reference: Householder, A. D., and Spring, J. +*A State-Based Model for Multi-Party Coordinated Vulnerability Disclosure +(MPCVD)*, CMU/SEI-2021-SR-021. + +Spec: `specs/cs-behavior.yaml` CSB-17. Decision: ADR-0060. +""" + +from collections.abc import Iterable, Sequence +from enum import StrEnum + +from vultron.core.states.cs import ( + CS, + CS_pxa, + CS_vfd, + PXA_Trigger, + VFD_Trigger, + is_pxa_exploit_public, + is_pxa_public_aware, + is_valid_pxa_transition, + is_valid_vfd_transition, + is_vfd_vendor_aware, +) +from vultron.errors import ( + VultronInvalidStateTransitionError, + VultronValidationError, +) + + +class CSEvent(StrEnum): + """The six CS transition events, in canonical dimension order. + + Each event flips exactly one CS dimension from its not-yet-happened value + to its happened value. Events are irreversible and occur at most once in + a case history. + """ + + V = "V" + """Vendor becomes aware.""" + F = "F" + """Fix becomes ready.""" + D = "D" + """Fix is deployed.""" + P = "P" + """Public becomes aware.""" + X = "X" + """Exploit is made public.""" + A = "A" + """Attacks are observed.""" + + +CS_EVENTS: tuple[CSEvent, ...] = tuple(CSEvent) +"""All six CS events in canonical (`VFDPXA`) order.""" + +VFD_EVENTS: frozenset[CSEvent] = frozenset({CSEvent.V, CSEvent.F, CSEvent.D}) +"""The CS events belonging to the vendor fix path (VFD) dimension.""" + +PXA_EVENTS: frozenset[CSEvent] = frozenset({CSEvent.P, CSEvent.X, CSEvent.A}) +"""The CS events belonging to the public/exploit/attack (PXA) dimension.""" + +CS_EVENT_TO_VFD_TRIGGER: dict[CSEvent, VFD_Trigger] = { + CSEvent.V: VFD_Trigger.V, + CSEvent.F: VFD_Trigger.F, + CSEvent.D: VFD_Trigger.D, +} +"""Maps a VFD-dimension CS event to the `VfdDimension.transition()` trigger. + +Exported for callers that drive the dimension machines directly rather than +going through `apply_cs_event` — e.g. emit-time guards that already hold a +`VfdDimension` and need the trigger for a CS event. +""" + +CS_EVENT_TO_PXA_TRIGGER: dict[CSEvent, PXA_Trigger] = { + CSEvent.P: PXA_Trigger.P, + CSEvent.X: PXA_Trigger.X, + CSEvent.A: PXA_Trigger.A, +} +"""Maps a PXA-dimension CS event to the `PxaDimension.transition()` trigger. + +The PXA counterpart of `CS_EVENT_TO_VFD_TRIGGER`; see that map for the +rationale. +""" + + +def cs_dimensions(state: CS) -> tuple[CS_vfd, CS_pxa]: + """Return the (VFD, PXA) dimension states of a compound CS state. + + Examples:: + + cs_dimensions(CS.vfdpxa) # (CS_vfd.vfd, CS_pxa.pxa) + cs_dimensions(CS.VFdPXa) # (CS_vfd.VFd, CS_pxa.PXa) + """ + compound = state.value + return compound.vfd_state, compound.pxa_state + + +def cs_from_dimensions(vfd_state: CS_vfd, pxa_state: CS_pxa) -> CS: + """Return the compound `CS` member for a (VFD, PXA) dimension pair. + + Every one of the 4 x 8 combinations is a valid compound state, so this + never fails for well-typed inputs. + + Examples:: + + cs_from_dimensions(CS_vfd.VFd, CS_pxa.Pxa) # CS.VFdPxa + """ + return CS[f"{vfd_state.name}{pxa_state.name}"] + + +def is_ephemeral_cs_state(state: CS) -> bool: + """Return True if *state* is transient and constrains the next event. + + A state is ephemeral when the formal model requires a specific event to + fire next: ``vP`` requires ``V``, ``pX`` requires ``P``. The two + conditions are mutually exclusive, because ``vP`` needs ``P`` set and + ``pX`` needs it unset. + + Examples:: + + is_ephemeral_cs_state(CS.vfdPxa) # True (public aware, vendor unaware) + is_ephemeral_cs_state(CS.vfdpXa) # True (exploit public, public unaware) + is_ephemeral_cs_state(CS.VfdPxa) # False + """ + return bool(required_next_cs_events(state)) + + +def required_next_cs_events(state: CS) -> frozenset[CSEvent]: + """Return the events that *state* permits as its immediate successor. + + An empty set means the state is not ephemeral and imposes no + next-event requirement of its own — any event still available under the + per-dimension machines may fire. + + Examples:: + + required_next_cs_events(CS.vfdPxa) # frozenset({CSEvent.V}) + required_next_cs_events(CS.vfdpXa) # frozenset({CSEvent.P}) + required_next_cs_events(CS.VfdPxa) # frozenset() + """ + vfd_state, pxa_state = cs_dimensions(state) + + # vP: the public is aware but the vendor is not -> V must fire next. + if is_pxa_public_aware(pxa_state) and not is_vfd_vendor_aware(vfd_state): + return frozenset({CSEvent.V}) + + # pX: an exploit is public but the public is not aware -> P must fire next. + if is_pxa_exploit_public(pxa_state) and not is_pxa_public_aware(pxa_state): + return frozenset({CSEvent.P}) + + return frozenset() + + +def cs_transition_event(src: CS, dst: CS) -> CSEvent | None: + """Return the single event that distinguishes *src* from *dst*. + + Returns None when the two states differ in zero or more than one + dimension bit, i.e. when no single event can account for the change. + + Examples:: + + cs_transition_event(CS.vfdpxa, CS.Vfdpxa) # CSEvent.V + cs_transition_event(CS.vfdpxa, CS.vfdpxa) # None (no change) + cs_transition_event(CS.vfdpxa, CS.VfdPxa) # None (two bits changed) + """ + changed = [ + event + for event, before, after in zip(CS_EVENTS, src.name, dst.name) + if before != after + ] + if len(changed) != 1: + return None + return changed[0] + + +def is_valid_cs_transition( + src: CS, dst: CS, *, allow_null: bool = False +) -> bool: + """Return True if (src -> dst) is a legal CS transition. + + A legal transition satisfies all of: + + 1. Exactly one of the six dimensions changes (Hamming distance 1). + 2. The change is monotone — from the not-yet-happened to the happened + value; CS events are irreversible. + 3. The changed dimension's per-machine transition table permits it + (`is_valid_vfd_transition` / `is_valid_pxa_transition`). + 4. If *src* is ephemeral, *dst* must be reached by its required event. + + Args: + src: the source compound state + dst: the destination compound state + allow_null: if True, treat ``src is dst`` as valid. Use for + same-state status re-assertions, which are bookkeeping rather + than transitions. + + Examples:: + + is_valid_cs_transition(CS.vfdpxa, CS.Vfdpxa) # True + is_valid_cs_transition(CS.vfdPxa, CS.vfdPxA) # False (vP needs V next) + is_valid_cs_transition(CS.vfdpXa, CS.vfdpXA) # False (pX needs P next) + is_valid_cs_transition(CS.Vfdpxa, CS.vfdpxa) # False (not monotone) + """ + if src is dst: + return allow_null + + event = cs_transition_event(src, dst) + if event is None: + return False + + src_vfd, src_pxa = cs_dimensions(src) + dst_vfd, dst_pxa = cs_dimensions(dst) + + if event in VFD_EVENTS: + if src_pxa is not dst_pxa: + return False + if not is_valid_vfd_transition(src_vfd, dst_vfd): + return False + else: + if src_vfd is not dst_vfd: + return False + if not is_valid_pxa_transition(src_pxa, dst_pxa): + return False + + required = required_next_cs_events(src) + if required and event not in required: + return False + + return True + + +def ensure_valid_cs_transition( + src: CS, dst: CS, *, allow_null: bool = False +) -> None: + """Raise unless (src -> dst) is a legal CS transition. + + Raises: + VultronInvalidStateTransitionError: if the transition is not legal. + """ + if is_valid_cs_transition(src, dst, allow_null=allow_null): + return + + required = required_next_cs_events(src) + if required: + detail = ( + f" — {src.name} is ephemeral and requires event(s)" + f" {sorted(e.value for e in required)} next" + ) + else: + detail = "" + raise VultronInvalidStateTransitionError( + f"CS: transition {src.name} -> {dst.name} is not permitted{detail}." + ) + + +def next_cs_states(state: CS) -> tuple[CS, ...]: + """Return every compound state reachable from *state* in one transition. + + Examples:: + + next_cs_states(CS.VFDPXA) # () — terminal state + len(next_cs_states(CS.vfdPxa)) # 1 — ephemeral, V only + """ + return tuple( + candidate + for candidate in CS + if is_valid_cs_transition(state, candidate) + ) + + +def _as_cs_event(value: CSEvent | str) -> CSEvent: + """Coerce *value* to a `CSEvent`. + + `CSEvent` is a `StrEnum`, so the single-letter strings of the legacy + string-pattern API (``"V"``, ``"F"``, ...) are accepted. This keeps the + bool-returning predicates total: callers migrating from + `case_states.validations` naturally pass strings, and a validator must + answer their question rather than raise on the input type. + + Raises: + VultronValidationError: if *value* is not a CS event. + """ + try: + return CSEvent(value) + except ValueError as exc: + raise VultronValidationError( + f"CS history: {value!r} is not a CS event; expected one of" + f" {[e.value for e in CS_EVENTS]}." + ) from exc + except TypeError as exc: + raise VultronValidationError( + f"CS history: {value!r} is not a CS event (unhashable or" + " non-string type)." + ) from exc + + +def apply_cs_event(state: CS, event: CSEvent | str) -> CS: + """Return the state reached by applying *event* to *state*. + + Args: + state: the current compound state + event: the event to apply, as a `CSEvent` or its string value + + Returns: + the resulting compound state + + Raises: + VultronValidationError: if *event* is not a CS event. + VultronInvalidStateTransitionError: if *event* cannot fire from + *state* — because it already happened, because its dimension + machine forbids it, or because *state* is ephemeral and requires + a different event next. + + Examples:: + + apply_cs_event(CS.vfdpxa, CSEvent.V) # CS.Vfdpxa + """ + event = _as_cs_event(event) + + for candidate in CS: + # `!=`, not `is not`: CSEvent is a StrEnum, so a plain string compares + # equal without being identical. + if cs_transition_event(state, candidate) != event: + continue + if is_valid_cs_transition(state, candidate): + return candidate + break + + raise VultronInvalidStateTransitionError( + f"CS: event '{event.value}' cannot fire from {state.name}:" + f" {_why_event_blocked(state, event)}." + ) + + +def _why_event_blocked(state: CS, event: CSEvent) -> str: + """Explain why *event* cannot fire from *state*. + + Only called on the failure path of `apply_cs_event`, where at least one of + three things is true: the event already happened, the state is ephemeral + and demands a different event, or the event's dimension prerequisite is + unmet (F before V, D before F). + + More than one can hold at once — e.g. ``F`` from ``vfdpXa`` is blocked both + by the ``pX`` ephemeral rule and by ``V`` not having occurred. The checks + run in order of proximity and report the first that applies, so the message + names the constraint the caller must satisfy first rather than every + constraint outstanding. + """ + if state.name[CS_EVENTS.index(event)].isupper(): + return "it has already occurred" + + required = required_next_cs_events(state) + if required: + return ( + f"{state.name} is ephemeral and requires event(s)" + f" {sorted(e.value for e in required)} next" + ) + + return "its dimension's prerequisite events have not occurred" + + +def _ensure_distinct_events(events: Sequence[CSEvent]) -> None: + seen: set[CSEvent] = set() + for event in events: + if event in seen: + raise VultronValidationError( + f"CS history: event '{event.value}' occurs more than once;" + " CS events are irreversible and happen at most once." + ) + seen.add(event) + + +def replay_cs_history( + events: Iterable[CSEvent | str], *, start: CS = CS.vfdpxa +) -> CS: + """Replay *events* from *start* and return the resulting state. + + This is the causal check over a whole trajectory: each event must be + legal given every event that preceded it. + + Args: + events: the ordered CS events to apply, as `CSEvent` members or their + string values (so ``"VFDPXA"`` and ``list(CSEvent)`` both work) + start: the state to replay from, default `CS.vfdpxa` + + Returns: + the compound state after the last event + + Raises: + VultronValidationError: if an event repeats or is not a CS event. + VultronInvalidStateTransitionError: if an event cannot fire at its + position in the sequence. + + Examples:: + + replay_cs_history([CSEvent.V, CSEvent.F]) # CS.VFdpxa + replay_cs_history("VF") # CS.VFdpxa + """ + sequence = [_as_cs_event(event) for event in events] + _ensure_distinct_events(sequence) + + state = start + for event in sequence: + state = apply_cs_event(state, event) + return state + + +def is_valid_cs_history_prefix( + events: Sequence[CSEvent | str], *, start: CS = CS.vfdpxa +) -> bool: + """Return True if *events* is a legal (possibly incomplete) trajectory. + + Use this on real case histories, which are usually still in progress. + An empty sequence is trivially valid. Events may be `CSEvent` members or + their string values. Anything that is not a CS event makes the sequence + invalid rather than raising — this is a predicate, so it always answers. + + Note that a prefix ending in an ephemeral state is accepted: the + ephemeral rules constrain what may come *next*, and nothing has come + next yet. + + Examples:: + + is_valid_cs_history_prefix([CSEvent.V, CSEvent.F]) # True + is_valid_cs_history_prefix([CSEvent.F]) # False + is_valid_cs_history_prefix([CSEvent.X]) # True (pX) + is_valid_cs_history_prefix([CSEvent.X, CSEvent.A]) # False + is_valid_cs_history_prefix("VF") # True + """ + try: + replay_cs_history(events, start=start) + except (VultronValidationError, VultronInvalidStateTransitionError): + return False + return True + + +def is_valid_cs_history(events: Sequence[CSEvent | str]) -> bool: + """Return True if *events* is a complete, legal case history. + + A complete history contains all six events exactly once and reaches + `CS.VFDPXA`. Equivalent to the ordering rules ``V ≺ F ≺ D``, + ``P ≺ X`` or ``XP`` adjacent, and ``V ≺ P`` or ``PV`` adjacent. + + Accepts the string form of the legacy `case_states.validations` API as + well as `CSEvent` members. + + Examples:: + + is_valid_cs_history(list(CSEvent)) # True (VFDPXA) + is_valid_cs_history("VFDPXA") # True + is_valid_cs_history([CSEvent.F, CSEvent.V, ...]) # False (F before V) + """ + if len(events) != len(CS_EVENTS): + return False + if set(events) != set(CS_EVENTS): + return False + return is_valid_cs_history_prefix(events) + + +def ensure_valid_cs_history(events: Sequence[CSEvent | str]) -> None: + """Raise unless *events* is a complete, legal case history. + + Raises: + VultronValidationError: if the history is incomplete, contains a + value that is not a CS event, or its events are not a + permutation of the six CS events. + VultronInvalidStateTransitionError: if the ordering is causally + impossible. + """ + sequence = [_as_cs_event(event) for event in events] + _ensure_distinct_events(sequence) + + missing = sorted(e.value for e in set(CS_EVENTS) - set(sequence)) + if missing: + raise VultronValidationError( + f"CS history: incomplete, missing event(s) {missing}." + ) + + replay_cs_history(sequence) + + +def valid_cs_histories() -> tuple[tuple[CSEvent, ...], ...]: + """Return every complete, legal case history. + + There are 70 of them. Ordering is deterministic (depth-first over `CS` + member order) so callers may rely on it in tests. + + Examples:: + + len(valid_cs_histories()) # 70 + """ + histories: list[tuple[CSEvent, ...]] = [] + + def walk(state: CS, acc: tuple[CSEvent, ...]) -> None: + if state is CS.VFDPXA: + histories.append(acc) + return + for candidate in next_cs_states(state): + event = cs_transition_event(state, candidate) + assert event is not None # guaranteed by is_valid_cs_transition + walk(candidate, acc + (event,)) + + walk(CS.vfdpxa, ()) + return tuple(histories)