fix: one canonical persisted shape for ParticipantStatus

docs/adr/0062-normalise-wire-to-core-at-both-ingress-and-persistence.md

@@ -0,0 +1,164 @@
+---
+status: accepted
+date: 2026-08-13
+deciders: Vultron maintainers
+consulted: Vultron maintainers
+informed: Vultron contributors
+---
+
+# Normalise Wire → Core at Ingress, and Enforce It Again at the Persistence Boundary
+
+## Context and Problem Statement
+
+`ParticipantStatus` and `CaseParticipant` each exist in two structurally
+incompatible shapes. The core types nest their dimensions
+(`rm: RmDimension`, `vfd: VfdDimension` — SDO-03-002, ADR-0036); the wire
+projections carry them flat (`rm_state`, `vfd_state`). Reading a nested
+dimension off a wire-shaped object yields `None`, and every core reader
+substituted an initial state for that `None` — silently resetting a
+participant's RM ladder (#2232, with #2264 as the symptom).
+
+`Record.from_obj()` was supposed to keep wire objects out of the store, but its
+guard was `type_.startswith("as_")` and wire vocabulary `type_` values are
+**bare** (`"CaseParticipant"`). Fifteen wire classes therefore shadowed a
+`CORE_VOCABULARY` entry and were written into core-typed rows unchallenged.
+
+So the question is not only *how* to normalise, but **where**: a wire-shaped
+object can enter the system at an HTTP ingress boundary, and it can reach the
+persistence boundary from several call paths. Enforcing in the wrong place
+either misses paths or breaks legitimate inbound traffic.
+
+## Decision Drivers
+
+- No wire-shaped row may exist in the DataLayer — that is the invariant #2232
+  asks for, and rows outlive whichever code wrote them.
+- A wire-shaped object arriving over HTTP is **legitimate inbound data**, not
+  corruption. Treating it as an error is a denial of service against the
+  protocol.
+- A shape mismatch discovered while reading a *stored* row **is** corruption and
+  must fail loudly (ARCH-15-001, ARCH-15-002) — the silent degrade is the whole
+  defect.
+- The received-side behavior tree must not abort because one embedded
+  participant is malformed; the HTTP inbox re-queues on exception, so an
+  escaping raise becomes an undrainable poison message.
+- Enforcement must be verifiable by a test, not by reviewer vigilance: 15 types
+  shadow a core type and the count will change.
+
+## Considered Options
+
+- Normalise at wire→core ingress only
+- Normalise at the persistence boundary only
+- Normalise at ingress, and enforce again at the persistence boundary
+- Unify the two shapes into one class
+
+## Decision Outcome
+
+Chosen option: **"Normalise at ingress, and enforce again at the persistence
+boundary"**, because the two placements answer different questions and neither
+subsumes the other. Ingress projection is what makes the *behaviour* correct —
+inbound data is converted where it arrives, so no core reader ever sees a wire
+shape and no reader has to degrade. Persistence-boundary normalisation is what
+makes the *invariant* hold — it is the single choke point every write passes
+through, so it can guarantee the stored row is canonical no matter which ingress
+path missed.
+
+Concretely:
+
+- **Ingress (primary).** `_project_to_core_participant()` in
+  `vultron/core/use_cases/received/case/_helpers.py` projects each embedded
+  participant of a received case snapshot via `to_core()` before anything reads
+  it. An unprojectable participant is logged at ERROR and **skipped**, not
+  raised: losing one malformed participant is strictly better than losing the
+  case.
+- **Persistence (backstop).** `_normalize_to_core()` in
+  `vultron/adapters/driven/db_record.py` projects the object *and its direct
+  children* for every `type_` in `_NORMALIZE_WIRE_TO_CORE`. Children matter
+  because a `VulnerabilityCase` row stores `case_participants` inline; one level
+  suffices because `to_core()` recurses.
+- **Readers stay strict.** `participant_status_rm_state()` and
+  `participant_status_vfd_state()` raise `VultronValidationError` on a non-core
+  shape. Absence (an empty status list) remains a legitimate `None`.
+
+Rejected for now: unifying the two shapes. It is the right end state — one class
+per concept, wire as a pure projection (ADR-0017) — but it is a breaking change
+across the AS2 vocabulary and the persisted-row format, and #2232 is a live data
+corruption bug. This ADR deliberately buys correctness now without foreclosing
+unification later; both enforcement points become redundant, and removable, once
+the shapes converge.
+
+### Consequences
+
+- Good, because the DataLayer invariant ("no wire-shaped row") holds regardless
+  of which write path is used, including paths not yet written.
+- Good, because inbound wire data still works: projection at ingress means
+  making readers strict does not break the protocol.
+- Good, because a projection failure now raises `VultronValidationError` rather
+  than a bare `ValueError`, so it cannot be absorbed by handlers written for
+  `crud.create()`'s duplicate-row `ValueError`.
+- Bad, because the same projection is expressed in two places, and a reader can
+  reasonably wonder which one is authoritative. Mitigated by
+  `notes/datalayer-design.md`, which names the persistence boundary as the
+  backstop.
+- Bad, because `_NORMALIZE_WIRE_TO_CORE` covers 2 of the 15 shadowing types. The
+  other 13 differ only by key spelling today, so they are misspelled rather than
+  unreadable — tracked in #2268.
+- Neutral, because per-write child projection costs one `model_fields` scan on
+  objects that are already being serialised.
+
+## Validation
+
+- `test/architecture/test_normalize_wire_to_core_ratchet.py` — grow-only ratchet
+  on `_NORMALIZE_WIRE_TO_CORE`, plus an exact enumeration of the un-normalised
+  shadowing types so a newly added one must be triaged.
+- `test/adapters/driven/test_db_record.py` — top-level and nested-child
+  normalisation; projection failure raises a non-`ValueError`.
+- `test/core/use_cases/received/case/test_helpers.py` — a wire-shaped incoming
+  participant against a core-shaped stored one: no raise escapes, the RM
+  regression guard still fires, and the persisted row is core-shaped.
+- `test/core/models/test_participant_status_shape.py` — both canonical readers
+  raise on a wire shape and on a present-but-unusable dimension.
+
+## Pros and Cons of the Options
+
+### Normalise at wire→core ingress only
+
+- Good, because it converts data where it arrives, which is where the type
+  information about "this came from the wire" actually exists.
+- Good, because it keeps the adapter free of shape-specific knowledge.
+- Bad, because it is an open set of call sites. #2232's first fix took this
+  reading of the issue, missed the received-case path, and turned every inbound
+  `Announce(VulnerabilityCase)` into an aborted behavior tree.
+- Bad, because it cannot state an invariant about stored rows.
+
+### Normalise at the persistence boundary only
+
+- Good, because it is one choke point and yields a checkable invariant.
+- Bad, because core readers still meet wire-shaped objects *before* the write —
+  which is exactly where the RM ladder was being reset.
+- Bad, because by the time an error surfaces the useful context (which activity,
+  which sender) is gone.
+
+### Normalise at ingress, and enforce again at the persistence boundary
+
+- Good, because behaviour and invariant are both covered, and each placement
+  fails safe for the failure mode it owns.
+- Neutral, because the redundancy is real but cheap and testable.
+- Bad, because two enforcement points must be kept in agreement.
+
+### Unify the two shapes into one class
+
+- Good, because it removes the defect class rather than guarding against it.
+- Bad, because it breaks the AS2 wire contract and the persisted-row format at
+  once, with no incremental path — not an acceptable shape for a bug fix.
+
+## More Information
+
+Related: issue #2232 (the shape duality), issue #2264 (initial-state
+substitution sites), issue #2268 (migrating the remaining 13 shadowing types).
+Related ADRs: ADR-0017 (wire is a projection of core), ADR-0034 (`dl.read()`
+returns core objects — this is its write-side counterpart), ADR-0036
+(dimension objects on `ParticipantStatus`).
+
+Generated spec requirements: none new — this decision implements existing
+`specs/architecture.yaml` ARCH-15-001, ARCH-15-002 and is the write-side
+counterpart to `specs/datalayer.yaml` DL-05-001 through DL-05-004.

docs/adr/index.md

@@ -129,6 +129,7 @@ General information about architectural decision records is available at <https:
 - [ADR-0057 Rename `CVDRole.OTHER` to `CVDRole.OBSERVER` and Define Observer Participant Semantics](0057-observer-participant-role.md)
 - [ADR-0058 Gate Demo Scenario Steps on Causal Preconditions, Not Temporal Order](0058-causal-gating-in-demo-scenarios.md) *(provisional)*
 - [ADR-0059 Buffer Pre-Genesis `Announce(CaseLedgerEntry)` and Drain on Case Seed](0059-buffer-pre-genesis-ledger-entries.md)
+- [ADR-0062 Normalise Wire → Core at Ingress, and Enforce It Again at the Persistence Boundary](0062-normalise-wire-to-core-at-both-ingress-and-persistence.md)
 
 ## Proposed ADRs
 

mkdocs.yml

@@ -277,6 +277,7 @@ nav:
       - Rename CVDRole.OTHER to CVDRole.OBSERVER and Define Observer Participant Semantics: 'adr/0057-observer-participant-role.md'
       - Gate Demo Scenario Steps on Causal Preconditions, Not Temporal Order: 'adr/0058-causal-gating-in-demo-scenarios.md'
       - Buffer Pre-Genesis Announce(CaseLedgerEntry) and Drain on Case Seed: 'adr/0059-buffer-pre-genesis-ledger-entries.md'
+      - Normalise Wire to Core at Ingress, and Enforce It Again at the Persistence Boundary: 'adr/0062-normalise-wire-to-core-at-both-ingress-and-persistence.md'
   - About:
     - Contributing: 'about/contributing.md'
     - FAQ: 'about/faq.md'

notes/datalayer-design.md

@@ -214,6 +214,48 @@ migrating it out of core is tracked as a **separate concern** (#1506, decided
 in ADR-0035), not part of the DL-05 entity work. Until then, the ratchet
 exemption set enumerates these Activity types explicitly so it can only shrink.
 
+## Write Path Normalises Wire → Core (#2232, ADR-0062)
+
+The read-path rule above says nothing about what gets *written*, and that gap
+was load-bearing. `Record.from_obj()` rejected objects whose `type_` starts with
+`as_` — but wire vocabulary `type_` values are **bare** (`"CaseParticipant"`, not
+`"as_CaseParticipant"`), so the guard never fired for the 15 wire classes that
+shadow a `CORE_VOCABULARY` entry. A wire-shaped object was written into a
+core-typed row, and whichever class read the row back decided what the data
+meant.
+
+For `ParticipantStatus` and `CaseParticipant` the two shapes are *structurally*
+incompatible — core nests `rm: RmDimension` where wire carries a flat `rm_state`
+— so a wire-shaped row makes `status.rm.state` yield `None` rather than merely
+misspell a key.
+
+**Rule:** `Record.from_obj()` normalises through `_normalize_to_core()`
+(`vultron/adapters/driven/db_record.py`) before serialising. The object **and its
+direct children** are projected via `to_core()`; one level of children is
+sufficient because `to_core()` recurses. Child projection is not optional
+polish: a `VulnerabilityCase` row stores its `case_participants` inline, so
+checking only the top level still persisted a flat `rm_state` inside a
+core-shaped case.
+
+`_NORMALIZE_WIRE_TO_CORE` enumerates the migrated types. It is the write-side
+analogue of `KNOWN_WIRE_ESCAPES` and ratchets the opposite way — it may only
+**grow** (`test/architecture/test_normalize_wire_to_core_ratchet.py`). The
+remaining 13 shadowing types differ only by key spelling today and are tracked
+in #2268; five of them (the actor types) have no `to_core()` at all yet.
+
+**A projection failure raises `VultronValidationError`, not `ValueError`.**
+`crud.create()` raises `ValueError` for an already-existing row and callers
+legitimately swallow *that*; sharing the type meant an unprojectable object was
+silently never stored and never logged. The two causes must stay distinguishable
+— see `_pre_store_nested_object` in
+`vultron/adapters/driving/fastapi/routers/actors/_inbox.py` for the correct
+two-branch handler.
+
+**This is defense in depth, not the primary boundary.** Projection belongs at
+wire→core ingress; the persistence boundary is the backstop that guarantees no
+wire-shaped row exists regardless of which ingress path missed it. ADR-0062
+records why both are kept.
+
 ## Activity Read-Back: Semantic Content vs. Envelope Reconstitution (ADR-0035, DL-06)
 
 **Decided (ADR-0035).** `dl.read(activity_id)` in `vultron/core/` is a

notes/domain-validation.md

@@ -114,6 +114,66 @@ canonical location instead.
 `behaviors/status/nodes/broadcast.py` was deleted in #1378 after its only
 content (`_find_case_manager_id`) was consolidated into `_resolve_case_manager_id`.
 
+### Exception: shape guards live in `models/_wire_spelling.py`
+
+`vultron/core/models/_helpers.py` cannot import from `vultron.core.states` —
+that is a circular import through `states/__init__.py`. Shape guards tend to
+grow state references (a guard that knows about `rm` eventually wants `RM`), so
+they live in `vultron/core/models/_wire_spelling.py` instead of being colocated
+with `_as_id()` and friends. This is a deliberate deviation from the rule above,
+not an oversight; it exists so the cycle cannot be reintroduced by the next
+guard that needs a state enum.
+
+---
+
+## Shape Guards: One Canonical Reader per Dimension (#2232)
+
+`ParticipantStatus` exists in two incompatible shapes: core nests
+`rm: RmDimension` / `vfd: VfdDimension` (SDO-03-002, ADR-0036), while the wire
+projection carries flat `rm_state` / `vfd_state`. Reading a dimension off the
+wrong shape yields `None` — which every reader then quietly substituted an
+initial state for, resetting the participant's ladder (#2264).
+
+**Read a dimension only through its canonical reader.** Both live in
+`vultron/core/models/participant_status.py`:
+
+| Reader | Returns | Raises |
+|---|---|---|
+| `participant_status_rm_state(status)` | the `RM` state | `VultronValidationError` on a non-core shape |
+| `participant_status_vfd_state(status)` | the `CS_vfd` state | `VultronValidationError` on a non-core shape |
+
+```python
+# Wrong — a wire-shaped status degrades to the initial state, silently.
+rm_dim = getattr(status, "rm", None)
+state = getattr(rm_dim, "state", None)
+if not isinstance(state, RM):
+    state = RM.START
+
+# Right — absence and shape mismatch are different outcomes.
+state = participant_status_rm_state(status)
+```
+
+This is the strict/loose rule applied to *shape*: an **empty** status list is a
+legitimate absence and callers must handle it (check `participant_statuses`
+before calling); a status that exists but exposes no usable dimension is a shape
+mismatch and must raise (ARCH-15-001, ARCH-15-002).
+
+**Where a raise is wrong.** At a wire→core ingress boundary, a wire-shaped
+status is *legitimate inbound data*, not a corrupt row. Those sites must
+**project** before reading — `as_ParticipantStatus.to_core()`, or
+`_project_to_core_participant()` in
+`vultron/core/use_cases/received/case/_helpers.py` — rather than let the reader
+raise. Making the reader strict without projecting at ingress first aborted the
+entire received-case behavior tree on every inbound `Announce`, which is how the
+first fix for #2232 regressed.
+
+The mirror-image guard is `reject_wire_spelled_keys()` in
+`vultron/core/models/_wire_spelling.py`: a core type validated against a
+wire-spelled (camelCase) payload drops every snake-only key in silence, because
+Pydantic v2 ignores unknown keys. It is computed per exact class, so a
+`CaseParticipant` role subclass that adds a field is covered without any
+registration step.
+
 ---
 
 ## Routing Failures vs. Validation Failures

plan/incoming/learnings/20260812-core-models-helpers-cannot-import-states.md

@@ -0,0 +1,40 @@
+---
+title: Concern — notes/domain-validation.md directs helpers to a module that cannot import core.states
+type: learning
+timestamp: 2026-08-12
+source: ISSUE-2232
+signal: concern
+---
+
+`notes/domain-validation.md` says canonical layer-neutral helpers belong in
+`vultron/core/models/_helpers.py`.  That guidance is **unfollowable for any
+helper that needs a state enum**, and the failure mode is a confusing
+`ImportError` rather than a clear rejection.
+
+`vultron/core/models/base.py` imports `_helpers` (for `_new_urn`/`_now_utc`), so
+`_helpers` is loaded *during* `models.base` initialisation.  Importing
+`vultron.core.states.rm` from `_helpers` first executes
+`vultron/core/states/__init__.py`, which pulls `states/cs.py` →
+`states/common.py` → back into `models.base` — still partially initialised:
+
+```text
+ImportError: cannot import name 'NonEmptyString' from partially initialized
+module 'vultron.core.models.base' (most likely due to a circular import)
+```
+
+The trap is that `states/rm.py`'s *own* imports are clean (logging, enum,
+transitions, `states.common`).  Inspecting the target module tells you nothing;
+the cycle runs through the package `__init__`.  So `_helpers.py` is usable only
+for helpers that depend on nothing outside stdlib and `TYPE_CHECKING`.
+
+`participant_status_rm_state` was instead placed in
+`vultron/core/models/participant_status.py`, next to the model it reads.  That
+is arguably the better home regardless — the canonical reader for a type lives
+with the type — but it was reached by hitting the wall, not by design.
+
+**Suggested fix:** amend `notes/domain-validation.md` to state the constraint
+explicitly, and to direct type-specific canonical readers to the type's own
+module. A shared BT-node wrapper (`read_rm_states`) went into
+`vultron/core/behaviors/helpers.py`, which has no such restriction.
+
+Related: #2269 (that placement was also forced by the `append.py` line ceiling).

plan/incoming/learnings/20260812-datalayer-write-path-has-no-shape-spec.md

@@ -0,0 +1,35 @@
+---
+title: Spec gap — DL-05 covers DataLayer read shape but nothing governs the write path
+type: learning
+timestamp: 2026-08-12
+source: ISSUE-2232
+signal: spec-gap
+---
+
+DL-05-001..004 and ADR-0034 specify that the DataLayer **port returns core
+objects**, and `test/architecture/test_dl_read_returns_core_objects.py`
+enforces it with a shrink-only `KNOWN_WIRE_ESCAPES` ratchet.  There is no
+corresponding requirement on the **write** side: nothing forbids storing a
+wire-shaped payload in a core-typed row.
+
+That asymmetry is what made #2232 possible.  The only write-path shape guard
+was `if obj.type_.startswith("as_")` in `Record.from_obj` — but wire
+vocabulary `type_` values are *bare* names (`"CaseParticipant"`), not
+`as_`-prefixed, so the guard never fires for the 15 wire classes whose `type_`
+is also a `CORE_VOCABULARY` key.  A wire `as_ParticipantStatus` (flat
+`rm_state`) was written into the `ParticipantStatus` table, and core readers
+doing `status.rm.state` got `None`.
+
+Enforcing "reads return core" without enforcing "writes store core" only
+guarantees the *type* of the object handed back, not that the row's field
+shape matches the class reading it.  A read-side ratchet cannot detect a
+malformed row; it can only confirm the class it instantiated.
+
+**Suggested spec addition** (companion to DL-05-001..004): a DataLayer write
+MUST store the canonical core field shape for any `type_` present in
+`CORE_VOCABULARY`, with a shrink-only exemption set mirroring
+`KNOWN_WIRE_ESCAPES`.  #2232 implemented this for `CaseParticipant` and
+`ParticipantStatus` (`_NORMALIZE_WIRE_TO_CORE` in
+`vultron/adapters/driven/db_record.py`); the remaining 13 types are tracked
+in #2268, which is where the architecture test asserting set completeness
+belongs.