Clarify checker store invariant

AGENTS.md

@@ -16,7 +16,7 @@ This document contains rules and conventions for AI agents working on this codeb
 6. **Every FSM `Registry.X.Get(...)` must have a matching preload, declared by the component that proposes the command** — The FSM apply path reads from the in-memory cache; a cache miss turns the read into a silent no-op. Each component that emits a proposal (the metadata converter, the index builder, the cluster-config reconciler, the idempotency-eviction scheduler, the mirror worker, admission) is responsible for declaring its own `preload.Needs` covering every key its apply path will read. There is NO central proposal→needs registry — coupling the preload package to every proposal type creates a single point that easily falls behind. The component knows what it reads; the component declares it. The shared `proposeTechnical` helper takes a `*preload.Needs` parameter the caller fills in (pass nil or an empty `Needs` when the apply path has no cache-keyed reads, e.g. cluster config / idempotency eviction). The preload populates the cache via `MirrorPreload` with a fresh value read at propose time (Pebble fallback on cache miss), and `PredictedIndex` catches mutations between propose and apply.
 7. **Never silently skip a "should not happen" branch** — A branch that is reachable only if an invariant is violated (nil where the contract says non-nil, a state we believe unreachable, a cache miss after a guaranteed preload, etc.) MUST surface a loud signal: `return fmt.Errorf("invariant: ...")` so it bubbles up, or `assert.Unreachable(...)` for SUT-level invariants exercised under antithesis. A silent `return nil` / `continue` on these branches hides real bugs — particularly catastrophic in the FSM apply path, where a no-op desyncs nodes from each other. Branches that represent genuine runtime conditions (cache miss as an expected outcome, stale proposal, deleted entity) keep their soft `return nil`. The distinction is whether the case is *expected* (soft skip OK) or *impossible by design* (must fail loudly). The comment must say *why* the case is impossible so a reader can decide whether to add a hard fail or relax the rule.
 
-8. **The audit log is the only source of truth — every other persisted dataset is a projection and must be verified by the checker** — Only `AuditEntry` (zone `Cold`, sub `Audit`) is cryptographically bound, via the hash chain that `state.BuildHashedHeaderPayload` + `processing.HashGenerator` produce and `checker.verifyAuditHashChain` verifies on every Check() run. Everything else stored in Pebble — `Log`, `AuditItem`, `AppliedProposal`, `LedgerLog.PurgedVolumes`, attribute caches (`Volume`, `Metadata`, `Transaction`, `Reference`, `Boundary`, etc.), reversion bitsets, idempotency keys, mirror cursors, chapters, bloom filters, signing keys, the read-side index — is a *projection* of orders that already live in the audit chain. Projections are rebuildable from the audit on demand, so we deliberately do NOT extend the hash chain to cover them (refactor over hash binding — see `feedback_audit_is_source_of_truth`). In exchange, **`internal/application/check/checker.go` MUST verify every projection it persists**: re-derive the value the projection should hold by replaying the audit (`ReplayLedgerLog`, `SimulateEphemeralPurge`, `partitionVolumes`, etc.) and compare to what is stored, emitting the matching `CHECK_STORE_ERROR_TYPE_*` event on divergence. A projection that the checker does not verify is a tampering vector — adding a new persisted projection without a matching compare* / collect* pass in the checker is the violation. The current passes are `compareVolumes`, `compareMetadata`, `compareTransactions`, `compareExclusionProjections` (AppliedProposal.TransientVolumes + LedgerLog.PurgedVolumes), `checkReversionInvariants`, `verifySealingHash`, `compareIdempotencyOutcomes` (frozen idempotency outcomes in SubIdempKeys vs the hash-chained AuditFailure/AuditSuccess that wrote them — the failure kind is re-derived from the chain-bound reason via `domain.KindForReason`, never stored), and `compareIndexes` (SubAttrIndex registry vs CreateIndex/DropIndex/RemovedMetadataFieldType/DeleteLedger logs — covers presence + identity; BuildStatus is intentionally excluded because it is purely informational on the cluster-wide registry entry — queries gate on the per-replica `IndexVersionState.CurrentVersion`, not on BuildStatus); extend the list as new persisted projections land.
+8. **The checker verifies the authoritative store, not every derived projection** — Only `AuditEntry` (zone `Cold`, sub `Audit`) is cryptographically bound, via the hash chain that `state.BuildHashedHeaderPayload` + `processing.HashGenerator` produce and `checker.verifyAuditHashChain` verifies on every Check() run. The checker must be able to validate the main persisted ledger state against the audit-derived state: when data is part of that authoritative store, or can affect correctness independently of the store it is derived from, `internal/application/check/checker.go` must re-derive the expected value from the audit chain or another already-verified source and compare it to what is stored, emitting the matching `CHECK_STORE_ERROR_TYPE_*` event on divergence. Persisted projections derived from the verified store — denormalized counters, rebuildable read models, informational build status, caches, and other refreshable datasets — do not each require their own compare pass merely because they are persisted. A replication/ingestion cursor is *technical* state, not a ledger business invariant: `MirrorCursor` (main store) is only the mirror worker's progress pointer into an external v2 source, and it is written **in the same atomic Raft-applied Pebble batch as the `MirrorIngest` orders it acknowledges** — the worker bundles the `MirrorSync` TU into the data proposal (`mirror/worker.go`), and `applyProposal` drains that proposal's orders + TUs through one `WriteSet.Merge` and one `batch.Commit()` (`machine.go`). A crash lands both or neither, so the persisted cursor can **never lag the ingested data** through any normal or crash path: the "cursor behind → re-emit already-applied logs" replay is *unreachable operationally* (postings can't be replayed without the cursor having advanced with them). The business effect of mirroring — the ingested transactions — is itself audit-bound (each `MirrorIngest` is an audit-chain order) and already checker-verified (`recordMirrorIngestMutations` feeding `compareVolumes`/`compareTransactions`), so the ledger's correctness holds regardless of the cursor. The only way to move the cursor off that value is to tamper the persisted cursor byte directly, and neither direction is a checker integrity concern. A cursor tampered **behind** self-heals and is checker-invisible: on the next tick the worker re-ingests, the cursor **re-advances atomically** to a value consistent with the (now-duplicated) audit, and the doubled volumes are themselves consistent with an audit that now legitimately contains the duplicate `MirrorIngest` orders — so a cursor compare and `compareVolumes`/`compareTransactions` (which replay the *same* audit the FSM applied) all pass; the real mitigation is FSM-level `v2LogId` idempotency in `processMirrorIngest` (skip an ingest whose `v2LogId` is already applied), a functional hardening tracked separately, not a checker pass. A cursor tampered **ahead** is stable and would make the worker silently skip source logs (v2→v3 completeness loss) — but that is a *parity* property against the external v2 source, verifiable only against the worker's tracked source-head (`SubPLMirrorSourceHead`), not against the audit the checker replays; its home is the mirror worker's recovery reconciliation, not an integrity compare pass. Classify such replication/sync cursors as technical state; the checker guards the *business* invariants of the main store (balances, transactions, metadata, reversions, idempotency, index registry), which are audit-derived and independent of these cursors. The rule for new persisted data is to classify it explicitly: if it is authoritative, security-sensitive, or can affect ledger business correctness independently of the store it is derived from, add a compare* / collect* pass or an equivalent recovery-side validation; it may be exempted as derived ONLY when it is deterministically rebuildable from still-retained verified state AND that rebuild is a tested, wired detect→reset→rebuild path that runs before the projection is served — a projection that is only rebuildable *in principle* (no executable rebuild yet) stays correctness-affecting and needs checker coverage. (Wiring the dedicated rebuild path for the currently-derived projections that lack one — e.g. `usagestore` — is tracked as a follow-up PR; until it lands, do not rely on the derived-exemption for those.) The current checker passes include `compareVolumes`, `compareMetadata`, `compareTransactions`, `compareExclusionProjections` (AppliedProposal.TransientVolumes + LedgerLog.PurgedVolumes), `checkReversionInvariants`, `verifySealingHash`, `compareIdempotencyOutcomes` (frozen idempotency outcomes in SubIdempKeys vs the hash-chained AuditFailure/AuditSuccess that wrote them — the failure kind is re-derived from the chain-bound reason via `domain.KindForReason`, never stored), and `compareIndexes` (SubAttrIndex registry vs CreateIndex/DropIndex/RemovedMetadataFieldType/DeleteLedger logs — covers presence + identity; BuildStatus is intentionally excluded because it is purely informational on the cluster-wide registry entry — queries gate on the per-replica `IndexVersionState.CurrentVersion`, not on BuildStatus); extend the checker when new authoritative persisted state lands.
 
 9. **Never bypass the FSM coverage gate** — Every cache-attribute read on the FSM hot path MUST go through `Scope.GetX(...)` so the per-order `coverage_bits` admit it. Reading the underlying `Registry.X.KeyStore().M` (or any other parent-cache iterator) directly skips the gate and produces non-deterministic FSM behavior: the gate is what binds the order to the admission-declared preload set, and a direct read silently sees keys the proposer never declared. There is NO documented exception — paths that need to iterate (e.g. cascade-on-delete) MUST either declare the relevant `preload.Needs` upfront, defer the work to a lifecycle path (`batch.deleteLedgerData` + `MarkLedgerForCleanup`), or be rejected at design review. New helpers that scan the parent KeyStore from inside an order/TU handler are the violation, even when wrapped in a method on `WriteSet`. The coverage gate exists precisely so admission's declared key set is the FSM's only legitimate read horizon — under no circumstances should the apply path widen it on the fly.
 

docs/technical/architecture/audit-vs-technical-state.md

@@ -52,7 +52,7 @@ non-deterministic map marshaling). So a value that is corrupted or tampered
 *before* it is proposed takes effect on every node identically, and no cross-node
 comparison — logical or byte-wise — can catch it. Where the sections below
 describe a persisted main-store projection that the checker does not yet verify
-(the mirror cursor, the readstore inverted-index contents, prepared queries,
+(the readstore inverted-index contents, prepared queries,
 persisted bloom blocks on the restart path, and the persisted governance
 projections — signing keys `SubGlobSigningKey`, signing config
 `SubGlobSigningConfig`, and maintenance mode `SubGlobMaintenanceMode`, each read
@@ -86,30 +86,50 @@ is correct: the worker proposes mirror ingest orders and a `MirrorSyncUpdate` in
 the same Raft proposal, and the FSM queues the cursor write through the same
 `WriteSet` so it only advances if the mirrored orders apply successfully.
 
-That atomicity protects normal failures, but it does not by itself prove that the
-stored cursor is still consistent with audited mirror orders after restore,
-manual repair, or future tooling. The cursor is wrong in **both** directions:
-
-- A cursor that advances **beyond** the highest audited `MirrorIngest.v2LogId`
-  makes the worker skip source logs without any audit entry showing the missing
-  proposals — silent under-ingestion.
-- A cursor that drops **below** the highest audited `v2LogId` is equally unsafe.
+That atomicity is the load-bearing property: because the `MirrorSyncUpdate` rides
+in the *same* proposal as the ingest orders and drains through the *same*
+`WriteSet.Merge` / `batch.Commit()`, the persisted cursor **can never lag or lead
+the ingested data through any normal or crash path** — a crash lands both or
+neither. So through normal operation the stored cursor always equals the highest
+audited `MirrorIngest.v2LogId`, and the only way to move it off that value is to
+tamper the persisted cursor byte directly (after restore, manual repair, or a
+Pebble-level edit). The two tamper directions behave very differently, and
+neither is an *integrity-checker* concern:
+
+- A cursor tampered **below** the highest audited `v2LogId` replays.
   `Worker.processBatch` (`internal/application/mirror/worker.go`) reloads the
   cursor and calls `FetchLogs(cursor, ...)`, which returns source logs strictly
   *after* the cursor. `TranslateBatch` then emits fresh `MirrorIngest` orders for
   logs that were already ingested, and `processMirrorCreatedTransaction`
   (`internal/domain/processing/processor_mirror.go`) reapplies every posting with
   `force=true`, overwriting the transaction state and re-adding the amounts to the
   volume pair via `applyPosting`. There is **no deduplication on `v2LogId`** and
-  volume mutation is additive, so a lowered cursor replays already-audited
-  transactions and applies their balance effects a second time — silent
-  double-application.
-
-Because it is only correct at one exact value, treat `MirrorCursor` as a
-persisted per-ledger projection (`ZonePerLedger` / `SubPLMirrorCursor`, in the
-main Pebble store) that must **equal** the highest audited `MirrorIngest.v2LogId`
-for the target ledger — not merely be less than or equal to it. Explicit edge
-cases:
+  volume mutation is additive, so the balance effects apply a second time — but
+  the replay **self-heals into an audit-consistent state**: the duplicate
+  `MirrorIngest` orders go through the real proposal path, so they are legitimately
+  hash-chained, the cursor re-advances atomically, and the doubled volumes match
+  an audit that now contains the duplicates. A `compare*` pass that re-derives
+  from the audit (`compareVolumes` / `compareTransactions`) therefore replays the
+  *same* orders the FSM applied and **cannot see the doubling**; a cursor-equality
+  pass also passes because the cursor has re-advanced. Verifying the cursor at
+  rest catches nothing here — the real mitigation is FSM-level `v2LogId`
+  idempotency in `processMirrorIngest` (skip an ingest whose `v2LogId` is already
+  applied), a functional hardening tracked separately.
+- A cursor tampered **beyond** the highest audited `v2LogId` makes the worker skip
+  source logs — silent v2→v3 under-ingestion. This *is* stable and at-rest
+  detectable, but only as a **parity** property against the external v2 source:
+  the expected bound is the worker's tracked source-head (`SubPLMirrorSourceHead`,
+  refreshed by `refreshSourceHead`), not anything the checker can re-derive from
+  the v3 audit alone (the skipped logs never entered the v3 audit, which stays
+  internally consistent). Its home is the mirror worker's recovery reconciliation
+  against the source-head, not an audit-vs-store integrity compare.
+
+`MirrorCursor` is therefore classified as **technical replication state**, not a
+checker business invariant: the ledger's business truth (balances, transactions,
+metadata) is carried by the audit-bound `MirrorIngest` orders and already verified
+by the normal checker passes, independent of the cursor. It lives at
+`ZonePerLedger` / `SubPLMirrorCursor` in the main Pebble store. Edge cases worth
+recording for the worker/recovery side (not the checker):
 
 - **Empty / never-ingested ledger:** the cursor is absent and reads back as `0`
   (`query.ReadMirrorCursor` defaults to `0`, and `processBatch` maps `0` to
@@ -130,13 +150,18 @@ cases:
   logged, purge not yet reached) as the transient case to exclude — not assume
   a deleted ledger keeps a stale cursor forever.
 
-This equality is **not** currently enforced: there is no `compare*` pass for
-`MirrorCursor` in `internal/application/check`, so per invariant #8 this is a
-known integrity gap, not an approved exemption. Raft replication does not close
-it — a cursor value corrupted or tampered before it is proposed takes effect on
-every node identically. Mirror recovery, repair tooling, or a checker pass
-that re-derives the expected cursor from the audited mirror-ingest logs must
-enforce the equality before the stored cursor can be trusted.
+There is deliberately **no** `compare*` pass for `MirrorCursor` in
+`internal/application/check`. Per invariant #8 the checker guards the business
+invariants of the main store against the audit; the cursor is not one of them
+(see the two tamper directions above — the *behind* case is checker-invisible and
+needs functional `v2LogId` idempotency, the *ahead* case is a v2-parity property
+owned by the mirror worker's source-head reconciliation). This is a classification
+as technical state, not an unapproved integrity gap: the audit-bound
+`MirrorIngest` orders and the existing volume/transaction passes already secure
+the business truth. Hardening the two tamper directions (ingest idempotency; a
+worker/recovery reconciliation of the cursor against source-head and the highest
+audited `v2LogId`) is tracked as separate functional follow-up, not as a checker
+compare pass.
 
 ## Readstore and Indexes