feat(EN-1550): idempotent mirror ingest on v2LogId

AGENTS.md

@@ -16,7 +16,7 @@ This document contains rules and conventions for AI agents working on this codeb
 6. **Every FSM `Registry.X.Get(...)` must have a matching preload, declared by the component that proposes the command** — The FSM apply path reads from the in-memory cache; a cache miss turns the read into a silent no-op. Each component that emits a proposal (the metadata converter, the index builder, the cluster-config reconciler, the idempotency-eviction scheduler, the mirror worker, admission) is responsible for declaring its own `preload.Needs` covering every key its apply path will read. There is NO central proposal→needs registry — coupling the preload package to every proposal type creates a single point that easily falls behind. The component knows what it reads; the component declares it. The shared `proposeTechnical` helper takes a `*preload.Needs` parameter the caller fills in (pass nil or an empty `Needs` when the apply path has no cache-keyed reads, e.g. cluster config / idempotency eviction). The preload populates the cache via `MirrorPreload` with a fresh value read at propose time (Pebble fallback on cache miss), and `PredictedIndex` catches mutations between propose and apply.
 7. **Never silently skip a "should not happen" branch** — A branch that is reachable only if an invariant is violated (nil where the contract says non-nil, a state we believe unreachable, a cache miss after a guaranteed preload, etc.) MUST surface a loud signal: `return fmt.Errorf("invariant: ...")` so it bubbles up, or `assert.Unreachable(...)` for SUT-level invariants exercised under antithesis. A silent `return nil` / `continue` on these branches hides real bugs — particularly catastrophic in the FSM apply path, where a no-op desyncs nodes from each other. Branches that represent genuine runtime conditions (cache miss as an expected outcome, stale proposal, deleted entity) keep their soft `return nil`. The distinction is whether the case is *expected* (soft skip OK) or *impossible by design* (must fail loudly). The comment must say *why* the case is impossible so a reader can decide whether to add a hard fail or relax the rule.
 
-8. **The audit log is the only source of truth — every other persisted dataset is a projection and must be verified by the checker** — Only `AuditEntry` (zone `Cold`, sub `Audit`) is cryptographically bound, via the hash chain that `state.BuildHashedHeaderPayload` + `processing.HashGenerator` produce and `checker.verifyAuditHashChain` verifies on every Check() run. Everything else stored in Pebble — `Log`, `AuditItem`, `AppliedProposal`, `LedgerLog.PurgedVolumes`, attribute caches (`Volume`, `Metadata`, `Transaction`, `Reference`, `Boundary`, etc.), reversion bitsets, idempotency keys, mirror cursors, chapters, bloom filters, signing keys, the read-side index — is a *projection* of orders that already live in the audit chain. Projections are rebuildable from the audit on demand, so we deliberately do NOT extend the hash chain to cover them (refactor over hash binding — see `feedback_audit_is_source_of_truth`). In exchange, **`internal/application/check/checker.go` MUST verify every projection it persists**: re-derive the value the projection should hold by replaying the audit (`ReplayLedgerLog`, `SimulateEphemeralPurge`, `partitionVolumes`, etc.) and compare to what is stored, emitting the matching `CHECK_STORE_ERROR_TYPE_*` event on divergence. A projection that the checker does not verify is a tampering vector — adding a new persisted projection without a matching compare* / collect* pass in the checker is the violation. The current passes are `compareVolumes`, `compareMetadata`, `compareTransactions`, `compareExclusionProjections` (AppliedProposal.TransientVolumes + LedgerLog.PurgedVolumes), `checkReversionInvariants`, `verifySealingHash`, `compareIdempotencyOutcomes` (frozen idempotency outcomes in SubIdempKeys vs the hash-chained AuditFailure/AuditSuccess that wrote them — the failure kind is re-derived from the chain-bound reason via `domain.KindForReason`, never stored), and `compareIndexes` (SubAttrIndex registry vs CreateIndex/DropIndex/RemovedMetadataFieldType/DeleteLedger logs — covers presence + identity; BuildStatus is intentionally excluded because it is purely informational on the cluster-wide registry entry — queries gate on the per-replica `IndexVersionState.CurrentVersion`, not on BuildStatus); extend the list as new persisted projections land.
+8. **The audit log is the only source of truth — every other persisted dataset is a projection and must be verified by the checker** — Only `AuditEntry` (zone `Cold`, sub `Audit`) is cryptographically bound, via the hash chain that `state.BuildHashedHeaderPayload` + `processing.HashGenerator` produce and `checker.verifyAuditHashChain` verifies on every Check() run. Everything else stored in Pebble — `Log`, `AuditItem`, `AppliedProposal`, `LedgerLog.PurgedVolumes`, attribute caches (`Volume`, `Metadata`, `Transaction`, `Reference`, `Boundary`, etc.), reversion bitsets, idempotency keys, mirror cursors, chapters, bloom filters, signing keys, the read-side index — is a *projection* of orders that already live in the audit chain. Projections are rebuildable from the audit on demand, so we deliberately do NOT extend the hash chain to cover them (refactor over hash binding — see `feedback_audit_is_source_of_truth`). In exchange, **`internal/application/check/checker.go` MUST verify every projection it persists**: re-derive the value the projection should hold by replaying the audit (`ReplayLedgerLog`, `SimulateEphemeralPurge`, `partitionVolumes`, etc.) and compare to what is stored, emitting the matching `CHECK_STORE_ERROR_TYPE_*` event on divergence. A projection that the checker does not verify is a tampering vector — adding a new persisted projection without a matching compare* / collect* pass in the checker is the violation. The current passes are `compareVolumes`, `compareMetadata`, `compareTransactions`, `compareExclusionProjections` (AppliedProposal.TransientVolumes + LedgerLog.PurgedVolumes), `checkReversionInvariants`, `verifySealingHash`, `compareIdempotencyOutcomes` (frozen idempotency outcomes in SubIdempKeys vs the hash-chained AuditFailure/AuditSuccess that wrote them — the failure kind is re-derived from the chain-bound reason via `domain.KindForReason`, never stored), `compareIndexes` (SubAttrIndex registry vs CreateIndex/DropIndex/RemovedMetadataFieldType/DeleteLedger logs — covers presence + identity; BuildStatus is intentionally excluded because it is purely informational on the cluster-wide registry entry — queries gate on the per-replica `IndexVersionState.CurrentVersion`, not on BuildStatus), and `compareMirrorV2LogID` (stored `LedgerBoundaries.last_mirror_v2_log_id` **==** `max(audited MirrorIngest.v2_log_id)` per ledger — a full equality check, `CHECK_STORE_ERROR_TYPE_MIRROR_V2LOGID_MISMATCH` on ANY divergence; the FSM enforces a contiguous applied prefix so at rest the two must be exactly equal, and the baseline floor is seeded from archived `Boundary` rows now included in the compact baseline snapshot; pre-field clusters are unsupported, no backfill leniency); extend the list as new persisted projections land.
 
 9. **Never bypass the FSM coverage gate** — Every cache-attribute read on the FSM hot path MUST go through `Scope.GetX(...)` so the per-order `coverage_bits` admit it. Reading the underlying `Registry.X.KeyStore().M` (or any other parent-cache iterator) directly skips the gate and produces non-deterministic FSM behavior: the gate is what binds the order to the admission-declared preload set, and a direct read silently sees keys the proposer never declared. There is NO documented exception — paths that need to iterate (e.g. cascade-on-delete) MUST either declare the relevant `preload.Needs` upfront, defer the work to a lifecycle path (`batch.deleteLedgerData` + `MarkLedgerForCleanup`), or be rejected at design review. New helpers that scan the parent KeyStore from inside an order/TU handler are the violation, even when wrapped in a method on `WriteSet`. The coverage gate exists precisely so admission's declared key set is the FSM's only legitimate read horizon — under no circumstances should the apply path widen it on the fly.
 

docs/technical/architecture/audit-vs-technical-state.md

@@ -93,17 +93,40 @@ manual repair, or future tooling. The cursor is wrong in **both** directions:
 - A cursor that advances **beyond** the highest audited `MirrorIngest.v2LogId`
   makes the worker skip source logs without any audit entry showing the missing
   proposals — silent under-ingestion.
-- A cursor that drops **below** the highest audited `v2LogId` is equally unsafe.
-  `Worker.processBatch` (`internal/application/mirror/worker.go`) reloads the
-  cursor and calls `FetchLogs(cursor, ...)`, which returns source logs strictly
-  *after* the cursor. `TranslateBatch` then emits fresh `MirrorIngest` orders for
-  logs that were already ingested, and `processMirrorCreatedTransaction`
-  (`internal/domain/processing/processor_mirror.go`) reapplies every posting with
-  `force=true`, overwriting the transaction state and re-adding the amounts to the
-  volume pair via `applyPosting`. There is **no deduplication on `v2LogId`** and
-  volume mutation is additive, so a lowered cursor replays already-audited
-  transactions and applies their balance effects a second time — silent
-  double-application.
+- A cursor that drops **below** the highest audited `v2LogId` makes
+  `Worker.processBatch` (`internal/application/mirror/worker.go`) reload the
+  cursor and call `FetchLogs(cursor, ...)`, which returns source logs strictly
+  *after* the cursor. `TranslateBatch` then re-emits `MirrorIngest` orders for
+  logs that were already ingested. **The FSM now enforces a contiguous applied
+  prefix (EN-1550).** `processMirrorIngest`
+  (`internal/domain/processing/processor_mirror.go`) records the highest applied
+  source id in `LedgerBoundaries.last_mirror_v2_log_id` — a true contiguous
+  prefix, since the worker ingests contiguously including `FillGap` orders — and,
+  *before* applying any posting or mutating any state, first rejects a malformed
+  `v2LogId == 0` (source v2 log ids are 1-based, so 0 is tamper/corruption) with
+  `ErrMirrorV2LogIDInvalid` (`KindInternal`) — it is never applied, since a 0 is
+  never recorded as `last` and a silently-applied 0 would be re-appliable forever.
+  It then decides three ways against the next slot (`expected = last + 1`):
+  - `v2LogId <= last`: already applied. Idempotent **no-op** — return `(nil, nil)`,
+    which `ProcessOrders` treats as "no log" (no sequence id consumed, no
+    audit-visible `Log`, no sink absorb). Postings are not re-forced, so balances
+    cannot double (pre-EN-1550 this replay caused silent **double-application** via
+    `force=true` additive volume mutation with no dedup on `v2LogId`).
+  - `v2LogId == expected`: the next contiguous log. Apply and advance `last`.
+  - `v2LogId > expected`: a **gap** — the cursor/high-water mark is ahead of the
+    applied prefix. Impossible under contiguous ingestion, so it is
+    corruption/tampering. The FSM **fails loud** (`ErrMirrorV2LogIDGap`,
+    `KindInternal`) and mutates nothing — it never silently applies past the gap
+    (which would desync nodes) or skips it. The rejection is deterministic (same
+    input → same rejection on every node); the worker surfaces it as a repeating
+    apply error rather than corrupting state.
+
+  The guard is a pure function of applied per-ledger state; v2 log ids are 1-based
+  and strictly increasing per source (`TranslateBatch`), so it is FSM-deterministic
+  and needs no new preload/coverage key (it lives inside the already-covered
+  boundaries). Only the **behind** direction is handled here; recovering a cursor
+  that is *ahead* of the true source head is worker-side source-head recovery and
+  remains out of scope.
 
 Because it is only correct at one exact value, treat `MirrorCursor` as a
 persisted per-ledger projection (`ZonePerLedger` / `SubPLMirrorCursor`, in the
@@ -138,6 +161,32 @@ every node identically. Mirror recovery, repair tooling, or a checker pass
 that re-derives the expected cursor from the audited mirror-ingest logs must
 enforce the equality before the stored cursor can be trusted.
 
+`LedgerBoundaries.last_mirror_v2_log_id` (the EN-1550 idempotency high-water
+mark) **is** covered by the checker as a full invariant-#8 **equality** check.
+`compareMirrorV2LogID` (`internal/application/check/checker.go`) verifies, per
+ledger, that the stored `last_mirror_v2_log_id` **equals** `max(audited
+MirrorIngest.v2_log_id)`, emitting `CHECK_STORE_ERROR_TYPE_MIRROR_V2LOGID_MISMATCH`
+on **any** divergence. Because the FSM enforces a contiguous applied prefix (the
+gap reject above), the persisted high-water mark must be exactly the max audited
+`v2_log_id` at rest — no more, no less:
+- `stored > max`: claims a source v2 log no audit entry recorded (future ingests
+  wrongly skipped).
+- `stored < max`: the projection lost applied ground (an already-applied ingest
+  would be re-applied).
+
+The audited max is derived from the live audit chain
+(`recordMirrorIngestMutations`) layered over a baseline floor
+(`foldBaselineBoundaries` seeds it from the archived
+`LedgerBoundaries.last_mirror_v2_log_id`, which the compact baseline snapshot now
+carries — `writeBaselineAttributes` includes `Boundary` rows), so a ledger whose
+mirror ingests live only in an archived chapter is not undercounted. A regular
+(never-mirrored) ledger has stored `0` and audited max `0` → equal → not flagged.
+
+There is **no** legacy / no-backfill leniency: existing pre-field clusters are
+**unsupported** — we do not ship a compat shim, and a store that predates
+`last_mirror_v2_log_id` would fail this equality (and simply re-mirror), which is
+acceptable pre-GA.
+
 ## Readstore and Indexes
 
 The index **registry** is business-visible and is already verified by the

docs/technical/architecture/subsystems/checker/checker.md

@@ -40,11 +40,13 @@ Each pass takes a persisted projection, re-derives the expected value by replayi
 | 7 | `compareExclusionProjections` | `AppliedProposal.TransientVolumes` and `LedgerLog.PurgedVolumes` agree with what `SimulateEphemeralPurge` would have produced | Replay + `SimulateEphemeralPurge` | `EXCLUSION_RECORD_MISMATCH` |
 | 8 | `compareIdempotencyOutcomes` | Frozen idempotency outcomes in `SubIdempKeys` match the outcome of the chain-bound `AuditSuccess` / `AuditFailure` that wrote them | Outcome map built during `verifyAuditHashChain` | `IDEMPOTENCY_MISMATCH` |
 | 9 | `compareIndexes` | `SubAttrIndex` registry matches the set derived from `CreateIndex` / `DropIndex` / `RemovedMetadataFieldType` / `DeleteLedger` logs (presence + identity only) | Replay of the index-affecting log types | `INDEX_MISMATCH` |
+| 10 | `compareMirrorV2LogID` | Stored `LedgerBoundaries.last_mirror_v2_log_id` **equals** the max audited `MirrorIngest.v2_log_id` per ledger (full equality) | Live audit chain (`recordMirrorIngestMutations`) over a baseline floor (`foldBaselineBoundaries`) | `MIRROR_V2LOGID_MISMATCH` (any divergence) |
 
 Notes:
 
 - **`compareIndexes` covers presence + identity** (`IndexID` match), **not `BuildStatus`** and **not `IndexVersionState`**. `BuildStatus` is informational and per-replica; `IndexVersionState` is by design local and may legitimately differ across nodes mid-rewrite (see [indexer / indexes.md](../indexer/indexes.md)).
 - **`compareIdempotencyOutcomes`** is the one pass that consumes a side-effect of `verifyAuditHashChain` (the expected-outcome map). The two are coupled by design — re-deriving the idempotency outcome means re-walking the chain anyway.
+- **`compareMirrorV2LogID` is a full equality check.** The FSM enforces a contiguous applied prefix (`processMirrorIngest` rejects any `v2LogId` gap with `ErrMirrorV2LogIDGap`), so at rest the stored EN-1550 high-water mark must be exactly `max(audited v2_log_id)`. Any divergence is flagged: `stored > max` claims a source log the audit never recorded; `stored < max` means the projection lost applied ground. A never-mirrored ledger has stored `0` and max `0` → equal → clean. There is no legacy/no-backfill leniency — pre-field clusters are unsupported. The audited max is seeded from a baseline floor (the compact baseline snapshot now includes `Boundary` rows) so archived-only mirror ingests are not undercounted.
 - **Order matters**: `verifyAuditHashChain` runs **first**. A broken chain stops the walk before any downstream pass — running them with a tampered chain would produce noise from already-detected corruption.
 
 ## Replay machinery
@@ -73,7 +75,8 @@ flowchart TB
     B --> H[verifySealingHash]
     B --> I[compareExclusionProjections]
     B --> J[compareIndexes]
-    D & E & F & G & H & I & J --> K[stream errors as they happen]
+    B --> L[compareMirrorV2LogID]
+    D & E & F & G & H & I & J & L --> K[stream errors as they happen]
     C --> K
 ```