feat(EN-1334): usagebuilder subsystem + event-counter cutover (EN-1420)

.golangci.yaml

@@ -163,6 +163,11 @@ linters:
     # (not OpenWriteSession on the main store).
     - path: internal/storage/readstore/
       linters: [forbidigo]
+    # The usagestore is a peer secondary store to readstore, dedicated to the
+    # usagebuilder projections. It manages its own Pebble DB and uses
+    # NewWriteSessionFromDB by the same rationale.
+    - path: internal/storage/usagestore/
+      linters: [forbidigo]
     # Tests use OpenWriteSession to seed data; the structural guarantee is on
     # production paths.
     - path: _test\.go$

cmd/ledgerctl/ledgers/stats.go

@@ -75,7 +75,6 @@ func runStats(cmd *cobra.Command, _ []string) error {
 	pterm.Printf("Postings:           %d\n", stats.GetPostingCount())
 	pterm.Printf("Logs:               %d\n", stats.GetLogCount())
 	pterm.Printf("Volumes:            %d\n", stats.GetVolumeCount())
-	pterm.Printf("Metadata:           %d\n", stats.GetMetadataCount())
 	pterm.Printf("References:         %d\n", stats.GetReferenceCount())
 	pterm.Printf("Reverts:            %d\n", stats.GetRevertCount())
 	pterm.Printf("Numscript execs:    %d\n", stats.GetNumscriptExecutionCount())

docs/technical/architecture/overview.md

@@ -154,17 +154,17 @@ Per-ledger boundaries use the `LedgerBoundaries` protobuf message from `raft_cmd
 message LedgerBoundaries {
   fixed64 next_transaction_id = 1;
   fixed64 next_log_id = 2;
-  fixed64 volume_count = 3;
-  fixed64 metadata_count = 4;
-  fixed64 reference_count = 5;
-  fixed64 posting_count = 6;
-  fixed64 ephemeral_evicted_count = 7;
-  fixed64 transient_used_count = 8;
-  fixed64 revert_count = 9;
-  fixed64 numscript_execution_count = 10;
 }
 ```
 
+> Every per-ledger counter — postings, reverts, numscript executions, references, ephemeral evictions, transient volumes, **and volumes** — lives in the usagebuilder side-store (`internal/application/usagebuilder`, `internal/storage/usagestore`). All counters are derived from the audit chain: event counts (posting / revert / numscript-exec / reference) come from replaying orders; transient counts come from `AppliedProposal.TransientVolumes`; ephemeral and volume counts come from three DISJOINT per-log lists that the FSM populates on every `LedgerLog`:
+>
+> - `purged_volumes` — draining evictions (was non-zero in Pebble, now zero, entry deleted at commit)
+> - `new_kept_volumes` — persistent-new (created + survives past commit)
+> - `ephemeral_volumes` — pure ephemeral (created + evicted same log)
+>
+> Volume count delta per log = `len(new_kept_volumes) − len(purged_volumes)`. Pure ephemeral tuples contribute +0 (was zero, is zero) and are tracked only for the index builder's skip logic and for the ephemeral-eviction counter. Splitting ephemeral out of `purged_volumes` avoids the 2× byte cost that a naive union-encoding would pay on ephemeral-heavy workloads. `LedgerBoundaries` carries only the two ID generators that drive apply-time allocation decisions. `metadata_count` is intentionally not exposed — the admission preload no longer injects old metadata values so cardinality cannot be reliably derived at the FSM level; it will come back on a sound foundation later.
+
 > **Note:** The `State` / `LedgerState` proto messages sometimes shown in older documentation are conceptual models, not actual proto definitions. The real FSM state is spread across the `Machine` struct fields and its `StateRegistry`.
 
 ### Benefits of Single Raft

docs/technical/architecture/subsystems/usage/README.md

@@ -0,0 +1,19 @@
+# Usage
+
+The background worker (`internal/application/usagebuilder`) that turns committed audit entries into per-ledger housekeeping counters — postings, reverts, numscript executions, references, ephemeral evictions, transient uses, live volume cardinality — plus per-Numscript-template invocation records. Runs on every node — leader and followers — independently, mirroring the indexer's decoupling from the FSM hot path: the FSM commits and signals; the usagebuilder reads its own Pebble read handle and writes to a dedicated secondary store (`internal/storage/usagestore`).
+
+The projections it maintains are exposed by the API — `GET /v3/{ledger}/stats` for the aggregate counters, `GET /v3/{ledger}/numscripts/{name}/usage` for per-template invocation counts.
+
+## Documents
+
+| Document | Description |
+|----------|-------------|
+| [usagebuilder.md](usagebuilder.md) | Pipeline mechanics: builder loop, audit-chain tailing, per-batch commit, log-payload consumption. |
+| [counters.md](counters.md) | Counter definitions, storage keys, `LedgerLog.{purged,new_kept,ephemeral}_volumes` split, formula for each counter. |
+
+## Related
+
+- [FSM](../fsm/) — emits the `LedgerLog.new_kept_volumes` / `ephemeral_volumes` / `purged_volumes` annotations the usagebuilder consumes. Depends on the volume preload contract (see AGENTS.md invariant #6).
+- [Indexer](../indexer/) — sibling subsystem with the same audit-tailing shape, but writes to a different secondary store (`readstore`).
+- [Checker](../checker/) — verifies the projections against the audit chain (`compareExclusionProjections` for ephemeral/transient tuples).
+- [Storage](../storage/) — the usage store is a peer Pebble instance to the readstore, WAL disabled, own comparer, own Pebble tuning.

docs/technical/architecture/subsystems/usage/counters.md

@@ -0,0 +1,144 @@
+# Counters and Storage Schema
+
+This page documents the projections the usagebuilder materialises: what each counter counts, how the FSM plumbs the source data through the log payload, and how the counters are keyed in the usagestore.
+
+These counters live in the **usagestore**, a peer secondary store. Per the checker-scope framing (CLAUDE.md invariant #8), a peer secondary store is out of the **main-store checker's** scope *by construction* — `Check()` never opens the usagestore — so the checker does **not** verify these counters. Their integrity is a per-replica rebuild-health concern (rebuildable from the audit on demand), not a main-store invariant-#8 projection.
+
+For the pipeline that populates these keys, see [usagebuilder.md](usagebuilder.md).
+
+## Counter Catalogue
+
+Every counter is keyed by `(ledger, counter_id)` in the usagestore (`internal/storage/usagestore/keys.go`) — one byte per counter, stable on-disk identifiers, never renumbered.
+
+| ID | Name | Source | Delta per event |
+|----|------|--------|-----------------|
+| `0x01` | `CounterPosting` | `Transaction.Postings` (`CreatedTransaction`) + `RevertTransaction.Postings` (`RevertedTransaction`) | `+len(Postings)` per applicable log |
+| `0x02` | `CounterRevert` | `RevertTransactionOrder` unmarshalled from `AuditItem.SerializedOrder` (covers both direct and mirror-ingested reverts) | `+1` per **successful** revert (gated on the produced log being a `RevertedTransaction`; a skipped order — `OrderSkipped` log — counts nothing) |
+| `0x03` | `CounterNumscriptExecution` | `CreateTransactionOrder` with a non-empty `Script.Plain` or a non-nil `NumscriptReference` | `+1` per **successful** order (gated on a produced `CreatedTransaction`; skips count nothing) |
+| `0x04` | `CounterReference` | `CreateTransactionOrder.Reference != ""` | `+1` per **successful** order (gated on a produced `CreatedTransaction`; skips count nothing) |
+| `0x05` | `CounterEphemeralEvicted` | `len(LedgerLog.EphemeralVolumes)` per log — the pure-ephemeral tuples (new + purged same log) | `+len(EphemeralVolumes)` |
+| `0x06` | `CounterTransientUsed` | `len(AppliedProposal.TransientVolumes[ledger].Volumes)` — batch-level, keyed by audit sequence | `+len(TransientVolumes[ledger])` per proposal |
+| `0x07` | `CounterVolume` | `len(LedgerLog.NewKeptVolumes) − len(LedgerLog.PurgedVolumes)` per log | net change in live volume-key cardinality |
+
+Missing keys read as `0`. Every counter clamps at zero on underflow via `applyDelta` — a subsystem bug that emits a spurious `−1` cannot drive the counter into an out-of-band representation.
+
+## Template Usage
+
+Per-Numscript-template records live under a separate key shape:
+
+```
+[usagestore prefix][ledger 64B][template_name]  → TemplateUsage { fixed64 count; Timestamp last_used }
+```
+
+Populated when a `CreateTransactionOrder` carries a non-nil `NumscriptReference`. `last_used` is the order's `Timestamp` (deterministic — same on every replica). Multiple invocations in the same batch fold via max on `last_used` and sum on `count`.
+
+Read path: `GET /v3/{ledger}/numscripts/{name}/usage` (`internal/adapter/http/handlers_get_numscript_usage.go`) — returns the persisted `TemplateUsage` proto or a zero-valued one when the template has never been invoked (not a 404).
+
+## The Log-Payload Contract
+
+Six of the seven counters plus template usage need FSM-side annotations on the log or the audit chain. Two categories:
+
+- **Straight from the raw order** (posting, revert, numscript-exec, reference, template usage): no FSM enrichment needed — the fields are already on `raftcmdpb.CreateTransactionOrder` / `RevertTransactionOrder`, which the usagebuilder unmarshalls from `AuditItem.SerializedOrder`.
+- **Volume annotations on the log** (ephemeral evicted, volume count): the FSM computes three DISJOINT per-log lists during `WriteSet.Merge` and injects them into the `LedgerLog` message.
+
+### `LedgerLog` volume annotations
+
+`misc/proto/common.proto` — three disjoint fields on `LedgerLog`:
+
+```protobuf
+message LedgerLog {
+  LedgerLogPayload data       = 1;
+  Timestamp        date       = 2;
+  fixed64          id         = 3;
+  repeated TouchedVolume purged_volumes    = 4;  // DRAINING only
+  repeated TouchedVolume new_kept_volumes  = 5;  // new + survives
+  repeated TouchedVolume ephemeral_volumes = 6;  // new + purged same log
+}
+```
+
+The three sets partition every volume update the log touched into DISJOINT categories:
+
+| Category | Prior value | Post-commit state | Field |
+|----------|-------------|-------------------|-------|
+| Draining | non-zero in Pebble | evicted (zero balance) | `purged_volumes` |
+| New + kept | undefined or zero placeholder | persisted | `new_kept_volumes` |
+| Pure ephemeral | undefined or zero placeholder | evicted (zero balance) | `ephemeral_volumes` |
+| Normal update | defined + non-zero | still persisted (updated value) | (none — no annotation needed) |
+| Transient | any | never persisted | (none — carried on `AppliedProposal.TransientVolumes` at batch level) |
+
+### Why disjoint (and not overlapping)
+
+An earlier encoding included ephemeral tuples in BOTH `purged_volumes` (as "evicted") and `new_volumes` (as "newly created"). Every ephemeral (account, asset) tuple paid its wire cost twice. On workloads with high ephemeral throughput (payout fan-out, escrow pass-throughs — 100+ ephemeral accounts per transaction is normal), the doubling adds a few MB/s of WAL / audit-chain growth for no functional benefit — the ephemeral net delta on `VolumeCount` is +0.
+
+The disjoint encoding pays exactly `len(ephemeral)` per log instead of `2 × len(ephemeral)`. The three lists are still complete: consumers that need "everything evicted from Pebble" take the union (see [indexer consumer](#indexer-consumer)); the volume-count formula becomes clean subtraction (`new_kept − purged`) with ephemeral contributing zero.
+
+### FSM emission
+
+`internal/infra/state/write_set.go` computes the three sets during `Merge`:
+
+1. `partitionVolumes(volumeUpdates)` (existing) yields `partResult.{kept, purged, transient}`.
+2. `splitPurged(partResult.purged)` (new, in `write_set_new_volumes.go`) partitions `purged` further into:
+   - **ephemeral**: `!Old.IsDefined() || isVolumePreloadZero(Old.Value())` — the key had no prior state, was touched, immediately purged.
+   - **draining**: `Old.IsDefined() && !isVolumePreloadZero(Old.Value())` — had a prior non-zero balance, drained to zero, evicted.
+3. `makeNewKeptKeySet(partResult.kept)` (new) yields the subset of `kept` where `Old` was undefined / zero-placeholder — i.e. new persistent volumes.
+4. `buildTouchedByLog(volumes.Slots(), setX)` (new, generalised from the previous `buildPurgedByLog`) intersects the per-order touched-volume tracking with each of the three sets to produce the per-log annotation lists. Deduplication + deterministic sort by (account, asset) keeps the log payload byte-identical across nodes.
+
+The three lists are injected into each `LedgerLog` inside the same `createdLogs` build loop that also injects `purged_volumes`. Note the audit hash chain does **not** cover `LedgerLog` content — it binds the audit header plus each item's order index, log sequence, and serialized order (see [Checker consumer](#checker-consumer) for what this means for tamper detection of the derived counters).
+
+### The preload contract this depends on
+
+The classification "new vs existing" is decided by the preloaded prior value at merge time — specifically `Update.Old.IsDefined()` combined with the zero-placeholder check. This is safe **because volume preload is structurally required by the FSM**: balance checks, Uint256 arithmetic and numscript resolution all read the current volume value, so admission has to preload every touched key. That contract is documented as invariant #6 in AGENTS.md.
+
+The comparable metadata preload was removed opportunistically once the indexer no longer needed it — the corresponding `MetadataCount` counter had to be dropped (see the EN-1420 commit) because `Old.IsDefined()` no longer distinguished "new key" from "overwrite" on the metadata merge. The volume analog holds because the FSM cannot function without those old values.
+
+## Usagestore Layout
+
+```
+[template_prefix][ledger 64B][template_name]   → TemplateUsage proto
+[counter_prefix][ledger 64B][counter_id 1B]    → uint64 BE
+[0xFE][0x01]                                    → progress cursor (uint64 BE, last consumed audit sequence)
+```
+
+Full keyspace conventions in `internal/storage/usagestore/keys.go`. The `[ledger 64B]` block is zero-padded fixed-width (`dal.LedgerNameFixedSize`) so the comparer can extract a per-ledger prefix for bloom-filter scoping — same technique as the readstore.
+
+## Consumers
+
+### API — `GetLedgerStats` reader
+
+`internal/application/ctrl/controller_default.go` — opens a single `usagestore.Snapshot` and routes all seven counter Gets through it. See [usagebuilder.md § Snapshot on the reader side](usagebuilder.md#snapshot-on-the-reader-side).
+
+### API — template usage endpoint
+
+`internal/adapter/http/handlers_get_numscript_usage.go` → `ctrl.Controller.GetTemplateUsage` → `usagestore.GetTemplateUsage`. Single point-read against the live store (no snapshot needed for a single-value query).
+
+### Indexer consumer
+
+`internal/application/indexbuilder/applied_proposal_sync.go`'s `extractPurgedVolumes` returns the UNION of `LedgerLog.PurgedVolumes ∪ LedgerLog.EphemeralVolumes` because both categories share the same downstream treatment (Pebble entry gone → skip acct→tx mapping). The protowire fast path (`protowire_postings.go`) parses field 6 alongside field 4 and exposes `GetEphemeralVolumes` on `parsedLog`.
+
+### Checker consumer
+
+`compareExclusionProjections` (`internal/application/check/checker.go`) accumulates `PurgedVolumes` and `EphemeralVolumes` from every log into the stored projection set, then compares against the exclusion set derived by replaying the audit chain (`AppliedProposal.TransientVolumes` union). Both eviction lists have identical semantics for the checker — the split is a pure log-payload compaction, not an invariant change. This pass verifies the exclusion projection *in the primary store* — the set the indexbuilder consumes.
+
+### Why the usagestore counters are not a checker target (design decision, not a gap)
+
+The checker (invariant #8) verifies projections persisted **in the primary Pebble store** — the store it operates on: `Volume`, `Metadata`, `Transaction`, `Reference`, `Boundary`, idempotency outcomes, the index registry, and the exclusion projection above. The usagestore is a **distinct, peer secondary Pebble instance** (`<data-dir>/usage/`) holding a *derived cache* of counters, rebuilt from the audit chain by the boot/tick fold, which folds forward from the persisted cursor (from an empty cursor, i.e. `usagestore.Reset()` or a wiped `<data-dir>/usage/`, that replays from audit sequence 0). There is no automatic rollback rewind — the audit chain is append-only, so the cursor never sits ahead of the head. Its authoritativeness is explicitly bounded by "eventually consistent with the FSM".
+
+Consequently, the three volume-annotation categories are deliberately **not** given a dedicated usagestore checker pass:
+
+- `NewKeptVolumes` has **no** primary-store consumer — it feeds only `CounterVolume` in the usagestore.
+- The primary-store-relevant signal, the exclusion set `PurgedVolumes ∪ EphemeralVolumes` consumed by the indexbuilder, **is** already verified by `compareExclusionProjections` above. The indexbuilder consumes the union and is indifferent to the Purged-vs-Ephemeral split, so the union is the correct verification granularity for the primary store.
+
+Corruption of a usagestore counter is therefore a *rebuild*, not an integrity incident: the recovery contract is "drop and replay from the audit chain", and the audit chain itself **is** cryptographically verified (`verifyAuditHashChain`). Serving a derived, rebuildable value through `GetLedgerStats` / `GetTemplateUsage` does not make it authoritative primary-store state. Extending audit-derived tamper coverage to the usagestore counters (which would require threading a new-volume collector through the shared `internal/domain/replay` package, also used by backup restore) is a separately-scoped effort tracked under EN-1422 — not a prerequisite for this subsystem.
+
+## Metrics
+
+Registered in `misc/devenv/monitoring-dashboards/jsonnet/lib/metrics.libsonnet`:
+
+| Metric | Description |
+|--------|-------------|
+| `usage.builder.last_indexed_sequence` | Highest audit sequence the builder has committed for this replica. |
+| `usage.builder.audit_last_sequence` | Highest audit sequence present in Pebble on this replica. |
+| `usage.builder.lag` | Difference between the two (indicator of the eventual-consistency window). |
+| `usagestore.level.bytes` / `memtable.bytes` / `cache.hits` / `cache.misses` | Pebble-internal metrics for the usagestore instance (parallel to the readindex namespace). |
+
+The three progress gauges are registered through `tailworker.RegisterTailGauges` — the same helper the audit indexer uses — so the naming pattern (`{ns}.last_indexed_sequence`, `{ns}.{source}_last_sequence`, `{ns}.lag`) stays consistent across every tail-worker subsystem.