feat(EN-1334): usagebuilder subsystem + event-counter cutover (EN-1420)

.golangci.yaml

@@ -163,6 +163,11 @@ linters:
     # (not OpenWriteSession on the main store).
     - path: internal/storage/readstore/
       linters: [forbidigo]
+    # The usagestore is a peer secondary store to readstore, dedicated to the
+    # usagebuilder projections. It manages its own Pebble DB and uses
+    # NewWriteSessionFromDB by the same rationale.
+    - path: internal/storage/usagestore/
+      linters: [forbidigo]
     # Tests use OpenWriteSession to seed data; the structural guarantee is on
     # production paths.
     - path: _test\.go$

AGENTS.md

@@ -14,6 +14,8 @@ This document contains rules and conventions for AI agents working on this codeb
 4. **Pebble writes only from the hot path or declared lifecycle paths** — `*dal.Store.OpenWriteSession()` is the only producer of `*dal.WriteSession`. Outside the FSM hot path, only declared lifecycle paths may call it: `internal/bootstrap/config_validation.go`, `internal/infra/backup/`, `internal/infra/attributes/prepare.go`. This is enforced by a `forbidigo` rule in `.golangci.yaml`; new call sites must be added to the exclusions block with a justification.
 5. **Never delete cache entries outside of rotations** — Cache entries must only be evicted during generation rotations (Gen0 → Gen1 → discard). Deleting individual entries breaks the cache prediction mechanism (bloom filters, tombstones).
 6. **Every FSM `Registry.X.Get(...)` must have a matching preload, declared by the component that proposes the command** — The FSM apply path reads from the in-memory cache; a cache miss turns the read into a silent no-op. Each component that emits a proposal (the metadata converter, the index builder, the cluster-config reconciler, the idempotency-eviction scheduler, the mirror worker, admission) is responsible for declaring its own `preload.Needs` covering every key its apply path will read. There is NO central proposal→needs registry — coupling the preload package to every proposal type creates a single point that easily falls behind. The component knows what it reads; the component declares it. The shared `proposeTechnical` helper takes a `*preload.Needs` parameter the caller fills in (pass nil or an empty `Needs` when the apply path has no cache-keyed reads, e.g. cluster config / idempotency eviction). The preload populates the cache via `MirrorPreload` with a fresh value read at propose time (Pebble fallback on cache miss), and `PredictedIndex` catches mutations between propose and apply.
+
+    **Volume preload is load-bearing for the `LedgerLog.new_kept_volumes` / `ephemeral_volumes` split.** Admission MUST preload the current value of every `(account, asset)` volume touched by an order, including postings resolved from numscript execution. The FSM classifies a volume as "newly created" by checking `Old.IsDefined()` (or the zero-placeholder via `isVolumePreloadZero`) at merge time — a silent cache miss on an existing volume produces a false-new classification, mis-routes the tuple between the `new_kept_volumes` / `ephemeral_volumes` / `purged_volumes` per-log lists, and inflates `usagestore.CounterVolume`. Because `usagestore` is a rebuildable peer side-store held **outside** checker scope (invariant #8 verifies only primary-Pebble-store projections), no checker pass would surface the drift; it persists until an explicit usage rebuild. That absence of a backstop makes the preload correctness even more load-bearing, not less. This is not opportunistic (unlike the metadata preload that was removed after the indexbuilder stopped needing it): balance checks, Uint256 arithmetic and numscript resolution all require the current volume value, so the preload is structurally mandatory. If a future refactor considers alleviating volume preload, `VolumeCount` must be re-hosted first (via computed-on-read or a subsystem-side seen-keys table) — see EN-1422 for the design rationale.
 7. **Never silently skip a "should not happen" branch** — A branch that is reachable only if an invariant is violated (nil where the contract says non-nil, a state we believe unreachable, a cache miss after a guaranteed preload, etc.) MUST surface a loud signal: `return fmt.Errorf("invariant: ...")` so it bubbles up, or `assert.Unreachable(...)` for SUT-level invariants exercised under antithesis. A silent `return nil` / `continue` on these branches hides real bugs — particularly catastrophic in the FSM apply path, where a no-op desyncs nodes from each other. Branches that represent genuine runtime conditions (cache miss as an expected outcome, stale proposal, deleted entity) keep their soft `return nil`. The distinction is whether the case is *expected* (soft skip OK) or *impossible by design* (must fail loudly). The comment must say *why* the case is impossible so a reader can decide whether to add a hard fail or relax the rule.
 
 8. **The audit log is the only source of truth — every other persisted dataset is a projection and must be verified by the checker** — Only `AuditEntry` (zone `Cold`, sub `Audit`) is cryptographically bound, via the hash chain that `state.BuildHashedHeaderPayload` + `processing.HashGenerator` produce and `checker.verifyAuditHashChain` verifies on every Check() run. Everything else stored in Pebble — `Log`, `AuditItem`, `AppliedProposal`, `LedgerLog.PurgedVolumes`, attribute caches (`Volume`, `Metadata`, `Transaction`, `Reference`, `Boundary`, etc.), reversion bitsets, idempotency keys, mirror cursors, chapters, bloom filters, signing keys, the read-side index — is a *projection* of orders that already live in the audit chain. Projections are rebuildable from the audit on demand, so we deliberately do NOT extend the hash chain to cover them (refactor over hash binding — see `feedback_audit_is_source_of_truth`). In exchange, **`internal/application/check/checker.go` MUST verify every projection it persists**: re-derive the value the projection should hold by replaying the audit (`ReplayLedgerLog`, `SimulateEphemeralPurge`, `partitionVolumes`, etc.) and compare to what is stored, emitting the matching `CHECK_STORE_ERROR_TYPE_*` event on divergence. A projection that the checker does not verify is a tampering vector — adding a new persisted projection without a matching compare* / collect* pass in the checker is the violation. The current passes are `compareVolumes`, `compareMetadata`, `compareTransactions`, `compareExclusionProjections` (AppliedProposal.TransientVolumes + LedgerLog.PurgedVolumes), `checkReversionInvariants`, `verifySealingHash`, `compareIdempotencyOutcomes` (frozen idempotency outcomes in SubIdempKeys vs the hash-chained AuditFailure/AuditSuccess that wrote them — the failure kind is re-derived from the chain-bound reason via `domain.KindForReason`, never stored), and `compareIndexes` (SubAttrIndex registry vs CreateIndex/DropIndex/RemovedMetadataFieldType/DeleteLedger logs — covers presence + identity; BuildStatus is intentionally excluded because it is purely informational on the cluster-wide registry entry — queries gate on the per-replica `IndexVersionState.CurrentVersion`, not on BuildStatus); extend the list as new persisted projections land.

cmd/ledgerctl/ledgers/stats.go

@@ -75,7 +75,6 @@ func runStats(cmd *cobra.Command, _ []string) error {
 	pterm.Printf("Postings:           %d\n", stats.GetPostingCount())
 	pterm.Printf("Logs:               %d\n", stats.GetLogCount())
 	pterm.Printf("Volumes:            %d\n", stats.GetVolumeCount())
-	pterm.Printf("Metadata:           %d\n", stats.GetMetadataCount())
 	pterm.Printf("References:         %d\n", stats.GetReferenceCount())
 	pterm.Printf("Reverts:            %d\n", stats.GetRevertCount())
 	pterm.Printf("Numscript execs:    %d\n", stats.GetNumscriptExecutionCount())

docs/technical/architecture/overview.md

@@ -154,17 +154,17 @@ Per-ledger boundaries use the `LedgerBoundaries` protobuf message from `raft_cmd
 message LedgerBoundaries {
   fixed64 next_transaction_id = 1;
   fixed64 next_log_id = 2;
-  fixed64 volume_count = 3;
-  fixed64 metadata_count = 4;
-  fixed64 reference_count = 5;
-  fixed64 posting_count = 6;
-  fixed64 ephemeral_evicted_count = 7;
-  fixed64 transient_used_count = 8;
-  fixed64 revert_count = 9;
-  fixed64 numscript_execution_count = 10;
 }
 ```
 
+> Every per-ledger counter — postings, reverts, numscript executions, references, ephemeral evictions, transient volumes, **and volumes** — lives in the usagebuilder side-store (`internal/application/usagebuilder`, `internal/storage/usagestore`). All counters are derived from the audit chain: event counts (posting / revert / numscript-exec / reference) come from replaying orders; transient counts come from `AppliedProposal.TransientVolumes`; ephemeral and volume counts come from three DISJOINT per-log lists that the FSM populates on every `LedgerLog`:
+>
+> - `purged_volumes` — draining evictions (was non-zero in Pebble, now zero, entry deleted at commit)
+> - `new_kept_volumes` — persistent-new (created + survives past commit)
+> - `ephemeral_volumes` — pure ephemeral (created + evicted same log)
+>
+> Volume count delta per log = `len(new_kept_volumes) − len(purged_volumes)`. Pure ephemeral tuples contribute +0 (was zero, is zero) and are tracked only for the index builder's skip logic and for the ephemeral-eviction counter. Splitting ephemeral out of `purged_volumes` avoids the 2× byte cost that a naive union-encoding would pay on ephemeral-heavy workloads. `LedgerBoundaries` carries only the two ID generators that drive apply-time allocation decisions. `metadata_count` is intentionally not exposed — the admission preload no longer injects old metadata values so cardinality cannot be reliably derived at the FSM level; it will come back on a sound foundation later.
+
 > **Note:** The `State` / `LedgerState` proto messages sometimes shown in older documentation are conceptual models, not actual proto definitions. The real FSM state is spread across the `Machine` struct fields and its `StateRegistry`.
 
 ### Benefits of Single Raft

docs/technical/architecture/subsystems/usage/README.md

@@ -0,0 +1,19 @@
+# Usage
+
+The background worker (`internal/application/usagebuilder`) that turns committed audit entries into per-ledger housekeeping counters — postings, reverts, numscript executions, references, ephemeral evictions, transient uses, live volume cardinality — plus per-Numscript-template invocation records. Runs on every node — leader and followers — independently, mirroring the indexer's decoupling from the FSM hot path: the FSM commits and signals; the usagebuilder reads its own Pebble read handle and writes to a dedicated secondary store (`internal/storage/usagestore`).
+
+The projections it maintains are exposed by the API — `GET /v3/{ledger}/stats` for the aggregate counters, `GET /v3/{ledger}/numscripts/{name}/usage` for per-template invocation counts.
+
+## Documents
+
+| Document | Description |
+|----------|-------------|
+| [usagebuilder.md](usagebuilder.md) | Pipeline mechanics: builder loop, audit-chain tailing, per-batch commit, log-payload consumption. |
+| [counters.md](counters.md) | Counter definitions, storage keys, `LedgerLog.{purged,new_kept,ephemeral}_volumes` split, formula for each counter. |
+
+## Related
+
+- [FSM](../fsm/) — emits the `LedgerLog.new_kept_volumes` / `ephemeral_volumes` / `purged_volumes` annotations the usagebuilder consumes. Depends on the volume preload contract (see AGENTS.md invariant #6).
+- [Indexer](../indexer/) — sibling subsystem with the same audit-tailing shape, but writes to a different secondary store (`readstore`).
+- [Checker](../checker/) — verifies the projections against the audit chain (`compareExclusionProjections` for ephemeral/transient tuples).
+- [Storage](../storage/) — the usage store is a peer Pebble instance to the readstore, WAL disabled, own comparer, own Pebble tuning.