Pre-prod hardening 1/4: schema & TimescaleDB foundation (destructive; fresh DB)

[aditya1702]

What

First of four pre-production hardening PRs (1/4 → this one, then #668 loadtest removal, #669 ingestion, #670 GraphQL/docs). This PR carries every destructive schema change — existing migration files are edited in place, so it must land before the production database is created. Everything here requires a fresh DB; the remaining PRs in the stack do not.

Why

wallet-backend is pre-release: this is the last window to fix schema-level issues without ALTER migrations. These changes come out of a full data-layer/TimescaleDB audit against a mainnet-data dev cluster (EXPLAIN plans and pg_stat/timescaledb_information measurements referenced in the commit messages).

Changes

• Storage retune + idempotent TSDB policy config: balance-table storage blocks retuned from measured churn (fillfactor 80→90, drop the autovacuum_vacuum_cost_limit no-op — per-table cost_delay=0 already exempts the worker from cost balancing); retention/compression policy setup made idempotent and applied in every ingestion mode.
• Time-pinned lateral probes for batch queries: the five dataloader batch queries carried no partition-column predicate, so each call decompressed all chunks (26–40s per batch on 9 days of mainnet data, growing with retention). They are now UNNEST + LATERAL per-key probes pinned to the parent's ledger time behind an OFFSET 0 fence: 2.5–18ms measured, constant with retention. Because relationship lookups pin on the parent row's ledger_created_at, every transaction/operation/state-change projection forces that column regardless of the client's field selection (regression-tested on minimal projections, end to end through the pinned child lookup). Also: bound array/limit parameters replace tuple IN-lists and interpolated LIMITs, root GetAll queries fail closed on non-positive limits, GetLedgerGaps scans only the requested window via chunk skipping.
• NUMERIC amount columns + deterministic state_change IDs: TEXT numeric columns (sac/sep41 balances, allowance amounts) become NUMERIC — writes cast once at the boundary, accumulation SQL runs natively, readers cast back so the GraphQL String contract is unchanged. state_change_id switches from crypto/rand to deterministic per-(to_id, operation_id) ordinals in namespaced ranges, so re-ingesting a ledger produces byte-identical rows and duplicates fail loudly on the PK.
• Per-processor sub-namespaces for state_change IDs: each emitting indexer processor (token transfers, effects, contract deploys, SAC events) numbers its own emissions inside its own sub-range (k<<28 within the indexer's namespace), declared via StateChangeSubBase() on the processor itself. IDs therefore never depend on processor registration/invocation order — adding or reordering processors cannot shift existing IDs, which would otherwise silently defeat the duplicate-insert PK backstop on replays. Within one processor, emission order derives from transaction meta (canonical on-chain data).
• Chunk skipping on transactions.ledger_number for windowed gap detection.
• Hypertable PK consolidation: PK column order matches the batch queries' access shape (equality column first, then the walk's sort key), making the separate API-sort indexes redundant — they're dropped; the TimescaleDB default partition-column indexes with zero consumers are dropped too.
• TimescaleDB pinned to 2.28.2 (Dockerfile, CI workflows, docker-compose).

Deploy notes

• Requires a fresh database (migrations edited in place; existing envs must be wiped + re-initialized).
• Fresh-apply is exercised end-to-end by the integration suite (fresh TimescaleDB container + all migrations).

🤖 Generated with Claude Code

[Copilot]

Pull request overview

Prepares the fresh pre-production database with optimized TimescaleDB storage, deterministic state-change identifiers, and partition-pinned queries.

Changes:

• Retunes schemas, indexes, numeric storage, policies, and TimescaleDB version.
• Reworks batch queries for chunk exclusion and bounded scans.
• Adds deterministic state-change IDs and regression coverage.

Reviewed changes

Copilot reviewed 52 out of 52 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
internal/services/sep41/processor.go	Assigns deterministic SEP-41 IDs.
internal/services/ingest_backfill.go	Bounds ledger-gap detection.
internal/serve/graphql/resolvers/transaction.resolvers.go	Passes transaction timestamps to loaders.
internal/serve/graphql/resolvers/transaction_resolvers_test.go	Updates transaction fixtures.
internal/serve/graphql/resolvers/test_utils.go	Shares fixture ledger timestamp.
internal/serve/graphql/resolvers/statechange.resolvers.go	Passes timestamps for relationships.
internal/serve/graphql/resolvers/statechange_resolvers_test.go	Updates state-change fixtures.
internal/serve/graphql/resolvers/resolver.go	Extends relationship loader keys.
internal/serve/graphql/resolvers/operation.resolvers.go	Pins operation state-change lookup.
internal/serve/graphql/resolvers/operation_resolvers_test.go	Updates operation fixtures.
internal/serve/graphql/dataloaders/transaction_loaders.go	Batches transaction timestamps.
internal/serve/graphql/dataloaders/statechange_loaders.go	Batches partition-pinned state changes.
internal/serve/graphql/dataloaders/operation_loaders.go	Batches partition-pinned operations.
internal/ingest/timescaledb.go	Converges TimescaleDB policies and jobs.
internal/ingest/timescaledb_test.go	Tests policy/job stability.
internal/ingest/ingest.go	Configures hypertables in every mode.
internal/indexer/types/types.go	Defines deterministic ID namespaces.
internal/indexer/types/types_test.go	Tests ordinal assignment.
internal/indexer/processors/state_change_builder.go	Documents deferred ID assignment.
internal/indexer/processors/contracts_test_utils.go	Updates state-change comparison notes.
internal/indexer/indexer.go	Assigns indexer state-change ordinals.
internal/db/migrations/2026-07-02.1-liquidity_pool_balances.sql	Retunes balance storage.
internal/db/migrations/2026-07-02.0-liquidity_pools.sql	Retunes pool autovacuum settings.
internal/db/migrations/2026-04-17.1-sep41_allowances.sql	Uses NUMERIC and removes indexes.
internal/db/migrations/2026-04-17.0-sep41_balances.sql	Uses NUMERIC and retunes storage.
internal/db/migrations/2026-01-16.0-sac-balances.sql	Converts SAC balances to NUMERIC.
internal/db/migrations/2026-01-15.0-native_balances.sql	Retunes native-balance storage.
internal/db/migrations/2026-01-12.0-trustline_balances.sql	Retunes trustline storage.
internal/db/migrations/2025-06-10.4-statechanges.sql	Reorders the state-change PK.
internal/db/migrations/2025-06-10.3-operations.sql	Consolidates operation indexes.
internal/db/migrations/2025-06-10.2-transactions.sql	Adds chunk skipping and indexes.
internal/data/transactions.go	Adds bounded and time-pinned queries.
internal/data/transactions_test.go	Tests transaction time pinning.
internal/data/statechanges.go	Reworks state-change queries and IDs.
internal/data/statechanges_test.go	Adds state-change regressions.
internal/data/sep41/balances.go	Performs native NUMERIC arithmetic.
internal/data/sep41/allowances.go	Casts allowance amounts at write/read boundaries.
internal/data/sac_balances.go	Supports binary NUMERIC writes.
internal/data/query_utils.go	Validates positive limits.
internal/data/protocols.go	Removes unused classified-protocol lookup.
internal/data/operations.go	Adds partition-pinned operation probes.
internal/data/operations_test.go	Tests operation query changes.
internal/data/mocks.go	Removes obsolete protocol mock method.
internal/data/ingest_store.go	Bounds ledger-gap scans.
internal/data/ingest_store_test.go	Tests bounded gap semantics.
docs/operations.md	Documents hypertable index auditing.
docs/data-migrations/adding-a-protocol.md	Updates storage guidance.
Dockerfile-timescale-cnpg	Pins TimescaleDB 2.28.2.
docker-compose.yaml	Updates local TimescaleDB image.
.github/workflows/go.yaml	Updates CI TimescaleDB image.
.github/workflows/build-cnpg-timescaledb-stg.yml	Updates staging image defaults.
.github/workflows/build-cnpg-timescaledb-dev.yml	Updates development image defaults.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[aristidesstaffieri]

Grouping ordinals by OperationID alone doesn't match the contract the registry promises — and the gap silently defeats the PK backstop for window-scoped emitters.

The namespace registry comment above says each emitter "numbers its own state changes 1..N in deterministic emission order within an (to_id, operation_id) group (see AssignStateChangeOrdinals)", but the implementation keys the counter map on OperationID only:

next := make(map[int64]int64, len(changes))
for i := range changes {
    opID := changes[i].OperationID
    next[opID]++
    changes[i].StateChangeID = base + next[opID]
}

Both current emitters are safe only incidentally:

• the indexer assigns per-transaction stream, so ToID is constant within a call;
• SEP-41 stages across a multi-ledger window, but its opID = toid(ledger, txIdx, opIdx+1) embeds the transaction and is never 0, so an OperationID group can never span ToIDs.

Nothing enforces either property at the call sites. The failing usage is exactly what the documented contract invites: a future window-scoped emitter (the Blend base is already reserved) staging transaction-level changes with OperationID = 0 for several transactions in one window. Those all share one counter bucket, so a given row's ordinal depends on which other ledgers happen to be in the window. Re-ingesting the same ledger under different window bounds (live vs. backfill, or a re-run migration) then produces different state_change_ids for the same logical rows — BatchCopy inserts silent duplicates instead of failing on the PK, which is precisely the corruption this scheme exists to surface.

Fix is one line of shape change — key the map on the pair, which also makes the code match its own doc:

type ordinalKey struct{ toID, opID int64 }

next := make(map[ordinalKey]int64, len(changes))
for i := range changes {
    k := ordinalKey{changes[i].ToID, changes[i].OperationID}
    next[k]++
    changes[i].StateChangeID = base + next[k]
}

(The function doc's "within each distinct OperationID group" wording would want the same update.)

Worth doing in this PR rather than later: once rows exist, any emitter that ever hit the single-key bucket has persisted IDs that a re-run under the fixed grouping won't reproduce — cheap now, messy after.

[aristidesstaffieri]

Removing the IngestionModeLive guard here trades a per-invocation property (idempotence) for one this call doesn't have: convergence across processes with different flags.

The new comment says "All settings applied here are idempotent, so running this on every start (live or backfill) is safe." That's true for one process re-running with the same flags — but the policies live in shared database state, and every process start now imposes its own flag values on them. A backfill job is typically launched ad hoc, without the live deployment's flag set, and the empty/zero defaults are interpreted as "disable"/"skip" rather than "leave unchanged". Three concrete failure modes fall out:

1. Backfill without --retention-period strips live retention. The flag defaults to "", and configureRetentionPolicy treats empty as disable: remove_retention_policy(...) runs on every hypertable, and configureReconciliationJob("") executes delete_job on reconcile_oldest_cursor. A one-off backfill on a production database silently disables retention and cursor reconciliation until the next live pod restart re-creates them — meanwhile chunks are never dropped and oldest_ingest_ledger goes stale.

2. Backfill with retention drops the history it's writing. The policy is now installed at startup, before the backfill writes. Backfilling a range older than RetentionPeriod (e.g. 90 days of history under a 30-day policy — the bootstrap case the new comment is motivated by) runs under an active policy_retention job that drops the freshly written chunks: the backfill logs success, ~60 days of it vanish. The progressive recompressor's compress_chunk($chunk) calls also race the concurrent chunk drops and error out, failing batches.

3. A backfill's compression cap ratchets live jobs down permanently. --compression-max-chunks 10 writes maxchunks_to_compress = 10 into every compression job's config, but the write is gated on maxChunksToCompress > 0 — so when live restarts with its default of 0, the block is skipped and the cap is never removed. Live compression is permanently limited to 10 chunks per run until someone alter_jobs by hand.

One fix shape covers all three. Either:

• restore the mode guard (backfill never touches policies), which gives up the stated motivation — a backfill-first bootstrap running under TimescaleDB's auto-created policy defaults; or
• keep the unconditional call but make it convergent: empty/zero flag values mean "leave existing configuration unchanged" instead of the current asymmetry where "" means disable for retention but 0 means skip for maxchunks. The bootstrap concern is then handled explicitly — e.g. only apply defaults-sensitive settings when the policy/job doesn't exist yet, so a backfill can create missing policies but never mutate or remove ones the live deployment configured.

The second keeps the bootstrap win and removes the operational footgun; the first is smaller but reintroduces the "backfill-first bootstrap runs uncapped" gap this PR set out to close.

[aristidesstaffieri]

The forced-column set here is missing the id columns the state-change resolvers build their loader keys from — nested operation { ... } silently resolves to null on minimal selections.

This line (correctly) forces to_id and ledger_created_at into the projection for the new time-pinned child lookups, but its sibling BatchGetByAccountAddress forces the full key set:

// BatchGetByAccountAddress (line ~35)
columns = prepareColumnsWithID(columns, types.StateChange{}, "", "to_id", "operation_id", "state_change_id", "account_id", "ledger_created_at")

// BatchGetAccountStateChangesByToIDs (this line)
columns = prepareColumnsWithID(columns, types.StateChange{}, "sc", "to_id", "ledger_created_at")

operation_id and state_change_id aren't just data fields — every state-change resolver feeds them into its dataloader key (resolver.go:136):

stateChangeKey := fmt.Sprintf("%d-%d-%d", toID, operationID, stateChangeID)

So for an account-scoped query whose selection doesn't happen to include a field mapping to operation_id — e.g. account → transactions → stateChanges { type, amount, operation { id } } — the rows come back with OperationID = 0, the loader key is "<toID>-0-0", and the operation lookup probes o.id = 0: no match, operation resolves to null even though it exists. As a second-order effect, all state changes of that transaction collapse onto the same "<toID>-0-0" key, so the dataloader dedups them into one shared (null) result.

Fix is to match the sibling's forced set on this line:

columns = prepareColumnsWithID(columns, types.StateChange{}, "sc", "to_id", "operation_id", "state_change_id", "account_id", "ledger_created_at")

Since this PR already added regression tests for minimal projections end-to-end through the pinned child lookup, extending one of those to assert operation { id } is non-null on a minimal account-scoped state-change selection would lock this in.