Pre-production hardening: schema, queries, ingestion, and API#665
Open
aditya1702 wants to merge 17 commits into
Open
Pre-production hardening: schema, queries, ingestion, and API#665aditya1702 wants to merge 17 commits into
aditya1702 wants to merge 17 commits into
Conversation
The generic one-to-many/one-to-one dataloader helpers applied keys[0]'s Columns/Limit/SortOrder/Cursor to every key in a batch window, so aliased fields with differing sub-selections received silently under-fetched columns, nested pages were clamped to a hidden cap of 10 (with cursors dropped and hasNextPage wrong) whenever >=2 parents batched together, and a nil limit would panic the whole batch. Batches are now grouped by QueryShape before fetching (mirroring the account-scoped loader), each group fetches with its own parameters, and cursored groups with >1 key fall back to per-key fetches. The hidden MaxOperationsPerBatch/MaxStateChangesPerBatch=10 caps are removed in favor of a documented maxNestedPageLimit=100 enforced at the resolvers (same reject policy as the account connections), and loaders fail closed on missing/non-positive limits.
The ingestion process ran on context.Background() and its signal handler only stopped the HTTP servers, so every SIGTERM (each K8s deploy) left the ingest loop running until the kubelet SIGKILLed it after the full grace period, and the wasm-extractor closer goroutine waited on a never-cancelled context so the wazero runtime was never released. Ingest() now derives its root context from signal.NotifyContext and threads it through setupDeps and Run. A shutdown-classified Run error exits 0 after an ordered synchronous teardown (ledger backend, wasm extractor, DB pool) via a cleanup func returned by setupDeps; genuine errors still fail the process. The HTTP servers shut down off the same root context, and the advisory-lock release defer detaches via context.WithoutCancel so the lock is actually freed when the context is already cancelled.
Two regression tests protect the production constraint that freighter's account-detail queries stay under GRAPHQL_COMPLEXITY_LIMIT=6000: the two heaviest real client query shapes (balances first:100 = 3901, transactions first:100 with embedded operations/stateChanges = 5601) are computed against the production complexity config and asserted under the limit, and a guard asserts AccountTransactionEdge.operations/stateChanges carry no complexity multiplier — with a break-detection case proving that adding the naive x50 multiplier pushes the transactions query to 196501.
…licy config Storage parameters (migrations edited in place, pre-release): - fillfactor 80->90 on 14 upsert tables; measured churn (~0.1%/day on the large balance tables) uses half the 20% reserve, and COPY-loaded heaps carry the reserve permanently. liquidity_pools keeps 80 (every-row-per-day churn). - autovacuum_vacuum_cost_limit removed everywhere: cost_delay=0 disables cost-based throttling entirely, making cost_limit a no-op. - The aggressive autovacuum block now applies only to usage-scaling tables; protocol-bounded tiny blend tables use defaults. - sep41_allowances drops its redundant owner index (PK-leading column) and unused spender index; the expiration index stays for the sweep with its non-HOT-update tradeoff documented. Root query indexes: transactions/operations/state_changes replace the auto-created single-column time index with composites matching the API keyset sort, turning root first pages from full-chunk seq scan + heapsort into an index walk. TSDB runtime config: configureHypertableSettings now runs in every ingestion mode (a backfill-first bootstrap previously ran TimescaleDB's auto-created columnstore policy with unlimited maxchunks and no retention), and retention/reconcile jobs converge in place via alter_job so job IDs and run history survive restarts.
The five dataloader batch queries (BatchGetByStateChangeIDs on transactions/operations, BatchGetByToIDs and BatchGetByOperationIDs on operations/state_changes) carried no ledger_created_at predicate, so every call hash-joined or band-joined against a full decompression of all chunks — 26-40s per batch on 9 days of mainnet data, growing linearly with retention. They are now UNNEST + CROSS JOIN LATERAL per-key probes with the partition column pinned from the parent's ledger time, projecting only the requested columns; the ORDER BY/LIMIT laterals sit behind an OFFSET 0 fence so the chunk-selecting scan keeps runtime chunk exclusion (measured on the dev replica: 2.5-18ms for 100-key batches with all 9 chunks excluded, constant with retention). Tuple IN-lists and interpolated limits become bound array/limit parameters. TransactionColumnsKey and the resolvers now thread the parent ledger time into the loader keys. Also: root GetAll queries fail closed on non-positive limits (a previously overflowable path silently dropped the LIMIT clause on state_changes); GetLedgerGaps takes the backfill window so gap detection scans only the requested ledger range via chunk-skipping, with open trailing gaps clipped to the window edge; the stateChanges txHash filter tolerates duplicate hashes (IN instead of a scalar subquery); dead ProtocolsModel.GetClassified removed.
…ange IDs Numeric amounts, rates, and prices previously stored as TEXT (sac/sep41 balances, sep41 allowance amounts, the blend position/reserve/backstop/ emission/oracle columns) become unconstrained NUMERIC: writes cast once at the ingestion boundary, accumulation SQL operates natively (no more text->numeric->text round-trips per write), binary COPY encodes via pgtype.Numeric (malformed decoder output now fails at write time), and readers cast back to text so Go structs and the GraphQL String! contract stay byte-identical. state_change_id switches from crypto/rand to a deterministic ordinal per (to_id, operation_id) assigned at emission in slice order, namespaced per emitter (indexer base 0, SEP-41 1<<40, Blend 2<<40) because all three write state changes for the same operations in independently-committed transactions. Reprocessing a ledger now yields byte-identical rows, an accidental duplicate insert fails loudly on the PK instead of silently duplicating, and cursor ordering within an operation reflects emission order.
…sactions Protocol classification previously ran inside the per-ledger persist transaction: a ledger deploying SEP-41/Blend contracts triggered simulate calls (batches of 20 with 2s inter-batch sleeps, 3x retries, 30s timeouts) while holding row locks on ingest_store cursors, contract_tokens, and blend_pools — an RPC outage could stall ingestion for minutes per ledger. Validators now split into Match (pure signature check), Prefetch (RPC only, before any transaction opens), and Apply (DB writes only — the signature carries no RPC service, making the guarantee compile-checked). The plan is computed once per ledger and reused verbatim across persist retries, so retries never re-issue RPC calls. protocol-setup shares the dispatcher and picks up the same fix. The checkpoint cold-start similarly fetched SAC metadata via RPC inside the single archive-load transaction; the load now commits with ledger-derived defaults and metadata enrichment runs in a short follow-up transaction whose failure logs rather than rolling back the completed load.
… pools Two defenses against concurrent ingesters after silent advisory-lock loss (a CNPG failover kills the lock session while the loop keeps writing through other pool connections): the loop now probes the lock-holding connection every ledger and exits fatally when the session is dead, and the latest-ledger cursor advances through a guarded update that refuses any value it doesn't own (ErrCursorGuardFailed) instead of blindly upserting — cursor regression is now impossible. Persist retries classify errors: SQLSTATE classes 22/23/42 and cursor guard failures fail on the first attempt instead of burning five retries before a crash loop; a self-cancelled datastore buffer (ErrBufferDead) exits immediately instead of a 10-attempt backoff ladder against a dead buffer. Protocol cursor CAS is gated on a startup snapshot (re-probed on the oldest-ledger sync cadence): never-initialized cursors skip their CAS entirely, so cursor_missing now fires only on the genuine existed-then- vanished incident and becomes alertable. Also: ledger-fetch duration/retry metrics now observed (help texts corrected); pond pools bounded (indexer 2xNumCPU, RPC-facing pools match the simulate batch size — the sep41 comment previously claimed a cap that did not exist); per-ledger insert/upsert detail logs demoted to debug; IndexerBuffer getter aliasing contracts documented.
…root time bounds Serving hardening: WriteTimeout/IdleTimeout on the HTTP server plus a 30s per-request context deadline (previously a slow query or client pinned a DB connection indefinitely against a small pool); a query-depth limit of 15 (complexity does not bound depth — first:1 chains cost ~1 per level); introspection now gated behind --graphql-introspection-enabled (default off); Prometheus operation labels derive from schema root field names instead of the client-controlled operationName (bounded cardinality). Error handling: the presenter masks any error without a known client-safe code as a generic internal error (raw SQL/driver text no longer reaches clients) and logs the original; pagination and cursor parse failures are coded BAD_USER_INPUT so they survive the masking; parse/validation-phase errors no longer panic the presenter (operation context is absent there). API contract: the three root connections cap pages at 100 (rejecting oversized and int32-overflow first/last values before any arithmetic) and accept since/until time bounds, defaulting to the last 7 days when unspecified — root pages now carry a chunk-excluding time predicate, and cursors page within the caller's window. UInt32 input parsing accepts all JSON numeric encodings with range checks.
…README sync Adds docs/operations.md covering the TimescaleDB upgrade runbook (single ALTER EXTENSION multi-minor jumps, the 2.27+ composite-bloom catalog rename, never landing on 2.28.0/.1), the production database checklist (chunk-skipping GUC required before first ingest, instance sizing and retention/compression flag values), alerting guidance (lag and restarts are the pager signals; retry-exhaustion is a post-mortem breadcrumb; cursor_missing is now alertable), and the connection-topology constraint for ever fronting ingest with a transaction-pooling proxy. TimescaleDB image pins move to 2.28.2 everywhere (docker-compose, the CNPG image build, both build workflows, and CI's test service). The GraphQL README drops a documented-but-nonexistent mutation and error code, references the complexity limit by its flag instead of a stale literal, adds the Blend queries, and documents the API behavior introduced by this hardening: since/until with the 7-day root default, uniform 100-per-page caps, the depth limit, gated introspection, error masking, and the request timeout. Ledger cursor casts widen to bigint.
…ap detection EXPLAIN verification of the windowed GetLedgerGaps showed the window was bounded only by the columnstore's batch min/max metadata — ledger_number carried no chunk-skipping ranges, so no chunks were excluded at plan time and every chunk paid at least a metadata pass. ledger_number rises monotonically with the partition column, making it an ideal skipping column: enabling it gives ledger-windowed queries plan-time chunk exclusion (ranges are recorded as chunks compress). The GetLedgerGaps doc comment now describes both bounding mechanisms accurately.
…saction timeout Between batch flushes the load transaction's connection sits idle while the next entries are decoded from the history archive stream. A production idle_in_transaction_session_timeout (which protects the vacuum horizon from abandoned transactions) would kill the legitimately long-lived cold-start load mid-way and force a full redo; SET LOCAL scopes the exemption to this transaction only.
Three hypertables carried a secondary index over the same column set as their primary key in a different order, and the two _accounts tables kept TimescaleDB's auto-created single-column time index that no query path uses. The PKs now use the column order their consumers need — the _accounts tables as (account_id, ledger_created_at, id) so one index serves both uniqueness and the account-history walk (B-tree scans run backward for DESC pages), state_changes as (ledger_created_at, to_id, operation_id, state_change_id) matching the root connection sort and decomposed cursor — and the redundant secondaries and dead time indexes are gone. Every remaining index has a nameable consumer. The uncompressed hot-day index working set drops from ~14GB to ~9.5GB and the two highest-insert- rate tables halve their random-B-tree maintenance per row. Verified by an EXPLAIN sweep of all sixteen consumer queries against the new layout on a seeded multi-chunk database: every account walk and root page is an index (only) scan with no sort or seq scan; reverse-lookup paths unchanged. The operations doc gains the hypertable index-usage measurement caveat: parent pg_stat_user_indexes always reads zero — usage accrues on chunk indexes in _timescaledb_internal and must be aggregated from there.
…metrics A GraphQL request's loader keys arrive in a near-instant burst (sibling resolvers fan out concurrently), but a default 50-parent page sits under the 100-key batch capacity and waited the full 5ms window at every nested loader level — a serialized ~5ms-per-level latency floor that often exceeded the batch SQL itself. The window drops to 1ms via shared constants (measured on a nested transactions->operations->stateChanges shape: 13.3ms -> 3.3ms mean), and batching efficiency becomes observable: every loader records batch size and fetch duration histograms so a shorter window producing partial batches — or a future N+1 regression — shows up on a dashboard instead of in a profile. A coalescing unit test pins that concurrent same-shape loads still collapse into one fetch.
The upgrade procedure is specific to the internal cluster deployment (image build pipeline, CNPG rollout, Database CRD reconciliation), not to operating this software; it lives with the deployment manifests. The operations reference keeps what any operator of wallet-backend needs: the database checklist, alerting guidance, and connection-topology constraints.
Bring blend_pool_claimed/blend_backstop_claimed in line with the other Blend tables: claimed_blnd/claimed_lp become NUMERIC (native accumulation, no text->numeric->text round-trips), fillfactor 90, drop the vacuum cost limit. Readers cast back to text so the Go string fields stay byte-identical.
aditya1702
force-pushed
the
blend/pr6-integration-tests
branch
from
6f67e5f to
39422d7
Compare
aditya1702
force-pushed
the
pre-prod-hardening
branch
from
f93f76c to
c4cda86
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Pre-production hardening pass across the database schema, data-layer queries, live ingestion, and the GraphQL API. The service is pre-release, so breaking changes land directly in the migration files (clean CREATEs for launch) and in data semantics where correctness demanded it.
Correctness fixes
hasNextPagewhenever two or more parents batched together. Batches are now grouped by query shape; nested pages honor a documented 100 cap.context.Background()until the kubelet SIGKILLed it after the full grace period every deploy. The root context now comes fromsignal.NotifyContext, shutdown finishes the in-flight ledger, and teardown closes the ledger backend, wasm extractor (previously never closed — leaked the wazero runtime), and DB pool in order.state_change_idwas a random int63 per row, so re-processing a ledger silently inserted duplicates. IDs are now deterministic ordinals per(to_id, operation_id), namespaced per emitter (indexer / SEP-41 / Blend write independently), so re-ingest produces byte-identical rows and accidental duplicates fail loudly on the PK.Query performance (measured on a mainnet-data environment)
CROSS JOIN LATERALper-key probes with the partition column pinned (plus anOFFSET 0fence on the ordered variants so chunk exclusion survives the ORDER BY/LIMIT): 2.5–18ms for 100-key batches with all historical chunks excluded at runtime, constant with retention.transactions/operations/stateChangesfirst pages did a full seq scan + heapsort of the day chunk (8–23s). They now have composite indexes matching the API sort keys, acceptsince/until(defaulting to the last 7 days), cap pages at 100, and fail closed on invalid limits (one overflow path previously stripped the LIMIT entirely).Schema & TimescaleDB
autovacuum_vacuum_cost_limitremoved (cost_delay=0 disables cost-based throttling), the aggressive autovacuum block dropped from protocol-bounded tiny tables, and two deadsep41_allowancesindexes removed.docs/operations.md(new) carries the upgrade runbook, production database checklist, alerting guidance, and connection-topology constraints.API hardening
Request/write/idle timeouts; a query depth limit (15); internal error masking (unknown errors return a generic
INTERNAL_SERVER_ERROR; pagination/cursor mistakes are codedBAD_USER_INPUT); introspection gated behind--graphql-introspection-enabled(default off); Prometheus operation labels bounded to schema root fields; tolerantUInt32input parsing. Two regression tests lock the freighter constraint: the full-detail queries compute 3901/5601 against the 6000 complexity limit, andAccountTransactionEdge.operations/stateChangesmust never gain a complexity multiplier.Retries & observability
Persist retries classify SQLSTATEs (data/integrity/schema errors fail fast instead of crash-looping through 5 retries); a dead datastore buffer exits immediately; the never-initialized-protocol-cursor case no longer floods the
cursor_missingmetric, making it alertable for real cursor loss; ledger-fetch duration/retries are now observed; worker pools are explicitly bounded.Verification
Full race-enabled unit suite and the integration suite (fresh DB, all migrations applied from scratch) pass;
make checkgreen at every commit. Query rewrites were verified withEXPLAIN (ANALYZE, BUFFERS)against production-scale data before/after.Breaking changes (pre-release)
Column types (TEXT→NUMERIC), state change ID values/ordering, root connections default to a 7-day window and cap pages at 100, introspection defaults off, and internal error text is no longer returned to clients.
🤖 Generated with Claude Code
Update — NUMERIC claimed tables
Brings the new
blend_pool_claimed/blend_backstop_claimedtables in line with the other Blend amount columns:claimed_blnd/claimed_lpbecomeNUMERIC(native accumulation, no text round-trips),fillfactor = 90, vacuum cost limit dropped; readers cast back to text so the Go/GraphQLStringcontract is unchanged.