Pre-prod hardening 3/4: ingestion lifecycle & resilience#669
Open
aditya1702 wants to merge 4 commits into
Open
Pre-prod hardening 3/4: ingestion lifecycle & resilience#669aditya1702 wants to merge 4 commits into
aditya1702 wants to merge 4 commits into
Conversation
aditya1702
force-pushed
the
hardening/pr3-ingestion
branch
from
July 16, 2026 19:53
4588921 to
f60a3a1
Compare
aditya1702
force-pushed
the
hardening/pr2-drop-loadtest
branch
from
July 16, 2026 19:53
90d864a to
7867d79
Compare
The ingestion process ran on context.Background() and its signal handler only stopped the HTTP servers, so every SIGTERM (each K8s deploy) left the ingest loop running until the kubelet SIGKILLed it after the full grace period, and the wasm-extractor closer goroutine waited on a never-cancelled context so the wazero runtime was never released. Ingest() now derives its root context from signal.NotifyContext and threads it through setupDeps and Run. A shutdown-classified Run error exits 0 after an ordered synchronous teardown (ledger backend, wasm extractor, DB pool) via a cleanup func returned by setupDeps; genuine errors still fail the process. The HTTP servers shut down off the same root context, and the advisory-lock release defer detaches via context.WithoutCancel so the lock is actually freed when the context is already cancelled.
…sactions Protocol classification previously ran inside the per-ledger persist transaction: a ledger deploying SEP-41/Blend contracts triggered simulate calls (batches of 20 with 2s inter-batch sleeps, 3x retries, 30s timeouts) while holding row locks on ingest_store cursors, contract_tokens, and blend_pools — an RPC outage could stall ingestion for minutes per ledger. Validators now split into Match (pure signature check), Prefetch (RPC only, before any transaction opens), and Apply (DB writes only — the signature carries no RPC service, making the guarantee compile-checked). The plan is computed once per ledger and reused verbatim across persist retries, so retries never re-issue RPC calls. protocol-setup shares the dispatcher and picks up the same fix. The checkpoint cold-start similarly fetched SAC metadata via RPC inside the single archive-load transaction; the load now commits with ledger-derived defaults and metadata enrichment runs in a short follow-up transaction whose failure logs rather than rolling back the completed load.
… pools Two defenses against concurrent ingesters after silent advisory-lock loss (a CNPG failover kills the lock session while the loop keeps writing through other pool connections): the loop now probes the lock-holding connection every ledger and exits fatally when the session is dead, and the latest-ledger cursor advances through a guarded update that refuses any value it doesn't own (ErrCursorGuardFailed) instead of blindly upserting — cursor regression is now impossible. Persist retries classify errors: SQLSTATE classes 22/23/42 and cursor guard failures fail on the first attempt instead of burning five retries before a crash loop; a self-cancelled datastore buffer (ErrBufferDead) exits immediately instead of a 10-attempt backoff ladder against a dead buffer. Protocol cursor CAS is gated on a startup snapshot (re-probed on the oldest-ledger sync cadence): never-initialized cursors skip their CAS entirely, so cursor_missing now fires only on the genuine existed-then- vanished incident and becomes alertable. Also: ledger-fetch duration/retry metrics now observed (help texts corrected); pond pools bounded (indexer 2xNumCPU, RPC-facing pools match the simulate batch size — the sep41 comment previously claimed a cap that did not exist); per-ledger insert/upsert detail logs demoted to debug; IndexerBuffer getter aliasing contracts documented.
…saction timeout Between batch flushes the load transaction's connection sits idle while the next entries are decoded from the history archive stream. A production idle_in_transaction_session_timeout (which protects the vacuum horizon from abandoned transactions) would kill the legitimately long-lived cold-start load mid-way and force a full redo; SET LOCAL scopes the exemption to this transaction only.
aditya1702
force-pushed
the
hardening/pr2-drop-loadtest
branch
from
July 16, 2026 20:41
7867d79 to
ef582cc
Compare
aditya1702
force-pushed
the
hardening/pr3-ingestion
branch
from
July 16, 2026 20:41
f60a3a1 to
50ca382
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Pre-production hardening 3/4 (stacked on #668). Live-ingestion lifecycle and resilience fixes; no schema changes.
Changes
context.Background()— SIGTERM only stopped the HTTP servers and the loop ran until SIGKILL). The current ledger finishes, then the advisory lock, ledger backend, wasm extractor, and DB pool are torn down cleanly — also fixing the wasm-extractor closer goroutine leak.🤖 Generated with Claude Code