Feat/hang ports - #11
Open
S0c5 wants to merge 11 commits into
Open
Conversation
Co-authored-by: Cursor <cursoragent@cursor.com>
Webhooks are now persisted to a new webhook_deliveries table and delivered by a background worker instead of inline in the monitor/sweeper. Failed deliveries (non-2xx, including 503) are retried with configurable limits, and a lease prevents duplicate POSTs. Adds POST /admin/retry_webhooks to re-queue permanently failed deliveries. Co-authored-by: Cursor <cursoragent@cursor.com>
Polygon fell thousands of blocks behind head because catch_up scanned one block at a time, each costing a full get_block_by_number plus a chain-wide (unfiltered) eth_getLogs call, with get_logs_with_retry treating an empty result as a failure and retrying up to 30x. - get_logs_with_retry now returns immediately on any Ok (including empty) and retries only on Err. - eth_getLogs is filtered to the chain's allowed token addresses instead of scanning every Transfer event on the chain. - When the gap between last-processed and head exceeds 10 blocks, catch_up switches to a batched path: one ranged, address-filtered eth_getLogs call per catch_up_chunk_size blocks (default 500), bisecting the range on provider errors (e.g. Alchemy's response-size cap) instead of retrying the identical request, plus concurrent eth_getBlockByNumber fetches (up to block_fetch_concurrency) for native transfers. Checkpoints once per chunk; replay is idempotent via the existing deposit-recording dedup. - Adds catch_up_chunk_size and block_fetch_concurrency chain config options. Covered by 7 new tests (regression, address filter, batch happy path, bisection, bisection floor, concurrent native fetch, idempotent replay). Co-authored-by: Cursor <cursoragent@cursor.com>
The monitor/sweeper/webhook-retry loops called rusqlite (via Db) directly from async fns with no spawn_blocking, and the r2d2 read pool had no connection_timeout (defaulting to 30s). During a large catch-up backlog, this let a busy chain pin a Tokio worker thread for tens of seconds, stalling everything else sharing that runtime. - Add Db::blocking, which offloads a Db closure onto spawn_blocking; route every DB call reached from async code (monitor, sweeper, webhook, register) through it. - Set an explicit 5s r2d2 connection_timeout instead of the 30s default. - Add tokio::task::yield_now() between blocks/logs in the batch catch-up path so one chunk can't monopolize a worker thread across many non-yielding blocking calls. Co-authored-by: Cursor <cursoragent@cursor.com>
The r2d2 read pool was built with an explicit 5s connection_timeout (da43b62) but never set .max_size(), so it silently used r2d2's default of 10 connections shared by every chain's monitor/sweeper/ webhook-retry loop plus inbound registrations. With multiple chains configured, that fixed ceiling can still be exhausted under a catch-up backlog or a flaky RPC provider, producing sustained "timed out waiting for connection" errors even with the fast-fail timeout. - Add Db::with_pool_size(url, max_size), with Db::new delegating to it using DEFAULT_READ_POOL_MAX_SIZE (10) so existing callers/tests are unaffected. - Add Config::db_read_pool_size, loaded from DB_READ_POOL_SIZE (default 20), and wire HotWalletService::new to use it. - Add a regression test proving the configured max_size is actually enforced by the pool. Co-authored-by: Cursor <cursoragent@cursor.com>
Make SQLite read-pool size configurable
Under high concurrency, the shared std::sync::Mutex<Connection> guarding all SQLite writes was a hard bottleneck: background monitor/sweeper/ webhook-retry loops and inbound HTTP writes all serialized on the same lock, and slow fsyncs on EFS (synchronous=FULL) meant a single blocked writer could stall requests for tens of seconds, surfacing as 60s timeouts upstream. - Replace the mutex with a dedicated writer thread fed by a two-lane queue (interactive: try-send + timeout, fail-fast; background: blocking send, batched into grouped transactions with early-commit when an interactive command arrives). - Route register_account, retry_erc20_deposit/retry_native_deposit, retry_webhook_delivery, and set_last_processed_block (interactive path) through the interactive lane so user-facing calls don't queue behind background batches. - Set PRAGMA synchronous=NORMAL (safe under WAL) to cut fsync overhead on EFS-backed storage. - Make queue capacities, batch size, and write timeout configurable via env vars, and log queue depth / interactive wait time when thresholds are exceeded. - On writer-thread death, flip a health flag surfaced via health() and abort the process rather than silently wedging. - Fix a P0 address-collision bug in register(): derivation index was a DefaultHasher hash of the account id (collision-prone and unstable across Rust versions); switch to a persisted sequential next_index counter allocated atomically on the writer thread (register_account_auto), seeded above the current max index by a new migration. - Add WriteQueueError so callers/HTTP layer can distinguish backpressure/timeout/writer-gone from generic internal errors. Covered by 15 new tests (pragma, batching/early-commit, timeout + late execution, saturation/backpressure on both lanes, writer death, priority bypass, collision fix, migration seeding, E2E saturation). Co-authored-by: Cursor <cursoragent@cursor.com>
A restart alone didn't recover from the 2026-07 write-queue-saturation incident because the writer never force-checkpoints: it just reopens the same oversized -wal file and resumes fighting auto-checkpoint attempts against it on every commit. - checkpoint_startup: force PRAGMA wal_checkpoint(TRUNCATE) after migrations, before the writer starts serving traffic, so every restart begins from a small WAL regardless of prior growth. - maybe_checkpoint: opportunistic, throttled PRAGMA wal_checkpoint(PASSIVE) after background batch commits, gated by a new EVM_CHECKPOINT_INTERVAL_SECS knob (default 30s) and skipped whenever an interactive write is waiting, so it never adds latency to a real request. - Log WAL byte size and the checkpoint's (busy, log_frames, checkpointed_frames) at both checkpoint points for CloudWatch visibility going forward. Co-authored-by: Cursor <cursoragent@cursor.com>
Replace single-writer mutex with priority-lane writer actor
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.