Accelerate Monitor catch-up for backlogged chains - #7
Merged
Conversation
Polygon fell thousands of blocks behind head because catch_up scanned one block at a time, each costing a full get_block_by_number plus a chain-wide (unfiltered) eth_getLogs call, with get_logs_with_retry treating an empty result as a failure and retrying up to 30x. - get_logs_with_retry now returns immediately on any Ok (including empty) and retries only on Err. - eth_getLogs is filtered to the chain's allowed token addresses instead of scanning every Transfer event on the chain. - When the gap between last-processed and head exceeds 10 blocks, catch_up switches to a batched path: one ranged, address-filtered eth_getLogs call per catch_up_chunk_size blocks (default 500), bisecting the range on provider errors (e.g. Alchemy's response-size cap) instead of retrying the identical request, plus concurrent eth_getBlockByNumber fetches (up to block_fetch_concurrency) for native transfers. Checkpoints once per chunk; replay is idempotent via the existing deposit-recording dedup. - Adds catch_up_chunk_size and block_fetch_concurrency chain config options. Covered by 7 new tests (regression, address filter, batch happy path, bisection, bisection floor, concurrent native fetch, idempotent replay). Co-authored-by: Cursor <cursoragent@cursor.com>
The monitor/sweeper/webhook-retry loops called rusqlite (via Db) directly from async fns with no spawn_blocking, and the r2d2 read pool had no connection_timeout (defaulting to 30s). During a large catch-up backlog, this let a busy chain pin a Tokio worker thread for tens of seconds, stalling everything else sharing that runtime. - Add Db::blocking, which offloads a Db closure onto spawn_blocking; route every DB call reached from async code (monitor, sweeper, webhook, register) through it. - Set an explicit 5s r2d2 connection_timeout instead of the 30s default. - Add tokio::task::yield_now() between blocks/logs in the batch catch-up path so one chunk can't monopolize a worker thread across many non-yielding blocking calls. Co-authored-by: Cursor <cursoragent@cursor.com>
Fix async-runtime stalls: move DB calls off async threads, yield during catch-up
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
catch_upscanned one block at a time, each doing a fullget_block_by_numberplus a chain-wide (unfiltered)eth_getLogscall, andget_logs_with_retrytreated an empty result as a failure, retrying up to 30x.get_logs_with_retrynow returns immediately on anyOk(including empty) and only retries onErr.eth_getLogsis filtered to the chain's allowed token addresses instead of scanning every Transfer event on chain.catch_upswitches to a batched path: one ranged, address-filteredeth_getLogscall percatch_up_chunk_sizeblocks (default 500), bisecting the range on provider errors (e.g. Alchemy's response-size cap) instead of retrying the identical request, plus concurrenteth_getBlockByNumberfetches (up toblock_fetch_concurrency) for native transfers. Checkpoints once per chunk; a replayed chunk is idempotent via the existing deposit-recording dedup.catch_up_chunk_size(default 500) andblock_fetch_concurrency(default 10) chain config options, documented inchains.toml.examplealong with a note on using separate Alchemy API keys per chain.Base branch note
This targets
feat/evm-hot-v2(open PR #5), since this branch was built directly on top of it and PR #5 hasn't landed onmainyet. This keeps the diff scoped to the catch-up work only.Test plan
cargo build(lib + both bins)cargo test— 76 passed, including 7 new tests: retry-on-empty regression, address filter presence, batch happy path, bisection on provider error, bisection floor (fail-fast, no infinite recursion), concurrent native fetch across multiple blocks, idempotent chunk replaycargo fmt --all -- --checkcargo clippy --all-targets -- -D warningsMade with Cursor