Feat/evm hot v2 - #5
Open
S0c5 wants to merge 8 commits into
Open
Conversation
Co-authored-by: Cursor <cursoragent@cursor.com>
Webhooks are now persisted to a new webhook_deliveries table and delivered by a background worker instead of inline in the monitor/sweeper. Failed deliveries (non-2xx, including 503) are retried with configurable limits, and a lease prevents duplicate POSTs. Adds POST /admin/retry_webhooks to re-queue permanently failed deliveries. Co-authored-by: Cursor <cursoragent@cursor.com>
Polygon fell thousands of blocks behind head because catch_up scanned one block at a time, each costing a full get_block_by_number plus a chain-wide (unfiltered) eth_getLogs call, with get_logs_with_retry treating an empty result as a failure and retrying up to 30x. - get_logs_with_retry now returns immediately on any Ok (including empty) and retries only on Err. - eth_getLogs is filtered to the chain's allowed token addresses instead of scanning every Transfer event on the chain. - When the gap between last-processed and head exceeds 10 blocks, catch_up switches to a batched path: one ranged, address-filtered eth_getLogs call per catch_up_chunk_size blocks (default 500), bisecting the range on provider errors (e.g. Alchemy's response-size cap) instead of retrying the identical request, plus concurrent eth_getBlockByNumber fetches (up to block_fetch_concurrency) for native transfers. Checkpoints once per chunk; replay is idempotent via the existing deposit-recording dedup. - Adds catch_up_chunk_size and block_fetch_concurrency chain config options. Covered by 7 new tests (regression, address filter, batch happy path, bisection, bisection floor, concurrent native fetch, idempotent replay). Co-authored-by: Cursor <cursoragent@cursor.com>
4 tasks
The monitor/sweeper/webhook-retry loops called rusqlite (via Db) directly from async fns with no spawn_blocking, and the r2d2 read pool had no connection_timeout (defaulting to 30s). During a large catch-up backlog, this let a busy chain pin a Tokio worker thread for tens of seconds, stalling everything else sharing that runtime. - Add Db::blocking, which offloads a Db closure onto spawn_blocking; route every DB call reached from async code (monitor, sweeper, webhook, register) through it. - Set an explicit 5s r2d2 connection_timeout instead of the 30s default. - Add tokio::task::yield_now() between blocks/logs in the batch catch-up path so one chunk can't monopolize a worker thread across many non-yielding blocking calls. Co-authored-by: Cursor <cursoragent@cursor.com>
Fix async-runtime stalls: move DB calls off async threads, yield during catch-up
Accelerate Monitor catch-up for backlogged chains
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
migrate_redb_to_sqliteimporter; includes swap fixes and a buffer-overrun panic fix.webhook_deliveriestable and delivered by a background worker. Non-2xx (incl. 503) retried with configurable limits; a lease prevents duplicate POSTs. AddsPOST /admin/retry_webhooks.Webhook details
pending→delivered(2xx) /failed(retries exhausted). Idempotent on(id, event).WEBHOOK_MAX_RETRIES,WEBHOOK_RETRY_DELAY_MS,WEBHOOK_RETRY_POLL_INTERVAL,WEBHOOK_RETRY_BATCH_SIZE,WEBHOOK_LEASE_SECONDS.Migrations
migrations/V2__webhook_deliveries.sqlisinclude_str!'d and applied on startup. Rebuild + restart to apply.Notes
POST /admin/retry_webhooksfor operational retries.Test plan
cargo build --releaseandcargo testpasswebhook_deliveries(PRAGMA user_version= 2)failedafter max retriesPOST /admin/retry_webhooksre-queues a failed delivery