perf(tests): parallelize e2e test suite, add sync RPC latency metric#1601
Merged
Conversation
… connections, reduce timeouts - Use Promise.all for cross-coin execution: coins run concurrently (they connect to independent backends). Each coin buffers its own output and flushes contiguously after completion. - Add preloadSamples() to TestContext: eagerly resolves block, tx, address, and fiat samples once at coin startup so downstream tests hit the cache instead of triggering redundant probe chains. - Replace per-call WebSocket connections with a wsPool: maintains 3 persistent connections, eliminating the 5s handshake overhead for every WS test. Pool acquires/releases per request with automatic dead-connection replacement. - Reduce timeouts: HTTP 30s->15s, WS dial 5s->3s, WS message 15s->10s.
…utput per-coin - Add a mandatory timeout to wsConnection.connect() using a 'settled' guard that rejects after wsDialTimeoutMs even when DNS or TCP stalls (the ws library's handshakeTimeout only applies after TCP connect). - Make wsPool.init() parallel (Promise.allSettled) so all connections are attempted concurrently; a partial pool is still usable. - Remove getSampleAddressTx from preloadSamples() — it probes up to 40 transactions with paginated lookups and is only needed by a few address-listing tests; keep it lazy. - Flush each coin's output immediately upon completion instead of waiting for Promise.all, so results from finished coins are visible even if another coin hangs. - Use Promise.allSettled instead of Promise.all so a rejection in one coin does not discard results from other coins.
… init race E2e-only remainder of the original review-findings commit; its tests/integration.go part (configurable init retry) was dropped along with the Go-side test parallelization.
…ions Promise.allSettled discarded runCoin rejections: a coin whose context creation or preload threw vanished from the summaries with its buffered output unflushed, and the run printed 'passed for N coin(s)' with exit code 0. Catch the rejection per coin and push a failed CoinRun summary so the aggregate failure path fires. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
preloadSamples() ran bare inside runCoin, outside the per-test try/catch, so one transient HTTP or schema-validation error during the probe chain rejected the whole coin. Handle it like the status preflight: emit a single SamplePreload failure with the buffered output preserved and stop the coin there. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
wsConnection already multiplexes concurrent requests by ID and tests within a coin run sequentially, so the 3-connection lease pool bought nothing over one persistent socket while adding an unbounded acquire retry loop and a permanently-cached failed init. One shared connection with redial-on-close keeps the handshake-per-coin win; every wait is bounded by the dial and message timeouts, and a failed dial is not cached so the next call retries. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
pragmaxim
force-pushed
the
tests/improve-preformance
branch
from
523e2f8 to
962dbf4
Compare
pragmaxim
changed the title
… and panel debug_traceBlockByHash is typically sub-100ms, reaches seconds only when the node is overloaded, and ~10-15s observations are trace_timeout / rpc_timeout expiries — not routine latency. Say so in the histogram help text, and exclude failed calls from the Grafana p50/p95 queries so timeout expiries do not skew the quantiles (failures are already charted in the sync RPC errors panel). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
cranycrane
approved these changes
Jul 3, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Part of #1600.
Implements the e2e-suite items from the test performance optimization plan:
cross-coin parallelization for the OpenAPI/e2e suite, proactive sample
preloading, a persistent WebSocket connection, and reduced timeouts. Also
adds Prometheus latency tracking for sync-internal RPC calls.
The Go integration-test parallelization originally in this PR was dropped:
running coins concurrently in one process races on the global
chaincfgregistry (unguarded map reads vs.
Registerwrites) and panics onduplicate testnet wire magics (bellcoin/gamecredits/groestlcoin all use
0x0709110b) once per-coinResetParams()is removed. Parallelizing atthe CI level (matrix jobs sharded per coin) avoids the shared-state
problem entirely and is left as follow-up.
OpenAPI/e2e tests
Cross-coin parallelization (
runner.ts)forloop → parallel execution across coins (they connect to independentBlockbook instances). Each coin buffers its output and flushes it upon
completion so finished coins stay visible even if another coin hangs.
preload) is recorded as a failed
CoinRunsummary — the run cannot gogreen while silently skipping a coin.
SamplePreloadcoin-levelfailure, mirroring the status preflight.
Proactive sample resolution (
context.ts)preloadSamples()eagerly resolves block, tx, address, and fiat samplesonce at coin startup.
getSampleAddressTxis deliberately skipped — itprobes up to 40 transactions with paginated lookups and stays lazy.
Persistent WebSocket connection (
context.ts)wsConnectionper coin, multiplexed by request ID. Eliminates the handshake overhead
every WS test previously paid.
cached, so a transient outage does not poison later WS tests.
connect()has asettled-guard timeout covering DNS/TCP stalls (wherethe ws library handshakeTimeout does not apply);
ws.send()failuresreject the pending promise immediately. All waits are bounded by the dial
and message timeouts.
TestContext.close()closes the connection so Node.js does not keep theevent loop alive on abnormal exit.
Reduced timeouts
Metrics: sync RPC latency (
rpc_sync_latency)Added a Prometheus histogram that tracks round-trip times for sync-internal
RPC calls (eth_getBlockByHash, eth_getLogs, debug_traceBlockByHash, etc.)
that were previously invisible in the
rpc_latencymetric (which only coversmethods going through the
blockChainWithMetricswrapper).Sync calls are typically sub-100ms; an overloaded node can take seconds,
and observations near 10-15s are
trace_timeout/rpc_timeoutexpiriesrather than real round trips. The existing
rpc_latencyhistogram cannotbe reused because its buckets max out at 5000ms and would fold that whole
overload/timeout tail into
+Inf.configs/metrics.yaml— newrpc_sync_latencyhistogram with[method, error]labels and wider millisecond buckets (up to 20s).common/metrics.go— newRPCSyncLatencystruct field.bchain/coins/eth/ethrpc.go— newobserveSyncRPCLatency()helper;instrumented call sites:
getBlockRaw(),processEventsForBlock(),getInternalDataForBlock(),GetChainInfo().configs/grafana/— newsync.rpc_sync_latencypanel (p50/p95 bymethod) in the Sync dashboard section. The quantile queries read successful
calls only (
error=""), so timeout expiries do not skew the latencydisplay; failures are charted in the sync RPC errors panel.
🤖 Generated with Claude Code