diff --git a/DB_OPTIMIZATION_PRIORITIES.md b/DB_OPTIMIZATION_PRIORITIES.md new file mode 100644 index 00000000000..1f479cc23b4 --- /dev/null +++ b/DB_OPTIMIZATION_PRIORITIES.md @@ -0,0 +1,147 @@ +# Database Optimization Priorities + +This document categorizes all DB-related pending items from the roadmap based on whether they require database resyncing and their potential performance impact. + +## Items Requiring Schema Modification (Require Resyncing) + +These items modify existing database tables/schema and require a full resync: + +| Section | Item | Priority | Description | +|---------|------|----------|-------------| +| IO | **Canonical tx index** | 1 | Add a canonical-tx index table or DUPSORT layout for O(1) lookups (currently O(k) prefix scans) | +| IO | **Split hot vs cold data** | 2 | Geth "freezer/ancients" pattern - store recent state in fast KV store, push old bodies/receipts to append-only ancient store | +| New Features | **Archive node** | — | Allow archive node mode - changes storage requirements/schema | +| New Features | **Pre merge blocks** | — | Be able to process pre merge blocks - requires schema changes to support different block formats | + +--- + +## Items NOT Requiring Schema Modification (No Resync) + +These items are optimizations and configurations that don't require resyncing, sorted by potential performance impact: + +### High Impact (Most Likely to Improve Performance) + +1. **Add Block Cache (RocksDB)** (#5935, P0) + - Currently relying only on OS page cache. Explicit block cache is fundamental for RocksDB performance + - Also try row cache + - **Why high impact**: Block cache is one of the most important RocksDB features for read performance + +2. **Use Two-Level Index (RocksDB)** (#5936, P0) + - Use Two-Level Index with Partitioned Filters + - **Why high impact**: Significantly reduces memory overhead and improves cache efficiency for large datasets + +3. **Use multiget on trie traversal** (#4949, P1) + - Using multiget on trie traversal might reduce read time + - **Why high impact**: Batching reads during trie traversal can dramatically reduce I/O latency + +4. **Bulk reads for block bodies** (P1) + - Implement `multi_get` for `get_block_bodies` and `get_block_bodies_by_hash` which currently loop over per-key reads + - Location: `crates/storage/store.rs:388-454` + - **Why high impact**: Substantial improvement for batch operations, reduces round-trips + +5. **Enable unordered writes for State (RocksDB)** (#5937, P0) + - For `ACCOUNT_TRIE_NODES, STORAGE_TRIE_NODES cf_opts.set_unordered_write(true);` + - Faster writes when we don't need strict ordering + - **Why high impact**: Can significantly speed up write-heavy operations during sync + +6. **Toggle compaction during sync** (P2) + - Disable RocksDB compaction during snap sync for higher write throughput, then compact after + - Nethermind pattern: Wire `disable_compaction/enable_compaction` into sync stages + - **Why high impact**: Proven pattern from other clients, can dramatically improve sync performance + +7. **Memory-Mapped Reads (RocksDB)** (#5943, P0) + - Can be an improvement on high-RAM systems + - **Why high impact**: Significant improvement by bypassing kernel page cache on systems with sufficient RAM + +### Medium-High Impact + +8. **Page caching + readahead** (#5940, P0) + - Use for trie iteration, sync operations + - **Why medium-high**: Reduces random I/O by prefetching related data + +9. **Reduce trie cache Mutex contention** (P1) + - `trie_cache` is behind `Arc>>` + - Use `ArcSwap` or `RwLock` for lock-free reads + - Location: `crates/storage/store.rs:159,1360` + - **Why medium-high**: High-frequency access point, lock contention can be significant bottleneck + +10. **Reduce LatestBlockHeaderCache contention** (P1) + - `LatestBlockHeaderCache` uses Mutex for every read + - Use `ArcSwap` for atomic pointer swaps + - Location: `crates/storage/store.rs:2880-2894` + - **Why medium-high**: Accessed on every read operation + +11. **Increase Bloom Filter (RocksDB)** (#5938, P0) + - Change and benchmark higher bits per key for state tables + - **Why medium-high**: Reduces unnecessary disk reads by improving filter accuracy + +12. **Use Bytes/Arc in trie layer cache** (P2) + - Trie layer cache clones `Vec` values on every read + - Use `Bytes` or `Arc<[u8]>` to reduce allocations + - Location: `crates/storage/layering.rs:57,63` + - **Why medium-high**: Reduces allocations in hot path + +13. **Optimize for Point Lookups (RocksDB)** (#5941, P0) + - Adds hash index inside FlatKeyValue for faster point lookups + - **Why medium-high**: Faster for common lookup patterns + +### Medium Impact + +14. **Consider LZ4 for State Tables (RocksDB)** (#5939, P0) + - Trades CPU for smaller DB and potentially better cache utilization + - **Why medium**: Depends on CPU vs I/O bottleneck and workload characteristics + +15. **Increase layers commit threshold** (#5944, P0) + - For read-heavy workloads with plenty of RAM + - **Why medium**: Reduces write amplification but only beneficial in specific scenarios + +16. **Configurable cache budgets** (P2) + - Expose cache split for DB/trie/snapshot as runtime config + - Currently hardcoded in ethrex + - **Why medium**: Allows tuning for specific hardware but requires user knowledge + +17. **Benchmark bloom filter** (#5946, P1) + - Review trie layer's bloom filter, remove it or test other libraries/configurations + - **Why medium**: May remove overhead if not beneficial, but needs measurement + +18. **Modify block size (RocksDB)** (#5942, P0) + - Benchmark different block size configurations + - **Why medium**: Workload dependent, requires benchmarking to determine optimal value + +### Lower Impact (But Still Useful) + +19. **Remove locks** (#5945, P1) + - Check if there are still some unnecessary locks, e.g. in the VM we have one + - **Why lower**: Limited scope, only affects specific components + +20. **geth db migration tooling** (P0, In Progress) + - As we don't support pre-merge blocks we need a tool to migrate other client's DB to ours at a specific block + - **Why lower**: More of a feature than performance improvement, enables compatibility + +21. **Migrations** (P4) + - Add DB Migration mechanism for ethrex upgrades + - **Why lower**: Infrastructure for future changes, not direct performance improvement + +--- + +## Summary + +- **4 items** require DB schema changes and resyncing +- **21 items** are DB-related optimizations/configurations that don't require resyncing +- Most high-priority (P0) DB work focuses on RocksDB tuning and configuration + +### Top Recommended Actions (No Resync Required) + +The top 7 items provide the most significant performance improvements: + +1. Add Block Cache (RocksDB) - fundamental for read performance +2. Use Two-Level Index (RocksDB) - reduces memory overhead +3. Use multiget on trie traversal - reduces I/O latency +4. Bulk reads for block bodies - improves batch operations +5. Enable unordered writes for State - speeds up writes during sync +6. Toggle compaction during sync - proven pattern for sync performance +7. Memory-Mapped Reads - significant improvement on high-RAM systems + +### Key Observation + +The explicit block cache (#5935) is likely the **single biggest win** since the system is currently relying only on OS page cache, which is a fundamental RocksDB optimization. diff --git a/crates/blockchain/blockchain.rs b/crates/blockchain/blockchain.rs index fbe50f5ba38..40ffb095505 100644 --- a/crates/blockchain/blockchain.rs +++ b/crates/blockchain/blockchain.rs @@ -2229,6 +2229,12 @@ impl Blockchain { transactions_count += block.body.transactions.len(); all_receipts.push((block.hash(), receipts)); + // Normalize VM cache for next block to prevent metadata pollution (issue #6467) + // This resets transient flags (exists, status) while preserving state changes + if i + 1 < blocks_len { + vm.normalize_cache_for_next_block(); + } + // Conversion is safe because EXECUTE_BATCH_SIZE=1024 log_batch_progress(blocks_len as u32, i as u32); tokio::task::yield_now().await; diff --git a/crates/blockchain/payload.rs b/crates/blockchain/payload.rs index c3cc0a0dc52..ea56051f199 100644 --- a/crates/blockchain/payload.rs +++ b/crates/blockchain/payload.rs @@ -833,10 +833,28 @@ pub fn apply_plain_transaction( // EIP-8037 (Amsterdam+): track regular and state gas separately let tx_state_gas = report.state_gas_used; let tx_regular_gas = report.gas_used.saturating_sub(tx_state_gas); - context.block_regular_gas_used = context + + // Compute new totals before committing them + let new_regular = context .block_regular_gas_used .saturating_add(tx_regular_gas); - context.block_state_gas_used = context.block_state_gas_used.saturating_add(tx_state_gas); + let new_state = context.block_state_gas_used.saturating_add(tx_state_gas); + + // EIP-8037 (Amsterdam+): post-execution block gas overflow check + // Reject the transaction if adding it would cause max(regular, state) to exceed the gas limit + if context.is_amsterdam && new_regular.max(new_state) > context.payload.header.gas_limit { + return Err(EvmError::Custom(format!( + "block gas limit exceeded (state gas overflow): \ + max({new_regular}, {new_state}) = {} > gas_limit {}", + new_regular.max(new_state), + context.payload.header.gas_limit + )) + .into()); + } + + // Commit the new totals + context.block_regular_gas_used = new_regular; + context.block_state_gas_used = new_state; if context.is_amsterdam { debug!( @@ -852,15 +870,14 @@ pub fn apply_plain_transaction( } // Update remaining_gas for block gas limit checks. - // EIP-8037 (Amsterdam+): per-tx check only validates regular gas against block limit. - // State gas is NOT checked per-tx; block-end validation enforces - // max(block_regular, block_state) <= gas_limit. + // EIP-8037 (Amsterdam+): remaining_gas reflects both regular and state gas dimensions. + // For pre-tx heuristic checks, this ensures we reject txs when either dimension is full. if context.is_amsterdam { context.remaining_gas = context .payload .header .gas_limit - .saturating_sub(context.block_regular_gas_used); + .saturating_sub(new_regular.max(new_state)); } else { context.remaining_gas = context.remaining_gas.saturating_sub(report.gas_used); } diff --git a/crates/vm/backends/levm/mod.rs b/crates/vm/backends/levm/mod.rs index 9500c2c83b7..0d46ef1d9a2 100644 --- a/crates/vm/backends/levm/mod.rs +++ b/crates/vm/backends/levm/mod.rs @@ -31,7 +31,6 @@ use ethrex_levm::account::{AccountStatus, LevmAccount}; use ethrex_levm::call_frame::Stack; use ethrex_levm::constants::{ POST_OSAKA_GAS_LIMIT_CAP, STACK_LIMIT, SYS_CALL_GAS_LIMIT, TX_BASE_COST, - TX_MAX_GAS_LIMIT_AMSTERDAM, }; use ethrex_levm::db::Database; use ethrex_levm::db::gen_db::{CacheDB, GeneralizedDatabase}; @@ -126,18 +125,13 @@ impl LEVM { })?; for (tx_idx, (tx, tx_sender)) in transactions_with_sender.into_iter().enumerate() { - // Pre-tx gas limit guard per EIP-8037/EIP-7825: - // Amsterdam: check min(TX_MAX_GAS_LIMIT, tx.gas) against regular gas only. - // State gas is NOT checked per-tx; block-end validation enforces - // max(block_regular, block_state) <= gas_limit. - // Pre-Amsterdam: check tx.gas against cumulative_gas_used (post-refund sum). - if is_amsterdam { - check_gas_limit( - block_regular_gas_used, - tx.gas_limit().min(TX_MAX_GAS_LIMIT_AMSTERDAM), - block.header.gas_limit, - )?; - } else { + // Pre-tx gas limit guard: + // Pre-Amsterdam: reject tx if cumulative post-refund gas + tx.gas > block limit. + // Amsterdam+: skip — EIP-8037's 2D gas model means cumulative gas (regular + + // state) can legally exceed the block gas limit as long as + // max(sum_regular, sum_state) stays within it. Block-level overflow is + // detected post-execution. + if !is_amsterdam { check_gas_limit(cumulative_gas_used, tx.gas_limit(), block.header.gas_limit)?; } @@ -189,6 +183,17 @@ impl LEVM { receipts.push(receipt); } + // EIP-7778 (Amsterdam+): block-level gas overflow check. + // Per-tx checks are skipped for Amsterdam because block gas is computed + // from pre-refund values; overflow can only be detected after execution. + if is_amsterdam && block_gas_used > block.header.gas_limit { + return Err(EvmError::Transaction(format!( + "Gas allowance exceeded: Block gas used overflow: \ + block_gas_used {block_gas_used} > block_gas_limit {}", + block.header.gas_limit + ))); + } + // Set BAL index for post-execution phase (requests + withdrawals, uint16) // Order must match geth: requests (system calls) BEFORE withdrawals. if is_amsterdam { @@ -424,18 +429,13 @@ impl LEVM { let mut tx_since_last_flush = 2; for (tx_idx, (tx, tx_sender)) in transactions_with_sender.into_iter().enumerate() { - // Pre-tx gas limit guard per EIP-8037/EIP-7825: - // Amsterdam: check min(TX_MAX_GAS_LIMIT, tx.gas) against regular gas only. - // State gas is NOT checked per-tx; block-end validation enforces - // max(block_regular, block_state) <= gas_limit. - // Pre-Amsterdam: check tx.gas against cumulative_gas_used (post-refund sum). - if is_amsterdam { - check_gas_limit( - block_regular_gas_used, - tx.gas_limit().min(TX_MAX_GAS_LIMIT_AMSTERDAM), - block.header.gas_limit, - )?; - } else { + // Pre-tx gas limit guard: + // Pre-Amsterdam: reject tx if cumulative post-refund gas + tx.gas > block limit. + // Amsterdam+: skip — EIP-8037's 2D gas model means cumulative gas (regular + + // state) can legally exceed the block gas limit as long as + // max(sum_regular, sum_state) stays within it. Block-level overflow is + // detected post-execution. + if !is_amsterdam { check_gas_limit(cumulative_gas_used, tx.gas_limit(), block.header.gas_limit)?; } @@ -497,6 +497,17 @@ impl LEVM { receipts.push(receipt); } + // EIP-7778 (Amsterdam+): block-level gas overflow check. + // Per-tx checks are skipped for Amsterdam because block gas is computed + // from pre-refund values; overflow can only be detected after execution. + if is_amsterdam && block_gas_used > block.header.gas_limit { + return Err(EvmError::Transaction(format!( + "Gas allowance exceeded: Block gas used overflow: \ + block_gas_used {block_gas_used} > block_gas_limit {}", + block.header.gas_limit + ))); + } + #[cfg(feature = "perf_opcode_timings")] { let mut timings = OPCODE_TIMINGS.lock().expect("poison"); @@ -972,32 +983,21 @@ impl LEVM { // balance in the BAL won't match execution that ran all txs). let mut block_regular_gas_used = 0_u64; let mut block_state_gas_used = 0_u64; - for (tx_idx, _, report, _, _, _) in &exec_results { - // Per-tx check: only regular gas is checked per-tx (EIP-8037/EIP-7825). - // State gas is validated at block end via max(regular, state) <= gas_limit. - let tx_gas_limit = txs_with_sender[*tx_idx].0.gas_limit(); - check_gas_limit( - block_regular_gas_used, - tx_gas_limit.min(TX_MAX_GAS_LIMIT_AMSTERDAM), - header.gas_limit, - )?; + for (_, _, report, _, _, _) in &exec_results { let tx_state_gas = report.state_gas_used; let tx_regular_gas = report.gas_used.saturating_sub(tx_state_gas); block_regular_gas_used = block_regular_gas_used.saturating_add(tx_regular_gas); block_state_gas_used = block_state_gas_used.saturating_add(tx_state_gas); - // Post-tx check: needed because all txs are already executed — if the last tx - // pushes actual gas over the limit, there's no next iteration to catch it - // like the sequential path does. - let running_block_gas_after = block_regular_gas_used.max(block_state_gas_used); - if running_block_gas_after > header.gas_limit { - return Err(EvmError::Transaction(format!( - "Gas allowance exceeded: \ - used {running_block_gas_after} > block limit {}", - header.gas_limit - ))); - } } let block_gas_used = block_regular_gas_used.max(block_state_gas_used); + // EIP-7778: block-level overflow check using pre-refund gas. + if block_gas_used > header.gas_limit { + return Err(EvmError::Transaction(format!( + "Gas allowance exceeded: Block gas used overflow: \ + block_gas_used {block_gas_used} > block_gas_limit {}", + header.gas_limit + ))); + } // 4. Per-tx BAL validation — now safe to run after gas limit is confirmed OK. // Also mark off storage_reads that appear in per-tx execution state. diff --git a/crates/vm/backends/mod.rs b/crates/vm/backends/mod.rs index 47c16578176..ec98049421f 100644 --- a/crates/vm/backends/mod.rs +++ b/crates/vm/backends/mod.rs @@ -195,6 +195,13 @@ impl Evm { LEVM::get_state_transitions(&mut self.db) } + /// Normalizes VM cache metadata between blocks in batch execution. + /// Fixes issue #6467 where transient account metadata (exists, status) leaks + /// between blocks, causing incorrect EIP-7702 gas refund calculations. + pub fn normalize_cache_for_next_block(&mut self) { + self.db.normalize_cache_for_next_block(); + } + /// Wraps [LEVM::process_withdrawals]. /// Applies the withdrawals to the state or the block_chache if using [LEVM]. pub fn process_withdrawals(&mut self, withdrawals: &[Withdrawal]) -> Result<(), EvmError> { diff --git a/crates/vm/levm/src/db/gen_db.rs b/crates/vm/levm/src/db/gen_db.rs index 9cf4458246f..9024cb5ca0e 100644 --- a/crates/vm/levm/src/db/gen_db.rs +++ b/crates/vm/levm/src/db/gen_db.rs @@ -409,6 +409,36 @@ impl GeneralizedDatabase { Ok(account_updates) } + /// Normalizes cached accounts between blocks in batch execution. + /// + /// After executing block N, accounts in current_accounts_state represent the + /// post-block state. These accounts will be the base for block N+1, but their + /// transient metadata (status, exists) must be reset to prevent pollution. + /// + /// This fixes issue #6467 where mark_modified() sets exists=true, which then + /// incorrectly persists into the next block, causing wrong EIP-7702 refunds. + /// + /// Called between blocks in add_blocks_in_batch. + pub fn normalize_cache_for_next_block(&mut self) { + // Normalize metadata in current_accounts_state (the working cache) + for (address, account) in self.current_accounts_state.iter_mut() { + // Reset status - these accounts are now the "unmodified" base for next block + account.status = AccountStatus::Unmodified; + + // Recalculate exists based on actual account content, not mark_modified flag. + // An account exists if it's non-empty (has balance, nonce, code, or storage). + // This matches the semantics of loading from DB (see From). + account.exists = !account.info.is_empty() || account.has_storage; + + // Ensure account exists in initial_accounts_state for storage access + // If it was created in a previous block of this batch, add it now + if !self.initial_accounts_state.contains_key(address) { + self.initial_accounts_state + .insert(*address, account.clone()); + } + } + } + pub fn get_state_transitions_tx(&mut self) -> Result, VMError> { let mut account_updates: Vec = vec![]; for (address, new_state_account) in self.current_accounts_state.drain() { diff --git a/mainnet_restore_commands.txt b/mainnet_restore_commands.txt new file mode 100644 index 00000000000..f40fe1c272f --- /dev/null +++ b/mainnet_restore_commands.txt @@ -0,0 +1,36 @@ +MAINNET RESTORE COMMANDS FOR admin@ethrex-mainnet-2 +===================================================== + +Date saved: 2026-04-13 +Purpose: Backup before switching to Holesky for snap sync testing + +LIGHTHOUSE (session: lighthouse): +--------------------------------- +tmux new-session -d -s lighthouse \ + lighthouse bn --network mainnet \ + --execution-endpoint http://localhost:8551 \ + --execution-jwt /home/admin/secrets/jwt.hex \ + --checkpoint-sync-url https://mainnet-checkpoint-sync.attestant.io \ + --port 9000 \ + --http --http-address 0.0.0.0 --http-port 5052 --http-allow-origin "*" \ + --metrics --metrics-port 5054 --metrics-address 0.0.0.0 \ + 2>&1 | tee ~/lighthouse.log + +ETHREX (session: 8): +-------------------- +# Note: Run from ethrex directory where ./target/release/ethrex exists +tmux new-session -d -s ethrex \ + ./target/release/ethrex --network mainnet \ + --http.addr 0.0.0.0 --http.port 8545 \ + --authrpc.port 8551 \ + --authrpc.jwtsecret /home/admin/secrets/jwt.hex \ + --p2p.port 30303 --discovery.port 30303 \ + --metrics --metrics.port 3701 \ + --log.dir /var/log/ethrex + +TO RESTORE: +----------- +1. Stop any running sessions: tmux kill-session -t lighthouse && tmux kill-session -t 8 +2. Run the lighthouse command above +3. Run the ethrex command above (from the ethrex directory) +4. Attach to sessions: tmux attach -t lighthouse (or -t ethrex) diff --git a/tooling/contract_metrics/Cargo.toml b/tooling/contract_metrics/Cargo.toml new file mode 100644 index 00000000000..0946dc847a4 --- /dev/null +++ b/tooling/contract_metrics/Cargo.toml @@ -0,0 +1,18 @@ +[package] +name = "contract_metrics" +version.workspace = true +edition.workspace = true +authors.workspace = true +documentation.workspace = true +license.workspace = true + +[[bin]] +name = "contract_metrics" +path = "src/main.rs" + +[dependencies] +clap.workspace = true +eyre.workspace = true +ethrex-common.workspace = true +ethrex-storage = { features = ["rocksdb"], workspace = true } +tokio = { features = ["full"], workspace = true } diff --git a/tooling/contract_metrics/src/main.rs b/tooling/contract_metrics/src/main.rs new file mode 100644 index 00000000000..2007e6d08a8 --- /dev/null +++ b/tooling/contract_metrics/src/main.rs @@ -0,0 +1,242 @@ +use clap::Parser; +use ethrex_common::{Address, types::TxKind}; +use ethrex_storage::{EngineType, Store}; +use std::{collections::HashMap, io::Write as _, path::PathBuf}; + +#[derive(Parser)] +#[command(about = "Extract contract deployment metrics from an ethrex mainnet database")] +struct Cli { + /// Path to the ethrex data directory (must contain a RocksDB store) + #[arg(long)] + datadir: PathBuf, + + /// First block to analyse (inclusive). Defaults to `end - 10000`. + #[arg(long)] + start: Option, + + /// Last block to analyse (inclusive). Defaults to the latest stored block. + #[arg(long)] + end: Option, + + /// Convenience: analyse the last M blocks ending at `end`. + /// Ignored if `--start` is given. + #[arg(long, default_value = "10000")] + blocks: u64, + + /// How many top-called contracts to print. + #[arg(long, default_value = "20")] + top_n: usize, + + /// Write per-block data as CSV to this file path. + #[arg(long)] + csv: Option, +} + +struct BlockStats { + block_number: u64, + total_txs: usize, + /// Transactions with TxKind::Create (attempted deployments). + create_txs: usize, + /// CREATE transactions whose receipt shows success. + successful_creates: usize, + /// Total gas used in the block (from the last receipt's cumulative field). + gas_used: u64, +} + +fn percentile(sorted: &[u64], p: f64) -> u64 { + if sorted.is_empty() { + return 0; + } + let idx = ((sorted.len() as f64 - 1.0) * p / 100.0).round() as usize; + sorted[idx] +} + +fn print_summary(label: &str, mut values: Vec) { + if values.is_empty() { + println!("{label}: no data"); + return; + } + values.sort_unstable(); + let sum: u64 = values.iter().sum(); + let mean = sum as f64 / values.len() as f64; + let min = values[0]; + let max = *values.last().unwrap(); + let p50 = percentile(&values, 50.0); + let p95 = percentile(&values, 95.0); + let p99 = percentile(&values, 99.0); + println!("{label}: min={min} mean={mean:.1} p50={p50} p95={p95} p99={p99} max={max}"); +} + +#[tokio::main] +async fn main() -> eyre::Result<()> { + let cli = Cli::parse(); + + let store = Store::new(&cli.datadir, EngineType::RocksDB)?; + + let end = match cli.end { + Some(n) => n, + None => store.get_latest_block_number().await?, + }; + let start = cli.start.unwrap_or_else(|| end.saturating_sub(cli.blocks)); + + println!( + "Scanning blocks {start}..={end} ({} blocks)", + end - start + 1 + ); + + let mut csv_writer: Option = if let Some(ref path) = cli.csv { + let f = std::fs::File::create(path)?; + Some(f) + } else { + None + }; + + if let Some(ref mut f) = csv_writer { + writeln!( + f, + "block_number,total_txs,create_txs,successful_creates,gas_used" + )?; + } + + let mut stats: Vec = Vec::with_capacity((end - start + 1) as usize); + let mut call_counts: HashMap = HashMap::new(); + let mut missing = 0u64; + + for block_num in start..=end { + let Some(header) = store.get_block_header(block_num)? else { + missing += 1; + continue; + }; + let block_hash = header.hash(); + + let Some(body) = store.get_block_body_by_hash(block_hash).await? else { + missing += 1; + continue; + }; + + let receipts = store + .get_receipts_for_block(block_hash) + .await? + .unwrap_or_default(); + + let total_txs = body.transactions.len(); + + // Pair transactions with their receipts. If receipts are missing or + // mismatched (should not happen on a healthy DB) we conservatively + // treat those txs as failed. + let create_txs = body + .transactions + .iter() + .filter(|tx| tx.is_contract_creation()) + .count(); + + let successful_creates = body + .transactions + .iter() + .zip(receipts.iter()) + .filter(|(tx, receipt)| tx.is_contract_creation() && receipt.succeeded) + .count(); + + let gas_used = receipts.last().map(|r| r.cumulative_gas_used).unwrap_or(0); + + for tx in &body.transactions { + if let TxKind::Call(to) = tx.to() { + *call_counts.entry(to).or_insert(0) += 1; + } + } + + if let Some(ref mut f) = csv_writer { + writeln!( + f, + "{block_num},{total_txs},{create_txs},{successful_creates},{gas_used}" + )?; + } + + stats.push(BlockStats { + block_number: block_num, + total_txs, + create_txs, + successful_creates, + gas_used, + }); + + if block_num % 1_000 == 0 { + eprintln!(" ... processed block {block_num}"); + } + } + + println!( + "\n=== Results over {} blocks ({missing} skipped) ===\n", + stats.len() + ); + + let total_creates: u64 = stats.iter().map(|s| s.successful_creates as u64).sum(); + let total_txs: u64 = stats.iter().map(|s| s.total_txs as u64).sum(); + println!("Total transactions : {total_txs}"); + println!("Total successful creates: {total_creates}"); + if total_txs > 0 { + println!( + "Deploy rate : {:.3}%", + total_creates as f64 / total_txs as f64 * 100.0 + ); + } + println!(); + + print_summary( + "creates/block (attempted) ", + stats.iter().map(|s| s.create_txs as u64).collect(), + ); + print_summary( + "creates/block (successful)", + stats.iter().map(|s| s.successful_creates as u64).collect(), + ); + print_summary( + "txs/block ", + stats.iter().map(|s| s.total_txs as u64).collect(), + ); + print_summary( + "gas_used/block ", + stats.iter().map(|s| s.gas_used).collect(), + ); + + // Top 10 blocks by deployment count. + let mut by_creates: Vec<&BlockStats> = stats.iter().collect(); + by_creates.sort_unstable_by_key(|s| std::cmp::Reverse(s.successful_creates)); + println!("\nTop 10 blocks by successful contract deployments:"); + println!( + "{:>12} {:>9} {:>8} {:>10}", + "block", "creates", "total_tx", "gas_used" + ); + for s in by_creates.iter().take(10) { + println!( + "{:>12} {:>9} {:>8} {:>10}", + s.block_number, s.successful_creates, s.total_txs, s.gas_used + ); + } + + // Top-N most-called contracts. + let total_calls: u64 = call_counts.values().sum(); + let mut by_calls: Vec<(Address, u64)> = call_counts.into_iter().collect(); + by_calls.sort_unstable_by_key(|(_, count)| std::cmp::Reverse(*count)); + + println!( + "\nTop {} most-called contracts ({} unique addresses, {} total calls):", + cli.top_n, + by_calls.len(), + total_calls + ); + println!( + "{:>5} {:<42} {:>10} {:>8}", + "rank", "address", "calls", "share%" + ); + for (rank, (addr, count)) in by_calls.iter().take(cli.top_n).enumerate() { + println!( + "{:>5} {addr:#x} {:>10} {:>7.3}%", + rank + 1, + count, + *count as f64 / total_calls as f64 * 100.0, + ); + } + + Ok(()) +} diff --git a/tooling/ipv6-test/docker-compose.yml b/tooling/ipv6-test/docker-compose.yml new file mode 100644 index 00000000000..ac86238612f --- /dev/null +++ b/tooling/ipv6-test/docker-compose.yml @@ -0,0 +1,85 @@ +networks: + eth-ipv6: + driver: bridge + enable_ipv6: true + ipam: + config: + - subnet: "172.28.0.0/16" + - subnet: "fd12:3456::/64" + +services: + node-a: + image: ethrex:local + build: + context: ../.. + dockerfile: Dockerfile + environment: + - RUST_LOG=ethrex_p2p=debug + networks: + eth-ipv6: + ipv4_address: "172.28.0.2" + ipv6_address: "fd12:3456::2" + volumes: + - ./genesis.json:/genesis.json + command: > + --network /genesis.json + --datadir memory + --p2p.addr :: + --p2p.port 30303 + --nat.extip fd12:3456::2 + --http.addr 0.0.0.0 + --http.port 8545 + + node-b: + image: ethrex:local + environment: + - RUST_LOG=ethrex_p2p=debug + networks: + eth-ipv6: + ipv4_address: "172.28.0.3" + ipv6_address: "fd12:3456::3" + volumes: + - ./genesis.json:/genesis.json + - ./start-node-b.sh:/start-node-b.sh + depends_on: + - node-a + entrypoint: ["/bin/bash", "/start-node-b.sh"] + + reth: + image: ghcr.io/paradigmxyz/reth:latest + networks: + eth-ipv6: + ipv4_address: "172.28.0.4" + ipv6_address: "fd12:3456::4" + volumes: + - ./genesis.json:/genesis.json + - ./start-reth.sh:/start-reth.sh + depends_on: + - node-a + entrypoint: ["/bin/bash", "/start-reth.sh"] + + geth: + image: ethereum/client-go:latest + networks: + eth-ipv6: + ipv4_address: "172.28.0.5" + ipv6_address: "fd12:3456::5" + volumes: + - ./genesis.json:/genesis.json + - ./start-geth.sh:/start-geth.sh + depends_on: + - node-a + entrypoint: ["/bin/sh", "/start-geth.sh"] + + besu: + image: hyperledger/besu:latest + networks: + eth-ipv6: + ipv4_address: "172.28.0.6" + ipv6_address: "fd12:3456::6" + volumes: + - ./genesis.json:/genesis.json + - ./start-besu.sh:/start-besu.sh + depends_on: + - node-a + entrypoint: ["/bin/bash", "/start-besu.sh"] diff --git a/tooling/ipv6-test/genesis.json b/tooling/ipv6-test/genesis.json new file mode 100644 index 00000000000..7792940f9b9 --- /dev/null +++ b/tooling/ipv6-test/genesis.json @@ -0,0 +1,36 @@ +{ + "config": { + "chainId": 3151908, + "homesteadBlock": 0, + "eip150Block": 0, + "eip155Block": 0, + "eip158Block": 0, + "byzantiumBlock": 0, + "constantinopleBlock": 0, + "petersburgBlock": 0, + "istanbulBlock": 0, + "berlinBlock": 0, + "londonBlock": 0, + "terminalTotalDifficulty": 0, + "terminalTotalDifficultyPassed": true, + "shanghaiTime": 0, + "cancunTime": 0, + "depositContractAddress": "0x00000000219ab540356cbb839cbe05303d7705fa", + "blobSchedule": { + "cancun": { + "target": 3, + "max": 6, + "baseFeeUpdateFraction": 3338477 + } + }, + "ethash": {} + }, + "nonce": "0x0", + "timestamp": "0x0", + "extraData": "0x", + "gasLimit": "0x17d7840", + "difficulty": "0x10000", + "mixHash": "0x0000000000000000000000000000000000000000000000000000000000000000", + "coinbase": "0x0000000000000000000000000000000000000000", + "alloc": {} +} diff --git a/tooling/ipv6-test/start-besu.sh b/tooling/ipv6-test/start-besu.sh new file mode 100755 index 00000000000..bb2b94e0d0f --- /dev/null +++ b/tooling/ipv6-test/start-besu.sh @@ -0,0 +1,37 @@ +#!/bin/bash +set -e + +apt-get update -qq && apt-get install -y -qq --no-install-recommends curl + +echo "[besu] Waiting for node-a HTTP..." +until curl -s --connect-timeout 1 http://node-a:8545 > /dev/null 2>&1; do + sleep 1 +done + +echo "[besu] Fetching node-a enode URL..." +RESPONSE=$(curl -s http://node-a:8545 \ + -H "Content-Type: application/json" \ + -d '{"jsonrpc":"2.0","method":"admin_nodeInfo","params":[],"id":1}') + +echo "[besu] admin_nodeInfo response: $RESPONSE" + +ENODE=$(echo "$RESPONSE" | grep -o '"enode":"[^"]*"' | sed 's/"enode":"//;s/"//') + +if [ -z "$ENODE" ]; then + echo "[besu] ERROR: could not parse enode from response" + exit 1 +fi + +echo "[besu] Connecting to: $ENODE" + +exec /opt/besu/bin/besu \ + --genesis-file=/genesis.json \ + --data-path=/data \ + --p2p-host=fd12:3456::6 \ + --p2p-interface=fd12:3456::6 \ + --nat-method=NONE \ + --p2p-port=30303 \ + --rpc-http-enabled \ + --rpc-http-host=0.0.0.0 \ + --rpc-http-port=8545 \ + --bootnodes="$ENODE" diff --git a/tooling/ipv6-test/start-geth.sh b/tooling/ipv6-test/start-geth.sh new file mode 100755 index 00000000000..ddd3845a99d --- /dev/null +++ b/tooling/ipv6-test/start-geth.sh @@ -0,0 +1,38 @@ +#!/bin/sh +set -e + +# Geth image is Alpine-based +apk add --no-cache curl + +echo "[geth] Initializing genesis..." +geth init --datadir /data /genesis.json + +echo "[geth] Waiting for node-a HTTP..." +until curl -s --connect-timeout 1 http://node-a:8545 > /dev/null 2>&1; do + sleep 1 +done + +echo "[geth] Fetching node-a enode URL..." +RESPONSE=$(curl -s http://node-a:8545 \ + -H "Content-Type: application/json" \ + -d '{"jsonrpc":"2.0","method":"admin_nodeInfo","params":[],"id":1}') + +echo "[geth] admin_nodeInfo response: $RESPONSE" + +ENODE=$(echo "$RESPONSE" | grep -o '"enode":"[^"]*"' | sed 's/"enode":"//;s/"//') + +if [ -z "$ENODE" ]; then + echo "[geth] ERROR: could not parse enode from response" + exit 1 +fi + +echo "[geth] Connecting to: $ENODE" + +exec geth \ + --datadir /data \ + --port 30303 \ + --nat extip:172.28.0.5 \ + --http --http.addr 0.0.0.0 --http.port 8545 \ + --bootnodes "$ENODE" \ + --verbosity 4 \ + --nodiscover=false diff --git a/tooling/ipv6-test/start-node-b.sh b/tooling/ipv6-test/start-node-b.sh new file mode 100644 index 00000000000..8d7db07c88e --- /dev/null +++ b/tooling/ipv6-test/start-node-b.sh @@ -0,0 +1,37 @@ +#!/bin/bash +set -e + +# curl is not in the base ubuntu image +echo "[node-b] Installing curl..." +apt-get update -qq && apt-get install -y -qq --no-install-recommends curl + +echo "[node-b] Waiting for node-a HTTP..." +until curl -s --connect-timeout 1 http://node-a:8545 > /dev/null 2>&1; do + sleep 1 +done + +echo "[node-b] Fetching node-a enode URL..." +RESPONSE=$(curl -s http://node-a:8545 \ + -H "Content-Type: application/json" \ + -d '{"jsonrpc":"2.0","method":"admin_nodeInfo","params":[],"id":1}') + +echo "[node-b] admin_nodeInfo response: $RESPONSE" + +ENODE=$(echo "$RESPONSE" | grep -o '"enode":"[^"]*"' | sed 's/"enode":"//;s/"//') + +if [ -z "$ENODE" ]; then + echo "[node-b] ERROR: could not parse enode from response: $RESPONSE" + exit 1 +fi + +echo "[node-b] Connecting to: $ENODE" + +exec /usr/local/bin/ethrex \ + --network /genesis.json \ + --datadir memory \ + --p2p.addr :: \ + --p2p.port 30303 \ + --nat.extip fd12:3456::3 \ + --http.addr 0.0.0.0 \ + --http.port 8545 \ + --bootnodes "$ENODE" diff --git a/tooling/ipv6-test/start-reth.sh b/tooling/ipv6-test/start-reth.sh new file mode 100755 index 00000000000..41eccb82799 --- /dev/null +++ b/tooling/ipv6-test/start-reth.sh @@ -0,0 +1,35 @@ +#!/bin/bash +set -e + +apt-get update -qq && apt-get install -y -qq --no-install-recommends curl + +echo "[reth] Waiting for node-a HTTP..." +until curl -s --connect-timeout 1 http://node-a:8545 > /dev/null 2>&1; do + sleep 1 +done + +echo "[reth] Fetching node-a enode URL..." +RESPONSE=$(curl -s http://node-a:8545 \ + -H "Content-Type: application/json" \ + -d '{"jsonrpc":"2.0","method":"admin_nodeInfo","params":[],"id":1}') + +echo "[reth] admin_nodeInfo response: $RESPONSE" + +ENODE=$(echo "$RESPONSE" | grep -o '"enode":"[^"]*"' | sed 's/"enode":"//;s/"//') + +if [ -z "$ENODE" ]; then + echo "[reth] ERROR: could not parse enode from response" + exit 1 +fi + +echo "[reth] Connecting to: $ENODE" + +exec reth node \ + --chain /genesis.json \ + --datadir /data \ + --addr :: \ + --port 30303 \ + --discovery.addr :: \ + --nat extip:fd12:3456::4 \ + --http --http.addr 0.0.0.0 --http.port 8545 \ + --bootnodes "$ENODE"