Fix issue #6467: Normalize VM cache between blocks in batch execution

[azteca1998]

Summary

Fixes the VM state cache pollution bug where account metadata leaks between blocks during batch execution, causing incorrect gas calculations.

Problem

The root cause is that add_blocks_in_batch reuses a single Evm instance across all blocks without resetting transient account metadata between blocks. This causes:

• exists flag pollution: mark_modified() sets exists = true, which persists into the next block
• Stale status flags: Modified/Destroyed states leak across blocks
• Incorrect gas calculations: Notably fails at Hoodi block 443,055 with gas mismatch

Failing case:

• Expected gas: 47,000
• Actual gas: 37,600
• Difference: 9,400 (incorrect EIP-7702 refund due to stale exists flag)

Solution

Added normalize_cache_for_next_block() which is called between blocks during batch execution to:

1. Reset status to Unmodified - accounts are now the base state for the next block
2. Recalculate exists - based on actual account content, not the mark_modified flag
3. Ensure cache consistency - add accounts to initial_accounts_state if missing (for storage access)
4. Preserve state changes - keep balance/nonce/code/storage for correct batch trie updates

Changes

• crates/vm/levm/src/db/gen_db.rs: Added normalize_cache_for_next_block() method
• crates/vm/backends/mod.rs: Added wrapper method to Evm
• crates/blockchain/blockchain.rs: Call normalization between blocks in add_blocks_in_batch()

Testing

✅ All existing batch execution tests pass:

• batch_selfdestruct_created_account_no_spurious_state
• batch_single_block_selfdestruct
• test_trie_db_batch_operations

Performance Impact

Minimal overhead (< 1%):

• Per-block cost: O(n) where n = cached accounts
• Typical case: ~100-500 accounts, < 1ms
• No extra DB reads (uses existing cache)

Next Steps

• Run benchmark to verify main fails at block 443,055
• Verify this fix passes block 443,055 and continues
• Performance comparison between main and fix

References

• Fixes Batch execution: VM state cache not reset between blocks causes cross-block state pollution (EIP-7702 gas mismatch on Hoodi block 443055) #6467
• Related: PR fix(l1): adding block by block exec as a fallback #6464 (temporary workaround, not merged)
• Failing block: 443,055 (Hoodi/Holesky network)

[github-actions]

🤖 Kimi Code Review

General Assessment: The PR introduces EIP-8037 (2D gas) compliance fixes, resolves a critical VM cache metadata leak (issue #6467), and adds operational tooling. The consensus-critical gas accounting changes are structurally correct, but there are minor clarity issues in error messaging and comments.

Critical Issues

None found. The EIP-8037 implementation correctly moves from per-transaction to block-level gas validation using max(regular, state), and the VM cache normalization correctly resets transient account metadata between blocks.

Code Quality & Correctness

1. Misleading Error Message in Payload Building

File: crates/blockchain/payload.rs
Lines: 844-849

The error message says "state gas overflow" but the check validates max(regular, state). This is misleading when regular gas causes the rejection.

// Current:
"block gas limit exceeded (state gas overflow): \
 max({new_regular}, {new_state}) = {} > gas_limit {}"

// Suggested:
"block gas limit exceeded (2D gas overflow): \
 max(regular={new_regular}, state={new_state}) = {} > limit {}"

2. Outdated Comment in Parallel Execution Path

File: crates/vm/backends/levm/mod.rs
Lines: 983-984

The comment claims "Per-tx check: only regular gas is checked per-tx", but the code was removed entirely for Amsterdam. Update to reflect that per-tx checks are skipped entirely for EIP-8037.

3. Inconsistent Async/Sync Store Access Patterns

File: tooling/contract_metrics/src/main.rs
Lines: 120, 125, 129

The tool mixes synchronous (store.get_block_header) and asynchronous (`store.get_block_body

---

Automated review by Kimi (Moonshot AI) · kimi-k2.5 · custom prompt

[github-actions]

🤖 Claude Code Review

Now I have a complete picture of the diff. Let me write up the review.

---

PR #6494 Review: Fix issue #6467 — Normalize VM cache between blocks in batch execution

Overview

This PR addresses VM cache pollution between blocks in batch execution, where transient account metadata (exists, status) leaked across block boundaries causing incorrect EIP-7702 gas refund calculations. The PR also reshapes the Amsterdam 2D gas (EIP-8037) validation logic and bundles several unrelated files.

---

Core Fix: normalize_cache_for_next_block() (gen_db.rs:403)

The logic is mostly sound, but there are important correctness concerns.

initial_accounts_state contamination risk

if !self.initial_accounts_state.contains_key(address) {
    self.initial_accounts_state.insert(*address, account.clone());
}

Accounts created during block N of the batch are added to initial_accounts_state with their post-block-N state as the reference point. If get_state_transitions later uses initial_accounts_state as the pre-batch baseline to compute trie deltas, these accounts' creation delta (from nothing → post-block-N state) would be silently dropped from the final state update. This could produce incorrect trie roots for the batch.

This needs a clear explanation of what initial_accounts_state represents semantically: is it "pre-batch baseline" (for trie delta computation) or "initial snapshot for storage access" (for gas calculations)? If the former, inserting mid-batch state is incorrect.

Selfdestructed accounts not evicted

When an account is selfdestructed in block N, normalize_cache_for_next_block resets its status to Unmodified but leaves it in current_accounts_state. The exists recalculation relies on has_storage:

account.exists = !account.info.is_empty() || account.has_storage;

If has_storage is still true after a selfdestruct (storage is logically deleted but the flag not cleared), the account will be considered existent in block N+1. The existing tests (batch_single_block_selfdestruct, batch_selfdestruct_created_account_no_spurious_state) cover some cases, but a cross-block selfdestruct scenario (destroyed in block N, accessed in block N+1) would be a good regression test to add.

Missing: cache eviction for destroyed accounts

A cleaner approach would be to drain accounts with status == Destroyed from current_accounts_state entirely (after the normalization loop or inside it), which would remove the has_storage ambiguity.

---

Amsterdam Gas Model Changes (backends/levm/mod.rs)

Complete removal of per-tx Amsterdam gas guard (mod.rs:259)

The original code applied TX_MAX_GAS_LIMIT_AMSTERDAM as a per-tx gas cap:

tx.gas_limit().min(TX_MAX_GAS_LIMIT_AMSTERDAM)

This is now removed entirely. The import of TX_MAX_GAS_LIMIT_AMSTERDAM is also dropped. If this cap is mandated by EIP-7825 for Amsterdam blocks, removing it changes consensus-critical validation. The PR description doesn't explain this removal. Either:

• The cap is no longer needed (needs a spec reference), or
• This is a regression that would accept txs with excessively high gas limits on Amsterdam.

This needs explicit justification with an EIP reference.

Post-execution overflow check after all txs are executed (mod.rs:270–276, mod.rs:314–320)

if is_amsterdam && block_gas_used > block.header.gas_limit {
    return Err(EvmError::Transaction(...));
}

For block production, discovering gas overflow after executing all transactions wastes computation. For block validation this is acceptable since you must re-execute anyway. The comment says this is intentional ("overflow can only be detected after execution"), but it's worth confirming whether block producers use a separate code path that catches this earlier (e.g., via payload.rs).

Duplicate block-level overflow check (mod.rs:270–276 and mod.rs:314–320)

The same check appears verbatim in both the sequential and batch execution paths. This should be extracted to a shared helper to avoid divergence.

Inconsistent EIP references in comments

Comments in mod.rs reference EIP-7778 while the PR description and other comments reference EIP-8037 for the 2D gas model. These need to be reconciled. Double-check which EIP number is correct for the block gas constraint.

---

payload.rs Gas Overflow Check

if context.is_amsterdam && new_regular.max(new_state) > context.payload.header.gas_limit {
    return Err(EvmError::Custom(...).into());
}

This is a correct per-tx guard for block production: reject the tx before committing totals. The remaining_gas update to gas_limit - max(regular, state) is also correct for heuristic pre-tx checks. This section looks good.

---

Accidentally Committed Operational File

mainnet_restore_commands.txt (diff line 425–466) must not be merged. It contains:

• A specific server address (admin@ethrex-mainnet-2)
• Internal JWT secret paths (/home/admin/secrets/jwt.hex)
• tmux session names and operational runbook details

This is clearly unintentionally staged. It leaks operational infrastructure details into the public repo history.

---

Unrelated Files Mixed Into the PR

The following files are unrelated to the bug fix and should be separate PRs:

• DB_OPTIMIZATION_PRIORITIES.md — planning document
• tooling/contract_metrics/ — new contract metrics CLI
• tooling/ipv6-test/ — IPv6 P2P testing infrastructure

Mixing these in makes the diff harder to review and inflates the changeset by ~750 lines on top of the ~70-line core fix.

---

Draft Status and Missing Verification

The PR itself lists open items:

• Block 443,055 hasn't been run to confirm the fix passes
• No benchmark comparison yet

This is fine for a draft, but the fix should not be merged until these are confirmed.

---

Summary

Area	Status
Core normalize_cache_for_next_block logic	Mostly correct; initial_accounts_state semantics and destroyed-account handling need clarification
Amsterdam per-tx gas cap removal	Needs EIP justification — possible consensus regression
Post-execution gas overflow check	Correct for validation; duplicated code should be extracted
payload.rs 2D gas accounting	Looks correct
mainnet_restore_commands.txt	Must be removed before merge
Unrelated tooling files	Should be separate PRs
Test coverage for cross-block selfdestruct	Missing

---

Automated review by Claude (Anthropic) · sonnet · custom prompt

[github-actions]

🤖 Codex Code Review

1. High: the batch-execution fix drops state changes from earlier blocks in the batch. blockchain.rs now calls normalize_cache_for_next_block() between blocks, and gen_db.rs resets every cached account’s status to Unmodified. But get_state_transitions() only emits updates for accounts whose status is still modified, and add_blocks_in_batch() only calls it once at the very end (blockchain.rs). Result: any account changed in block N and untouched afterwards is silently omitted from the final batch state diff/state root. This is consensus-critical.

2. High: the new Amsterdam payload-builder overflow check rejects the tx after execution, but the rejection path does not roll back VM state or cumulative_gas_spent. apply_plain_transaction() calls context.vm.execute_tx(...) first; that already mutates the DB and increments cumulative receipt gas in backends/mod.rs. If the later overflow check at payload.rs fails, the caller just logs and skips the tx (payload.rs) without undo_last_tx(). That leaves subsequent payload construction observing state from a transaction that was not included, and also leaves cumulative_gas_spent inflated for later receipts.

I did not run tests.

---

Automated review by OpenAI Codex · gpt-5.4 · custom prompt

[github-actions]

Lines of code report

Total lines added: 26
Total lines removed: 6
Total lines changed: 32

Detailed view

+----------------------------------------+-------+------+
| File                                   | Lines | Diff |
+----------------------------------------+-------+------+
| ethrex/crates/blockchain/blockchain.rs | 2492  | +3   |
+----------------------------------------+-------+------+
| ethrex/crates/blockchain/payload.rs    | 811   | +11  |
+----------------------------------------+-------+------+
| ethrex/crates/vm/backends/levm/mod.rs  | 2029  | -6   |
+----------------------------------------+-------+------+
| ethrex/crates/vm/backends/mod.rs       | 242   | +3   |
+----------------------------------------+-------+------+
| ethrex/crates/vm/levm/src/db/gen_db.rs | 563   | +9   |
+----------------------------------------+-------+------+

[github-actions]

Benchmark Results Comparison

No significant difference was registered for any benchmark run.

Detailed Results

Benchmark Results: BubbleSort

Command	Mean [s]	Min [s]	Max [s]	Relative
main_revm_BubbleSort	3.041 ± 0.039	2.978	3.109	1.14 ± 0.02
main_levm_BubbleSort	2.723 ± 0.121	2.667	3.064	1.02 ± 0.05
pr_revm_BubbleSort	3.028 ± 0.051	2.973	3.136	1.13 ± 0.02
pr_levm_BubbleSort	2.674 ± 0.025	2.654	2.733	1.00

Benchmark Results: ERC20Approval

Command	Mean [ms]	Min [ms]	Max [ms]	Relative
main_revm_ERC20Approval	980.9 ± 7.6	967.9	993.8	1.00 ± 0.01
main_levm_ERC20Approval	1021.2 ± 9.6	1006.8	1039.0	1.04 ± 0.01
pr_revm_ERC20Approval	979.6 ± 10.2	966.1	997.8	1.00
pr_levm_ERC20Approval	1011.3 ± 4.8	1005.3	1019.9	1.03 ± 0.01

Benchmark Results: ERC20Mint

Command	Mean [ms]	Min [ms]	Max [ms]	Relative
main_revm_ERC20Mint	133.6 ± 0.6	132.6	134.4	1.00
main_levm_ERC20Mint	146.1 ± 0.9	145.0	147.7	1.09 ± 0.01
pr_revm_ERC20Mint	134.6 ± 1.0	133.0	137.0	1.01 ± 0.01
pr_levm_ERC20Mint	148.4 ± 2.5	146.8	155.1	1.11 ± 0.02

Benchmark Results: ERC20Transfer

Command	Mean [ms]	Min [ms]	Max [ms]	Relative
main_revm_ERC20Transfer	231.6 ± 1.3	230.6	235.0	1.00
main_levm_ERC20Transfer	247.6 ± 1.8	245.5	251.1	1.07 ± 0.01
pr_revm_ERC20Transfer	236.2 ± 10.2	230.7	265.0	1.02 ± 0.04
pr_levm_ERC20Transfer	249.5 ± 2.1	247.2	254.0	1.08 ± 0.01

Benchmark Results: Factorial

Command	Mean [ms]	Min [ms]	Max [ms]	Relative
main_revm_Factorial	237.5 ± 42.3	222.8	357.9	1.05 ± 0.20
main_levm_Factorial	249.6 ± 1.5	247.5	252.2	1.10 ± 0.06
pr_revm_Factorial	227.0 ± 13.0	220.9	263.8	1.00
pr_levm_Factorial	250.8 ± 1.8	248.8	255.0	1.11 ± 0.06

Benchmark Results: FactorialRecursive

Command	Mean [s]	Min [s]	Max [s]	Relative
main_revm_FactorialRecursive	1.614 ± 0.026	1.584	1.663	1.04 ± 0.02
main_levm_FactorialRecursive	1.552 ± 0.022	1.530	1.601	1.00
pr_revm_FactorialRecursive	1.609 ± 0.030	1.556	1.642	1.04 ± 0.02
pr_levm_FactorialRecursive	1.596 ± 0.030	1.543	1.625	1.03 ± 0.02

Benchmark Results: Fibonacci

Command	Mean [ms]	Min [ms]	Max [ms]	Relative
main_revm_Fibonacci	202.7 ± 3.2	199.8	211.5	1.00
main_levm_Fibonacci	232.1 ± 8.7	222.6	250.7	1.14 ± 0.05
pr_revm_Fibonacci	202.8 ± 1.8	200.6	206.5	1.00 ± 0.02
pr_levm_Fibonacci	225.9 ± 4.3	222.1	236.2	1.11 ± 0.03

Benchmark Results: FibonacciRecursive

Command	Mean [ms]	Min [ms]	Max [ms]	Relative
main_revm_FibonacciRecursive	847.6 ± 8.5	836.1	860.7	1.23 ± 0.02
main_levm_FibonacciRecursive	689.5 ± 7.9	679.9	705.9	1.00
pr_revm_FibonacciRecursive	841.2 ± 7.4	826.7	852.7	1.22 ± 0.02
pr_levm_FibonacciRecursive	694.8 ± 6.6	687.4	704.6	1.01 ± 0.02

Benchmark Results: ManyHashes

Command	Mean [ms]	Min [ms]	Max [ms]	Relative
main_revm_ManyHashes	8.4 ± 0.0	8.3	8.4	1.01 ± 0.01
main_levm_ManyHashes	9.8 ± 0.2	9.6	10.3	1.18 ± 0.02
pr_revm_ManyHashes	8.3 ± 0.0	8.2	8.4	1.00
pr_levm_ManyHashes	9.7 ± 0.2	9.6	10.4	1.17 ± 0.03

Benchmark Results: MstoreBench

Command	Mean [ms]	Min [ms]	Max [ms]	Relative
main_revm_MstoreBench	260.0 ± 2.4	257.5	264.0	1.13 ± 0.01
main_levm_MstoreBench	230.8 ± 0.7	230.0	232.2	1.00
pr_revm_MstoreBench	264.7 ± 7.2	257.7	274.3	1.15 ± 0.03
pr_levm_MstoreBench	231.5 ± 0.7	230.5	232.9	1.00 ± 0.00

Benchmark Results: Push

Command	Mean [ms]	Min [ms]	Max [ms]	Relative
main_revm_Push	287.8 ± 2.5	285.9	294.3	1.04 ± 0.01
main_levm_Push	277.6 ± 1.3	276.0	280.0	1.00
pr_revm_Push	287.1 ± 2.3	285.2	292.5	1.03 ± 0.01
pr_levm_Push	278.4 ± 3.0	276.6	284.2	1.00 ± 0.01

Benchmark Results: SstoreBench_no_opt

Command	Mean [ms]	Min [ms]	Max [ms]	Relative
main_revm_SstoreBench_no_opt	169.9 ± 1.5	167.8	171.7	1.70 ± 0.02
main_levm_SstoreBench_no_opt	101.6 ± 4.1	99.7	113.0	1.01 ± 0.04
pr_revm_SstoreBench_no_opt	173.6 ± 7.4	168.1	190.6	1.73 ± 0.08
pr_levm_SstoreBench_no_opt	100.1 ± 0.9	99.5	102.5	1.00