Replay issue

[QuantumExplorer]

Cannot sync past mainnet block 24175: app hash mismatch from an ungated protocol-version-3 behavior change

Summary

An evonode cannot sync Platform past mainnet (evo1) block 24175. Tenderdash aborts on replay with an app hash mismatch and is restarted by Docker every ~10s indefinitely.

Block 24175 is a protocol-version-3-era block (~Nov 2024) being executed by protocol-version-13 code. Every Drive version that can read the state produces the same "wrong" hash, and every v1.x Drive refuses to open the state at all — with an error showing the persisted state is tagged version 3 while containing bytes no v3-era Drive accepts.

This looks like a behavior change that was made without gating it on protocol version, so historical blocks can no longer be replayed under their historical rules.

Error

ERROR: failed to start node: error on replay: wrong Block.Header.AppHash at state height 24175, block 24175.
  Expected AAEAA0E96117BC0310C2D8F82D5EDA0959333BC4F71751B11BAB0A60E2F2D68C
  got      402A8119ACA6E6CA1DEBE635848FD1CD3ADD99CB7AEF7705A2798D317418AB58

Expected is the value in the committed block header (network consensus). got is what this node computes.

Handshake and replay state on every attempt:

ABCI Handshake App Info   hash C19201BB13D377DF1208C1BF7BD16EF8E9526835407DE4901ADD3C4D0709B311
                          height 24174  protocol-version 13  software-version 4.1.1

ABCI Replay Blocks        appHeight 24174   stateHeight 24174   storeHeight 24175
                          stateCoreChainLockedHeight 2157965
                          storeCoreChainLockedHeight 2157967

Environment

Core is fully synced and healthy throughout; masternode state READY, PoSe penalty 0.

The block

Block 24175 is empty, on round 1:

Processed proposal with 0 transactions for height: 24175, round: 1
invalid_tx_count: 0   valid_tx_count: 0   storage_fees: 0   processing_fees: 0

Chain app version in that era:

"Version":{"consensus":{"block":"14","app":"3"},"software":"1.7.0"},"ChainID":"evo1","InitialHeight":1

So the chain was at app/protocol version 3, while the Drive replaying it reports protocol-version 13.

The block's only distinguishing action is advancing the core chain-locked height 2157965 → 2157967. With zero transactions, zero fees and no state transitions, the candidate surface for the divergence is limited to block-level work: chain-lock advance, masternode list / quorum rotation, epoch handling, withdrawal processing.

This node happens to be the deterministic proposer for (24175, round 1):

height 24175  round 1  node_proTxHash EE22F5  proposer_proTxHash EE22F5
"propose step; our turn to propose"

Reproduction

Reproduced from a completely clean slate:

1. dashmate reset --platform at 2026-08-18T03:31:47Z — both platform volumes recreated empty.
2. Block-synced genesis → 24174 in ~2h45m (Drive RocksDB writes from 03:31 to 06:16).
3. Failed at 24175 with byte-identical Expected/got hashes.

The same failure had already occurred on an earlier independent from-genesis sync (Jul 4 – Aug 8 2026). Two independent syncs, same two hashes.

Restarting does not help

There are reports of operators getting past this block by restarting. That did not reproduce here across ~36,900 attempts:

• Docker's restart loop only cycles Tenderdash. drive_abci had RestartCount=0 across 36,894 Tenderdash restarts, so Drive's in-memory state was never cleared.
• block execution context already existed, but we are running it again for same height 24175/round 1 is logged once per Tenderdash restart (36,894×). This is cleared by restarting Drive, but does not change the result.
• A full dashmate restart --platform (bouncing drive_abci) was tested. The first ProcessProposal after the clean start shows no context warning — and still produced 402A….

Drive version bisect

Same on-disk state at height 24174, only the Drive image swapped:

Image	drive-abci	Image date	Result
794702d0831e	(2024-11-04 build)	2024-11-04	panic — cannot open platform
dashpay/drive:1.5.0-rc.1	1.5.0-rc.1	2024-10-30	panic — cannot open platform
dashpay/drive:1	1.8.0	2025-01-16	panic — cannot open platform
dashpay/drive:2	2.1.3	2025-10-29	402A8119…
dashpay/drive:3	3.0.1	2026-02-06	402A8119…
dashpay/drive:4	4.1.1	2026-08-18	402A8119…

The v1.x panic is the key evidence

All three v1.x builds fail identically:

ERROR drive_abci::execution::storage::fetch_platform_state::v0:
  Unable deserialize platform state for version 3,
  bytes: "01000100fd00000192a6b39b2cfb5e6efc0020ed8d05c1920…"

ERROR drive_abci: panic, panicked at packages/rs-drive-abci/src/main.rs:144:18:
  Failed to open platform: Protocol(PlatformDeserializationError(
    "unable to deserialize PlatformStateForSaving:
     UnexpectedVariant { type_name: \"SignatureVerificationQuorumSetForSaving\", allowed: Range… }"))
  at packages/rs-drive-abci/src/main.rs:444

The persisted PlatformStateForSaving is tagged version 3, but contains a SignatureVerificationQuorumSetForSaving variant outside the range any v3-era Drive accepts. Modern code writes a different shape of data under the same protocol version number — the version is recorded but not honoured.

Notably, the type that breaks compatibility is the signature-verification quorum set, and the one thing block 24175 does is advance the core chain-locked height, i.e. exactly when quorum-set state rotates. The incompatible struct and the divergent block point at the same subsystem.

What is and isn't established

Established:

• 2.1.3, 3.0.1 and 4.1.1 all produce 402A…. The divergence exists no later than 2.1.3 (Oct 2025) and has been carried forward through three major versions.
• The persisted v3-tagged platform state is not readable by v3-era Drive.
• The failure is fully reproducible from an empty data directory and is not caused by corruption, unclean shutdown, host issues, or stale in-memory state.

Not established:

• The v1.x results are inconclusive, not refutations — those binaries never got far enough to execute the block. Testing them properly needs state written by v1.x, i.e. a from-genesis sync on the old binary.
• 2.1.3 and 3.0.1 read state written by 4.1.1, so strictly this shows agreement among 2.x+ rather than that 2.1.3 would produce 402A… from its own state.
• The exact commit is not pinned; it is bracketed to 1.8.0 → 2.1.3.
• I have not verified that other operators' reports concern this same block.

Impact

• The node cannot sync past 24175; Platform has never been operational on it.
• Any node syncing evo1 from genesis would be expected to hit the same wall.
• There is no built-in bypass: dashmate 4.1.1 hardcodes enable = false for Tenderdash statesync in its config.toml template, and regenerates that file on every start.

Suggested fix

Gate the changed quorum-set serialisation/handling on protocol version, so blocks from the v3 era replay under v3 rules. Failing that, a supported snapshot / state-sync bootstrap path would let new nodes skip the affected range.

Useful next step

A from-genesis sync on Drive 1.8.0 would produce a genuine v3-era state and either confirm that 1.8.0 reproduces AAEA…D68C or push the bracket earlier. I can run that if it would help pin the commit.