fix(restore): force joiners of a restored cluster through checkpoint sync

docs/ops/backup-restore.md

@@ -122,7 +122,7 @@ The temporary checkpoint is removed from the leader's filesystem after the backu
 
 Backup preparation is performed on the **restore side** (during `FinalizeRestore` or `store bootstrap`), not during backup. It resets cluster-local and checkpoint-era zones and leaves the attribute zone **byte-for-byte intact** — there is no attribute compaction, because each canonical key holds exactly one Pebble entry (no per-index history to fold):
 
-1. **Reset lastAppliedIndex**: The Raft applied index is reset to 0, so the restored cluster starts fresh.
+1. **Pin lastAppliedIndex to 1**: The restored store is the FSM state the new Raft log builds on; the restored bootstrap plants its WAL snapshot at this index, so the new log starts at 2 and any fresh peer is routed through the snapshot → checkpoint-sync path (plain log replay from index 1 would land on an empty store and miss the restored state). Source-cluster Raft indices are discarded either way.
 2. **Remove persisted config**: Node and cluster IDs are stripped for portability.
 3. **Wipe the cluster-transient zone**: In-flight-only tracking (e.g. running backup jobs) has no meaning on the restored cluster.
 4. **Drop persisted bloom blocks**: Stale bloom blocks are cleared so the booting node rebuilds the bloom from a full attribute scan using its own config.
@@ -142,7 +142,7 @@ The backup is a complete Pebble database that contains:
 | Per-Ledger | `0x03` | Per-ledger data |
 | Cold | `0x04` | Transaction logs (`{0x04, 0x01}`), audit entries (`{0x04, 0x02}`) |
 | Idempotency | `0x05` | Idempotency keys |
-| Global | `0x06` | Last applied index (reset to 0), last applied timestamp, signing keys, signing config, chapters, sink configs, sink cursors, sink statuses |
+| Global | `0x06` | Last applied index (pinned to 1 on restore), last applied timestamp, signing keys, signing config, chapters, sink configs, sink cursors, sink statuses |
 
 > **Note**: If chapters have been archived before the backup, the archived logs and audit entries are no longer in the backup (they have been purged to cold storage). Attributes remain.
 
@@ -381,11 +381,12 @@ ledgerctl restore finalize --yes
 
 Calls `RestoreService.FinalizeRestore` (unary). This commits the staged backup as live data:
 
-1. Opens the staging directory read-only to extract `lastAppliedIndex` and `lastAppliedTimestamp`.
-2. Writes the **RESTORED marker** JSON file to `{dataDir}/RESTORED`.
-3. Creates `{dataDir}/checkpoints/` directory.
-4. **Atomically** hard-links the staging directory to `{dataDir}/checkpoints/0` (using a temp directory + `os.Rename` for crash safety).
-5. Removes the staging directory.
+1. Prepares the staged store ([Backup Preparation on Restore](#backup-preparation-on-restore): applied index pinned to 1, persisted config/bloom/peers/cache reset).
+2. Reads back `lastAppliedIndex` and `lastAppliedTimestamp` from the prepared store.
+3. Writes the **RESTORED marker** JSON file to `{dataDir}/RESTORED`.
+4. Creates `{dataDir}/checkpoints/` directory.
+5. **Atomically** hard-links the staging directory to `{dataDir}/checkpoints/0` (using a temp directory + `os.Rename` for crash safety).
+6. Removes the staging directory.
 
 On next startup, `ScanLatestCheckpointID()` scans the `checkpoints/` directory for the highest numbered subdirectory to find the active checkpoint.
 
@@ -464,7 +465,7 @@ ledgerctl store bootstrap --driver s3 --s3-bucket my-bucket --s3-region us-east-
 4. **Preview**: Opens the staging as a read-only Pebble database, reads metadata (last applied index, timestamp, ledger list), and displays a summary table.
 5. **Validate** (optional): If `--validate` is set, runs the full integrity checker (`check.Checker`) -- the same checker used by `store check` and `restore validate`.
 6. **Confirm**: Unless `--yes` is set, prompts for user confirmation.
-7. **Prepare**: Prepares attributes for backup (Global-zone resets: applied index → 0, persisted config stripped, persisted bloom blocks dropped); the attribute zone is left intact.
+7. **Prepare**: Prepares attributes for backup (Global-zone resets: applied index pinned to 1 — the restored genesis' WAL-snapshot index — persisted config stripped, persisted bloom blocks dropped); the attribute zone is left intact.
 8. **Finalize**: Hard-links staging to `{data-dir}/checkpoints/0`, writes the `RESTORED` marker JSON.
 9. **Cleanup**: Removes the staging directory.
 
@@ -510,7 +511,7 @@ On startup, the node detects the `RESTORED` marker in `NewNode()`:
      - `nextSequenceID` from the last log sequence
      - `lastAuditHash` and `nextAuditSequenceID` from the last audit entry
    - Creates an FSM snapshot (`fsm.CreateSnapshot()`)
-   - Creates a WAL snapshot at `marker.LastAppliedIndex` with `ConfState{Voters: [nodeID]}` (single-node bootstrap)
+   - Creates a WAL snapshot at `marker.LastAppliedIndex` with `ConfState{Voters: [nodeID]}` (single-node bootstrap); a marker carrying 0 is refused — the snapshot must sit above the log start so joiners are forced through checkpoint sync
    - Removes the RESTORED marker
    - Continues with normal Raft startup
 
@@ -524,12 +525,12 @@ The `RESTORED` file is a JSON file written to the data directory during `Finaliz
 
 ```json
 {
-  "lastAppliedIndex": 12345,
+  "lastAppliedIndex": 1,
   "lastAppliedTimestamp": 1700000000000000
 }
 ```
 
-- `lastAppliedIndex`: The Raft index of the last applied entry in the backup (reset to 0 during backup preparation, so this is always 0 in practice).
+- `lastAppliedIndex`: The Raft index the restored state occupies in the new log (pinned to 1 during backup preparation, so this is always 1 in practice); the restored bootstrap creates its WAL snapshot at this index.
 - `lastAppliedTimestamp`: The HLC timestamp (microseconds) of the last applied entry.
 
 **File**: `internal/infra/node/restored_marker.go`
@@ -542,7 +543,7 @@ The `RESTORED` file is a JSON file written to the data directory during `Finaliz
 |-----------|-----------|
 | **Consistent snapshot** | Backup checkpoint is created as a direct Pebble checkpoint. Boundaries are always up-to-date in Pebble (written on every commit), so the checkpoint is consistent without Raft consensus or FSM gating. |
 | **Incremental efficiency** | SST files are immutable — same name means same content. Only new/changed files are uploaded; stale files are deleted. |
-| **Self-contained on restore** | During restore finalize, the applied index, persisted config and bloom blocks are reset (Global-zone); the attribute zone is preserved byte-for-byte. No dependency on the original cluster's Raft indices. |
+| **Self-contained on restore** | During restore finalize, the applied index is pinned to 1 and persisted config and bloom blocks are reset (Global-zone); the attribute zone is preserved byte-for-byte. No dependency on the original cluster's Raft indices. |
 | **Data integrity (content)** | `ValidateRestore` runs the full integrity checker: log sequence continuity, volume balance verification, metadata consistency. |
 | **Fresh directory required** | Restore mode refuses to start if existing checkpoints are found in `checkpoints/`, preventing accidental overwrites. |
 | **Atomic finalize** | Checkpoint placement uses `HardLink()` (temp directory + atomic `os.Rename`) for crash safety. |

docs/ops/cli.md

@@ -2147,7 +2147,7 @@ ledgerctl store bootstrap --driver s3 --s3-bucket <bucket> --data-dir /path/to/d
 4. Opens the staging as a read-only Pebble database and displays a preview (ledger count, timestamps)
 5. If `--validate` is set, runs the full integrity checker (same as `store check`); aborts before finalizing if any integrity error is found
 6. Prompts for confirmation (unless `--yes`)
-7. Prepares attributes for backup (Global-zone resets): resets applied index to 0, strips persisted config, drops persisted bloom blocks; the attribute zone is left intact
+7. Prepares attributes for backup (Global-zone resets): pins the applied index to 1 (the WAL-snapshot index the restored genesis occupies, so joiners must sync a checkpoint), strips persisted config, drops persisted bloom blocks; the attribute zone is left intact
 8. Hard-links staging to `checkpoints/0`, writes `RESTORED` marker
 9. Cleans up the staging directory
 

docs/technical/architecture/subsystems/fsm/deterministic-fsm.md

@@ -460,7 +460,7 @@ FSM flushes at apply time (batch, no sync):
 
 ### 12.3 Backup preparation (backup only)
 
-There is exactly one entry per canonical key, so there is no attribute compaction. Backup preparation (`PrepareForBackup`) leaves the attribute zone byte-for-byte intact and only resets cluster-local and checkpoint-era state: applied index → 0, persisted config deleted, cluster-transient zone wiped, persisted bloom blocks and Raft peers dropped, and the cache zone cleared (checkpoint-era cache rows predate the replayed delta and would otherwise serve stale CacheHits on the restored node).
+There is exactly one entry per canonical key, so there is no attribute compaction. Backup preparation (`PrepareForBackup`) leaves the attribute zone byte-for-byte intact and only resets cluster-local and checkpoint-era state: applied index pinned to 1 (the restored genesis’ WAL-snapshot index — the new log starts at 2, forcing joiners through checkpoint sync), persisted config deleted, cluster-transient zone wiped, persisted bloom blocks and Raft peers dropped, and the cache zone cleared (checkpoint-era cache rows predate the replayed delta and would otherwise serve stale CacheHits on the restored node).
 
 ### 12.4 Pebble key layout
 

internal/adapter/grpc/server_restore.go

@@ -325,7 +325,9 @@ func (s *RestoreServiceServerImpl) FinalizeRestore(_ context.Context, _ *restore
 
 	stagingDir := s.stagingDir()
 
-	// Reset Global-zone keys so the staged backup is restartable on a fresh cluster.
+	// Prepare Global-zone keys (applied index pinned to the restore-genesis
+	// index, cluster-local state reset) so the staged backup is restartable
+	// on a fresh cluster.
 	s.logger.Infof("Preparing backup for restore compatibility")
 
 	if err := attributes.PrepareForBackup(store); err != nil {

internal/infra/attributes/prepare.go

@@ -1,6 +1,7 @@
 package attributes
 
 import (
+	"encoding/binary"
 	"fmt"
 
 	"github.com/cockroachdb/pebble/v2"
@@ -18,8 +19,8 @@ import (
 // versions to fold. The attribute zone is left byte-for-byte intact.
 //
 // The six resets are:
-//  1. lastAppliedIndex -> 0, so the restored cluster starts fresh without
-//     raft-index conflicts.
+//  1. lastAppliedIndex -> 1, the index the restored FSM genesis occupies in
+//     the new raft log (see below), fresh of any source-cluster raft index.
 //  2. persisted config (nodeId, clusterId) deleted, so the backup is portable
 //     to any cluster.
 //  3. ZoneClusterTransient wiped — in-flight-only tracking (backup jobs) has
@@ -38,11 +39,20 @@ import (
 func PrepareForBackup(s *dal.Store) error {
 	batch := s.OpenWriteSession()
 
-	// Reset lastAppliedIndex to 0 so the restored cluster starts fresh.
-	if err := batch.SetBytes([]byte{dal.ZoneGlobal, dal.SubGlobLastAppliedIndex}, make([]byte, 8)); err != nil {
+	// Pin lastAppliedIndex to 1: the restored store is the FSM state the new
+	// raft log builds on, and the RESTORED bootstrap plants its WAL snapshot
+	// at this index, so the log starts at 2 and raft must route any fresh
+	// peer through the snapshot → checkpoint-sync path. At 0 the snapshot is
+	// empty and the log claims completeness from index 1 — a learner joining
+	// before the first post-restore raft snapshot is then "caught up" by
+	// plain log replay onto an empty store, missing every restored row.
+	appliedIndex := make([]byte, 8)
+	binary.BigEndian.PutUint64(appliedIndex, 1)
+
+	if err := batch.SetBytes([]byte{dal.ZoneGlobal, dal.SubGlobLastAppliedIndex}, appliedIndex); err != nil {
 		_ = batch.Cancel()
 
-		return fmt.Errorf("resetting applied index: %w", err)
+		return fmt.Errorf("pinning applied index: %w", err)
 	}
 
 	// Remove persisted config (nodeId, clusterId) so the backup is portable to any cluster.

internal/infra/attributes/prepare_test.go

@@ -158,7 +158,7 @@ func TestPrepareForBackupPreservesAttributesByteForByte(t *testing.T) {
 }
 
 // TestPrepareForBackupResetsGlobalZone asserts the three Global-zone resets:
-// applied index -> 0, persisted config deleted, persisted bloom blocks dropped.
+// applied index -> 1, persisted config deleted, persisted bloom blocks dropped.
 func TestPrepareForBackupResetsGlobalZone(t *testing.T) {
 	t.Parallel()
 
@@ -186,7 +186,8 @@ func TestPrepareForBackupResetsGlobalZone(t *testing.T) {
 
 	idx, err := readLastAppliedIndex(s)
 	require.NoError(t, err)
-	require.Equal(t, uint64(0), idx, "applied index must be reset to 0")
+	require.Equal(t, uint64(1), idx,
+		"applied index must be pinned to 1 — the WAL-snapshot index the restored FSM genesis occupies")
 
 	_, _, err = s.Get([]byte{dal.ZoneGlobal, dal.SubGlobPersistedConfig})
 	require.ErrorIs(t, err, pebble.ErrNotFound, "persisted config must be deleted")
@@ -237,7 +238,7 @@ func TestPrepareForBackupClearsCacheZone(t *testing.T) {
 
 // TestPrepareForBackupRestorableOnFreshCluster runs the full backup->restore
 // pipeline (write -> prepare -> tar -> extract -> reopen) and asserts the
-// attribute values survive and the applied index is 0 on the fresh cluster.
+// attribute values survive and the applied index is pinned to 1 on the fresh cluster.
 func TestPrepareForBackupRestorableOnFreshCluster(t *testing.T) {
 	t.Parallel()
 
@@ -317,7 +318,8 @@ func TestPrepareForBackupRestorableOnFreshCluster(t *testing.T) {
 
 		idx, err := readLastAppliedIndex(s)
 		require.NoError(t, err)
-		require.Equal(t, uint64(0), idx, "applied index must be 0 on the fresh cluster")
+		require.Equal(t, uint64(1), idx,
+			"applied index must be 1 on the fresh cluster — the restored genesis' WAL-snapshot index")
 	}()
 }
 

internal/infra/node/node.go

@@ -284,10 +284,14 @@ func NewNode(
 		}
 
 		if marker != nil {
-			// Restore mode: bootstrap from restored data.
-			// The backup was compacted: all attribute indices are 0 and lastAppliedIndex is 0.
-			// We need to recover the FSM counters (nextLedgerID, nextSequenceID, etc.)
-			// from the Pebble data before creating the WAL snapshot.
+			// Restore mode: bootstrap from restored data. The restored store
+			// carries lastAppliedIndex 1 (PrepareForBackup pins it), so the WAL
+			// snapshot below lands at 1 and the new log starts at 2: raft then
+			// has to route any fresh peer through the snapshot → checkpoint-sync
+			// path instead of "catching it up" by replaying the log onto an
+			// empty store, which would miss the entire restored FSM genesis.
+			// FSM counters (nextLedgerID, nextSequenceID, etc.) are recovered
+			// from Pebble before creating the WAL snapshot.
 			logger.WithFields(map[string]any{
 				"lastAppliedIndex":     marker.LastAppliedIndex,
 				"lastAppliedTimestamp": marker.LastAppliedTimestamp,
@@ -297,6 +301,16 @@ func NewNode(
 				return nil, fmt.Errorf("recovering FSM state from store: %w", err)
 			}
 
+			// A snapshot at 0 is no snapshot: the new log would claim
+			// completeness from index 1 and a fresh learner could be caught
+			// up by plain log replay onto an empty store, missing the whole
+			// restored genesis. PrepareForBackup pins the index to >= 1; a 0
+			// here means a marker written by another tool or by hand.
+			if marker.LastAppliedIndex == 0 {
+				return nil, fmt.Errorf(
+					"invariant: RESTORED marker carries lastAppliedIndex 0; the restored genesis must occupy index >= 1 so joiners are forced through checkpoint sync — re-run restore finalize / store bootstrap to regenerate the marker")
+			}
+
 			initialConfState = &raftpb.ConfState{
 				Voters: []uint64{cfg.NodeID},
 			}

internal/infra/node/restored_marker.go

@@ -12,6 +12,10 @@ const restoredMarkerFile = "RESTORED"
 
 // RestoredMarker contains the metadata written by FinalizeRestore.
 type RestoredMarker struct {
+	// LastAppliedIndex is the raft index the restored genesis occupies in the
+	// new log — always >= 1 (PrepareForBackup pins it): bootstrap plants the
+	// WAL snapshot at this index so joiners must sync a checkpoint, and
+	// refuses a marker carrying 0.
 	LastAppliedIndex     uint64 `json:"lastAppliedIndex"`
 	LastAppliedTimestamp uint64 `json:"lastAppliedTimestamp"`
 }