test(model): exercise RebuildDelta via a model-test restore cycle

.claude/skills/antithesis-run/SKILL.md

@@ -98,13 +98,19 @@ docker build --platform linux/amd64 \
   -t "$ANTITHESIS_REPOSITORY/ledger-operator:$TAG" misc/operator
 docker push "$ANTITHESIS_REPOSITORY/ledger-operator:$TAG"
 
-# 4. Build & push workload.
+# 4. Build & push workload (+ the sidecar variant, which ships without the
+#    test tree so the Composer never schedules test commands into it).
 ( cd tests/antithesis/workload && \
   docker build --platform linux/amd64 \
     --build-arg GOARCH=amd64 --build-arg GOOS=linux \
     -f Dockerfile \
     -t "$ANTITHESIS_REPOSITORY/workload-ledger-v3:$TAG" ../../.. && \
-  docker push "$ANTITHESIS_REPOSITORY/workload-ledger-v3:$TAG" )
+  docker push "$ANTITHESIS_REPOSITORY/workload-ledger-v3:$TAG" && \
+  docker build --platform linux/amd64 \
+    --build-arg GOARCH=amd64 --build-arg GOOS=linux \
+    -f Dockerfile --target sidecar \
+    -t "$ANTITHESIS_REPOSITORY/workload-ledger-v3-sidecar:$TAG" ../../.. && \
+  docker push "$ANTITHESIS_REPOSITORY/workload-ledger-v3-sidecar:$TAG" )
 ```
 
 **Compose mode** — same image build/push, plus the etcd retag:

docs/technical/contributing/testing.md

@@ -384,6 +384,26 @@ default. Common tunables (full list in the script header):
 | `MODEL_DUMP_BATCHES` | Log every submitted bulk (`[batch-dump]` lines) for deterministic offline replay through `tests/oracle/cmd/replay`. |
 | `RESTART_INTERVAL` / `DEAD_TIME` | Cluster restart cadence and how long a killed node stays down. |
 | `COMPACTION_MARGIN` | Raft entries between snapshots; low values force snapshot recovery. |
+| `RESTORE_INTERVAL` | Seconds between backup/restore cycles with `--restore`. |
+
+#### Restore cycles (`--restore`, single node)
+
+`run_model_test.sh --restore` periodically quiesces the driver, takes an
+incremental backup, kills the node, bootstraps a fresh store from the backup
+(running the `RebuildDelta` replay), and relaunches on it — the driver then
+resumes and its ordinary checks validate the rebuilt state. The run fails if
+no cycle completed or any cycle failed. The driver-side knobs are
+`MODEL_RESTORE_INTERVAL` and `MODEL_RESTORE_TIMEOUT` (seconds; the timeout is
+the per-cycle lease after which the driver gives up waiting and resumes).
+
+On Antithesis (k8s template) the same driver rendezvous is serviced by the
+`restore-orchestrator` sidecar in the workload pod
+(`tests/antithesis/workload/restore-orchestrator.sh`), which drives the
+operator through a full disaster-recovery pass: quiesce-point backup, Cluster
+teardown (CR + PVCs), restore-mode round-trip (`ledgerctl restore download` /
+`finalize`), and the flip back to a full cluster. The
+`singleton_driver_model: restore cycle completed` Sometimes assertion in the
+report is the proof the path actually ran.
 
 #### Maintaining the model
 

tests/antithesis/k8s/cluster.yaml

@@ -76,3 +76,25 @@ spec:
     value: minioadmin
   - name: AWS_SECRET_ACCESS_KEY
     value: minioadmin
+---
+# The ledger image is referenced only by the Cluster CR's spec.image fields,
+# which the Antithesis environment builder does not scan — it mirrors images
+# from pod templates. This no-op Job carries the image in a pod template so it
+# is baked into the hermetic environment for the operator's pods to use.
+apiVersion: batch/v1
+kind: Job
+metadata:
+  name: ledger-image-preload
+spec:
+  backoffLimit: 0
+  template:
+    spec:
+      restartPolicy: Never
+      containers:
+      - name: preload
+        image: __REGISTRY__/__IMAGE_NAME__:__TAG__
+        command: ["/bin/true"]
+        resources:
+          requests:
+            cpu: 1m
+            memory: 8Mi

tests/antithesis/k8s/workload.yaml

@@ -11,14 +11,26 @@ kind: ClusterRole
 metadata:
   name: workload-ledger-updater
 rules:
+# The restore-orchestrator sidecar deletes and re-creates the Cluster (restore
+# mode round-trip), on top of the drivers' rolling updates.
 - apiGroups: ["ledger.formance.com"]
   resources: ["clusters"]
-  verbs: ["get", "list", "watch", "update", "patch"]
+  verbs: ["get", "list", "watch", "update", "patch", "create", "delete"]
 # Operational drivers (rolling_restart, quorum_recovery) delete pods and
 # observe the StatefulSet they belong to.
 - apiGroups: [""]
   resources: ["pods"]
   verbs: ["get", "list", "watch", "delete"]
+# The restore-orchestrator sidecar execs ledgerctl inside ledger pods
+# (backup at the quiesce point, restore download/finalize).
+- apiGroups: [""]
+  resources: ["pods/exec"]
+  verbs: ["create"]
+# The restore teardown deletes the cluster's volume claims so the re-created
+# cluster starts from the restored store, not stale disks.
+- apiGroups: [""]
+  resources: ["persistentvolumeclaims"]
+  verbs: ["get", "list", "delete"]
 - apiGroups: ["apps"]
   resources: ["statefulsets"]
   verbs: ["get", "list", "watch"]
@@ -93,6 +105,21 @@ spec:
         # MODEL_DUMP_BATCHES env at config-image build.
         - name: MODEL_DUMP_BATCHES
           value: "__MODEL_DUMP_BATCHES__"
+        # Restore-cycle rendezvous with the restore-orchestrator sidecar (only
+        # the model driver writes requests; other templates leave it idle).
+        # The interval and lease are k8s-sized: one cycle is a full cluster
+        # teardown + restore + rejoin, minutes under fault injection.
+        - name: MODEL_RESTORE_REQ
+          value: /var/run/restore/req
+        - name: MODEL_RESTORE_RESP
+          value: /var/run/restore/resp
+        - name: MODEL_RESTORE_INTERVAL
+          value: "180"
+        - name: MODEL_RESTORE_TIMEOUT
+          value: "900"
+        volumeMounts:
+        - name: restore-rendezvous
+          mountPath: /var/run/restore
         # Readiness probe ensures Antithesis waits for the ledger cluster
         # to be reachable before starting the Test Composer.
         readinessProbe:
@@ -106,3 +133,37 @@ spec:
             memory: 128Mi
           limits:
             memory: 512Mi
+      # Services the model driver's restore requests by driving the operator:
+      # quiesce-point backup, cluster teardown, restore-mode round-trip, and
+      # the flip back to normal. See restore-orchestrator.sh. The sidecar image
+      # variant carries no /opt/antithesis/test tree, so the Composer never
+      # schedules test commands into this container (they would run without
+      # the workload env above).
+      - name: restore-orchestrator
+        image: __REGISTRY__/workload-ledger-v3-sidecar:__TAG__
+        command: ["/opt/antithesis/restore-orchestrator.sh"]
+        env:
+        - name: NAMESPACE
+          valueFrom:
+            fieldRef:
+              fieldPath: metadata.namespace
+        - name: CLUSTER_NAME
+          value: ledger
+        - name: CLUSTER_REPLICAS
+          value: "3"
+        - name: MODEL_RESTORE_REQ
+          value: /var/run/restore/req
+        - name: MODEL_RESTORE_RESP
+          value: /var/run/restore/resp
+        volumeMounts:
+        - name: restore-rendezvous
+          mountPath: /var/run/restore
+        resources:
+          requests:
+            cpu: 10m
+            memory: 64Mi
+          limits:
+            memory: 128Mi
+      volumes:
+      - name: restore-rendezvous
+        emptyDir: {}