Skip to content

fix(bootstrap): keep the cold-storage module out of restore mode#1594

Merged
Azorlogh merged 1 commit into
release/v3.0from
fix/restore-mode-cold-storage
Jul 15, 2026
Merged

fix(bootstrap): keep the cold-storage module out of restore mode#1594
Azorlogh merged 1 commit into
release/v3.0from
fix/restore-mode-cold-storage

Conversation

@Azorlogh

Copy link
Copy Markdown
Contributor

Found by the Antithesis model test's restore cycle (#1593): every restore-mode pod crash-looped and the cycle died at ledgerctl restore download with "container not found".

Bug

cmd/server/server.go selects RestoreModule() vs the full Module() by mode, but appended bootstrap.ColdStorageModule(...) unconditionally whenever the driver wasn't none. That module's Archiver constructor consumes the runtime graph — *dal.Store, *state.Machine, ctrl.Admission, *node.Node — none of which restore mode provides ("no Raft"), so fx refused the graph and the server exited at boot:

Error: missing types: *dal.Store; *state.Machine; ctrl.Admission; *node.Node

Any deployment with cold storage enabled cannot enter restore mode — a disaster-recovery blocker, not just a chaos-suite failure. It went unnoticed because every restore exercise (the restore e2e suites, the operator's restore-incremental chainsaw test) runs with the driver at none; the Antithesis cluster spec is production-shaped with S3 cold storage, and its restore cycle was the first thing to combine the two.

Bisection confirmed cold storage is the only trigger — --sentinel-mode and --receipt-signing-key boot fine in restore mode.

Fix

ColdStorageModule(driver, restore) contributes nothing in restore mode: a restoring server neither archives nor reads cold data, and backup downloads carry their own storage configuration per request.

Tests

fx.ValidateApp regression pair in internal/bootstrap/module_restore_test.go: the restore-mode graph with cold storage enabled must build, and the counterfactual (ungated) graph must fail with the exact missing-types error. Also verified end-to-end locally: ledger-server run --restore --cold-storage-driver ... now boots and serves the restore API.

The server assembles RestoreModule or the full Module by mode, but
appended ColdStorageModule unconditionally whenever the driver was not
"none". The module's Archiver consumes the runtime graph (*dal.Store,
*state.Machine, ctrl.Admission, *node.Node) that restore mode
deliberately does not provide, so fx refused the whole graph and the
server exited at boot — any deployment with cold storage enabled could
not enter restore mode at all. Found by the Antithesis model test's
restore cycle: the restore-mode pod crash-looped and every
`ledgerctl restore download` exec failed with "container not found".

ColdStorageModule now takes the restore flag and contributes nothing in
restore mode: a restoring server neither archives nor reads cold data,
and backup downloads carry their own storage configuration per request.

The regression test fx-validates the restore-mode graph with cold
storage enabled (the gap every restore e2e shared — they all ran with
the driver at "none"), plus the counterfactual proving the ungated
graph is unbuildable.
@coderabbitai

coderabbitai Bot commented Jul 15, 2026

Copy link
Copy Markdown
Contributor

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

🗂️ Base branches to auto review (1)
  • main

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 065458f2-2b88-4727-9253-56d3adff90a8

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix/restore-mode-cold-storage

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@NumaryBot

Copy link
Copy Markdown
Contributor

✅ Approve — automated review

The change correctly gates the cold-storage fx module out of restore mode and updates the only call site. The added regression test covers both the fixed restore graph and the counterfactual ungated failure.

No findings.

@codecov

codecov Bot commented Jul 15, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 74.98%. Comparing base (1b502c4) to head (0d197a7).

Additional details and impacted files
@@               Coverage Diff                @@
##           release/v3.0    #1594      +/-   ##
================================================
+ Coverage         74.73%   74.98%   +0.24%     
================================================
  Files               430      430              
  Lines             45722    45722              
================================================
+ Hits              34172    34283     +111     
+ Misses             8501     8384     -117     
- Partials           3049     3055       +6     
Flag Coverage Δ
e2e 74.98% <100.00%> (+0.24%) ⬆️
scenario 74.98% <100.00%> (+0.24%) ⬆️
unit 74.98% <100.00%> (+0.24%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@Azorlogh Azorlogh merged commit 322286d into release/v3.0 Jul 15, 2026
22 of 24 checks passed
@Azorlogh Azorlogh deleted the fix/restore-mode-cold-storage branch July 15, 2026 17:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants