-
-
Notifications
You must be signed in to change notification settings - Fork 1.3k
feat: redundancy awareness & cross-volume file comparison #3053
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
+3,143
−171
Merged
Changes from 8 commits
Commits
Show all changes
17 commits
Select commit
Hold shift + click to select a range
1eea61b
feat: add redundancy awareness & cross-volume file comparison
jamiepine 1c07bf6
Merge remote-tracking branch 'origin/main' into spacedrive-redundancy
jamiepine 2926675
redundancy UI improvements + ZFS volume detection fix
jamiepine f824bd4
Merge branch 'main' into spacedrive-redundancy
jamiepine 2e573ba
fix ZFS pool capacity reporting and stats filtering
jamiepine f027b49
show capacity and visibility in sd volume list
jamiepine 7699a2e
document filesystem support matrix and detection
jamiepine e05db09
WIP: redundancy filter wiring across search, CLI, and UI
jamiepine 29267ce
add web context menu renderer and UI polish
jamiepine 30e5216
sd-server dev workflow: auto web build, shutdown watchdog, stable dat…
jamiepine b3b1be0
add Sources space item alongside Redundancy in default library layout
jamiepine dad50ec
add TrueNAS native build script
jamiepine 8fde30a
never block RPC on synchronous statistics calculation
jamiepine 452fbf4
self-heal protocol handler registration on re-init
jamiepine 1086efc
fall back to pkarr+DNS discovery when mDNS port is unavailable
jamiepine 777b345
succeed pairing if either mDNS or relay discovery wins
jamiepine 3ee0f43
Merge main into spacedrive-redundancy
jamiepine File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,70 @@ | ||
| use clap::Args; | ||
| use uuid::Uuid; | ||
|
|
||
| use sd_core::ops::redundancy::summary::RedundancySummaryInput; | ||
|
|
||
| #[derive(Args, Debug)] | ||
| pub struct SummaryArgs { | ||
| /// Restrict summary to specific volume UUIDs (can be specified multiple times). | ||
| /// Omit to summarize the entire library. | ||
| #[arg(long = "volume", value_name = "VOLUME_UUID")] | ||
| pub volumes: Option<Vec<Uuid>>, | ||
| } | ||
|
|
||
| impl From<SummaryArgs> for RedundancySummaryInput { | ||
| fn from(args: SummaryArgs) -> Self { | ||
| Self { | ||
| volume_uuids: args.volumes, | ||
| } | ||
| } | ||
| } | ||
|
|
||
| #[derive(Args, Debug)] | ||
| pub struct AtRiskArgs { | ||
| /// Only return files present on this volume (UUID) | ||
| #[arg(long, value_name = "VOLUME_UUID")] | ||
| pub volume: Option<Uuid>, | ||
|
|
||
| /// Show redundant files (content on 2+ volumes) instead of at-risk (content on 1 volume) | ||
| #[arg(long)] | ||
| pub redundant: bool, | ||
|
|
||
| /// Max number of files to show | ||
| #[arg(long, default_value = "50")] | ||
| pub limit: u32, | ||
|
|
||
| /// Pagination offset | ||
| #[arg(long, default_value = "0")] | ||
| pub offset: u32, | ||
| } | ||
|
|
||
| #[derive(Args, Debug)] | ||
| pub struct CompareArgs { | ||
| /// First volume UUID | ||
| pub volume_a: Uuid, | ||
|
|
||
| /// Second volume UUID | ||
| pub volume_b: Uuid, | ||
|
|
||
| /// What to compare | ||
| #[arg(long, value_enum, default_value = "unique-a")] | ||
| pub mode: CompareMode, | ||
|
|
||
| /// Max number of files to show | ||
| #[arg(long, default_value = "50")] | ||
| pub limit: u32, | ||
|
|
||
| /// Pagination offset | ||
| #[arg(long, default_value = "0")] | ||
| pub offset: u32, | ||
| } | ||
|
|
||
| #[derive(clap::ValueEnum, Clone, Debug)] | ||
| pub enum CompareMode { | ||
| /// Files present on volume A but not on volume B | ||
| UniqueA, | ||
| /// Files present on volume B but not on volume A | ||
| UniqueB, | ||
| /// Files present on both volumes | ||
| Shared, | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,314 @@ | ||
| //! Redundancy CLI domain | ||
| //! | ||
| //! Commands for inspecting data redundancy across volumes: | ||
| //! - `summary` — per-volume at-risk vs redundant breakdown + replication score | ||
| //! - `at-risk` — list files whose content lives on only one volume | ||
| //! - `compare` — diff two volumes (unique-to-A, unique-to-B, shared) | ||
|
|
||
| mod args; | ||
|
|
||
| use anyhow::Result; | ||
| use clap::Subcommand; | ||
| use comfy_table::{presets::UTF8_BORDERS_ONLY, Attribute, Cell, Table}; | ||
|
|
||
| use crate::context::Context; | ||
| use crate::util::prelude::*; | ||
|
|
||
| use sd_core::ops::redundancy::summary::{RedundancySummaryInput, RedundancySummaryOutput}; | ||
| use sd_core::ops::search::{ | ||
| input::{ | ||
| FileSearchInput, PaginationOptions, SearchFilters, SearchMode, SearchScope, SortDirection, | ||
| SortField, SortOptions, | ||
| }, | ||
| output::FileSearchOutput, | ||
| }; | ||
|
|
||
| use self::args::*; | ||
|
|
||
| #[derive(Subcommand, Debug)] | ||
| pub enum RedundancyCmd { | ||
| /// Show redundancy summary (replication score + per-volume breakdown) | ||
| Summary(SummaryArgs), | ||
| /// List at-risk files (content that only lives on one volume) | ||
| AtRisk(AtRiskArgs), | ||
| /// Compare two volumes (unique-to-A, unique-to-B, or shared) | ||
| Compare(CompareArgs), | ||
| } | ||
|
|
||
| pub async fn run(ctx: &Context, cmd: RedundancyCmd) -> Result<()> { | ||
| match cmd { | ||
| RedundancyCmd::Summary(args) => run_summary(ctx, args).await, | ||
| RedundancyCmd::AtRisk(args) => run_at_risk(ctx, args).await, | ||
| RedundancyCmd::Compare(args) => run_compare(ctx, args).await, | ||
| } | ||
| } | ||
|
|
||
| async fn run_summary(ctx: &Context, args: SummaryArgs) -> Result<()> { | ||
| let input: RedundancySummaryInput = args.into(); | ||
| let out: RedundancySummaryOutput = execute_query!(ctx, input); | ||
|
|
||
| print_output!(ctx, &out, |o: &RedundancySummaryOutput| { | ||
| render_summary(o); | ||
| }); | ||
| Ok(()) | ||
| } | ||
|
|
||
| async fn run_at_risk(ctx: &Context, args: AtRiskArgs) -> Result<()> { | ||
| let input = FileSearchInput { | ||
| query: String::new(), | ||
| scope: SearchScope::Library, | ||
| mode: SearchMode::Fast, | ||
| filters: SearchFilters { | ||
| at_risk: Some(!args.redundant), | ||
| on_volumes: args.volume.map(|v| vec![v]), | ||
| ..Default::default() | ||
| }, | ||
| sort: SortOptions { | ||
| field: SortField::Size, | ||
| direction: SortDirection::Desc, | ||
| }, | ||
| pagination: PaginationOptions { | ||
| limit: args.limit, | ||
| offset: args.offset, | ||
| }, | ||
| }; | ||
|
|
||
| let out: FileSearchOutput = execute_query!(ctx, input); | ||
| let label = if args.redundant { "redundant" } else { "at-risk" }; | ||
|
|
||
| print_output!(ctx, &out, |o: &FileSearchOutput| { | ||
| render_file_list(o, label); | ||
| }); | ||
| Ok(()) | ||
| } | ||
|
|
||
| async fn run_compare(ctx: &Context, args: CompareArgs) -> Result<()> { | ||
| let (on, not_on, min_count, label) = match args.mode { | ||
| CompareMode::UniqueA => ( | ||
| Some(vec![args.volume_a]), | ||
| Some(vec![args.volume_b]), | ||
| None, | ||
| "unique to A", | ||
| ), | ||
| CompareMode::UniqueB => ( | ||
| Some(vec![args.volume_b]), | ||
| Some(vec![args.volume_a]), | ||
| None, | ||
| "unique to B", | ||
| ), | ||
| CompareMode::Shared => ( | ||
| Some(vec![args.volume_a, args.volume_b]), | ||
| None, | ||
| Some(2u32), | ||
| "shared", | ||
| ), | ||
| }; | ||
|
|
||
| let input = FileSearchInput { | ||
| query: String::new(), | ||
| scope: SearchScope::Library, | ||
| mode: SearchMode::Fast, | ||
| filters: SearchFilters { | ||
| on_volumes: on, | ||
| not_on_volumes: not_on, | ||
| min_volume_count: min_count, | ||
| ..Default::default() | ||
| }, | ||
| sort: SortOptions { | ||
| field: SortField::Size, | ||
| direction: SortDirection::Desc, | ||
| }, | ||
| pagination: PaginationOptions { | ||
| limit: args.limit, | ||
| offset: args.offset, | ||
| }, | ||
| }; | ||
|
|
||
| let out: FileSearchOutput = execute_query!(ctx, input); | ||
|
|
||
| print_output!(ctx, &out, |o: &FileSearchOutput| { | ||
| render_file_list(o, label); | ||
| }); | ||
| Ok(()) | ||
| } | ||
|
|
||
| // ─── rendering helpers ──────────────────────────────────────────────────────── | ||
|
|
||
| fn render_summary(o: &RedundancySummaryOutput) { | ||
| let totals = &o.library_totals; | ||
|
|
||
| // Library-wide header | ||
| let mut overview = Table::new(); | ||
| overview.load_preset(UTF8_BORDERS_ONLY); | ||
| overview.set_header(vec![ | ||
| Cell::new("Redundancy Summary").add_attribute(Attribute::Bold), | ||
| Cell::new(""), | ||
| ]); | ||
| overview.add_row(vec![ | ||
| Cell::new("Unique content"), | ||
| Cell::new(format_bytes_i64(totals.total_unique_content_bytes)), | ||
| ]); | ||
| overview.add_row(vec![ | ||
| Cell::new("At risk"), | ||
| Cell::new(format!( | ||
| "{} ({})", | ||
| format_bytes_i64(totals.total_at_risk_bytes), | ||
| percent_of(totals.total_at_risk_bytes, totals.total_unique_content_bytes), | ||
| )), | ||
| ]); | ||
| overview.add_row(vec![ | ||
| Cell::new("Redundant"), | ||
| Cell::new(format!( | ||
| "{} ({})", | ||
| format_bytes_i64(totals.total_redundant_bytes), | ||
| percent_of(totals.total_redundant_bytes, totals.total_unique_content_bytes), | ||
| )), | ||
| ]); | ||
| overview.add_row(vec![ | ||
| Cell::new("Replication score"), | ||
| Cell::new(format!( | ||
| "{:.2} ({:.1}%)", | ||
| totals.replication_score, | ||
| totals.replication_score * 100.0 | ||
| )), | ||
| ]); | ||
| println!("{}", overview); | ||
| println!(); | ||
|
|
||
| // Per-volume breakdown | ||
| if o.volumes.is_empty() { | ||
| println!("No volumes with indexed content found."); | ||
| return; | ||
| } | ||
|
|
||
| let mut table = Table::new(); | ||
| table.load_preset(UTF8_BORDERS_ONLY); | ||
| table.set_header(vec![ | ||
| Cell::new("Volume").add_attribute(Attribute::Bold), | ||
| Cell::new("Total").add_attribute(Attribute::Bold), | ||
| Cell::new("At-Risk").add_attribute(Attribute::Bold), | ||
| Cell::new("Redundant").add_attribute(Attribute::Bold), | ||
| Cell::new("Files").add_attribute(Attribute::Bold), | ||
| ]); | ||
|
|
||
| for v in &o.volumes { | ||
| let name = v | ||
| .display_name | ||
| .as_deref() | ||
| .map(|n| n.to_string()) | ||
| .unwrap_or_else(|| v.volume_uuid.to_string()); | ||
|
|
||
| let at_risk = format!( | ||
| "{} ({})", | ||
| format_bytes_i64(v.at_risk_bytes), | ||
| percent_of(v.at_risk_bytes, v.total_bytes), | ||
| ); | ||
| let redundant = format!( | ||
| "{} ({})", | ||
| format_bytes_i64(v.redundant_bytes), | ||
| percent_of(v.redundant_bytes, v.total_bytes), | ||
| ); | ||
| let files = format!( | ||
| "{} ({} at-risk / {} redundant)", | ||
| v.total_file_count, v.at_risk_file_count, v.redundant_file_count | ||
| ); | ||
|
|
||
| table.add_row(vec![ | ||
| Cell::new(name), | ||
| Cell::new(format_bytes_i64(v.total_bytes)), | ||
| Cell::new(at_risk), | ||
| Cell::new(redundant), | ||
| Cell::new(files), | ||
| ]); | ||
|
|
||
| // UUID on a subline for precision | ||
| table.add_row(vec![ | ||
| Cell::new(format!(" {}", v.volume_uuid)), | ||
| Cell::new(""), | ||
| Cell::new(""), | ||
| Cell::new(""), | ||
| Cell::new(""), | ||
| ]); | ||
| } | ||
|
|
||
| println!("{}", table); | ||
| } | ||
|
|
||
| fn render_file_list(o: &FileSearchOutput, label: &str) { | ||
| if o.files.is_empty() { | ||
| println!("No {} files found.", label); | ||
| return; | ||
| } | ||
|
|
||
| println!( | ||
| "Showing {} of {} {} file(s) ({}ms)", | ||
| o.files.len(), | ||
| o.total_found, | ||
| label, | ||
| o.execution_time_ms | ||
| ); | ||
| println!(); | ||
|
|
||
| let mut table = Table::new(); | ||
| table.load_preset(UTF8_BORDERS_ONLY); | ||
| table.set_header(vec![ | ||
| Cell::new("#").add_attribute(Attribute::Bold), | ||
| Cell::new("Name").add_attribute(Attribute::Bold), | ||
| Cell::new("Size").add_attribute(Attribute::Bold), | ||
| Cell::new("Ext").add_attribute(Attribute::Bold), | ||
| Cell::new("Modified").add_attribute(Attribute::Bold), | ||
| Cell::new("Path").add_attribute(Attribute::Bold), | ||
| ]); | ||
|
|
||
| for (i, f) in o.files.iter().enumerate() { | ||
| table.add_row(vec![ | ||
| Cell::new((i + 1).to_string()), | ||
| Cell::new(truncate(&f.name, 48)), | ||
| Cell::new(format_bytes_u64(f.size)), | ||
| Cell::new(f.extension.clone().unwrap_or_default()), | ||
| Cell::new(f.modified_at.format("%Y-%m-%d %H:%M").to_string()), | ||
| Cell::new(truncate(&f.sd_path.display().to_string(), 60)), | ||
| ]); | ||
| } | ||
|
|
||
| println!("{}", table); | ||
| } | ||
|
|
||
| fn percent_of(part: i64, total: i64) -> String { | ||
| if total <= 0 { | ||
| return "0%".into(); | ||
| } | ||
| format!("{:.1}%", (part as f64 / total as f64) * 100.0) | ||
| } | ||
|
|
||
| fn format_bytes_i64(bytes: i64) -> String { | ||
| if bytes < 0 { | ||
| return format!("-{}", format_bytes_u64(bytes.unsigned_abs())); | ||
| } | ||
| format_bytes_u64(bytes as u64) | ||
| } | ||
|
|
||
| fn format_bytes_u64(bytes: u64) -> String { | ||
| const UNITS: &[&str] = &["B", "KB", "MB", "GB", "TB", "PB"]; | ||
| let mut size = bytes as f64; | ||
| let mut unit = 0; | ||
| while size >= 1024.0 && unit < UNITS.len() - 1 { | ||
| size /= 1024.0; | ||
| unit += 1; | ||
| } | ||
| if unit == 0 { | ||
| format!("{} {}", bytes, UNITS[unit]) | ||
| } else { | ||
| format!("{:.1} {}", size, UNITS[unit]) | ||
| } | ||
| } | ||
|
|
||
| fn truncate(s: &str, max: usize) -> String { | ||
| if s.chars().count() <= max { | ||
| s.to_string() | ||
| } else { | ||
| let mut out: String = s.chars().take(max.saturating_sub(1)).collect(); | ||
| out.push('…'); | ||
| out | ||
| } | ||
| } | ||
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🧩 Analysis chain
🏁 Script executed:
Repository: spacedriveapp/spacedrive
Length of output: 2202
🏁 Script executed:
Repository: spacedriveapp/spacedrive
Length of output: 640
🏁 Script executed:
Repository: spacedriveapp/spacedrive
Length of output: 1849
🏁 Script executed:
Repository: spacedriveapp/spacedrive
Length of output: 794
🏁 Script executed:
Repository: spacedriveapp/spacedrive
Length of output: 560
🏁 Script executed:
Repository: spacedriveapp/spacedrive
Length of output: 5070
🏁 Script executed:
Repository: spacedriveapp/spacedrive
Length of output: 298
🏁 Script executed:
Repository: spacedriveapp/spacedrive
Length of output: 2578
🏁 Script executed:
Repository: spacedriveapp/spacedrive
Length of output: 1111
Sharedmode can return files that are not actually on both volumes.on_volumesuses OR semantics at the SQL layer (v.uuid IN (...)means "present on any of these volumes"), soon_volumes=[A, B]withmin_volume_count=2will match a file that exists on{A, C}(on A ✓, on 2 volumes ✓) and report it as shared, even though it does not exist on B. To correctly identify content present on both A and B, you need AND semantics—e.g., two separateon_volumespredicates—rather than OR + volume count. As written, the CLI will over-report the shared set whenever a third volume exists.🤖 Prompt for AI Agents