Skip to content

Add HomeStore superblock decoder and validator#873

Open
xiaoxichen wants to merge 1 commit intoeBay:masterfrom
xiaoxichen:xiaoxi
Open

Add HomeStore superblock decoder and validator#873
xiaoxichen wants to merge 1 commit intoeBay:masterfrom
xiaoxichen:xiaoxi

Conversation

@xiaoxichen
Copy link
Copy Markdown
Collaborator

This tool decodes and validates HomeStore superblock structures from disk dumps.
It checks bitmap consistency, CRC integrity, and VDev allocation.

Usage:
# Quick validation summary
python3 decode_superblock.py <disk_dump>

# Detailed verbose output
python3 decode_superblock.py <disk_dump> --verbose

# Inspect specific chunk
python3 decode_superblock.py <disk_dump> chunk 512

# Inspect specific vdev
python3 decode_superblock.py <disk_dump> vdev 0
python3 decode_superblock.py <disk_dump> vdev meta

What it validates:

  • First block magic number and metadata
  • Bitmap vs chunk array consistency
  • Chunk CRC checksums (detects corruption)
  • VDev allocation and utilization
  • Orphan chunks (corrupted vdev_id)

Exit codes:
0 = All validations passed 1 = Issues detected (CRC errors, bitmap inconsistencies, orphans)

Common issues detected:

  • CRC mismatch: Chunk data corrupted on disk
  • Bitmap inconsistency: Bitmap/chunk allocation out of sync
  • Orphan chunks: Chunks with invalid/corrupted vdev_id

  This tool decodes and validates HomeStore superblock structures from disk dumps.
  It checks bitmap consistency, CRC integrity, and VDev allocation.

  Usage:
    # Quick validation summary
    python3 decode_superblock.py <disk_dump>

    # Detailed verbose output
    python3 decode_superblock.py <disk_dump> --verbose

    # Inspect specific chunk
    python3 decode_superblock.py <disk_dump> chunk 512

    # Inspect specific vdev
    python3 decode_superblock.py <disk_dump> vdev 0
    python3 decode_superblock.py <disk_dump> vdev meta

  What it validates:
  - First block magic number and metadata
  - Bitmap vs chunk array consistency
  - Chunk CRC checksums (detects corruption)
  - VDev allocation and utilization
  - Orphan chunks (corrupted vdev_id)

  Exit codes:
    0 = All validations passed
    1 = Issues detected (CRC errors, bitmap inconsistencies, orphans)

  Common issues detected:
  - CRC mismatch: Chunk data corrupted on disk
  - Bitmap inconsistency: Bitmap/chunk allocation out of sync
  - Orphan chunks: Chunks with invalid/corrupted vdev_id

Signed-off-by: Xiaoxi Chen <xiaoxchen@ebay.com>
@xiaoxichen
Copy link
Copy Markdown
Collaborator Author

This is a BORING code written by AI , lets focus more on functional

☆SDDZ☆xiaoxchen@QFHYQ56F7P HomeStore % python3 tools/superblock_checker.py ./sm1.data.32m

HomeStore Superblock Summary
================================================================================
Product: HomeStore4x v5
System UUID: 84ba0a2b-b9f4-459f-8a69-6d575eba4ac1
Formatting: Complete
Device: pdev_id=1, size=111.76 TB, data_offset=1.75 GB

VDevs (4 total):
  [0] META_VDEV    'meta    '   5721.96 GB  1,430 chunks    100% util  ✓
  [1] LOGDEV_VDEV  'LogDev  '      2.88 GB     92 chunks    100% util  ✓
  [2] INDEX_VDEV   'index   '   5721.96 GB  1,431 chunks    100% util  ✓
  [3] DATA_VDEV    'blkdata '  90406.00 GB  45,203 chunks    100% util  ✓

Chunks:
  Total allocated: 48,157 / 65,536 (73.5%)
  Valid checksums: 48,156 (100.0%)
  Invalid:         1

Issues:
  ✗ 1 chunks with CRC errors:
  ✗ 1 orphan chunks (corrupted vdev_id):
    Slot 512 (vdev_id=1547590718 INVALID): CRC error: stored=0x0000 calc=0x5d31

Status: ⚠ ISSUES DETECTED

@codecov-commenter
Copy link
Copy Markdown

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 48.22%. Comparing base (1a0cef8) to head (014a977).
⚠️ Report is 325 commits behind head on master.
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files
@@            Coverage Diff             @@
##           master     #873      +/-   ##
==========================================
- Coverage   56.51%   48.22%   -8.30%     
==========================================
  Files         108      110       +2     
  Lines       10300    12894    +2594     
  Branches     1402     6198    +4796     
==========================================
+ Hits         5821     6218     +397     
+ Misses       3894     2561    -1333     
- Partials      585     4115    +3530     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@xiaoxichen xiaoxichen requested review from Besroy and yuwmao March 26, 2026 02:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants