fix(appliance): rebuild a container store left corrupt by an interrupted write (#1029)

[VijitSingh97]

Closes #1029.

Validated end to end on the physical HP bench today: a stick built from this branch installed the appliance to its internal NVMe and brought the full stack up (8 containers). That is the flow #1029 said was impossible.

The reported root cause was wrong

The issue blamed an unguarded mkdir -p "$PWD/data" against a read-only root. That is a reproduction artifact, and the evidence is in the issue's own transcript:

• /opt/pithead/data does not exist on the box, and the unit runs ExecStart=/data/pithead/pithead with WorkingDirectory=/data/pithead — a real rw ext4 mountpoint (pithead-mount-generator mounts partition 4 of the disk the system booted from, which on a stick boot is the stick's own).
• An earlier unguarded write, mkdir -p "$spool" at pithead:2066, would have aborted before the Loading this build's container images line the transcript shows. /data/pithead/data/firstboot exists on disk owned 1000:1000. It succeeded.
• mkdir -p reports the deepest component it could not create, so both writes produce a byte-identical message. The string alone cannot identify the line.

The by-hand repro was run from /opt/pithead, which is not where the service runs.

What actually breaks it

Running the wizard as the unit does, and a bare podman run with no wizard involved, both give:

Error: readlink /data/containers/storage/overlay: invalid argument

The store held seven zero-length lower files. containers/storage splits an empty-but-present lower on : into one empty element, joins it onto the graph root and readlinks that directory — hence the error naming overlay itself. A correct base layer carries no lower file (f2ec4de… correctly had none), so zero-length is damage, never valid. Signature of an interrupted write.

Why it was permanent — the actual product defect. load_baked_images had two guards: the digest record matched, and image exists returned true (the image was there, just unrunnable). Both passed, so the archive was never reloaded and one unclean reset bricked the medium across every later boot, with nothing on the console.

The fix

Detect the defect before trusting any digest record, tear the store down, reload. Keyed on the defect itself, so a healthy store is untouched and a provisioned machine never re-pulls for nothing.

Proven by live repair before writing it: wiping the store and reloading brought the wizard up on the bricked box.

Verification

Gate	Result
Real hardware	Install-from-stick completed to internal NVMe; 8 containers up
New unit tests	4, covering damaged and healthy stores
Mutation check	With the repair disabled, both damage tests fail — the tests are not vacuous
make lint	0 errors
tests/stack	2320 passed, 0 failed

Scope

Does not address what interrupted the write — filed as #1030 (the journal is volatile, so the failing boot's evidence was gone; that is #1030's first recommendation). The unguarded $PWD writes at pithead:2066/6605 remain real latent bugs, just not this one.

Base is develop-v2; retarget before merge if auto-close is wanted.