fix(cloud-backups): checksum Azure uploads server-side (CRC64), at every payload size

[logicflakes]

Problem

The S3 upload path opts into server-side integrity verification, so a completed upload means the stored bytes are the bytes we sent. The Azure path asked for nothing — Azure received no checksum at all, and a "successful" upload proved only that bytes arrived.

That matters here because audit-rotate DROPs the source table once an upload verifies, and the default gate is existence + size — which a length-preserving corruption passes. It's also why the docs told Azure users to run --verify-restore (full re-download + decrypt + pg_restore -l) on every weekly run: several GB of billed egress and ~10 extra minutes per run, to compensate for a missing header.

Change

Set TransactionalValidation so the SDK computes a CRC64 and Azure validates it server-side, rejecting the payload on mismatch.

That alone is not sufficient, and this is the part worth reviewing carefully. azblob's UploadStream short-circuits to a single-shot Put Blob whenever the whole payload fit in block 0 and is smaller than BlockSize:

// blockblob/chunkwriting.go, commitBlocks
if bt.maxBlockNum == 0 && len(bt.firstBlock) < int(bt.options.BlockSize) {
    up, err := to.Upload(ctx, streaming.NopCloser(...), bt.options.getUploadOptions())

…and UploadStreamOptions.getUploadOptions() (blockblob/models.go) copies Tags/Metadata/Tier/HTTPHeaders/CPK/AccessConditions — not TransactionalValidation. So every .sha256 sidecar (65 bytes) and every archive under 10 MiB would have uploaded unchecksummed while the docs claimed otherwise — worse than the status quo, because it would also have removed the --verify-restore recommendation that was the only mitigation for exactly those payloads.

blockblob.Upload does honour the option, so UploadStream now buffers at most one block to decide which path applies and routes undersized payloads through it.

Evidence

Asserted on the wire, not structurally. TestAzureUploadStreamChecksumsEverySize drives the real provider against an httptest server through a non-seekable pipe (like the real pg_dump | age source) and requires x-ms-content-crc64 on every payload request:

payload	path	header
65 B (sidecar)	single-shot Put Blob	present
BlockSize - 1	single-shot Put Blob	present
BlockSize	1 staged block + blocklist	present
BlockSize + 1 KiB	2 staged blocks + blocklist	present

Negative control — with the size split reverted, the test fails on exactly the two undersized cases while the options-struct assertion stays green:

--- FAIL: .../sha256_sidecar_sized
    1 of 1 payload request(s) carried NO x-ms-content-crc64: [comp="" len=65]
--- FAIL: .../just_under_one_block
    1 of 1 payload request(s) carried NO x-ms-content-crc64: [comp="" len=10485759]

That's why the test checks the wire rather than the options struct.

Memory

ComputeCRC64 does io.ReadAll, which is only safe because it's applied per block — peak memory is flat in archive size (a 4.5 GB archive costs the same as a 100 MB one; every staged request body is exactly 10 MiB regardless of payload). Measured peak RSS, fresh process per case:

payload= 100 MiB  validation=false   42 MiB   |  validation=true  139 MiB
payload=1000 MiB  validation=false   44 MiB   |  validation=true  142 MiB
payload=4500 MiB  validation=false   44 MiB   |  validation=true  199 MiB

The delta is ~+100–200 MiB, not the naive BlockSize * Concurrency = 30 MiB, because ReadAll's append growth re-buffers each block at roughly 2x. Comfortably inside the cronjob's limits.memory: 1Gi; the code comment records the measured figure rather than the naive one. Possible follow-up in rearm-saas: requests.memory: 256Mi is now marginal alongside the pg_dump subprocess in the same cgroup — worth ~384Mi, but that's a different repo and not needed for correctness.

Verified clean (SDK source + empirical)

• Streaming contract — no seekable reader required; the SDK hands StageBlock a NopCloser(bytes.NewReader(...)) over a buffer it had to materialize anyway, and Apply returns a fresh reader at position 0 of identical length, so Content-Length stays correct.
• Retry — with the server 500-ing the first attempt at each block, the retry resends an identical CRC and full body. CRC computation happens before the policy pipeline, so it doesn't consume the try budget (~10 ms per 10 MiB block).
• Fail-closed — a simulated 400 Md5Mismatch on a staged block surfaces as a hard error, so the archive is never dropped.
• AAD ClientSecretCredential — orthogonal; the bearer-challenge policy replays the same seekable body with the header attached.

Before merge

I could not exercise real Azure from here — the provider builds its endpoint as https://<account>.blob.core.windows.net with AAD auth, so Azurite isn't reachable through it, and my kubectl only reaches the rhythm sandbox. Recommend one live smoke upload against a real storage account, with both a >10 MiB and a small object, since if anything about the header were wrong the failure mode is fail-closed but total for the Azure backup path.

Informational, no action: blockblob/client.go StageBlock does if err != nil { return StageBlockResponse{}, nil } after Apply — it swallows that error. Only reachable now that TransactionalValidation is non-nil; in practice Apply reads an in-memory bytes.Reader and can't fail, and a missing block would be rejected at CommitBlockList anyway.

Notes

Independent of #10 (which narrows the S3 IAM surface) and reviewable separately, per request. Both touch adjacent README.md lines, so whichever merges second may need a trivial git merge origin/main on its branch — happy to do that.

CI: branch runs in this repo are red for unrelated reasons (10 jobs for other images fail in the legacy reliza-docker-action getversion step against app.relizahub.com; every branch run here fails that way while main is green). build-cloud-backups — the only job building anything this PR touches — passes.

🤖 Generated with Claude Code