feat(cloud-backups): retention-gated drop for pg audit-rotate

[logicflakes]

What

Reworks pg audit-rotate from keep-copy + same-run drop to retention-gated whole-archive drop ("Approach B"). Each run rotates the write-only audit table aside as an immutable timestamp-named archive, backs it up to a permanent bucket, retains it for AUDIT_RETENTION_DAYS (default 30), and DROPs it whole once aged. A two-pass reconcile recovers interrupted backups (reuse-if-present via HeadObject, never a blind re-dump) and drops aged-out archives.

Highlights

• Drop-gate oracle = rotation timestamp parsed from the archive name; parse failure fails safe (retain, never the time.Parse zero-value).
• Cheap pre-drop gate by default (dump + .sha256 both present via Head); --verify-restore adds re-download + pg_restore -l + SHA-256.
• Preflight guards (both refuse-by-default):
  • frozen entity_name='instances' rows still read by InstanceService → --drop-instance-rows to override (not data loss — backed up — but app instance-revision reads go empty ~retention days post-cutover).
  • non-owner GRANTs that CREATE TABLE LIKE won't reproduce → split-role safety (would otherwise break audit writes after the first rotation).
• --drain-backlog cutover drops the new archive immediately (first-run only).
• pg backup --exclude-table so the full-DB backup skips the resident archive tables (avoids re-dumping them into the expiring bucket + a DROP-vs-pg_dump race).
• audit_rotate_summary log (dropped/retained/recovered/quarantined, oldest_archive_age_days) for dead-man / disk alerting.
• Removed the vestigial --no-drop/--drop-pending staging flags — retention is the inspect-before-drop buffer now. Renamed keep-tail-days → audit-retention-days.
• Typed storage.ErrNotFound (S3 + Azure) so the drop gate distinguishes a definitive absence from a transient error.

Testing

Unit tests (gate, oracle, mappings, exclude-table, ACL/instances SQL) + an 8-scenario stress test against Postgres 17 + MinIO + a 15 GB real-data cutover (rotate catalog-only ~30 ms → 4.47 GB encrypted → verify-restore → DROP reclaimed ~15 GB). Reviewed across correctness/security, conventions, SRE, DBA, and holistic-simplicity lenses.

Rollout

Paired with the rearm-core helm PR (values/cronjob + EXCLUDE_TABLE + a bucket-collision fail assert). Merge this and build the image before rolling the helm change. Enabling in prod also requires external dead-man + disk-free alerting (the reclaim-nothing failure is otherwise silent) and a separate no-expiry bucket (GOVERNANCE-not-COMPLIANCE / versioned+noncurrent-expiry — a deny-overwrite WORM bucket is incompatible with recovery).

🤖 Generated with Claude Code