Skip to content

fix(cloud-backups): alert on uncovered base paths, not absent months - #15

Open
logicflakes wants to merge 1 commit into
mainfrom
2026-08-oci-backup-uncovered-base-paths
Open

fix(cloud-backups): alert on uncovered base paths, not absent months#15
logicflakes wants to merge 1 commit into
mainfrom
2026-08-oci-backup-uncovered-base-paths

Conversation

@logicflakes

Copy link
Copy Markdown
Collaborator

Follow-up to #14. That PR's missing-target guard is firing as a false positive on customer prod, and will keep doing so on every run until this lands.

- error = 1 of 4 target(s) produced no backup because the repository was reported absent:
          rearm-artifacts/downloadable-artifacts-2026-08
- msg   = backup_targets_missing_from_registry
- time  = 2026-08-15T08:26:16Z

#14 merged at 00:52 UTC that day; this alert is the merged code.

Why it fires

Under rolling months the tool does no discovery. It fabricates <base>-<current month> and <base>-<previous month> from the clock and attempts each, learning of absence by failing to back the target up.

Meanwhile a monthly repository is never created explicitly -- registries create one implicitly on first push, and both writers derive the same name from the UTC clock at push time (rebom-backend ociService.getMonthlyRepositoryName, rearm-core ArtifactService.getMonthlyRepositoryName). So a base path that receives no artifacts in a month has no repository for that month at all.

SBOMs land constantly, so rebom-artifacts-2026-08 always exists. Downloadable artifacts only land when a release publishes one -- none had been published in August, so that repo had never been created. #14's guard assumed a current-month target must exist: true for the first path, false for the second. It would page every run until the month's first downloadable, then again every new month, for any path without guaranteed monthly traffic.

The fix

Alert per base path, not per target: did this base path produce any backup at all?

One month absent is indistinguishable from a quiet month. A base path producing nothing is unambiguous, and is exactly what a renamed, deleted or mistyped path looks like. Under explicit paths a base path expands to one target, so the strict behaviour there is preserved by construction rather than by a special case. SkipIsExpected and the whole current-vs-previous month distinction are deleted.

Why not an existence check

Considered seriously and rejected as unreachable, not merely undesirable. Distinguishing "never created" from "deleted" needs state the credential envelope does not contain:

  • The registry cannot enumerate repositories without catalogue scope.
  • The destination cannot be read back under the write-only credential that storage.Provider's contract deliberately preserves.
  • Conditional writes do leak existence using write permission alone (S3 If-None-Match: * -> 412, Azure -> 409 BlobAlreadyExists) -- but the probe that answers "absent" succeeds, leaving an undeleteable 0-byte object at the real backup key. It destroys the answer it returns. If-Match: * is worse: it overwrites a live backup. A marker-key variant fails identically.

Worth recording for whoever revisits that decision: on Azure the write-only ideal in the Provider comment is not achievable with RBAC anyway -- there is no built-in write-only blob role, so the SP must hold Storage Blob Data Contributor, which already grants read. A probe would be zero-elevation on that deployment; the constraint is real only for S3.

Residual gap, stated deliberately

A current-month repo deleted while the previous month survives will not alert -- because with the available information that is indistinguishable from a quiet month. Closing it needs either s3:ListBucket / blob read, or an explicit per-path requireMonthly flag in the chart. Both remain open; neither is needed for the reported failure.

Verification

Rig reproduces the production shape exactly (rebom present for both months, downloadable present only for the previous month):

build ERROR lines to operator
merged code (b5e60ae) 1 -- reproduces the customer alert verbatim
this branch 0
base path absent in both months 1, naming the path

go build / go vet / go test -race ./... clean, gofmt clean, added lines pure ASCII.

Note on the last check: my first attempt to prove the alert still fires used curl -X DELETE against registry:2, which returns 405 because delete is disabled by default -- the repo never went away, so that run's "0 errors" was correct but meaningless. Re-verified with a base path that genuinely has no repositories.

🤖 Generated with Claude Code

The previous commit's missing-target guard fired on customer prod on
2026-08-15 for rearm-artifacts/downloadable-artifacts-2026-08, and it was
a false positive of exactly the kind this branch exists to remove.

Why it fired. Under rolling months the tool does no discovery: it
FABRICATES <base>-<current month> and <base>-<previous month> from the
clock and attempts each, so absence is discovered by failing to back the
target up. Meanwhile a monthly repository is never created explicitly --
registries create one implicitly on first push, and both writers derive
the same name from the UTC clock at push time (rebom-backend
ociService.getMonthlyRepositoryName, rearm-core ArtifactService
getMonthlyRepositoryName). So a base path that receives no artifacts in a
given month has no repository at all for that month. SBOMs land
constantly, so rebom-artifacts always exists; downloadable artifacts only
land when a release publishes one. The guard assumed a current-month
target must exist, which is true for the first path and false for the
second, and it would have paged every run until the month's first
downloadable was published, then again each new month.

What replaces it. The guard now asks the strongest question the tool's
own credentials can actually answer: did this BASE PATH produce any
backup at all? One month absent is indistinguishable from a quiet month;
a base path producing nothing is unambiguous, and is what a renamed,
deleted or mistyped path looks like. Under explicit paths a base path
expands to exactly one target, so the strict behaviour there is
unchanged by construction rather than by a special case. The
current-vs-previous month distinction, and SkipIsExpected with it, are
gone.

An existence check against the destination was considered and rejected as
unreachable, not merely undesirable. Distinguishing "never created" from
"deleted" needs state the credential envelope does not contain: the
registry cannot enumerate repositories without catalogue scope, and the
destination cannot be read back under the write-only credential that
storage.Provider's contract deliberately preserves. Conditional writes
do leak existence using write permission alone (S3 If-None-Match, Azure
409 BlobAlreadyExists), but the probe that answers "absent" succeeds, and
so leaves an undeleteable 0-byte object at the backup key -- it destroys
the answer it returns. Closing the residual gap (a current-month repo
deleted while the previous month survives) therefore requires either
s3:ListBucket / blob read, or an explicit per-path expectation in the
chart. Both remain open; neither is needed for the reported failure.

Verified against the production shape on the rig -- rebom present for
both months, downloadable present only for the previous month: the
shipped binary reproduces the customer alert verbatim, and this build
emits zero ERROR lines. A base path absent in both months still alerts,
naming it. go test -race clean.

Co-Authored-By: Claude <noreply@anthropic.com>
ReARM-Agent: 1420896f-adf5-4843-896f-d863cfcc6528
ReARM-Agentic-Session: 94afdb9d-05d3-44ad-abcd-92e753ddf2df
ReARM-Agentic-Session: c378a7d7-3ac8-4441-8a9f-df5402cf6fcd
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant