fix(cloud-backups): alert on uncovered base paths, not absent months - #15
Open
logicflakes wants to merge 1 commit into
Open
fix(cloud-backups): alert on uncovered base paths, not absent months#15logicflakes wants to merge 1 commit into
logicflakes wants to merge 1 commit into
Conversation
The previous commit's missing-target guard fired on customer prod on 2026-08-15 for rearm-artifacts/downloadable-artifacts-2026-08, and it was a false positive of exactly the kind this branch exists to remove. Why it fired. Under rolling months the tool does no discovery: it FABRICATES <base>-<current month> and <base>-<previous month> from the clock and attempts each, so absence is discovered by failing to back the target up. Meanwhile a monthly repository is never created explicitly -- registries create one implicitly on first push, and both writers derive the same name from the UTC clock at push time (rebom-backend ociService.getMonthlyRepositoryName, rearm-core ArtifactService getMonthlyRepositoryName). So a base path that receives no artifacts in a given month has no repository at all for that month. SBOMs land constantly, so rebom-artifacts always exists; downloadable artifacts only land when a release publishes one. The guard assumed a current-month target must exist, which is true for the first path and false for the second, and it would have paged every run until the month's first downloadable was published, then again each new month. What replaces it. The guard now asks the strongest question the tool's own credentials can actually answer: did this BASE PATH produce any backup at all? One month absent is indistinguishable from a quiet month; a base path producing nothing is unambiguous, and is what a renamed, deleted or mistyped path looks like. Under explicit paths a base path expands to exactly one target, so the strict behaviour there is unchanged by construction rather than by a special case. The current-vs-previous month distinction, and SkipIsExpected with it, are gone. An existence check against the destination was considered and rejected as unreachable, not merely undesirable. Distinguishing "never created" from "deleted" needs state the credential envelope does not contain: the registry cannot enumerate repositories without catalogue scope, and the destination cannot be read back under the write-only credential that storage.Provider's contract deliberately preserves. Conditional writes do leak existence using write permission alone (S3 If-None-Match, Azure 409 BlobAlreadyExists), but the probe that answers "absent" succeeds, and so leaves an undeleteable 0-byte object at the backup key -- it destroys the answer it returns. Closing the residual gap (a current-month repo deleted while the previous month survives) therefore requires either s3:ListBucket / blob read, or an explicit per-path expectation in the chart. Both remain open; neither is needed for the reported failure. Verified against the production shape on the rig -- rebom present for both months, downloadable present only for the previous month: the shipped binary reproduces the customer alert verbatim, and this build emits zero ERROR lines. A base path absent in both months still alerts, naming it. go test -race clean. Co-Authored-By: Claude <noreply@anthropic.com> ReARM-Agent: 1420896f-adf5-4843-896f-d863cfcc6528 ReARM-Agentic-Session: 94afdb9d-05d3-44ad-abcd-92e753ddf2df ReARM-Agentic-Session: c378a7d7-3ac8-4441-8a9f-df5402cf6fcd
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #14. That PR's missing-target guard is firing as a false positive on customer prod, and will keep doing so on every run until this lands.
#14 merged at 00:52 UTC that day; this alert is the merged code.
Why it fires
Under rolling months the tool does no discovery. It fabricates
<base>-<current month>and<base>-<previous month>from the clock and attempts each, learning of absence by failing to back the target up.Meanwhile a monthly repository is never created explicitly -- registries create one implicitly on first push, and both writers derive the same name from the UTC clock at push time (
rebom-backendociService.getMonthlyRepositoryName,rearm-coreArtifactService.getMonthlyRepositoryName). So a base path that receives no artifacts in a month has no repository for that month at all.SBOMs land constantly, so
rebom-artifacts-2026-08always exists. Downloadable artifacts only land when a release publishes one -- none had been published in August, so that repo had never been created. #14's guard assumed a current-month target must exist: true for the first path, false for the second. It would page every run until the month's first downloadable, then again every new month, for any path without guaranteed monthly traffic.The fix
Alert per base path, not per target: did this base path produce any backup at all?
One month absent is indistinguishable from a quiet month. A base path producing nothing is unambiguous, and is exactly what a renamed, deleted or mistyped path looks like. Under explicit paths a base path expands to one target, so the strict behaviour there is preserved by construction rather than by a special case.
SkipIsExpectedand the whole current-vs-previous month distinction are deleted.Why not an existence check
Considered seriously and rejected as unreachable, not merely undesirable. Distinguishing "never created" from "deleted" needs state the credential envelope does not contain:
storage.Provider's contract deliberately preserves.If-None-Match: *-> 412, Azure -> 409BlobAlreadyExists) -- but the probe that answers "absent" succeeds, leaving an undeleteable 0-byte object at the real backup key. It destroys the answer it returns.If-Match: *is worse: it overwrites a live backup. A marker-key variant fails identically.Worth recording for whoever revisits that decision: on Azure the write-only ideal in the
Providercomment is not achievable with RBAC anyway -- there is no built-in write-only blob role, so the SP must hold Storage Blob Data Contributor, which already grants read. A probe would be zero-elevation on that deployment; the constraint is real only for S3.Residual gap, stated deliberately
A current-month repo deleted while the previous month survives will not alert -- because with the available information that is indistinguishable from a quiet month. Closing it needs either
s3:ListBucket/ blob read, or an explicit per-pathrequireMonthlyflag in the chart. Both remain open; neither is needed for the reported failure.Verification
Rig reproduces the production shape exactly (rebom present for both months, downloadable present only for the previous month):
b5e60ae)go build/go vet/go test -race ./...clean, gofmt clean, added lines pure ASCII.Note on the last check: my first attempt to prove the alert still fires used
curl -X DELETEagainstregistry:2, which returns 405 because delete is disabled by default -- the repo never went away, so that run's "0 errors" was correct but meaningless. Re-verified with a base path that genuinely has no repositories.🤖 Generated with Claude Code