Override PMM-T526's health start-period so the unhealthy verdict lands - #1181
Open
claude[bot] wants to merge 1 commit into
Open
Override PMM-T526's health start-period so the unhealthy verdict lands#1181claude[bot] wants to merge 1 commit into
claude[bot] wants to merge 1 commit into
Conversation
The server image widens HEALTHCHECK --start-period to 720s to cover SEP provisioning's Grafana wait, and Docker keeps FailingStreak at 0 for the whole start period, so a container with a broken prometheus.base.yml reports 'health: starting' rather than 'unhealthy' for ~12 minutes. Override only --health-start-period on this one container, leaving the image's own healthcheck command to decide, and widen the wait to 90s. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RmR9KJ9cm6gTU2mP798crG Signed-off-by: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Failures fixed (investigator)
CLI / Integration tests / CLI / Integration / pmm-server container(job 94528832645)cli/tests/pmm-server-only.spec.ts:120/@server-only— PMM-T526 "Use Invalid Prometheus Custom Config File to Check if Container is unhealthy"What failed
pmm-server containerwas one of two red CLI jobs in that run. Its ownRun CLI testsstepends in
|| true, so the job went red one step later onlaunchable gate(
Actionable Failures | 1):The container never reported
unhealthy— it was stillhealth: starting.Root cause — Docker's start-period, not a product bug
percona/pmm#5768 (PMM-15331, newly added to the
PMM-15205 SEP stack this FB builds) widens the image's
HEALTHCHECK --start-periodfrom25s to 720s, to cover
grafana-sep's nominal 600s Grafana wait plus its bounded tokenvalidation. Verified on the exact image CI used
(
perconalab/pmm-server-fb:PR-4500-77eee52, digestsha256:f7b078d4…, the same digestLaunchable recorded for the failing session):
Docker does not increment
FailingStreakduring the start period, so it withholds theunhealthyverdict for its whole duration. Observed directly on the FB image, after the samebroken-config-plus-restart the test performs — every probe exits 1, yet the streak never moves
and the status never leaves
starting:So the product is behaving as #5768 intends and PMM still detects the broken
prometheus.base.ymlcorrectly (readyzfails, every probe fails). Only the rendering ofthat into a container-level
unhealthyis now deferred by ~12.2 minutes (720s + 3 × 4s), wellpast the test's 45s window. This is our test's assumption, not a regression.
The fix
Override only
--health-start-periodon this one container, back to the 25s the image usedbefore #5768, and widen the step's wait from 45s to 90s so the verdict window isn't marginal.
docker run --health-start-periodreplaces that single field; the image's own/opt/healthcheck.shstill decides, so the test keeps asserting the real, user-visibleunhealthytransition rather than a proxy for it. Nothing is loosened — a server that stopsgoing unready on a broken Prometheus config still fails this test.
The alternative, waiting out the shipped 720s, would add ~12.5 min to a job that currently
runs in 2 min, and would break again on the next budget change.
Verification
Reproduced and fixed on a throwaway Linode VM following
.github/workflows/runner-integration-cli-tests.yml, with the failing run's ownPMM_SERVER_IMAGE=perconalab/pmm-server-fb:PR-4500-77eee52andWIZARD_ARGS=--help:main):1 failed— same assertion, same message,Up 44 seconds (health: starting). Identical to CI.1 passed. With the override the container reachesunhealthy35s after the restart, and pre-restart it still reacheshealthyfirst, sothere is no stale-verdict race across the restart.
@server-onlysuite on this branch against the FB image — the same 9 tests CI ran:9 passed (2.0m).mainships today, sothe fixed test passes on both. Confirmed against
perconalab/pmm-server:3-dev-latest(
StartPeriod: 25000000000):1 passed. This is a normal PR, not a blocked draft.npx eslint tests/pmm-server-only.spec.tsclean.Not covered by this PR
The other red job in that run,
CLI / Integration / Generic(PMM-T2237's 47-vs-48 summary-zipcount), is already tracked by the blocked draft #1166 — untouched here.
This PR no longer exercises the shipped 720s start-period end to end; whether a never-ready
server is eventually marked
unhealthyat ~12 min is left to #5768's own review.Generated by Claude Code