Skip to content

Override PMM-T526's health start-period so the unhealthy verdict lands - #1181

Open
claude[bot] wants to merge 1 commit into
mainfrom
claude/kind-meitner-h2tynd
Open

Override PMM-T526's health start-period so the unhealthy verdict lands#1181
claude[bot] wants to merge 1 commit into
mainfrom
claude/kind-meitner-h2tynd

Conversation

@claude

@claude claude Bot commented Aug 13, 2026

Copy link
Copy Markdown
Contributor

Failures fixed (investigator)

  • source: Percona-Lab/pmm-submodules PR #4500 — run 31724325468, check CLI / Integration tests / CLI / Integration / pmm-server container (job 94528832645)
  • tests:
    • cli/tests/pmm-server-only.spec.ts:120 / @server-only — PMM-T526 "Use Invalid Prometheus Custom Config File to Check if Container is unhealthy"

What failed

pmm-server container was one of two red CLI jobs in that run. Its own Run CLI tests step
ends in || true, so the job went red one step later on launchable gate
(Actionable Failures | 1):

Error: Stdout does not contain unhealthy, !

expect(received).toContain(expected) // indexOf
Expected substring: "unhealthy"
Received string:    "d46c109374c7 perconalab/pmm-server-fb:PR-4500-77eee52 … Up 43 seconds (health: starting) … PMM-T526"

- Timeout 45000ms exceeded while waiting on the predicate
  at cli/tests/pmm-server-only.spec.ts:141:12

The container never reported unhealthy — it was still health: starting.

Root cause — Docker's start-period, not a product bug

percona/pmm#5768 (PMM-15331, newly added to the
PMM-15205 SEP stack this FB builds) widens the image's HEALTHCHECK --start-period from
25s to 720s, to cover grafana-sep's nominal 600s Grafana wait plus its bounded token
validation. Verified on the exact image CI used
(perconalab/pmm-server-fb:PR-4500-77eee52, digest sha256:f7b078d4…, the same digest
Launchable recorded for the failing session):

{"Test":["CMD","/opt/healthcheck.sh"],"Interval":4000000000,"Timeout":2000000000,"StartPeriod":720000000000,"Retries":3}

Docker does not increment FailingStreak during the start period, so it withholds the
unhealthy verdict for its whole duration. Observed directly on the FB image, after the same
broken-config-plus-restart the test performs — every probe exits 1, yet the streak never moves
and the status never leaves starting:

t=5s  status=starting logEntries=5 failingStreak=0
…
t=80s status=starting logEntries=5 failingStreak=0
--- last probe exit codes ---
1 ''
1 ''

So the product is behaving as #5768 intends and PMM still detects the broken
prometheus.base.yml correctly (readyz fails, every probe fails). Only the rendering of
that into a container-level unhealthy is now deferred by ~12.2 minutes (720s + 3 × 4s), well
past the test's 45s window. This is our test's assumption, not a regression.

The fix

Override only --health-start-period on this one container, back to the 25s the image used
before #5768, and widen the step's wait from 45s to 90s so the verdict window isn't marginal.
docker run --health-start-period replaces that single field; the image's own
/opt/healthcheck.sh still decides, so the test keeps asserting the real, user-visible
unhealthy transition rather than a proxy for it. Nothing is loosened — a server that stops
going unready on a broken Prometheus config still fails this test.

The alternative, waiting out the shipped 720s, would add ~12.5 min to a job that currently
runs in 2 min, and would break again on the next budget change.

Verification

Reproduced and fixed on a throwaway Linode VM following
.github/workflows/runner-integration-cli-tests.yml, with the failing run's own
PMM_SERVER_IMAGE=perconalab/pmm-server-fb:PR-4500-77eee52 and WIZARD_ARGS=--help:

  • Before (spec at main): 1 failed — same assertion, same message, Up 44 seconds (health: starting). Identical to CI.
  • After (this branch, same VM): 1 passed. With the override the container reaches
    unhealthy 35s after the restart, and pre-restart it still reaches healthy first, so
    there is no stale-verdict race across the restart.
  • Full @server-only suite on this branch against the FB image — the same 9 tests CI ran:
    9 passed (2.0m).
  • Not blocked on #5768. The override sets 25s, which is exactly what main ships today, so
    the fixed test passes on both. Confirmed against perconalab/pmm-server:3-dev-latest
    (StartPeriod: 25000000000): 1 passed. This is a normal PR, not a blocked draft.
  • npx eslint tests/pmm-server-only.spec.ts clean.

Not covered by this PR

The other red job in that run, CLI / Integration / Generic (PMM-T2237's 47-vs-48 summary-zip
count), is already tracked by the blocked draft #1166 — untouched here.

This PR no longer exercises the shipped 720s start-period end to end; whether a never-ready
server is eventually marked unhealthy at ~12 min is left to #5768's own review.


Generated by Claude Code

The server image widens HEALTHCHECK --start-period to 720s to cover SEP
provisioning's Grafana wait, and Docker keeps FailingStreak at 0 for the
whole start period, so a container with a broken prometheus.base.yml
reports 'health: starting' rather than 'unhealthy' for ~12 minutes.

Override only --health-start-period on this one container, leaving the
image's own healthcheck command to decide, and widen the wait to 90s.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RmR9KJ9cm6gTU2mP798crG
Signed-off-by: Claude <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant