PMM-15441 Fit the pmm-ha CI install back onto the Kind node - #946
Merged
Conversation
The pmm-client StatefulSet added in PMM-14665 (percona#913) has no override in ci/ci-values.yaml, so CI runs 3 Client pods at 100m CPU each. On the single-node Kind cluster that leaves nothing for the third PMM replica: pmm-ha-<release>-2 stays Pending with "Insufficient cpu" and the install hits the 600s timeout. Cap the Clients at 1 and shrink their requests. Also fix clickhouse.keeper.replicas, a key the chart never reads - it is replicasCount - so CI has been running 3 Keepers instead of 1 since January. Together this frees ~450m of requests; the stuck replica needs 200m. Signed-off-by: theTibi <tkorocz@gmail.com>
theTibi
requested review from
JiriCtvrtka and
ademidoff
and removed request for
a team
September 3, 2026 19:58
theTibi
added a commit
to theTibi/percona-helm-charts
that referenced
this pull request
Sep 3, 2026
Two conflicts, both from work that landed on PMM-HA-GA after this branch forked: - charts/pmm-ha/Chart.yaml: percona#937 bumped 1.6.1 -> 1.6.2 and appVersion to "3.9.1" while this branch bumped to 1.7.0. Kept 1.7.0 (still ahead of 1.6.2, and the render-breaking even-replicas check justifies the minor) with appVersion "3.9.1". - charts/pmm-ha/templates/haproxy-init-configmap.yaml: percona#937 (PMM-15394) deleted the file; this branch had fixed its pmm_host loop to use pmm.fullname. Took the deletion - the fix has nothing left to apply to. Nothing references the ConfigMap, the init container or wait-for-pmm.sh any more. Silent, non-conflicting overlaps checked by hand: - _helpers.tpl pmm.haPeers: percona#868 (PMM-15151) landed the same .Release.Name -> pmm.fullname fix, and this branch had already adopted that block verbatim, so the merge is a no-op there. The peer-addressing suite stays as a regression guard. - README.md: dropped the upgrade bullet announcing the PMM_HA_PEERS change. It ships in 1.6.2 via percona#868, so attributing it to 1.7.0 is now wrong. The even-replicas breaking change is unchanged. - ci/ci-values.yaml: percona#946 (PMM-15441) rewrote the file and already sets clickhouse.keeper.replicasCount, so this branch's fix for the ignored clickhouse.keeper.replicas key is subsumed. - The new examples/ and docs/SIZING.md from percona#935 set no PMM replicas value, so the new validators do not reject them. helm unittest: 3 suites, 27 tests passed. Renders verified against the merged tree - replicas=4, replicas=12 and clickhouse.keeper.replicasCount=2 all fail with the intended messages. Signed-off-by: theTibi <tkorocz@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Jira: https://perconadev.atlassian.net/browse/PMM-15441
Problem
Every PR against
PMM-HA-GAis red onPMM-HA PR checks / lint-test. Lint itself passes (1 chart(s) linted, 0 chart(s) failed) — the failing step isRun chart-testing (install pmm-ha):The Kind cluster is a single ~4 vCPU node. The PMM StatefulSet is ordered, so replica
-2is only created ~6 minutes in, by which point every other pod has claimed the node's CPU. It staysPendingand Helm's--timeout 600sexpires.Cause
0766844(PMM-14665, #913) addedcharts/pmm-ha/templates/pmm-client-statefulset.yamlwithpmmClient.replicas: 3atcpu: 100meach, and no matching override incharts/pmm-ha/ci/ci-values.yaml— +300m of requests on a node with no headroom. #913's own branch was already failing with this exact error before it merged (runs on 2026-08-30 and 2026-09-03).Contributing:
ci/ci-values.yamlsetsclickhouse.keeper.replicas: 1, but the chart readsclickhouse.keeper.replicasCount(templates/clickhouse-keeper.yaml#L10). That override has been a silent no-op sincea116efb(2026-01-09), so CI has been running 3 Keepers instead of 1 — another +200m.Base branch was green at
c94c0ef(06:49 UTC, 2026-09-03) and red from0766844(07:27 UTC) onward.Fix
CI values only — chart defaults are unchanged. The Client pod's init container has hardcoded 50m requests, so a pod's effective request is
max(init, containers):The stuck replica needs 200m.
Not the fix: raising the
cttimeout (the pod isPending, it would never schedule), or a multi-node Kind cluster (each Kind node reports the host's full 4 vCPU, so that would over-subscribe and trade a hard failure for a starvation flake).Verification
yqconfirms every override now resolves to a key the chart actually reads —pmmClient.replicas,pmmClient.resources(pmm-client-statefulset.yaml#L13,L107) andclickhouse.keeper.replicasCount— and that the deadclickhouse.keeper.replicaskey is gone. The real check is this PR's ownlint-testrun going green.Unblocks #865, #868, #919, #935, #936, #937, #938, #940, #941, #944.