Skip to content

PMM-14738 Validate PMM HA replica counts and publish the supported range - #936

Open
theTibi wants to merge 6 commits into
percona:PMM-HA-GAfrom
theTibi:PMM-14738
Open

PMM-14738 Validate PMM HA replica counts and publish the supported range#936
theTibi wants to merge 6 commits into
percona:PMM-HA-GAfrom
theTibi:PMM-14738

Conversation

@theTibi

@theTibi theTibi commented Aug 27, 2026

Copy link
Copy Markdown
Contributor

PMM-14738

Adds the validation and documentation for PMM HA scaling that the chart never had, plus the test suite that guards it.

Nothing checked the two values that can break a cluster

Rendered at replicas 1 through 12, everything was accepted silently:

replicas Raft peers HAProxy slots Warning
2 2 10 none
4 4 10 none
12 12 10 none
  • Even counts. Raft elects by majority, so 4 replicas need more votes to elect a leader without surviving more losses than 3 do, and replicas=2 survives none — one pod restart stops the cluster.
  • Above maxReplicas. HAProxy renders server-template pmm 1-<maxReplicas> and fills the slots from a headless-service DNS answer in arbitrary order, then marks a backend UP only when it answers /v1/server/leaderHealthCheck. A pod without a slot is invisible to HAProxy — and if the Raft leader lands there, every backend is DOWN and PMM serves 503. Not a degraded pod: an outage.

pmm.replicas.validate and pmm.keeper.validate now reject both, plus malformed input. The regex is strict on purpose: sprig's int parses base 0, so "010" would have validated as 8, and anything wider than int64 overflows to 0 — either way the error would quote a number nobody typed.

Suggestions are clamped to the ceiling, so the error never names a value the next check rejects:

replicas=2   Use 1 or 3.
replicas=4   Use 3 or 5.
replicas=10  Use 9.
replicas=12  maxReplicas is 10, so the largest supported value is 9.

clickhouse.keeper.replicasCount is a Raft ensemble too. values.yaml has always said it "should be odd"; that is now enforceable rather than advisory.

Smaller fixes

  • maxReplicas had its | default 10 duplicated between the validator and haproxy-configmap.yaml. Both now read pmm.maxReplicas, so they cannot drift — and because default treats 0 as empty, maxReplicas: 0 used to become 10 silently.
  • The workflow's paths: filter did not include the workflow file, so a PR editing only its assertions ran no job.
  • azure/setup-helm v4.2.0 / Helm 3.15.4 -> v5.0.0 / Helm 3.21.0, which is what the helm-unittest plugin needs to load.

Tests

charts/pmm-ha/tests/ — 27 helm-unittest cases, no cluster required, runnable with make test-pmm-ha:

Test Suites: 3 passed, 3 total
Tests:       27 passed, 27 total

Scoping a suite to templates/statefulset.yaml skips the lookup in pg-user-credentials-secrets.yaml entirely, so these do not depend on Helm's template ordering the way a helm template assertion would.

Mutation-tested against the merged tree — every regression is caught:

Mutation Result
revert pmm.haPeers to .Release.Name 2 failed
revert the server-template slot count to a literal 10 2 failed
drop the even-count check 5 failed
make pmm.validate.oddCount always reject 20 failed, 5 errored
drop the maxReplicas ceiling check 2 failed

Merged with PMM-HA-GA

Merging the current base branch dropped two changes this PR originally carried, because equivalent work landed first:

  • The PMM_HA_PEERS fix. pmm.haPeers built peer hostnames from .Release.Name while the StatefulSet is named after pmm.fullname, and the two are equal only when the release name already contains the chart name — so helm install pmm-prod percona/pmm-ha gave Raft peers that never resolved. PMM-15151: make PG and ClickHouse operators watch all namespaces #868 (PMM-15151) shipped the same fix, so there is nothing left to change here. tests/ha-peers_test.yaml stays as the regression guard, including a case asserting that a release named pmm-ha still renders byte-identically.
  • ci/ci-values.yaml. This PR removed clickhouse.keeper.replicas, a key nothing reads (the template reads replicasCount). PMM-15441 Fit the pmm-ha CI install back onto the Kind node #946 (PMM-15441) rewrote the file and already sets replicasCount.

templates/haproxy-init-configmap.yaml was deleted by #937 (PMM-15394), and this PR's pmm.fullname fix to its host loop went with it. The chart version conflict resolved to 1.7.0, keeping appVersion "3.9.1" from #937.

Upgrade impact

Breaking for anyone running an even replicas. 2 or 4 was accepted before and now fails the render; those users must set an odd value in the same helm upgrade, which recreates the pods. The README carries an upgrade note.

Chart 1.6.2 -> 1.7.0. Reviewers: a case can be made for 2.0.0 — a ^1.6.2 consumer will otherwise adopt a render-breaking change automatically. Happy to bump if you prefer.

Related

Raft elects by majority, so an even replica count needs more votes to elect a
leader without surviving more failures, and replicas=2 survives none. HAProxy
renders only maxReplicas server-template slots and marks a backend UP only when
it answers /v1/server/leaderHealthCheck, so a replica above that ceiling is
invisible to HAProxy - and if the Raft leader lands there, no backend is UP at
all. Neither was checked or documented.

pmm.replicas.validate and pmm.keeper.validate reject both, along with malformed
input: sprig's int parses base 0, so "010" would have validated as 8, and
anything wider than int64 overflows to 0.

Also fixes pmm.haPeers and the HAProxy readiness probe, which built pod
hostnames from .Release.Name while the StatefulSet is named after pmm.fullname.
The two are equal only when the release name already contains the chart name, so
any other release never formed a quorum. Release "pmm-ha" renders identically,
so existing installs see no change.

maxReplicas now has a single definition, pmm.maxReplicas, shared by the
validator and the ConfigMap that consumes it. ci-values.yaml set
clickhouse.keeper.replicas, a key nothing reads, so CI ran three Keepers instead
of one.

Covered by helm unittest suites under charts/pmm-ha/tests: 28 cases, no cluster
required, runnable with make test-pmm-ha.

Signed-off-by: theTibi <tkorocz@gmail.com>
The helm-unittest plugin.yaml now carries a platformHooks field that Helm 3.15
cannot parse, so the plugin installs but is unusable and make helm-unittest
exits 1. test.yaml already runs its unittest jobs on azure/setup-helm@v5.0.0
with Helm v3.21.0; match it rather than pinning the plugin to an older release,
since the Makefile target is shared with the pxc suites.

Signed-off-by: theTibi <tkorocz@gmail.com>
percona#868 (PMM-15151) and percona#919 (PMM-13858) carry the same .Release.Name ->
pmm.fullname fix for the Raft peer list. All three were already functionally
identical; this PR differed only in where the $fullname assignment sat and in
the wording of the comment, which made _helpers.tpl conflict four ways for no
functional reason.

Identical text on both sides of a merge resolves cleanly, so the three PRs no
longer collide here whichever one lands first. tests/ha-peers_test.yaml still
covers the fix.

Signed-off-by: theTibi <tkorocz@gmail.com>
percona#937 (PMM-15394) removes templates/haproxy-init-configmap.yaml and the
haproxy.initContainers readiness gate. This PR referenced it in three places,
so merging both would have failed the ha-peers suite and left stale docs:

- tests/ha-peers_test.yaml: dropped the template from the suite and the
  data["wait-for-pmm.sh"] assertion.
- values.yaml, README.md: the HAProxy ConfigMap note now describes the one
  ConfigMap left after percona#937 - haproxy.cfg, which carries maxReplicas.

After percona#937 only one value is embedded in an HAProxy ConfigMap, so the guidance
gets simpler rather than weaker.

Signed-off-by: theTibi <tkorocz@gmail.com>
percona#938 (PMM-15393) removes the required podAntiAffinity that capped
haproxy.replicaCount at the worker-node count, so stating that cap here becomes
wrong the moment percona#938 lands - and it conflicted with percona#938, which leaves this
line at its original text and documents the new behaviour beside it.

Restores the base wording.

Signed-off-by: theTibi <tkorocz@gmail.com>
Two conflicts, both from work that landed on PMM-HA-GA after this branch forked:

- charts/pmm-ha/Chart.yaml: percona#937 bumped 1.6.1 -> 1.6.2 and appVersion to "3.9.1"
  while this branch bumped to 1.7.0. Kept 1.7.0 (still ahead of 1.6.2, and the
  render-breaking even-replicas check justifies the minor) with appVersion "3.9.1".

- charts/pmm-ha/templates/haproxy-init-configmap.yaml: percona#937 (PMM-15394) deleted the
  file; this branch had fixed its pmm_host loop to use pmm.fullname. Took the
  deletion - the fix has nothing left to apply to. Nothing references the ConfigMap,
  the init container or wait-for-pmm.sh any more.

Silent, non-conflicting overlaps checked by hand:

- _helpers.tpl pmm.haPeers: percona#868 (PMM-15151) landed the same .Release.Name ->
  pmm.fullname fix, and this branch had already adopted that block verbatim, so the
  merge is a no-op there. The peer-addressing suite stays as a regression guard.
- README.md: dropped the upgrade bullet announcing the PMM_HA_PEERS change. It ships
  in 1.6.2 via percona#868, so attributing it to 1.7.0 is now wrong. The even-replicas
  breaking change is unchanged.
- ci/ci-values.yaml: percona#946 (PMM-15441) rewrote the file and already sets
  clickhouse.keeper.replicasCount, so this branch's fix for the ignored
  clickhouse.keeper.replicas key is subsumed.
- The new examples/ and docs/SIZING.md from percona#935 set no PMM replicas value, so the
  new validators do not reject them.

helm unittest: 3 suites, 27 tests passed. Renders verified against the merged tree -
replicas=4, replicas=12 and clickhouse.keeper.replicasCount=2 all fail with the
intended messages.

Signed-off-by: theTibi <tkorocz@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant