PMM-15394 Let HAProxy start without waiting for every PMM replica

[theTibi]

PMM-15394

Removes the HAProxy init container. That is the whole change — init-addr last,libc,none is already in haproxy.cfg today, so nothing is added.

The gate turns a healthy cluster into no ingress

templates/haproxy-init-configmap.yaml required every PMM replica to answer HTTP 200 before HAProxy could start, then exit 1 after 60 attempts at 5s. The pod CrashLoopBackOffs at Init:0/1 and HAProxy never starts.

Every fresh install races this, and the chart guarantees it will. The StatefulSet has no podManagementPolicy, so it defaults to OrderedReady: pmm-ha-1 is created only once pmm-ha-0 is Ready, and pmm-ha-2 only once pmm-ha-1 is. For most of the five-minute budget the later replicas do not exist — no pod, no IP, no DNS record — and curl returns 000. PMM Server is a multi-GB image that also runs database migrations on first boot, so three replicas started one after another routinely exceed it.

PMM-15202 is this failure: all three HAProxy pods at Init:0/1 with ~100 restarts over 11 hours. Closed as Not a Bug after the reporter reinstalled with more resources.

It was also redundant

server-template pmm 1-N monitoring-service.<ns>.svc.cluster.local:8443 \
  check ssl verify none resolvers k8s init-addr last,libc,none

init-addr … none holds a slot with no address instead of aborting startup when the name does not resolve; resolvers k8s picks it up later; http-check expect status 200 against /v1/server/leaderHealthCheck brings it UP only once it is genuinely serving. HAProxy running before any PMM pod exists is the designed behaviour — it returns 503 until a backend is up, which beats not running.

The init container added no capability. It only converted "HAProxy up, no backends yet" into "no HAProxy at all".

Verified on a live cluster

Before — HAProxy was already serving with 7 of 10 slots holding unresolvable names:

pmm1     MAINT (resolution)
pmm3     UP                    ← the Raft leader
pmm5..10 MAINT (resolution)
BACKEND  UP

The gate, run unmodified from the live ConfigMap against two absent replicas:

✅ Instance 0 is ready (1/5)   ✅ Instance 1 (2/5)   ✅ Instance 2 (3/5)
❌ Connection check failed for pmm-ha2-3... (HTTPS status: 000)
❌ Connection check failed for pmm-ha2-4... (HTTPS status: 000)
❌ Timeout: PMM instances did not become ready within 5 minutes  →  exit 1

Three healthy replicas serving, and it still fails.

After — deleted a follower and restarted HAProxy in the same moment:

pmm-ha-haproxy-...-4q9qx   1/1  Running   8s     ← up while a replica was gone
pmm-ha-haproxy-...-rqw85   1/1  Running   10s
pmm-ha2-2                  2/2  Terminating

Rollout complete in ~47s, zero restarts, readyz and leaderHealthCheck 200 throughout. A follower was targeted deliberately so Raft quorum held at 2/3.

Changes

File	Change
templates/haproxy-init-configmap.yaml	deleted
values.yaml	drop haproxy.initContainers and haproxy.extraVolumes — that volume only ever fed the script
Chart.yaml	version bump

Verified locally: helm lint clean, rendered manifests drop 38 → 37 (the removed ConfigMap), the HAProxy Deployment renders no initContainers, and both init-addr last,libc,none and the leader health check are still present.

Notes for reviewers

Version. 1.6.0 → 1.6.2. .github/ct.yaml sets target-branch: main, so ct lint --check-version-increment=true compares against main, which is already at 1.6.1 — a bump to 1.6.1 here would fail the check, and 1.6.0 would read as a decrement. Worth noting that #935 currently carries no Chart.yaml bump and last passed CI before main moved to 1.6.1; it will likely hit the same check on its next run.

Interaction with #936. That PR (targeting main) fixes a line in the file this one deletes — its HAProxy readiness probe addressed pods by .Release.Name rather than the StatefulSet name. Whichever lands second resolves trivially in favour of the deletion.

Test coverage. The helm unittest harness lives in #936 against main. Once the bases converge I will add a suite asserting HAProxy renders no init container and that init-addr … none plus the leader health check stay in haproxy.cfg, so this cannot silently come back.

Also fixed for free

The init container was alpine:latest running apk add --no-cache curl at every start, so HAProxy could not start without egress to the Alpine package mirrors. Air-gapped and restricted clusters are affected.

Not in scope

podManagementPolicy: Parallel would remove the serialized startup that makes the budget impossible in the first place, and is arguably right for a Raft cluster where pmm-ha-0 currently has to reach Ready with no peers. It is an immutable StatefulSet field, so flipping it breaks helm upgrade on every existing install and needs a documented --cascade=orphan recreate. Worth its own ticket; this change does not depend on it.

🤖 Generated with Claude Code