Skip to content

PMM-14989 Retain state on client pod restart #954

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
ademidoff wants to merge 21 commits into from
+17 −4
Open

PMM-14989 Retain state on client pod restart #954

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
7216cca
chore: add PMM_HA_NAMESPACE to statefulset
fabio-silva Aug 10, 2026
5f72753
chore: use Downward API to expose namespace to pods
fabio-silva Aug 10, 2026
d61f8b3
Merge pull request #924 from percona/PMM-13860-finalize-ha-overview-d…
fabio-silva Aug 11, 2026
71e656d
Merge pull request #925 from percona/PMM-15309-add-ch-datasource-user
ademidoff Aug 16, 2026
331ccf5
Merge remote-tracking branch 'origin/main' into PMM-HA-GA
ademidoff Sep 3, 2026
c94c0ef
Update secret retrieval command in values.yaml
ademidoff Sep 3, 2026
0766844
PMM-14665 Mark the PMM PostgreSQL cluster nodes as internal (#913)
ademidoff Sep 3, 2026
267b0b7
Fix typo in PMM Client section of values.yaml
ademidoff Sep 3, 2026
f6089a6
docs: clarify pmmClient replica guidance
Copilot Sep 3, 2026
3f0694e
PMM-15441 Fit the pmm-ha CI install back onto the Kind node (#946)
theTibi Sep 3, 2026
e15035d
PMM-15151: make PG and ClickHouse operators watch all namespaces (#868)
theTibi Sep 3, 2026
1832c26
PMM-15149: support installing PMM-HA into multiple namespaces (#865)
theTibi Sep 3, 2026
88ca133
PMM-14403 Finalize the HA resource limitations (#935)
theTibi Sep 3, 2026
924116e
PMM-15394 Let HAProxy start without waiting for every PMM replica (#937)
theTibi Sep 3, 2026
bd385e9
PMM-15393 Let HAProxy scale beyond the worker-node count (#938)
theTibi Sep 4, 2026
1b3de77
PMM-15301 Disable QAN for PMM HA's bundled PostgreSQL cluster (#940)
theTibi Sep 4, 2026
f37bb12
PMM-14989 Document the PMM Client chart parameters
ademidoff Sep 4, 2026
6d80155
PMM-14989 Document the PMM Client chart parameters
ademidoff Sep 4, 2026
683312b
Enhance pmmClient.forceRegistration description
ademidoff Sep 7, 2026
eee5963
PMM-14899 Refactor initContainer registration logic for PMM Client to…
ademidoff Sep 7, 2026
fa4dfac
Merge branch 'PMM-14989-retain-state-on-client-pod-restart' of ssh://…
ademidoff Sep 7, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 17 additions & 0 deletions charts/pmm-ha/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -349,6 +349,23 @@ To create additional service tokens manually, see the [PMM documentation on serv
| `readyProbeConf.failureThreshold` | When a probe fails, Kubernetes will try failureThreshold times before giving up | `6` |


### PMM Client

| Name | Description | Value |
| ------------------------------------ | ---------------------------------------------------------------- | ---------------------------------------------------------------------------------------------- |
| `pmmClient.replicas` | Number of PMM Client pods carrying the delegated monitoring | `3` |
| `pmmClient.image.repository` | PMM Client image repository | `percona/pmm-client` |
| `pmmClient.image.pullPolicy` | PMM Client image pull policy | `IfNotPresent` |
| `pmmClient.image.tag` | PMM Client image tag, defaults to the chart appVersion | `3.9.1` |
| `pmmClient.forceRegistration` | Register the Node even when one with the same name exists (WARNING: PMM Server may remove the existing Node and its configured Services) | `false` |
| `pmmClient.storage.size` | Size of the volume holding the Agent identity and metrics buffer | `2Gi` |
| `pmmClient.storage.storageClassName` | Storage class of that volume, cluster default if empty | `""` |
| `pmmClient.resources` | Resources requested for the PMM Client container | `{"requests": {"memory": "200Mi", "cpu": "100m"}, "limits": {"memory": "1Gi", "cpu": "500m"}}` |
| `pmmClient.nodeSelector` | Node labels for the PMM Client pods | `{}` |
| `pmmClient.tolerations` | Tolerations for the PMM Client pods | `[]` |
| `pmmClient.affinity` | Affinity rules for the PMM Client pods | `{}` |


### PMM secrets

| Name | Description | Value |
Expand Down
4 changes: 0 additions & 4 deletions charts/pmm-ha/templates/pmm-client-statefulset.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -54,10 +54,6 @@ spec:
- -c
- |
set -e
if [ -s "${PMM_AGENT_CONFIG_FILE}" ]; then
echo "This Node is registered already, pmm-agent keeps its identity."
exit 0
fi
Comment on lines -57 to -60

@JiriCtvrtka JiriCtvrtka Sep 9, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These four lines are what make a client pod restart survivable — removing them regresses the case the PR title names.

With the default forceRegistration: false, the init container now runs pmm-agent setup on every pod recreation (a helm upgrade touching the pod template, node drain, eviction, scale down-and-up). The node name $(POD_NAMESPACE)-$(POD_NAME) is stable and the PVC is retained, so pmm-managed returns AlreadyExists (managed/services/management/node.go:45-47) and setup calls os.Exit(1) (agent/commands/setup.go:163-164). The pod sets no restartPolicy, so kubelet retries the init container forever: Init:CrashLoopBackOff, the pmm-client container never starts, every Service on that Node goes dark.

forceRegistration: true isn't an escape — this gate ran before that flag was consulted, which is what confined it to the lost-volume recovery case it documents. Without it, Reregister: true makes pmm-managed run models.RemoveNode(..., models.RemoveCascade) (node.go:48) on every pod start, dropping the Node with every Service and Agent on it. So neither value works: false wedges the pod, true silently discards the delegated services.

The pinned image can't cover for it either: values.yaml:157 pins tag: "3.9.1", and v3.9.1:agent/commands/setup.go still has the bare if !cfg.Setup.SkipRegistration { register(cfg, l) }. PMM-15260 is unmerged.

Suggest restoring the gate and landing the removal separately, against a release that carries PMM-15260 with pmmClient.image.tag bumped to match. Note that PMM_AGENT_SETUP_SKIP_REGISTRATION=1 isn't a safe substitute: the admin→service_token swap happens inside register() and env overrides the stored file, so skipping it persists server.username: admin plus the admin password in plaintext to the volume.

Also worth updating in the same commit, since they document the behavior this removes: the comment at lines 44-48 ("only happens when the configuration file is missing") and values.yaml:166-167 ("a restarted pod keeps its Node and the Services configured on it"). And the PR body says "No template, value or default is changed here", which doesn't match this hunk.

echo "Registering ${PMM_AGENT_SETUP_NODE_NAME} with ${PMM_AGENT_SERVER_ADDRESS}..."
pmm-agent setup
env:
Expand Down
Loading