PMM-14956 HA alert templates - #5730
Draft
4nte wants to merge 14 commits into
Draft
Conversation
WalkthroughPMM HA now normalizes peer configuration, exposes the expected node count through HA metrics, and adds five alert templates for cluster health conditions. Documentation describes template setup, alert-rule creation, remediation, and detection limitations. ChangesPMM HA alerting and metrics
Possibly related PRs
🚥 Pre-merge checks | ✅ 4✅ Passed checks (4 passed)
Comment |
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #5730 +/- ##
==========================================
+ Coverage 43.59% 45.21% +1.61%
==========================================
Files 415 218 -197
Lines 43134 27867 -15267
==========================================
- Hits 18804 12599 -6205
+ Misses 22454 13915 -8539
+ Partials 1876 1353 -523
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
catalinaadam
temporarily deployed
to
PMM-14956-ha-alert-templates - pmm-doc-3.9.0 PR #5730
August 4, 2026 19:12 — with 
Render
Destroyed
Render
Destroyed
Contributor
|
@coderabbitai full review |
✅ Action performedFull review finished. |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@managed/cmd/pmm-managed/main.go`:
- Around line 1245-1256: Update parseHAPeers to accept the existing
*logrus.Entry main logger, and replace the package-level logrus.Warnf duplicate
warning with a structured field for the peer while preserving the warning
message and duplicate-handling behavior. Update its caller to pass main.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 83f38b89-81fb-4540-b835-65783dc99e13
📒 Files selected for processing (13)
documentation/docs/alert/templates_list.mddocumentation/docs/install-pmm/install-HA-clustered.mdmanaged/cmd/pmm-managed/main.gomanaged/cmd/pmm-managed/main_test.gomanaged/data/alerting-templates/ha_leader_flapping.ymlmanaged/data/alerting-templates/ha_no_leader.ymlmanaged/data/alerting-templates/ha_node_unreachable.ymlmanaged/data/alerting-templates/ha_quorum_at_risk.ymlmanaged/data/alerting-templates/ha_split_brain.ymlmanaged/services/ha/ha.gomanaged/services/ha/ha_metrics.gomanaged/services/ha/ha_metrics_test.gomanaged/services/ha/haservice.go
Comment on lines
+1245
to
+1256
| func parseHAPeers(peers string) []string { | ||
| var nodes []string | ||
| seen := make(map[string]struct{}) | ||
|
|
||
| for node := range strings.SplitSeq(peers, ",") { | ||
| node = strings.TrimSpace(node) | ||
| if node == "" { | ||
| continue | ||
| } | ||
| if _, ok := seen[node]; ok { | ||
| logrus.Warnf("Ignoring duplicate entry %q in PMM_HA_PEERS.", node) | ||
| continue |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟠 Major | ⚡ Quick win
Use a structured logrus.Entry for duplicate-peer warnings.
Line 1255 uses the package-level logrus.Warnf. Pass the existing main entry to parseHAPeers. Log the peer as a field so the message retains component context and supports structured filtering.
As per coding guidelines, use “structured logging, such as s.l.WithField("key", value).Error("message"), and pass *logrus.Entry rather than *logrus.Logger.”
Proposed fix
- nodes := parseHAPeers(*haPeers)
+ nodes := parseHAPeers(l, *haPeers)
-func parseHAPeers(peers string) []string {
+func parseHAPeers(l *logrus.Entry, peers string) []string {
...
- logrus.Warnf("Ignoring duplicate entry %q in PMM_HA_PEERS.", node)
+ l.WithField("peer", node).Warn("Ignoring duplicate entry in PMM_HA_PEERS.")📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| func parseHAPeers(peers string) []string { | |
| var nodes []string | |
| seen := make(map[string]struct{}) | |
| for node := range strings.SplitSeq(peers, ",") { | |
| node = strings.TrimSpace(node) | |
| if node == "" { | |
| continue | |
| } | |
| if _, ok := seen[node]; ok { | |
| logrus.Warnf("Ignoring duplicate entry %q in PMM_HA_PEERS.", node) | |
| continue | |
| func parseHAPeers(l *logrus.Entry, peers string) []string { | |
| var nodes []string | |
| seen := make(map[string]struct{}) | |
| for node := range strings.SplitSeq(peers, ",") { | |
| node = strings.TrimSpace(node) | |
| if node == "" { | |
| continue | |
| } | |
| if _, ok := seen[node]; ok { | |
| l.WithField("peer", node).Warn("Ignoring duplicate entry in PMM_HA_PEERS.") | |
| continue |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@managed/cmd/pmm-managed/main.go` around lines 1245 - 1256, Update
parseHAPeers to accept the existing *logrus.Entry main logger, and replace the
package-level logrus.Warnf duplicate warning with a structured field for the
peer while preserving the warning message and duplicate-handling behavior.
Update its caller to pass main.
Source: Coding guidelines
Both halves of the expression now sit behind an 'and on() (count < expected)' gate, so removing a node on purpose and updating PMM_HA_PEERS clears the alert instead of leaving it stuck for the rest of the 6h lookback. The gate has to stay a bare comparison: '< bool' always yields a series, and 'and' filters on existence rather than value. Correct the partition note in the docs accordingly: a surviving-majority partition fires none of these alerts, because the isolated node keeps writing metrics and keeps reporting itself as a voter from its local Raft membership. Leader flapping is what actually fires there. Signed-off-by: Ante Gulin <ante.gulin@percona.com>
Add templates for VictoriaMetrics, ClickHouse, Grafana and qan-api2, each firing on the up metric PMM already scrapes. The VictoriaMetrics selector covers both the standalone job and the vmselect/vminsert/vmstorage/vmagent jobs used in a clustered deployment. Document them under a new "PMM internal component templates" section, including the two blind spots: VictoriaMetrics and Grafana cannot observe their own total failure, since they respectively serve and evaluate the rules. Signed-off-by: Ante Gulin <ante.gulin@percona.com>
Grafana may be reading the provisioning directory while pmm-managed writes to it, so the file has to be replaced rather than truncated and rewritten. Nothing in pmm-managed did temp-file-plus-rename before this. Signed-off-by: Ante Gulin <ante.gulin@percona.com>
Applying a provisioning change means restarting Grafana, and deciding whether to means knowing whether it is running. reload() cannot be reused: it is reread plus update, a no-op when the unit file itself has not changed. supervisorctl status exits non-zero for a stopped program, so the status is parsed from its output rather than its exit code. Signed-off-by: Ante Gulin <ante.gulin@percona.com>
Adds the frozen catalog of nine rules across two bundles and the renderer that turns the shipped templates into a Grafana alerting provisioning file, with golden fixtures. The UIDs are frozen and append-only: removing one leaves the rule behind forever on every server that ever created it. The catalog is validated against the invariants Grafana enforces at startup, where a violation stops Grafana booting rather than reporting a bad rule. Signed-off-by: Ante Gulin <ante.gulin@percona.com>
Every node renders and writes the file; only the leader restarts Grafana. Alert rules live in the Grafana database all nodes share, so one ingestion serves the cluster, but a node that stopped writing would revert it the next time its own Grafana restarted. The Metrics datasource UID cannot be guessed. Grafana only began deriving it from the datasource name in 8.3.4, and PMM shipped older Grafanas until 2.28.0, so servers first installed before then store a random one. A wrong UID leaves rules reporting healthy while querying nothing, so resolution fails closed: a missing row or absent table means a first boot and derives, anything else refuses to render. A rule UID is only PMM's while PMM provisions it, so UIDs owned by someone else are omitted from both the group and the deletions. Claiming one back kills Grafana on a provenance mismatch, and deleting one silently destroys a user's rule. Metrics report what PMM did, not what Grafana holds: PMM cannot know a file was ingested and does not claim to. Signed-off-by: Ante Gulin <ante.gulin@percona.com>
PMM_ENABLE_HA_ALERTS and PMM_ENABLE_COMPONENT_ALERTS, both defaulting to true and read once at start, following the PMM_HA_ENABLE pattern. This breaks the PMM_ENABLE_* family convention, where every other variable has a settings, API and UI presence. It is a deliberate trade: the settings half was dropped to keep the change small, and a bundle toggle is not something that has to change at runtime. Signed-off-by: Ante Gulin <ante.gulin@percona.com>
/srv is writable under an arbitrary UID and persists across container recreation, which the image's own provisioning directory does not. Grafana ships that directory non-empty, and ansible will not convert a directory into a symlink, so it is removed first. Signed-off-by: Ante Gulin <ante.gulin@percona.com>
Covers both bundles, the environment variables that turn them off, and the HA values. Records that upgrading starts four PMM Server component alerts, so users with contact points configured will begin receiving them. Signed-off-by: Ante Gulin <ante.gulin@percona.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
PMM-14956
Feature build
What this adds
Five built-in alert templates for a PMM Server High Availability cluster:
pmm_ha_no_leaderpmm_ha_split_brainpmm_ha_leader_flappingpmm_ha_node_unreachablePMM_HA_PEERSdeclarespmm_ha_quorum_at_riskDocumentation
documentation/docs/alert/templates_list.md- new "PMM High Availabilitytemplates" section with the table, the steps to create rules from the templates,
and the coverage limitations.
documentation/docs/install-pmm/install-HA-clustered.md- short pointer to thatsection from the HA install page.
Implementation details
last_over_time(...[2m])on every query, not a bare selector. The metricsdatabase's own staleness window is longer than the alert needs. Without the
bounded lookback a departed leader's final sample keeps counting, which delays
pmm_ha_no_leaderand can make an ordinary failover look like a split brain.The 2m window is deliberately shorter than every
for:.max by(node_id)before aggregating. Collapses duplicate series for one node,so extra scrape labels cannot make a single node count twice and raise a split
brain on its own.
< bool/> boolrather than a bare comparison. A bare comparison filtersthe series away and Grafana evaluates the empty result as a label-less NoData
instance, which breaks alert identity and grouping. The
boolform keeps oneinstance carrying its labels.
pmm_ha_quorum_at_risksuppresses itself below three nodes viaand on() (... > 2). On one- and two-node clusters the condition is permanentlytrue, so it would be noise.
pmm_ha_node_unreachablecovers those sizes.pmm_ha_node_unreachableusesunlessplus anabsent()-guarded fallback.A node that is down emits nothing, so it cannot be named from a live series. The
first term names nodes seen in the last 6h but not the last 2m; the fallback
catches a node that never reported or has been gone longer than 6h, and reports it
without a name. The guard stops both terms firing for the same node.
pmm_ha_uprather than reporting 0. Thatis why the counting alerts compare against
pmm_ha_expected_nodesinstead ofsumming a health gauge.
templates silent on standalone PMM.
Summary by CodeRabbit
New Features
pmm_ha_expected_nodesmetric to report configured cluster size.Bug Fixes
Documentation