PMM-14956 HA alert templates

.mockery.yaml

@@ -48,6 +48,9 @@ packages:
   github.com/percona/pmm/managed/services/alerting:
     interfaces:
       grafanaClient:
+      grafanaProvisioningClient:
+      supervisordService:
+      leaderService:
   github.com/percona/pmm/managed/services/management/backup:
     interfaces:
       awsS3:

build/ansible/roles/grafana/tasks/main.yml

@@ -48,6 +48,27 @@
     - plugins
     - dashboards
 
+# pmm-managed renders the built-in alert rules into /srv/grafana/provisioning/alerting at runtime
+# and creates that directory itself, because /srv is emptied at the end of the image build and
+# repopulated by the entrypoint. Only the link has to be baked in. A link with no target yet is
+# fine: Grafana logs that it cannot read the directory and carries on.
+# Grafana's own package ships an empty alerting/ directory here, and ansible refuses to convert a
+# directory into a symlink even with force, so it has to go first.
+- name: Remove Grafana's stock alerting provisioning directory
+  file:
+    path: /usr/share/grafana/conf/provisioning/alerting
+    state: absent
+
+- name: Link Grafana's alerting provisioning directory to /srv
+  file:
+    src: /srv/grafana/provisioning/alerting
+    dest: /usr/share/grafana/conf/provisioning/alerting
+    owner: pmm
+    group: root
+    state: link
+    force: true
+    follow: false
+
 - name: Copy change-admin-password command
   copy:
     src: change-admin-password

documentation/docs/alert/disable_alerts.md

@@ -6,4 +6,21 @@ If for some reason you want to disable PMM Alert templates and keep only Grafana
 {.power-number}
 
 1. Go to **Configuration > Settings > Advanced settings**.
-2. Disable the **Percona Alerting** option. The **Alerts** menu will now display only Grafana-managed alert rules.
+2. Disable the **Percona Alerting** option. The **Alerts** menu will now display only Grafana-managed alert rules.
+## Disable the built-in alert rules
+
+PMM creates and maintains two sets of alert rules of its own: the [PMM component alerts](templates_list.md#pmm_component_alerts), on every server, and the [High Availability alerts](templates_list.md#pmm_ha_alerts), on a clustered one. Each has its own environment variable, so you can turn one off and keep the other:
+
+| Variable | Default | Controls |
+|----------|---------|----------|
+| `PMM_ENABLE_COMPONENT_ALERTS` | `true` | The PMM component alerts, on every server |
+| `PMM_ENABLE_HA_ALERTS` | `true` | The High Availability alerts, on a clustered server |
+
+Set either to `false` and recreate the server to turn that set off. In Docker that means recreating the container with the variable set; with the Helm chart, change the value and upgrade the release.
+
+Keep the following in mind:
+
+- **These take effect when the server starts.** PMM writes the rules while it starts up, so a change needs the server recreated - which is what changing an environment variable requires anyway. To quieten a rule right now, without restarting anything, add a [silence](silence_alerts.md) instead.
+- **Turning a set off deletes its rules.** Any silences you created keep working if you turn the set back on, because they match on labels and the labels do not change. The rules' own state does not survive, so anything that was firing is evaluated afresh and notifies again once its `for` duration has passed.
+- **Your own rules are never touched.** Rules you created from these templates are yours, including copies you made to change a threshold.
+- **Disabling Percona Alerting removes both sets** as well, since these rules are built from Percona alert templates. That switch is in **Configuration > Settings > Advanced settings**. It needs no server restart, but it is not instant either: PMM notices it on its next check, within a few minutes.

documentation/docs/alert/templates_list.md

@@ -6,6 +6,8 @@ The table below lists all the alert templates available in Percona Monitoring an
 
 - [Operating System templates](#os_alerts)
 - [PMM templates](#pmm_alerts)
+- [PMM High Availability templates](#pmm_ha_alerts)
+- [PMM internal component templates](#pmm_component_alerts)
 - [MongoDB templates](#mongodb_alerts)
 - [PBM templates](#pbm_alerts)
 - [MySQL templates](#mysql_alerts)
@@ -29,6 +31,73 @@ The table below lists all the alert templates available in Percona Monitoring an
 | PMM | **PMM agent down** | Monitors PMM Agent status and alerts when an agent becomes unreachable, indicating potential host or agent issues. | MySQL, MongoDB, PostgreSQL, ProxySQL |
 | PMM | **Backup failed [Technical Preview]** | Monitors backup processes and raises alerts on failures. Provides details about the failed backup artifact and affected service to ensure data safety and recovery readiness. This template is currently in [Technical Preview](../reference/glossary.md) and is intended for testing purposes only, as it is subject to change. | MySQL, MongoDB, PostgreSQL, ProxySQL |
 
+<a id="pmm_ha_alerts"></a>
+### PMM High Availability templates
+
+These templates monitor a PMM Server [High Availability cluster](../install-pmm/install-HA-clustered.md). They never raise alerts on a standalone (non-HA) PMM installation, because the underlying HA metrics are only exposed when HA mode is enabled.
+
+| Area | Template name | Description | Database technology |
+| :----|:------------- | :---------- | :------------------ |
+| PMM HA | **PMM HA cluster has no active leader** | Alerts when no node in the cluster holds the Raft leader lease, which means that leader-only work such as advisor checks, backups, telemetry and scheduled tasks has stopped. | PMM |
+| PMM HA | **PMM HA split-brain detected** | Alerts when more than one node claims Raft leadership at the same time, which means the nodes have formed separate Raft clusters instead of one. | PMM |
+| PMM HA | **PMM HA leader is flapping** | Alerts when the Raft term on a node changes more than 5 times (default threshold) within 10 minutes, which indicates an unstable network or a leader that keeps restarting. | PMM |
+| PMM HA | **PMM HA node unreachable** | Alerts when fewer nodes report HA metrics than the number configured in `PMM_HA_PEERS`, which indicates that at least one PMM Server node is down or isolated. | PMM |
+| PMM HA | **PMM HA quorum at risk** | Alerts when the number of live Raft voters has fallen to or below the smallest majority that still forms a quorum. Applies to clusters of three nodes or more. | PMM |
+
+#### These alerts are created for you
+
+Unlike other alert templates, these five do not wait for you to build rules from them. On a PMM Server running in High Availability mode, PMM creates and maintains the rules itself, in the **PMM High Availability** folder and the **PMM Managed** evaluation group. All you need to do is configure a [contact point](./contact_points.md) so the alerts reach you.
+
+Two things follow from PMM owning these rules:
+
+- **They cannot be edited in the interface.** They always match what your PMM version ships, so a fix Percona makes to an expression reaches your server when you upgrade, with nothing to migrate by hand.
+- **To use different values, make your own copy.** Create a rule from the same template with the threshold or duration you want, then turn the built-in set off as described in [Disable alerts](./disable_alerts.md) so that you are not notified twice.
+
+If you created rules from these templates by hand in an earlier version, they are still there and you will now be notified twice for the same problem. Delete your copies, or turn the built-in set off. PMM does not touch rules you made: it counts them in the `pmm_alerting_provisioning_duplicate_rules` metric and logs them once, so you can find them.
+
+To turn the built-in High Availability rules off, set `PMM_ENABLE_HA_ALERTS=false` and recreate the server. On a standalone server the variable has no effect, because these rules are only created for a cluster. See [Disable alerts](disable_alerts.md#disable-the-built-in-alert-rules).
+
+#### Coverage limitations
+
+Keep the following in mind when you rely on these alerts:
+
+- **A complete cluster outage cannot be detected from inside the cluster.** Each node's metrics are collected only by the monitoring agent running on that same node, so when every node is down there is nothing left to report it and all HA alerts fall silent. Monitor the load balancer endpoint from outside the cluster to cover this case.
+- **Split-brain detection requires the isolated node to still reach shared storage.** If a network partition also cuts a node off from the shared VictoriaMetrics storage, its metrics never arrive and the second leader stays invisible.
+- **An ordinary network partition does not cause a split brain.** Raft is designed to prevent two leaders: a node in a minority partition cannot win an election. If no side of the partition holds a majority, the cluster is left with no leader at all and *PMM HA no active leader* fires. If a majority survives, it elects a new leader within seconds and none of these alerts fire, including *PMM HA quorum at risk* and *PMM HA node unreachable*. The isolated node keeps writing metrics and keeps reporting itself as a voter, because each node reads its Raft membership from its own local copy, which the partition does not change. So from the metrics alone the cluster still looks complete. What does change is the isolated node's Raft term, which climbs as it repeatedly fails to win elections, so *PMM HA leader is flapping* is the alert most likely to fire in that situation. The split-brain alert covers the rarer case where nodes end up in separate clusters, for example when they cannot discover each other at startup and each bootstraps its own.
+- **The node unreachable alert names one node at a time.** A node that stopped within the last 6 hours is named in the alert. A node that has never reported since the cluster started, or that has been down for longer than 6 hours, cannot be named, because no metrics remain to identify it; the alert then fires without a node name. These two cases do not combine: if any node can be named, the alert names it and does not separately report the ones it cannot. Use the **High Availability** page to see the full picture whenever more than one node is missing.
+- **The quorum alert does not apply to one- and two-node clusters.** On those the condition would be permanently true, since with two nodes quorum is two and both nodes are always essential, so the template suppresses itself and stays silent. *PMM HA node unreachable* does cover them, so rely on it instead.
+- **Changing `PMM_HA_PEERS` raises alerts until the rollout finishes.** The expected node count is the highest value any node reports, so while some nodes carry the new peer list and others still carry the old one, the cluster looks smaller than expected. *PMM HA node unreachable* and *PMM HA quorum at risk* can both fire for the duration of a rolling restart, in either direction. Let the rollout finish before treating either as a real failure.
+
+<a id="pmm_component_alerts"></a>
+### PMM internal component templates
+
+These templates monitor the components that make up PMM Server itself, rather than the databases PMM monitors. They alert when a component stops responding to the health check that PMM already collects, so no extra configuration is needed.
+
+| Area | Template name | Description | Database technology |
+| :----|:------------- | :---------- | :------------------ |
+| PMM | **PMM VictoriaMetrics is down** | Alerts when a VictoriaMetrics component stops responding. Covers the single instance used by a standalone PMM Server and the vmselect, vminsert, vmstorage and vmagent components used by a clustered deployment. | PMM |
+| PMM | **PMM ClickHouse is down** | Alerts when the ClickHouse instance that stores Query Analytics data stops responding. Query Analytics stops collecting while this lasts; metrics are unaffected. | PMM |
+| PMM | **PMM Grafana is down** | Alerts when a Grafana instance stops responding. The user interface, dashboards and alert rule evaluation all depend on it. | PMM |
+| PMM | **PMM Query Analytics API is down** | Alerts when the qan-api2 component stops responding. Query Analytics stops accepting new query data and its page cannot load. | PMM |
+
+#### These alerts are created for you
+
+PMM creates and maintains these four rules itself, on every server, in the **PMM Server** folder and the **PMM Managed** evaluation group. Configure a [contact point](./contact_points.md) and you will be told when part of PMM stops responding; there is nothing else to set up.
+
+As with the High Availability rules, PMM owns them: they cannot be edited in the interface, they follow whatever your PMM version ships, and you make your own copy from the same template if you want different values. If you upgraded from a version without them and had already built your own rules from these templates, delete your copies to avoid being notified twice.
+
+A component this deployment does not scrape produces no data, and a rule with no data stays silent rather than firing, so these rules are safe on any topology. To turn them off, set `PMM_ENABLE_COMPONENT_ALERTS=false` and recreate the server. See [Disable alerts](disable_alerts.md#disable-the-built-in-alert-rules).
+
+#### Coverage limitations
+
+These alerts are evaluated by Grafana querying VictoriaMetrics, which means two of them cannot observe their own total failure. Keep the following in mind:
+
+- **VictoriaMetrics being down completely cannot be detected by this alert.** The rule has to query VictoriaMetrics to run, so when VictoriaMetrics is gone the query cannot execute and the expression never evaluates. What the alert does cover is a single component failing while the rest still answers queries, which is the common case in a clustered deployment. A complete outage instead shows up as *every* alert rule reporting a datasource error at once. Monitor PMM Server from outside to cover this case.
+- **Grafana being down everywhere cannot be detected either**, because Grafana is what evaluates the alert rules. In a clustered deployment the alert still catches Grafana failing on one node while another node keeps evaluating.
+- **ClickHouse and Query Analytics are fully covered.** VictoriaMetrics stays up to record their failure, so a total outage of either is detected normally.
+- **In a clustered deployment, coverage depends on the Helm chart.** A standalone PMM Server scrapes these components itself. In a cluster, VictoriaMetrics and ClickHouse run as separate workloads scraped by the chart, so these alerts only see them if the chart uses the same job names. Where a component is not scraped at all, the alert stays silent rather than firing falsely.
+- **These alerts cover PMM Server only.** For a PMM Client that has stopped reporting, use *PMM agent down* instead.
+
 <a id="mongodb_alerts"></a>
 ### MongoDB templates
 

documentation/docs/install-pmm/install-HA-clustered.md

@@ -715,12 +715,25 @@ You can't swap these databases for ones you run yourself, like an existing Click
 
 If you're used to standalone PMM, note that `PMM_DISABLE_BUILTIN_CLICKHOUSE` and `PMM_DISABLE_BUILTIN_POSTGRES` work differently here. In standalone, they let you switch to your own database. In HA, they're part of the internal setup and must stay set to `1`.
 
+### Built-in alert rules
+
+A clustered deployment creates its own alert rules on first start: five covering the cluster itself, in the **PMM High Availability** folder, and four covering PMM's own components, in the **PMM Server** folder. See [Alert templates](../alert/templates_list.md#pmm_ha_alerts) for what each one detects. Configure a [contact point](../alert/contact_points.md) and they will reach you; nothing else is needed.
+
+To start a cluster without them, add either of these to `pmmEnv`:
+
+```yaml
+pmmEnv:
+  PMM_ENABLE_HA_ALERTS: "false"          # do not create the High Availability rules
+  PMM_ENABLE_COMPONENT_ALERTS: "false"   # do not create the PMM component rules
+```
+
+During a rolling upgrade the nodes briefly run different PMM versions. If a release changes one of these rules, it can alternate between the old and the new definition until every node has been rolled, which is expected and resolves itself once the rollout finishes.
+
 #### Adjust data retention and other settings
 
 Set customizable variables in your `values.yaml` to match your monitoring requirements. 
 
 Choose a retention period that matches your compliance requirements and storage capacity. For example, `720h` keeps 30 days of data and `4320h` keeps 180 days:
-
 ```yaml
 pmmEnv:
   PMM_DATA_RETENTION: "2160h"  # Adjust based on your retention policy (default: 90 days)
@@ -838,6 +851,12 @@ View detailed role and health information for all PMM nodes in one place.
     - **Follower** status: which nodes are on standby
     - **Health** status: whether each node is responding
 
+### Get alerted about cluster problems
+
+Checking the High Availability page tells you the state of the cluster right now, but it does not notify you when that state changes. PMM creates and maintains alert rules for the failure modes of an HA cluster: no active leader, split-brain, a flapping leader, an unreachable node, and a quorum at risk.
+
+There is nothing to create: configure a [contact point](../alert/contact_points.md) so the notifications reach you. To turn the rules off, set `PMM_ENABLE_HA_ALERTS=false` and recreate the server. For the template list and the coverage limitations, see [PMM High Availability templates](../alert/templates_list.md#pmm_ha_alerts).
+
 ### Scale your deployment
 
 #### Scale PMM server replicas

documentation/docs/install-pmm/install-pmm-server/deployment-options/docker/env_var.md

@@ -45,6 +45,8 @@ Enable or disable specific PMM features:
 | `PMM_ENABLE_TELEMETRY` | `true` | Enables usage data collection |
 | `PMM_ENABLE_ALERTING` | `true` | Enables Percona Alerting system |
 | `PMM_ENABLE_BACKUP_MANAGEMENT` | `true` | Enables backup features |
+| `PMM_ENABLE_COMPONENT_ALERTS` | `true` | Keeps the built-in alert rules for PMM's own components (VictoriaMetrics, ClickHouse, Grafana, Query Analytics). Set to `false` to remove them |
+| `PMM_ENABLE_HA_ALERTS` | `true` | Keeps the built-in alert rules for a High Availability cluster. No effect on a standalone server |
 | `PMM_ENABLE_AZURE_DISCOVER` | `false` | Enables Azure database discovery |
 | `PMM_ENABLE_INTERNAL_PG_QAN` | `0` (disabled) | Enables Query Analytics for PMM Server's internal PostgreSQL. Useful for troubleshooting or HA scenarios. Set to `1` to enable. Can also be controlled via **Configuration > Settings > Advanced settings**. See [QAN for PMM Server's internal PostgreSQL](../../../../use/qan/QAN-stored-metrics.md#monitor-pmm-servers-internal-postgresql)