From 510a6f42a9b5432e597db52cd92c4cfa5b68bbb7 Mon Sep 17 00:00:00 2001 From: Matej Kubinec Date: Tue, 25 Aug 2026 13:42:19 +0200 Subject: [PATCH 01/15] PMM-14912 Add dynamic thresholds design documents Carry the design of record and the performance-testing results onto a branch based on main, so the backend can be implemented clean rather than on top of the proof-of-concept. The decision document records the design and the live measurements behind it; the performance document records the collector benchmarks that settled override-only emission over emitting a threshold for every target. Full authoring history for both remains on PMM-14912-dynamic-thresholds, along with the proof-of-concept they were written against. Co-Authored-By: Claude Opus 5 (1M context) Signed-off-by: Matej Kubinec --- dynamic-thresholds-main-decision.md | 1131 +++++++++++++++++++++ dynamic-thresholds-performance-testing.md | 356 +++++++ 2 files changed, 1487 insertions(+) create mode 100644 dynamic-thresholds-main-decision.md create mode 100644 dynamic-thresholds-performance-testing.md diff --git a/dynamic-thresholds-main-decision.md b/dynamic-thresholds-main-decision.md new file mode 100644 index 0000000000..857d459edd --- /dev/null +++ b/dynamic-thresholds-main-decision.md @@ -0,0 +1,1131 @@ +# Dynamic Alert Thresholds — decision and implementation plan (against `main`) + +Per-node (and later per-service, per-cluster) threshold overrides for alert rules created from +templates, specified against **`main` at `a807f56d8`**. Every file and line reference below was checked +against `main`, not against any feature branch. + +**Scope of this document** + +| In scope | Out of scope | +|---|---| +| Migration, reform models, CRUD helpers | UI (thresholds modal, hooks, Grafana-side trigger) | +| Threshold metric collector | Dedicated metrics endpoint + own scrape job (costed as a follow-up) | +| Rule builder: threshold-query injection and single-expression desugaring | Grafana `UpdateRule`/`DeleteRule` RPCs | +| gRPC/REST API for reading and writing overrides | Service and cluster **behaviour** (schema and API are generalised now; only node scope is implemented) | +| Reconciler for orphaned rule registry rows | | + +Written greenfield: it assumes nothing exists beyond `main`. + +> **On evidence.** Where this document states a fact as *measured*, it was measured on a live PMM +> server (Grafana 12.4.5, VictoriaMetrics v1.147.0). Where it *extrapolates* from those measurements, it +> says so. Unverified assumptions are listed in §8, and decisions still needed are in §9. +--- + +## Summary + +**Verdict.** Per-target threshold overrides are delivered as **data, not as rule edits**: the Grafana +rule is written once at creation and never rewritten, and changing a threshold is a single Postgres row. +Postgres is the source of truth, VictoriaMetrics is a derived transport carrying **only the overrides**, +and the default is fanned out at query time over PMM's existing node-inventory metric. This keeps +tuning one target from disturbing alert state on every other target of the same rule, which is the +failure mode that rules out the obvious alternative of rendering thresholds into the query. Estimated +3–5 weeks for the node-only increment; the schema and API are generalised for service and cluster scope +from the start, because routes and proto fields are additive-only. + +### How it fits together + +``` + PMM UI / API pmm-managed VictoriaMetrics Grafana + ──────────── ─────────── ─────────────── ─────── + set threshold ──POST──▶ ┌─────────────────────┐ + │ alert_rule_ │ + │ threshold_overrides│◀── canonical + └──────────┬──────────┘ + │ read once per scrape + ▼ + ┌─────────────────────┐ scrape ┌──────────────────┐ + │ threshold collector │────────────▶│ pmm_alert_ │ + │ (overrides only) │ /debug/ │ threshold_ │ + └─────────────────────┘ metrics │ override │ + └────────┬─────────┘ + ┌─────────────────────┐ scrape ┌────────┴─────────┐ + │ inventory collector │────────────▶│ pmm_managed_ │ + │ (already exists) │ │ inventory_nodes │ + └─────────────────────┘ └────────┬─────────┘ + │ + T_ reads both + ▼ + ┌──────────────────┐ + │ alert rule │──▶ Alertmanager + │ A / T / C │ + └──────────────────┘ +``` + +The rule never changes after creation. The only write path for a threshold change is the leftmost arrow. + +### Where state lives, and what is authoritative + +``` + ┌──────────────────────────┐ ┌──────────────────────────┐ ┌──────────────────────────┐ + │ PostgreSQL │ │ VictoriaMetrics │ │ Grafana rule │ + ├──────────────────────────┤ ├──────────────────────────┤ ├──────────────────────────┤ + │ AUTHORITATIVE for │ │ DERIVED transport │ │ AUTHORITATIVE for │ + │ · override values │ │ · one series per │ │ · which rules exist │ + │ · per-param snapshot │ │ override (tens) │ │ · title, folder, group │ + │ (default, join label, │ │ · rebuilt every scrape │ │ · template_name label │ + │ scopes, unit, range) │ │ · never authoritative │ │ DERIVED │ + │ │ │ │ │ · default as a literal │ + │ Lose it → overrides gone │ │ Lose it → falls back to │ │ · rule_id label │ + │ │ │ defaults, keeps alerting │ │ Written once │ + └──────────────────────────┘ └──────────────────────────┘ └──────────────────────────┘ +``` + +Nothing needs both stores to agree to be correct: a threshold missing from VictoriaMetrics degrades to +the rule's default rather than to silence, and a Postgres row whose rule or target no longer exists is +inert because the emit path resolves it away. + +### For implementers — the shape to build + +- **Metric:** `pmm_alert_threshold_override{rule_id, param, target}`, one series per override, value = effective threshold after precedence is resolved **in Go** (§4.6). +- **Rule:** three steps — `A` observed, `T_` threshold, `C` math `$A > $T_`. `T_` is **always emitted**, even with no overrides (§5.1). +- **Threshold expression:** `max by () (label_replace(override…)) or (max by () () * 0 + )` — every clause justified in §4.3. The default fans out over the **observed expression**, not over an inventory metric, so no `last_over_time` and no window (§4.7). +- **Clearing:** tombstone the row, never delete it — 14–21 s instead of 309 s (§4.4). +- **Schema:** `alert_rules` (5 columns) + `alert_rule_threshold_overrides` keyed `(rule_id, param_name, scope, target)` — §4.4. +- **Cleanup:** override rows are deleted by the node/service removal API, not a sweep (§4.8); the reconciler handles only registry rows for rules deleted in Grafana (§4.9). +- **Ten implementation steps with file references and effort: §6.** + +--- + +## 1. The problem + +Alert rules created from templates bake the threshold into the query (`$A > 80`). Changing the threshold +for one node means editing or recreating the rule. We want a per-node override that is a **data change**, +not a rule change. + +## 2. Recommendation in one paragraph + +Emit a **VictoriaMetrics gauge carrying only the overrides**, materialise the **default at query time by +fanning out over the rule's own observed expression**, and have the rule compare its observed query +against that combination. The Grafana rule is written **once at creation and never rewritten**, so tuning +a threshold never disturbs alert state. Postgres is the single source of truth; VictoriaMetrics is a +derived transport. + +```promql +# the injected threshold step, T_ +max by (node_name) ( + label_replace(pmm_alert_threshold_override{rule_id="", param=""}, + "node_name", "$1", "target", "(.*)") +) +or +(max by (node_name) () * 0 + ) +``` + +Two properties of this shape are load-bearing, and both are measured (§8): + +- **The default is fanned out over the observed expression, not over an inventory metric.** The threshold + can then never outlive or predecease `$A`, because both read the same series. This removes the + 15-minute silent-stop cliff *structurally* rather than bounding it, and it deletes `last_over_time` + from the query — there is no window left to tune. Cost: the observed expression is evaluated twice, + measured at **~1.25×**, not 2×. +- **Clearing an override is a value change, never a disappearance.** The row is tombstoned rather than + deleted (§4.4), so the series keeps being emitted and simply carries the resolved default. Measured + **14–21 s**, against **309 s** when a clear is signalled by absence. + +## 3. What `main` already provides + +This is the part that makes the estimate small. On `main` today: + +| Foundation | Where | +|---|---| +| **Multi-expression templates** (`queries:` / `expressions:` / `condition:`) | `managed/pi/alert/query.go:25` (`TemplateQuery`), `:38` (`UsesMultipleExpressions`), `:55`–`:131` (validation) | +| Rule builder with a multi-expression path | `managed/services/alerting/rule_builder.go:58` (`buildGrafanaRuleData`), `:81` (`buildMultiExpressionRuleData`) | +| Prom-query and math-expression step builders | `rule_builder.go:141` (`newPromQueryData`), `:165` (`newMathExpressionData`) | +| Filter application | `rule_builder.go:120` (`fillAndFilterExpr`) | +| Rule creation flow | `managed/services/alerting/service.go:679` (`CreateRule`), annotations filled at `:744` | +| Grafana rule creation | `managed/services/grafana/client.go:712` (`CreateAlertRule`) | +| **A per-node inventory metric, already scraped** | `managed/services/inventory/inventory_metrics.go:77-80` → `pmm_managed_inventory_nodes{node_id, node_type, node_name, container_name}` | +| A per-service/agent inventory metric | `inventory_metrics.go:71-75` → `pmm_managed_inventory_agents{…, service_id, service_name, node_id, node_name, …}` | +| Collector registration + `/debug/metrics` exposition | `managed/cmd/pmm-managed/main.go:942`, `debugAddr` at `:123` | +| That endpoint already scraped by VM | `managed/services/victoriametrics/scrape_configs.go:80` (job `pmm-managed`, interval `MR` = 10 s per `managed/models/settings.go:209`, timeout `0.9 × MR` per `scrape_configs.go:41`) | +| Stable identifiers on observed series | `Node.UnifiedLabels()` `managed/models/node_model.go:118`, `Service.UnifiedLabels()` `managed/models/service_model.go:115` | +| Leader-election hook for background work | `managed/services/ha/haservice.go:569` (`AddLeaderService`), `:597` (`IsLeader` — true when HA is disabled) | +| Migrations up to **118** | `managed/models/database.go:1184` → this feature takes **119** | + +What is **absent** on `main` and must be built: the `overridable` param flag, both tables, the collector, +threshold-query injection, the API, the reconciler, and (per the scope decision) single-expression +desugaring. `main` has **no** `UpdateAlertRule`, `ListAlertRules` or `DeleteAlertRule` — and the +recommended design needs only `ListAlertRules`, for the reconciler. + +Template inventory on `main`: **42 templates — 38 single-expression, 4 multi-expression.** +`pmm_node_high_cpu_load` is already multi-expression, so it becomes overridable with one YAML line. + +## 4. Design + +### 4.1 Rule shape + +Every overridable rule compiles to the same steps, whatever the template looked like: + +| refId | datasource | body | +|---|---|---| +| `A`, `B`, … | Metrics (VM) | the template's observed queries, unchanged | +| `T_` | Metrics (VM) | the threshold expression from §2, one per overridable param | +| `C` | `__expr__` math | the template's expression, with `[[ .param ]]` swapped for `$T_` | + +Written once. A threshold change touches one Postgres row and nothing in Grafana. + +### 4.2 Metric shape + +``` +pmm_alert_threshold_override{rule_id, param, target} # gauge +``` + +Exactly three labels. Value = the effective threshold for that target, in the param's native unit, +**after precedence resolution in Go**. + +| Label | Why | +|---|---| +| `rule_id` | Scopes the selector so one rule cannot pick up another's series | +| `param` | Required for multi-param rules | +| `target` | The **value of the rule's join label** (`node_name` / `service_name` / cluster value), resolved in Go from the ID stored in Postgres | + +Excluded deliberately: **`scope`** (precedence cannot be expressed in PromQL — §4.6), the join label by +name (its *name* varies per rule and a fixed `prom.NewDesc` cannot vary label names, so generic `target` +plus one `label_replace` keeps a **checked** collector), the default value (a literal in the rule), +and rule metadata (`template_name`, `rule_title` — they churn on rename; see §4.4 for why they are not +stored at all). + +```go +desc: prom.NewDesc( + "pmm_alert_threshold_override", + "Effective alert threshold override for a rule parameter and target. Emitted only where an "+ + "override or a tombstone exists; targets without either fall back to the rule's default, "+ + "which the rule query materialises by fanning out over its own observed expression.", + []string{"rule_id", "param", "target"}, + nil, +), +``` + +Implement `Describe` directly (`ch <- c.desc`) rather than via `prom.DescribeByCollect`, which would run +a full `Collect` — and therefore a database query — merely to describe the collector. + +**Cardinality:** one series per `(rule, param, target-covered-by-an-override-or-tombstone)`. Because +precedence is resolved in Go, a coarse-scope override **expands** — a cluster override over 200 nodes +would emit 200 series. Bounded by "targets actually covered, plus targets ever tuned", which is why this +never approaches `rules × params × nodes`. That bound is the whole point: §4.12 measures what happens when +it is removed. + +### 4.3 Why each clause of the threshold query + +| Clause | Job | +|---|---| +| `label_replace(…, "", "$1", "target", "(.*)")` | Maps generic `target` onto whichever label this rule joins on. One fixed descriptor serves every scope. | +| `max by ()` | Three jobs: **strips `instance`/`job`** (the scrape target is `pmm-server`/`pmm-managed`, which can never match an observed series' `instance`); **reduces both sides of `or` to identical label sets**, without which `or` returns both instead of preferring the left; and **collapses HA duplicates**. Measured cost: none — `samplesScanned` is identical with and without it. | +| `or` | Set union preferring the left: "override if present, else default". Requires identical label sets, which `max by` guarantees. | +| `* 0 + ` | Preserves the label set, replaces the value → one default series per target **that has observed data**. | + +> **There is no `last_over_time` in this query, and no window to choose.** An earlier revision fanned the +> default out over `pmm_managed_inventory_nodes` and needed `last_over_time(…[15m])` to keep it resolving +> across a pmm-managed restart. That made the window a *reliability* parameter and produced the cliff +> in §4.7. Fanning out over the observed expression removes the need entirely: if `$A` resolves, so does +> `T`. Verified live — see §8. + +> **Do not resurrect `samplesScanned` as the cost argument.** A previous revision justified `[15m]` with +> per-series scan counts (bare **61**, `[15m]` **91**, `[1h]` **361**, `[7d]` **11,246**) taken on VM +> v1.147.0. On **v1.149.0**, which PMM 3.7.1 ships, that counter under-reports and cannot distinguish the +> functions: `count_over_time(…[1h])` provably reads all 360 samples per series yet reports the **same +> 151** as `last_over_time(…[1h])`. Only `bare = 61` still reproduces. Cost claims must come from +> wall-clock or `vm_rows_read_per_query`, not from `samplesScanned`. + +> **Window length turned out not to matter anyway.** Measured at 1000 nodes with 2 h of history: +> `[15m]` **3.8 ms**, `[2h]` **4.5 ms**, bare **3.8 ms** — against a 60 s evaluation interval. Any scan +> argument for or against a window is noise at PMM's scale; decide these questions on failure modes. + +> **Never wrap the override series in `last_over_time`.** With tombstones (§4.4) a cleared override is a +> *value change*, so a window is unnecessary; and if a row is ever hard-deleted, a window would keep the +> deleted value resolving for its whole length. + +### 4.4 Postgres schema (migration 119) + +Generalised for the confirmed roadmap. The decisive constraint: **`cluster` is a label value, not an +entity** — there is no `clusters` table, so it can never have a foreign key, and the same holds for +`environment`, `replication_set` and custom labels. Keeping FKs for node/service but not cluster would +force per-scope branching through every query and handler, so the target column is polymorphic. + +```sql +CREATE TABLE alert_rules ( + rule_id VARCHAR NOT NULL, -- PMM-minted; the identity + grafana_rule_uid VARCHAR CHECK (grafana_rule_uid <> ''), -- cached handle, NOT identity + params JSONB NOT NULL, -- see below + created_at TIMESTAMP NOT NULL, + updated_at TIMESTAMP NOT NULL, + PRIMARY KEY (rule_id), + UNIQUE (grafana_rule_uid) +); + +CREATE TABLE alert_rule_threshold_overrides ( + id VARCHAR NOT NULL, + rule_id VARCHAR NOT NULL REFERENCES alert_rules (rule_id) ON DELETE CASCADE, + param_name VARCHAR NOT NULL CHECK (param_name <> ''), + scope VARCHAR NOT NULL CHECK (scope <> ''), -- 'node' | 'service' | 'cluster' + target VARCHAR NOT NULL CHECK (target <> ''), -- node_id | service_id | cluster label value + value DOUBLE PRECISION NOT NULL, + cleared_at TIMESTAMP, -- non-NULL => tombstone, see below + created_at TIMESTAMP NOT NULL, + updated_at TIMESTAMP NOT NULL, + PRIMARY KEY (id), + UNIQUE (rule_id, param_name, scope, target) +); + +CREATE INDEX alert_rule_threshold_overrides_target_idx + ON alert_rule_threshold_overrides (scope, target); +``` + +`CHECK (x <> '')` follows repo convention (55 such constraints in `database.go`). **No +`CHECK (scope IN (…))`** — validate the enum in Go so adding a scope needs no migration. + +#### Clearing is a tombstone, not a delete + +`cleared_at` exists because **absence is a slow signal**. If clearing an override deletes the row, the +series stops being emitted, and a stopped series stays queryable for VM's whole lookbehind — measured +**309 s** end to end through a real rule. If clearing instead marks the row, the series continues and its +*value* changes, which is visible on the next scrape: measured **14–21 s**. Same rule, same environment, +~15× apart (§8). + +Three rules make this safe: + +1. **The tombstone stores no value.** `value` keeps whatever the override was, for audit; the collector + ignores it once `cleared_at` is set and resolves the default from `params[p].default` instead. Storing + the default here instead would silently freeze cleared targets at the *old* default the next time a + default changes — and changing a default is a supported operation (§4.4), so this is a real defect, not + a hypothetical. +2. **The resolver decides, not the writer.** The emitted target set widens from "targets with an override" + to "targets with an override **or** a tombstone"; the §4.10 resolver then runs unchanged. For a + tombstoned target it recomputes across the remaining scopes and falls back to the default **only if + nothing survives**. So clearing a node override that sits under a cluster override correctly yields the + cluster value, not the default — no new precedence logic. +3. **Tombstones are never garbage-collected.** Once the value is not stored, a retained tombstone + self-corrects on a default change. And sweeping would be behaviourally invisible anyway, since the + tombstone's emitted value is identical to the fan-out default it would fall through to — so a sweeper + buys nothing but a leader-election question. Growth is bounded by *targets ever tuned*, not by + inventory. + +`ListNodeThresholds` and the UI **must** treat `cleared_at IS NOT NULL` as "not overridden", or every +target ever tuned will read as tuned forever. That is the likeliest place for this to ship a bug. + +#### The `params` snapshot + +Rule metadata that Grafana already holds — title, folder, rule group, template name — is **not** stored; +it is read from the rule, which is authoritative (§4.4). `params` is the one thing that is irreducible. It +is a snapshot, keyed by param name: + +```json +{ + "threshold": { + "default": 80, + "join_label": "node_name", + "scopes": ["node"], + "unit": "%", + "summary": "A percentage from configured maximum", + "min": 0, + "max": 100 + } +} +``` + +Nothing else can supply it. The default *is* in the rule query, but as a PromQL literal, and recovering it +means parsing generated text — rejected in §7. Taking it from the *current* template instead lets the API +report a default the rule does not actually use, if the template was edited after the rule was created. +And it cannot live on the overrides table, because a param with **no** override has no row there. + +#### This is also what makes changing a default possible later + +Because the effective default is stored *and* rendered into the rule, the two can be reconciled: update +`params[p].default`, re-render the `T_` step, `PUT` the rule. It is deliberately **not cheap** — a +rule-definition change resets alert state for that rule (§7) — but that is acceptable for a rare, +deliberate administrative act, unlike per-target tuning. Without the snapshot there would be nothing to +change *from*, only a literal buried in a query. + +If dynamic defaults ever become routine rather than rare, the answer is not to optimise this path but to +move the default into data as well — see §10, which treats that as the trigger to revisit the +thresholds-datasource alternative. + +#### How `scope` and `target` are used + +`scope` and `target` are not just descriptive — together they drive three different code paths. + +**Write path (API).** `scope` says which kind of thing is being targeted, `target` identifies it. +`SetNodeThreshold(node_id, …)` is the node-scope alias: it becomes `scope='node'`, +`target=` (§4.5). Validation checks that `scope` is legal for that param per its `scopes` list, +and that `target` exists where existence is checkable. `UNIQUE (rule_id, param_name, scope, target)` allows +a node override *and* a cluster override to coexist for the same param — precedence decides which wins. + +**Emit path (collector).** `scope` selects what to do with `target`: + +| `scope` | `target` holds | Collector action | +|---|---|---| +| `node` | `node_id` | resolve → `node_name` via `nodes`; skip if it no longer resolves | +| `service` | `service_id` | resolve → `service_name` via `services`; skip if it no longer resolves | +| `cluster` | the `cluster` label value | no lookup — but **expand** onto the param's join label (below) | + +The skip-if-unresolvable behaviour is what makes a stale row inert (§4.8), and the emitted label is always +the *resolved* value, never the stored ID, because the rule joins on names. + +**Read path (API).** `(scope, target)` is the lookup key: "all thresholds for this node" is +`WHERE scope='node' AND target=$1`, which is what the `(scope, target)` index exists for. The same +precedence function then produces the effective value (§4.10). + +##### Which scopes are legal for a param — a rule, not a list + +A coarse scope has to be projected onto the param's join label, and that projection must be +**unambiguous**. The general rule: + +> A scope `S` is legal for a param whose join label is `L` **iff the mapping `L → S` is a function** in the +> inventory — i.e. every value of `L` maps to at most one target at scope `S`. + +Applying it: + +| join label | scope | `L → S` | Legal? | +|---|---|---|---| +| `node_name` | `node` | `node_name → node_id` | ✅ one-to-one | +| `service_name` | `service` | `service_name → service_id` | ✅ one-to-one | +| `service_name` | `cluster` | `service_name → cluster` | ✅ each service has exactly one cluster, so a cluster override expands onto its services without conflict | +| `node_name` | `cluster` | `node_name → cluster` | ❌ a node can host services from several clusters | +| `node_name` | `service` | `node_name → service_id` | ❌ a node hosts many services, so two service overrides could claim the same node | + +So a node-joined param accepts `{node}`; a service-joined param accepts `{service, cluster}`. This is a +**parse-time validation** on `override_scopes`, not a runtime tie-break — the illegal rows must never +exist. It also generalises to labels not yet supported (`environment`, `replication_set`) without new +case-by-case reasoning. + +> **Consequence for the collector.** Expanding a coarse scope requires knowing the param's `join_label`, +> so the collector reads `alert_rules.params` **only once multi-scope behaviour ships**. In the node-only +> first increment every override is `scope='node'` onto `node_name`, resolution is driven by `scope` +> alone, and the collector touches only the overrides table plus inventory. + +#### `grafana_rule_uid` — a cache, never the identity + +`pmm_rule_id` stays the identity: PMM mints it, stamps it as a rule label, and it is stable by +construction. The Grafana UID is stored **only as a handle** for cheap direct addressing — +`GET/PUT/DELETE /api/v1/provisioning/alert-rules/{uid}` needs no folder or group, which is exactly what a +future default-change or delete path wants. + +It must be treated as a cache. Note first what is *not* the reason: PMM has no delete-and-recreate +lifecycle — no `DeleteRule` RPC, no `DeleteAlertRule` on the client, and this design never rewrites a rule +after creation. Nor is a plain user delete-then-create a problem, because the replacement carries no +`pmm_rule_id` label, so the old rule is simply *gone* and its registry row is garbage rather than "the +same rule with a new UID". + +The staleness paths that do exist are **Grafana-side copy operations that preserve labels while minting a +new UID**: + +- duplicating a rule in the Grafana UI; +- alert-rule export/import, including provisioning-file restore; +- Grafana backup/restore. + +So the handle can go stale, which is why: + +- the column is **nullable** — the value may not be known yet, and code must work without it; +- reads that matter fall back to matching on the `pmm_rule_id` label; +- a `404` on direct addressing means "refresh the handle", not "the rule is gone" — **re-resolve by label and update the stored UID**, which is self-healing and free, because the reconcile/read pass already holds both the UID and the label for every rule. + +#### The same copy operations break `pmm_rule_id` uniqueness — and mostly that is fine + +Duplicating a PMM rule in Grafana produces **two rules carrying the same `pmm_rule_id`**. +`UNIQUE (grafana_rule_uid)` does not help: the collision is on the identity *label*, inside Grafana, where +PMM can impose no constraint. + +The important realisation is that **the coupling lives in the query text, not in the label.** The copy's +`T_` step literally contains `rule_id=""`, so a duplicated rule evaluates against the +original's overrides no matter what identity scheme is used. No scheme can prevent that, because copying +copies the query. Treat it as intended behaviour: **duplicated rules share thresholds.** + +What must change is every place that assumed a 1:1 mapping: + +| Concern | Resolution | +|---|---| +| Caching the UID | If a `pmm_rule_id` maps to more than one Grafana rule, store **`NULL`** — refuse to cache an ambiguous handle rather than picking one arbitrarily. | +| Reconciliation | It only asks *"is this `pmm_rule_id` present at all?"* — a boolean, so duplicates are harmless. Do not add "which one". | +| `ListNodeThresholds` | May legitimately return the same threshold under two rule titles. That is honest — there really are two rules — so present both rather than silently collapsing them. | +| Operator visibility | Count duplicated `pmm_rule_id`s during the reconcile pass and expose the count as a gauge (plus a log line), so this is observable instead of surprising. | + +None of this requires preventing duplication, which is not preventable without read-only rules — rejected +in §7 as group-wide and irreversible. + +**Capturing it costs nothing.** The Ruler API POST already returns the UIDs it created — +`{"message":"rule group updated successfully","created":["ffvj7sosfptdsd"]}` — and `CreateAlertRule` +(`managed/services/grafana/client.go:712`) currently returns only `error`, discarding that body. Have it +parse and return the created UID instead of adding a second round-trip. Note an *update* returns +`"updated"` rather than `"created"`, so handle both. + +**What `target` holds:** the stable **ID** for node/service (`node_id`, `service_id` — never reused), and +the **label value** for cluster (which has no ID). The query joins on the *name*, because that is what +templates aggregate by, so the collector resolves ID → name at emit time, bounded by the override count: + +```go +overrides, _ := models.FindThresholdOverrides(tx.Querier) // V rows, one indexed query +nodes, _ := models.FindNodesByIDs(tx.Querier, nodeIDsOf(overrides)) // WHERE node_id IN (…) +// scope='cluster' needs no lookup: target IS the label value. +``` + +> **In the node-only increment the collector never reads `alert_rules`.** It emits only overrides, takes +> each value from the override row, and picks the resolution table from that row's `scope` — so it needs +> neither defaults nor `params`. (Coarse-scope expansion later requires `params[p].join_label`; see "Which +> scopes are legal for a param" below.) Either way, nothing in the emit path knows whether the rule still +> exists — which is exactly why an orphaned registry row keeps producing series. See §4.9. + +That resolution doubles as cleanup: an override whose target no longer resolves emits nothing, so a stale +row is inert and invisible, and GC becomes tidiness rather than correctness. + +**Note on `node_name` reuse.** `UNIQUE (node_name)` (`database.go:117`) guarantees uniqueness at any +instant but **not across time** — rebuild a host, re-register with the same hostname, and you get the same +`node_name` with a fresh `node_id`. Storing `node_id` and rendering `node_name` is what keeps a stale +override from silently attaching to a different machine. + +### 4.5 Value and API validation + +`DOUBLE PRECISION` accepts non-finite values, and this would be **`main`'s first float column**, so there +is no existing precedent to inherit. Guard at both layers. + +Database: + +```sql +CHECK (value = value -- rejects NaN (NaN <> NaN) + AND value > '-Infinity'::float8 + AND value < 'Infinity'::float8) +``` + +API, in this order, each with a defined code: + +| Check | Code | +|---|---| +| `value` is finite (not NaN, not ±Inf) | `InvalidArgument` | +| `value` within the param's declared `range` | `InvalidArgument` | +| param exists on the rule | `NotFound` | +| param is `overridable` | `FailedPrecondition` | +| `scope` is a supported value, and legal for this param per `override_scopes` | `InvalidArgument` | +| `target` exists, where existence is checkable (node/service; not cluster) | `NotFound` | +| rule exists in the registry | `NotFound` | + +**Alias conflict.** The API carries both `node_id` and the general `scope`/`target` pair (§6 step 7). +`node_id` is an alias for `scope='node', target=`. If both are supplied and disagree, reject with +`InvalidArgument` — do not silently prefer one. Freeze this rule in the proto comments before the API +ships, since the fields are permanent once released. + +### 4.6 Precedence is resolved in Go, not PromQL + +Three measurements settle this: + +| Test | Result | +|---|---| +| `or` across **different** label sets | returns **both** series — no precedence | +| `or` across **identical** label sets | left wins — so `or` precedence needs same-label-set terms | +| node override 50 + cluster override 90 under `max by (node_name)` | **90** — `max` picks the larger value, not the more specific scope | + +Reducing to a common label set is required for `or` to mean "prefer the left", but that reduction is +exactly what destroys the scope information needed to rank by. So the collector emits only the effective +value per target, and **`ListNodeThresholds` and the collector must share one precedence function** — +PromQL provides no backstop if they diverge. Proposed order: `node` → `service` → `cluster`. + +### 4.7 Semantics that fall out for free + +- **Set an override** → row inserted → series appears within one scrape → `or` prefers it. +- **Clear an override** → row **tombstoned**, not deleted (§4.4) → the series keeps being emitted and its value changes to the resolved default. Measured latency **14–21 s**. Deleting the row instead signals the clear by absence, which measured **309 s** — the same ~5 minutes an earlier revision of this document accepted as unavoidable. It is not unavoidable; it is a consequence of encoding "cleared" as absence. +- **New node** → the fan-out picks it up; nothing to write. +- **Node or service deleted** → **there is no cascade for the target.** The only foreign key is `rule_id → alert_rules`, so deleting a *rule* cascades its overrides; deleting a *node or service* does not. The polymorphic `target` column (§4.4) cannot carry an FK, because `cluster` targets have no referent table. Two consequences, and both matter: + - **Immediately harmless:** ID → name resolution fails for the deleted entity, so the collector emits nothing for it. The row is inert from the next scrape onward — no phantom series, no wrong threshold. + - **But the rows must still be cleared.** Left behind, they accumulate silently, they show up in any admin/debug listing of overrides, and they make `ListNodeThresholds`-style queries return entries for entities that no longer exist. **Removing a node or a service must clear its overrides** — see the garbage-collection spec below. +- **pmm-managed down** → **defaults keep resolving indefinitely; only overrides degrade.** Because the + default is fanned out over the observed expression (§2), `T` cannot go empty while `$A` still resolves. + One step of degradation remains, and it is bounded and safe-direction: + +| Elapsed | Behaviour | +|---|---| +| 0 – ~5 min | Overrides and defaults both resolve. Normal. | +| ~5 min | Override series age out of VM's lookbehind → **overridden targets revert to their defaults**. Behaviour change #1, and the only one. | +| indefinitely | Defaults keep resolving from the observed expression. Alerting continues, at defaults, for as long as `$A` flows. | + + **This is measured, not argued.** Two rules were run side by side against the same observed series while + the fan-out metric was cut off. The inventory-fan-out rule lost its raw series at **259 s**, lost `T` + at **856 s** when the `[15m]` window expired, and at **905 s** reported `state=inactive, health=ok` — + a green, healthy, non-alerting rule with live data breaching its threshold. The observed-query fan-out + rule held `T = 80` and stayed `firing` for the full 26-minute run (§8). + + **The old shape had a second, worse trigger than a crash.** The threshold collector and the inventory + collector share one `/debug/metrics` endpoint and one `0.9 × MR = 9 s` timeout. A threshold collector + slow enough to blow that budget takes `pmm_managed_inventory_nodes` down **with** it — measured, see + §4.12 — which under the old fan-out silently disabled *every* rule within 15 minutes. Fanning out over + the observed expression severs that coupling: the rule no longer reads anything pmm-managed publishes. + + Detection still needs no new plumbing: **`up{job="pmm-managed"}` already exists on `main`** and covers + the remaining override-revert step. The dedicated endpoint's `up{job="pmm-thresholds"}` would later + attribute it more precisely. + +### 4.8 Clearing overrides when nodes and services are removed + +`target` carries no foreign key, so deletion of the referenced entity is handled **in the removal API +itself** — not by a background sweep. PMM already does dependant cleanup this way, which makes the hook +idiomatic rather than novel. + +> **Two different operations, deliberately.** A user *clearing* an override **tombstones** the row so the +> series keeps resolving and the clear lands in 14–21 s (§4.4). A *node or service being removed* +> **hard-deletes** the rows, because there is no longer any target to emit for and nothing will ever query +> it again — a tombstone there would be pure residue. Clearing is a value change; removal is a deletion. + +**The mechanism: delete inside the existing removal transaction.** + +```go +// managed/models/node_helpers.go, in RemoveNode +DeleteThresholdOverridesForTarget(q, ThresholdScopeNode, id) + +// managed/models/service_helpers.go, in RemoveService +DeleteThresholdOverridesForTarget(q, ThresholdScopeService, id) +``` + +Three properties of the existing code make this sufficient on its own: + +1. **The DB restricts rather than cascades.** `services.node_id` is a plain `FOREIGN KEY (node_id) REFERENCES nodes (node_id)` with no `ON DELETE CASCADE`, so PMM must remove dependants explicitly — which is exactly why these chokepoints exist: `RemoveNode` (`node_helpers.go:255`), `RemoveService` (`service_helpers.go:354`), `RemoveAgent` (`agent_helpers.go:1444`), each taking a `RemoveMode`. +2. **The helpers compose.** `RemoveNode` with `RemoveCascade` finds the services on that node and calls `RemoveService(…, RemoveCascade)` for each, which in turn routes through `RemoveAgent`. So deleting a node clears its own node-scoped override **and** the service-scoped overrides of every service on it, with no second code path and no gap. +3. **It is atomic.** The delete runs in the same reform transaction as the removal, so there is no window in which overrides outlive their entity — unlike a sweep, which is eventually consistent by construction. + +It also sidesteps the awkwardness a sweep would have had: no leader-election question (the delete happens +wherever the removal API is served) and no lag before the UI reflects reality. + +**Retained as a free backstop, not as the mechanism:** the collector resolves `target` IDs to join-label +names and **skips anything that no longer resolves**, so a row missed by any future removal path — or +inserted by direct SQL — can never produce a wrong threshold. It is simply inert. + +**Never delete `scope='cluster'` rows.** There is no "delete a cluster" operation to hook, and more +importantly a cluster override with no matching services is **dormant, not stale**: services may be added +to that cluster later, and the override should then apply. Expansion yielding nothing already makes it +inert, so keeping the row is correct rather than untidy. + +**Tests:** removing a node deletes its node-scoped override in the same transaction; removing a node with +cascade also deletes the service-scoped overrides of its services; a cluster override survives every +removal; an override whose target was deleted out-of-band emits nothing. + +### 4.9 Cross-store creation and reconciliation + +Rule creation spans Grafana and Postgres, so the ordering and failure behaviour are part of the design: + +1. **Mint `rule_id` first.** It is the idempotency key for every subsequent step; a retry is a plain upsert on the same id. +2. **Write Postgres, then Grafana.** If the Grafana call fails, the registry row is orphaned — which is already the reconciler's job, so no bespoke compensation path is needed. +3. **Grafana rule without a registry row** (the reverse failure, possible if a row is lost) is not silently harmful: the default is a literal in the rule, so it keeps evaluating correctly at defaults, and only the override API fails. Repair from the `pmm_rule_id` and `template_name` labels already stamped on the rule. + +**The reconciler is a garbage collector, not a consistency mechanism — and it never writes to Grafana.** +Because Grafana is authoritative for which rules exist (§4.4) and both the API join and the collector's +ID→name resolution drop unresolvable rows at read time, it has **no correctness duty for alerting +behaviour**. It has two jobs, and only one of them costs anything real: + +There are two kinds of orphan, and only one of them is the reconciler's problem: + +| Orphan | Cleaned by | Cost of not cleaning it | +|---|---|---| +| Override rows whose **node/service** is gone | **The removal API hooks (§4.8)** — atomic, immediate | n/a — handled at the source | +| **Registry rows** whose Grafana **rule** is gone | **The reconciler.** Irreducible: rule deletion happens *in Grafana*, where PMM has no hook and gets no notification | **Real.** The collector keeps emitting `pmm_alert_threshold_override` series for a rule that no longer exists. Nothing queries them, but they consume VM ingestion and cardinality and grow without bound as rules churn | + +So the reconciler has exactly **one** deletion job, plus a refresh: + +It should also **refresh `grafana_rule_uid`** while it is there, since the listing already pairs each UID +with its `pmm_rule_id`. + +An optional periodic sweep for stray override rows (direct SQL, or a removal path that forgets the hook) +is cheap — one `DELETE … WHERE scope='node' AND target NOT IN (SELECT node_id FROM nodes)` — but it is +belt-and-braces, not required, because read-time resolution already makes such rows inert. It must never +touch `scope='cluster'` (§4.8). + +(The "an identical group re-POST is a no-op" measurement quoted in §7 belongs to the *rejected* inline +design, where a reconciler would re-render rules. It is not a licence for this one to write.) + +> **A background job may not be needed at all.** The read path already fetches the authoritative rule +> list from Grafana, so it could delete registry rows it finds absent as a side effect — no ticker. +> Trade-off: cleanup then happens only when someone exercises the API, and it still needs the leader check +> (`IsLeader()`), because a follower must not write. Worth weighing rather than assuming the ticker. + +Deletion safety rules: + +- Require a **successful, complete** listing. On any error or partial response, skip the cycle — never delete on incomplete data. +- Require absence in **K consecutive cycles** (K ≥ 2) before deleting, to survive transient inconsistency and creation races. +- **Use a single global listing** (`GET /api/ruler/grafana/api/v1/rules` returns every folder) and match on the `pmm_rule_id` label alone. **Do not scope the check to a per-rule folder** — that would delete live data: a user can move a rule between folders in Grafana, after which a folder-scoped check reports it absent and the reconciler removes the overrides of a rule that still exists. This is also the second reason `folder_uid` is not stored. + +### 4.10 Multi-scope resolution — the shared resolver + +Only node scope is implemented in the first increment, so the full resolver is a gate for the +**service/cluster increment**, not for the first merge. What must exist from day one is the **shared +function boundary**, so the API and the collector cannot drift: + +```go +// ResolveEffective returns exactly one value per target, precedence applied. +// The collector and ListNodeThresholds MUST both call this. Nothing else may +// implement precedence. +func ResolveEffective(rule *models.AlertRule, param string, + overrides []*models.AlertRuleThresholdOverride, + inv Inventory) map[string]float64 // join-label value -> effective threshold +``` + +Invariants to assert and table-test: + +- **Exactly one emitted series per `(rule, param, target)`.** This is not a tie-break preference: if the resolver ever emits two series with identical labels, the Prometheus gatherer errors and the **entire** `/metrics` response fails. `node_name` and `service_name` are both `UNIQUE` (`database.go:117`, `:139`), so live targets cannot collide — the risk is a resolver bug, so assert it. +- **Precedence `node` → `service` → `cluster`**, most specific first. A param legal at both service and cluster scope with both present resolves to **service**. +- **Cluster scope is only legal for params whose join label is service-level.** A node can host services from several clusters, so "the cluster override for this node" has no unique answer. This is a validation rule on `override_scopes`, not a runtime tie-break. +- **Unresolvable targets are skipped**, never defaulted to something else. + +### 4.11 High availability + +Every pmm-managed node registers collectors unconditionally and scrapes its own localhost, labelled +`instance = PMM_HA_NODE_ID`, so an N-node cluster emits N copies. Postgres is shared/external in HA, so +the copies are identical and `max by ()` collapses them. + +> **Invariant:** never drop `max by ()`. It looks redundant once the metric is override-only, +> but it is what makes the design HA-safe — and its absence only manifests with more than one node. + +The **reconciler must be leader-only**: +`haService.AddLeaderService(ha.NewContextService("threshold-reconciler", fn))` (`haservice.go:569`). +`IsLeader()` returns true when HA is disabled, so single-node deployments are unaffected. + +### 4.12 The collector that already exists on this branch — and what to change + +`main` has none of this, but **this branch already carries a working collector** — +`managed/services/alerting/threshold_metrics.go`, registered at `managed/cmd/pmm-managed/main.go:1034`. +It is not the design above. It implements **emit-everything**: for every rule, for every param, for every +node in inventory, it emits `pmm_alert_threshold{rule_id, param, node_name}` — its own doc comment says +"emitting a value for every node ensures unoverridden nodes still evaluate against the default". So the +recommendation in §2 is a **change away from what is built**, and the emit-everything row in §7 is a +measurement of live code, not a thought experiment. + +**Measured on that collector**, 1011 nodes seeded into `nodes`, scrape timeout `0.9 × MR = 9 s`: + +| nodes × rules × params | series per scrape | median | worst | % of 9 s budget | +|---|---|---|---|---| +| 1011 × 1 × 1 | 1,011 | 76 ms | 85 ms | 0.9 % | +| 1011 × 20 × 1 | 21,231 | 385 ms | 514 ms | 5.7 % | +| 1011 × 20 × 5 | 102,111 | 1,870 ms | 2,070 ms | 23 % | +| 1011 × 42 × 5 | 213,321 | 4,121 ms | 4,511 ms | 50 % | +| 1011 × 20 × 10 | 203,211 | 4,395 ms | **6,209 ms** | **69 %** | +| 1011 × 42 × 20 | 850,251 | **26,964 ms** | — | **blown** | + +At the scenario §7 quotes — 1000 nodes × 20 rules, one param each — emit-everything costs **385 ms** and +is genuinely fine. The exposure is **multi-param rules**: `main` ships 42 templates, so the 50–69 % rows +are reachable rather than hypothetical, and `worst` is what matters because one blown scrape drops the +whole endpoint. + +**What a blown scrape actually does**, observed at 42 × 20: + +``` +scrape_duration_seconds{job="pmm-managed"} 9.001 +up{job="pmm-managed"} 0 +count(pmm_managed_inventory_nodes) EMPTY <- collateral +``` + +The threshold collector takes the **inventory collector down with it** — one endpoint, one timeout. Under +the inventory-fan-out shape this was a compounding failure: a slow threshold collector silently disabled +every rule within 15 minutes (§4.7). The observed-query fan-out removes that path. + +**Three defects to fix regardless of which emission policy wins:** + +1. **No timeout.** `Collect` uses bare `context.Background()`. The inventory collector next door bounds + itself with `requestTimeout = 3 * time.Second` (`inventory_metrics.go:33`). That missing bound is *why* + the scrape runs 27 s instead of returning partial data, and why the blast radius reaches other + collectors. +2. **`Describe` uses `prom.DescribeByCollect`** (`threshold_metrics.go:67`) — exactly what §4.2 says not + to do, since it runs a full `Collect`, and therefore a database query, merely to describe the collector. +3. **Node scope only.** Overrides are keyed on `o.NodeID`; there is no polymorphic `target`, no `scope`, + and no shared resolver, so §4.4, §4.6 and §4.10 are unbuilt. + +Confirmed from the source, matching §7's estimate: `2 + R` queries per scrape — `FindAlertRules`, +`FindNodes`, then `FindThresholdOverridesByRule` once per rule, all inside one transaction. + +Switching it to override-only is a change to the same `Collect` loop — emit where an override or tombstone +exists, drop the per-node fan-out — which makes an apples-to-apples comparison cheap on the same code +path, same DB, same endpoint. + +--- + +## 5. Worked examples + +### 5.1 Multi-expression template — one line to make it overridable + +`pmm_node_high_cpu_load` is already multi-expression on `main`. The whole template-side change: + +```yaml + params: + - name: threshold + summary: A percentage from configured maximum + unit: "%" + type: float + range: [0, 100] + value: 80 + overridable: true # <-- the only addition + override_scopes: [node] # <-- optional; defaults to [node] / node_name +``` + +Generated rule, with one override (`node-02` → 90) and default 80: + +| refId | body | +|---|---| +| `A` | `(1 - avg by(node_name) (rate(node_cpu_seconds_total{mode="idle"}[5m]))) * 100` *(unchanged)* | +| `T_threshold` | `max by (node_name) (label_replace(pmm_alert_threshold_override{rule_id="7f3a…", param="threshold"}, "node_name", "$1", "target", "(.*)")) or (max by (node_name) ((1 - avg by(node_name) (rate(node_cpu_seconds_total{mode="idle"}[5m]))) * 100) * 0 + 80)` | +| `C` | `$A > $T_threshold` | + +Resolves to `node-02 = 90`, every other **reporting** node `= 80`. Note the second clause is `A`'s own +expression with its value discarded by `* 0` — that is what makes the threshold share `$A`'s fate instead +of depending on a metric pmm-managed publishes (§4.7). The duplication is generated, never hand-written, +and §5.2 already parses the template's query on the AST for the single-expression case. + +> **The `T_` step is always emitted, including when no override rows exist.** Omitting it for +> untuned rules looks like a free optimisation — the rule would stay byte-identical to today's output — +> and it is **wrong**: a rule created without the step needs its definition changed to add one when the +> first override arrives, which resets alert state for every instance on that rule, the precise failure +> this design exists to avoid. With no overrides the step is still present and simply resolves to the +> default for every target. +> +> The honest cost: **every overridable rule pays the fan-out scan (~N × 91 samples per evaluation) +> whether or not anyone has tuned it.** The "untuned rules cost nothing" property does not exist. + +### 5.2 Single-expression template — desugared + +`pmm_mysql_too_many_connections` on `main` is single-expression: + +```yaml + expr: |- + max_over_time(mysql_global_status_threads_connected[5m]) / ignoring (job) + mysql_global_variables_max_connections + * 100 + > bool [[ .threshold ]] +``` + +Marking its param `overridable: true` triggers desugaring at build time — the template file keeps its +`expr:` form; only the generated rule changes: + +| refId | body | +|---|---| +| `A` | `max_over_time(mysql_global_status_threads_connected[5m]) / ignoring (job) mysql_global_variables_max_connections * 100` | +| `T_threshold` | the threshold expression (join label `service_name`, fan-out over `pmm_managed_inventory_agents{service_name!=""}`) | +| `C` | `$A > $T_threshold` | + +Two mechanical details: + +1. **`> bool` disappears** — Grafana math comparisons already yield 0/1. +2. **`{{ $value }}` must become `{{ printf "%.2f" $values.A.Value }}`** — with multiple refIDs `$value` is no longer a single scalar. This template's `description` uses `{{ $value }}`, so shipping desugaring without the rewrite ships broken alert text. Rewrite it during desugaring (annotations are filled at `service.go:744`). + +Constraints inherited from parsing an `expr:`: the token must be the RHS of the **final** comparison, and +a single expression admits **one** overridable param. Reject at parse time otherwise. + +#### Do the split on the AST, not with a regular expression + +A permissive regex split is the wrong tool: it mishandles parentheses, comparison modifiers (`bool`), +vector-matching clauses (`on`/`ignoring`, `group_left`), and nested comparisons, and it fails *silently* +by producing a plausible-but-wrong `A`. `github.com/prometheus/prometheus v0.313.0` is **already a direct +dependency on `main`** (`go.mod:61`), so the parser is available with no new dependency. + +The one wrinkle is that `[[ .threshold ]]` is not valid PromQL, so it must be replaced before parsing. +Pad the replacement to the **same byte length** as the token and AST positions map 1:1 onto the original +text — which means `A` can be sliced out of the *original* string, preserving the author's formatting +exactly: + +```go +// splitSingleExpr splits a single-expression template into the observed query and +// the comparison operator, so the builder can emit A + T_ + math C. +// +// The [[ .param ]] token is replaced by a byte-length-padded numeric sentinel so +// that PromQL positions map 1:1 onto the original text. +func splitSingleExpr(expr, paramName string) (lhs string, op parser.ItemType, isBool bool, err error) { + token := paramTokenRegexp(paramName).FindString(expr) + if token == "" { + return "", 0, false, fmt.Errorf("param %q is not referenced in the expression", paramName) + } + + // "0" plus spaces to the token's exact byte length: parseable, and position-preserving. + sentinel := "0" + strings.Repeat(" ", len(token)-1) + probe := strings.Replace(expr, token, sentinel, 1) + + parsed, err := parser.ParseExpr(probe) + if err != nil { + return "", 0, false, fmt.Errorf("failed to parse expression: %w", err) + } + + // The threshold must be the RHS of the outermost comparison. + bin, ok := parsed.(*parser.BinaryExpr) + if !ok || !bin.Op.IsComparisonOperator() { + return "", 0, false, errors.New("an overridable param must be the right-hand side of the expression's top-level comparison") + } + if num, ok := bin.RHS.(*parser.NumberLiteral); !ok || num.Val != 0 { + return "", 0, false, errors.New("an overridable param must be compared directly, not used inside a larger expression") + } + + // Vector matching on the comparison itself cannot survive the split into a + // Grafana math step, so reject it rather than silently changing semantics. + if bin.VectorMatching != nil { + return "", 0, false, errors.New("vector matching on the threshold comparison is not supported for overridable params") + } + + // Positions are byte offsets into probe, which is the same length as expr. + r := bin.LHS.PositionRange() + return strings.TrimSpace(expr[r.Start:r.End]), bin.Op, bin.ReturnBool, nil +} +``` + +Everything the regex approach would have to special-case is now either handled by the parser or rejected +with a precise message. Note `bin.ReturnBool` captures the `bool` modifier — which is then **dropped**, +because Grafana math comparisons already yield 0/1 (§5.2, gotcha 1). + +**Verification must precede implementation.** The filter-vs-0/1 semantic difference is not proven for +real alert lifecycle behaviour — `for:` pending/firing transitions, NoData, recovery, annotation +rendering. Desugaring an existing template before that check risks changing *alerting behaviour* while +believing you only changed *threshold delivery*. See §9, gate 3. + +### 5.3 Two params at two different scopes — validated live + +The case that proves the design generalises. Both params in one rule, with different join labels: + +```yaml + queries: + - ref_id: A # service-scoped (carries node_name too) + expr: |- + sum by (service_name, node_name) (pg_stat_database_numbackends) + / on (service_name, node_name) group_left() + max by (service_name, node_name) (pg_settings_max_connections) * 100 + - ref_id: B # node-scoped + expr: |- + (1 - avg by(node_name) (rate(node_cpu_seconds_total{mode="idle"}[5m]))) * 100 + expressions: + - ref_id: C + type: math + expression: "$A > [[ .connections_threshold ]] && $B > [[ .cpu_threshold ]]" + condition: C + params: + - name: connections_threshold + value: 80 + overridable: true + override_scopes: [service, cluster] # -> join label service_name + - name: cpu_threshold + value: 80 + overridable: true + override_scopes: [node] # -> join label node_name +``` + +Deployed as four rules on a live server (actuals: connections ≈ 0.7 %, CPU ≈ 5.4 %): + +| Case | Overrides | Effective thresholds | Result | +|---|---|---|---| +| none | — | conn **0.50** / cpu **3.00** (both fan-out defaults) | **fires** | +| node only | `cpu_threshold`@node = 90 | conn 0.50 / cpu **90** | does not fire | +| service only | `connections_threshold`@service = 90 | conn **90** / cpu 3.00 | does not fire | +| node **+** service | cpu@node = 1, conn@service = 0.1 | conn **0.10** / cpu **1.00** | **fires** | + +The last row is the load-bearing one: its **defaults are 90/90**, so it could never fire on defaults. It +fired, annotated `conn=0.70% (thr 0.10) cpu=5.36% (thr 1.00)` — two overrides at two different scopes +applied simultaneously in one rule. + +This also proved **Grafana math subset-matches across differing label sets**: `$A{node,service}` against +`$T{service}`, `$B{node}` against `$T{node}`, then `&&` across the two results. That had been an +assumption in every prior write-up. + +### 5.4 Annotations + +`{{ printf "%.0f" $values.T_.Value }}` renders the **effective per-node** threshold — verified, +including on a multi-param rule. Templates should use it rather than interpolating `[[ .threshold ]]`, +which freezes the default at creation time and is then wrong for every overridden target. Word it +neutrally (`"CPU load is X%, threshold Y%"`), because annotations are also rendered for tracked +non-firing instances. + +--- + +## 6. Implementation plan + +Ordered so each step is independently reviewable. All paths are `main` paths. + +| # | Step | Files | Effort | +|---|---|---|---| +| 1 | `overridable` + `override_scopes` on `Parameter`; validation (float only, must be a `[[ .name ]]` token in an expression step, at most one for a desugared single-expression template) | `managed/pi/alert/parameter.go:25`, `managed/pi/alert/query.go` validation, `template.go:113` | S | +| 2 | Migration 119 (**including `cleared_at`**, §4.4) + reform models + helpers (`FindThresholdOverrides`, `FindThresholdOverridesByTarget`, `UpsertThresholdOverride`, `ClearThresholdOverride` — tombstones, does **not** delete, `DeleteThresholdOverridesForTarget`, `FindAlertRules`, `CreateAlertRule`, `DeleteAlertRule`) | `managed/models/database.go:1184`, new `alert_rule*_model.go` + `alert_rule_helpers.go`, `make gen` | M | +| 3 | Threshold collector — **rewrite, not create: it already exists on this branch and emits every node** (§4.12). Switch to override-or-tombstone emission; one indexed query + bounded ID→name resolution; fixed 3-label desc. Three defects to fix in the same pass, independent of emission policy: **bound `Collect` with a 3 s timeout** (it uses bare `context.Background()`, which is why a slow scrape reached 27 s and took the inventory collector with it), replace **`prom.DescribeByCollect`** with `ch <- c.desc`, and add `scope` handling | `managed/services/alerting/threshold_metrics.go` (exists), registered `main.go:1034` | S–M | +| 4 | Rule builder: allocate `T_` refIDs, inject the threshold step, swap `[[ .param ]]` → `$T_`; thread `ruleID` through | `rule_builder.go:58`/`:81`, reusing `newPromQueryData:141` and `newMathExpressionData:165` | M | +| 5 | `CreateRule`: mint `rule_id`, stamp a `pmm_rule_id` label, persist the registry row with the `params` snapshot, return the id. Also have `CreateAlertRule` parse and return the created Grafana UID (already in the POST response body) and store it as `grafana_rule_uid` | `service.go:679`, `grafana/client.go:712` | S | +| 6 | Single-expression desugaring: split `expr` into `A` + `T_` + math `C`; rewrite `{{ $value }}` → `{{ $values.A.Value }}` | new `managed/pi/alert/overridable.go`, `rule_builder.go`, annotations at `service.go:744` | S–M | +| 7 | API: `overridable` on the param-definition message; `rule_id` on `CreateRuleResponse`; list/set/clear threshold RPCs with `scope`/`target` fields present but node-only behaviour — **clear tombstones the row rather than deleting it**, and list must report `cleared_at IS NOT NULL` as *not overridden* (§4.4); shared precedence function. **Route shape is pending §10 Q1** — node-centric vs generic — and that answer is needed before the proto freezes | `api/alerting/v1/alerting.proto`, new `managed/services/alerting/threshold_overrides.go`, `make gen` | M | +| 8 | `ListAlertRules` on the Grafana client + leader-only reconciler (registry-row orphans only). Separately, hook `DeleteThresholdOverridesForTarget` into `RemoveNode` / `RemoveService` (§4.8) | `managed/services/grafana/client.go:712` area, `deps.go`, `haservice.go:569`, `models/node_helpers.go:255`, `models/service_helpers.go:354` | S | +| 9 | Mark built-in templates overridable — `node_high_cpu_load.yml` (one line) and, once step 6 lands, `mysql_too_many_connections.yml` | `managed/data/alerting-templates/` | XS | +| 10 | Add `cluster` to the inventory services descriptor, before anything depends on that label set | `managed/services/inventory/inventory_metrics.go:84` | XS | + +**3–5 weeks** for one engineer for the node-only increment. + +The earlier figure of 2–3 weeks covered the ten steps above with unit tests and silently excluded +everything else. Stated properly: + +| Included | Excluded | +|---|---| +| The ten steps, with unit tests | UI | +| `make gen` cycles for proto and reform | Service and cluster **behaviour** (schema/API only) | +| AST-based desugaring (§5.2) and its rejection cases | Scale re-measurement on a large inventory | +| Cross-store failure handling and the reconciler (§4.9) | HA cluster testing beyond the `max by` invariant | +| GC for stale targets (§4.8) | API review turnaround | + +Assumes the §9 decision gates are resolved **in parallel** with implementation, not serially. If gate 1 +(NoData/availability) or gate 5 (precedence sign-off) blocks, the critical path extends by that wait +rather than by engineering time. + +Deliberately **not** needed: `UpdateAlertRule`, a per-group write mutex, `uid`-preserving JSON patching, +Grafana provenance, PromQL parsing of generated text, a VM write path, or a keep-alive writer. + +**Tombstones *are* needed** — an earlier revision listed them here as avoided. They are what makes a clear +take 14–21 s instead of 309 s (§4.4), and they cost one nullable column plus a filter in the list path, not +a writer or a sweeper. + +### Follow-ups, costed + +| Follow-up | Effort | Why not now | +|---|---|---| +| Dedicated endpoint + own scrape job, giving `up{job="pmm-thresholds"}` | XS–S | Isolation and hygiene, not behaviour. `promscrape.yml` is generated in pmm-managed (`victoriametrics.go` `populateConfig`), so it stays in-tree. | +| Service and cluster **behaviour** | M | Schema and API already generalised; needs the precedence function plus the `cluster` inventory label from step 10. | +| UI | M | Separate deliverable. | + +--- + +## 7. Alternatives rejected + +Each with the measurement that settled it. + +| Alternative | Why not | +|---|---| +| **Inline the overrides as literals in the rule query** (no metric) | A rule-definition change **resets alert state for every instance on that rule** — `activeAt` moved `14:09Z → 14:18Z` for a node whose own threshold was untouched, with `uid` *and* `guid` preserved. Under `for: 5m`, tuning one node blinds every other node on that rule for five minutes. Batching, one-rule-per-group, shortening `for:` and `keep_firing_for` all fail to preserve the timer, because Grafana keys state on the rule definition. | +| **Put the canonical state inside the rule** (no tables) | Same state-reset cost, plus no contractual home: either parseable PromQL (symmetric escaping of user-controlled `node_name`, stable float formatting, permanent support for old shapes) or a JSON sidecar in the query model. The sidecar survives both an API round-trip and a UI edit on Grafana 12.4.5 — but rests on an undocumented implementation detail that an upgrade could remove silently. | +| **Read-only rules via Grafana provenance** | Group-wide and irreversible: claiming provenance makes the Ruler API reject the **whole group** (`400`), and release fails with `409 alerting.provenanceMismatch`. Only exit is delete-and-recreate, minting a new UID and breaking silences. | +| **Emit a threshold for every node** (default or override) | `rules × params × nodes`, ~99 % of it identical defaults. **Not rejected on query cost** — measured at 1000 nodes it is 4.5 ms, indistinguishable from the recommendation (§4.3). Rejected on **scrape cost and blast radius**, measured on the collector already on this branch (§4.12): fine at one param per rule (385 ms), 69 % of the 9 s budget at 20 rules × 10 params, and at 42 × 20 it blows the timeout and takes `pmm_managed_inventory_nodes` down with it. Also loses the in-rule literal fallback, and turns an orphaned `alert_rules` row from one inert row into \|inventory\| live series. | +| **Push overrides into VM on change** (import API / remote-write) | Buys only propagation latency — immediate vs ≤ 1 scrape — which is noise against a 60 s evaluation interval. Costs a keep-alive writer and retry/ordering logic. (The tombstone this row once counted against it is now adopted anyway — §4.4 — so it is no longer a differentiator; the writer and the ordering logic still are.) | +| **Cache overrides in a `GaugeVec` mutated on write** | Needs restart warm-up and produces N duplicated series in HA, for no gain: after override-only emission the read is one indexed query over a tens-of-rows table. | +| **A Prometheus-compatible "PMM Thresholds" datasource** | Architecturally cleanest — zero duplication, immediate clears, precedence purely in Go — but the largest new surface (a hand-written API shim whose contract with Grafana's Prometheus datasource is version-sensitive), it touches `build/ansible`, and it puts a synchronous Postgres round-trip inside every rule evaluation. Revisit if thresholds must change without an API call (schedules, computed baselines). | +| **Exception-rule splitting** / thresholds as scrape-target labels | Rule splitting rewrites the base rule anyway and changes alert identity when a target moves between rules, breaking dedup and silences. The label approach needs a vmagent reload per override change and pollutes every scrape target. | + +### Known costs of the recommendation + +- **Two stores.** A rule deleted in Grafana orphans an `alert_rules` row; paid for by the reconciler (step 8), which is safe to run on a timer because an identical group re-POST is a genuine no-op (`"no changes detected in the rule group"`, `activeAt` preserved). +- **The observed expression is evaluated twice per rule evaluation** — once as `$A`, once as the fan-out that manufactures the default's label set. Measured **~1.25×** on a real heavy query (`rate(node_cpu_seconds_total[5m])`, 9.4 → 11.8 ms), not 2×. Re-measure for the heaviest overridable template before GA. +- **The default's target set is "has observed data", not "is in inventory".** A node in inventory whose exporter is down gets no threshold — and no `$A` either, so no alert instance. §8 already accepted that as harmless, but it is a semantic change, not a no-op. +- **A tombstone row per target ever tuned**, never garbage-collected (§4.4). Bounded by human action rather than inventory, but monotonic. +- **The default is a literal in the rule**, so changing a default for everyone is a rule edit. +- **Threshold history lives only as long as VM retention** (~30 days). + +--- + +## 8. Verification + +Verified on two live servers. **Env A:** Grafana 12.4.5, VM v1.147.0, 3 nodes / 1 service. **Env B** +(2026-08-20/21, all figures below unless marked A): PMM 3.7.1, Grafana 12.4.5, **VM v1.149.0**, MR = 10 s, +`latencyOffset=5s`, `disableCache=true`, 11 real nodes / 40 k real series, plus 1000 synthetic nodes with +2 h of backfilled history for the scale figures. + +| Verified | Result | +|---|---| +| Threshold expression: no overrides → all targets at default; one override → that target only | ✅ | +| Override that **raises** the bar suppresses a target the default would have fired | ✅ | +| Target in inventory but reporting no data → no alert instance | ✅ harmless | +| Node + service overrides in one rule, two join labels | ✅ §5.3 | +| Grafana math subset-matches across differing label sets | ✅ | +| Override series expire independently per `(rule, param, target)` | ✅ A | +| `$values.T_.Value` in annotations | ✅ A, multi-param | +| **Clear by absence** (row deleted) | ✅ **309 s**, reproducing env A's 280–300 s | +| **Clear by value change** (row tombstoned, §4.4) | ✅ **14–21 s**, two independent runs — ~15× faster | +| **Inventory fan-out under loss of the fan-out metric** | ✅ raw series gone at 259 s, `T` empty at 856 s, rule `state=inactive health=ok` at **905 s** while `$A` still breached — the silent stop, reproduced | +| **Observed-query fan-out under the same loss** | ✅ `T` held at 80, rule stayed `firing` for the full 26 min | +| Window cost at 1000 nodes: bare / `[15m]` / `[2h]` | ✅ 3.8 / 3.8 / **4.5 ms** — window length is immaterial | +| Observed expression evaluated twice | ✅ 9.4 → **11.8 ms** (~1.25×, not 2×) | +| `samplesScanned` as a cost proxy on VM v1.149.0 | ❌ **invalid** — `count_over_time[1h]` reads 360 samples/series yet reports the same 151 as `last_over_time[1h]`; only `bare = 61` reproduces | +| Emit-everything collector scrape cost vs the 9 s timeout | ✅ §4.12 — 385 ms at 20×1, 69 % of budget at 20×10, blown at 42×20 | +| A blown pmm-managed scrape takes the inventory metric with it | ✅ `up=0`, `scrape_duration=9.001`, `count(pmm_managed_inventory_nodes)` EMPTY | + +Still to verify: + +1. **Filter vs 0/1 semantics after desugaring** — a PromQL comparison without `bool` filters series, Grafana math yields 0/1. The firing set should match; confirm against a real rule including `for:` and NoData. Live check, not a unit test. +2. **Behaviour at scale — query side now measured, collector side partly.** The 1000-node figures above are real, but the nodes were synthetic rows and synthetic series, not registered inventory with live agents. Still open: the override-only collector's scrape cost at scale (§4.12 measured emit-everything; the override-only variant has not been built), and the whole picture under HA with more than one pmm-managed. +3. **Three scopes on a single param**, resolved most-specific-first in Go. +4. **`-promscrape.config.dryRun` acceptance** of a new scrape job, if the dedicated-endpoint follow-up is taken. + +Procedure: [`dev/docs/process/running-and-verifying-locally.md`](dev/docs/process/running-and-verifying-locally.md). + +--- + +## 9. Decision gates + +Promoted out of "open questions" because each can invalidate GA behaviour. None is an engineering +unknown; each needs a decision or a test result. + +| # | Gate | Owner | Blocks | +|---|---|---|---| +| 1 | **`no_data_state` / availability — largely retired by the §2 fan-out change, confirm and close.** The failure was real and measured: a rule with a missing threshold reports `state=inactive, health=ok` at 905 s while `$A` is still breaching. Fanning the default out over the observed expression makes `T` unable to vanish while `$A` resolves, so the cliff is removed structurally rather than bounded by a window — verified over 26 min (§8). Flipping to `NoData` remains non-viable (it would fire for every legitimately absent target). What is left to decide: whether the remaining step — overridden targets reverting to defaults after ~5 min of pmm-managed downtime, detected by `up{job="pmm-managed"}` — is acceptable for GA. | Product + Eng | GA | +| 2 | **Stale-target GC** implemented and tested per §4.8 | Eng | merge | +| 3 | **Desugaring semantics verified live** — firing, pending under `for:`, recovery, NoData, annotation rendering — *before* step 6 is implemented | Eng | step 6 | +| 4 | **Cross-store creation semantics** implemented per §4.9 (ordering, idempotency key, K-cycle absence rule) | Eng | merge | +| 4b | **Registry-row orphan GC** — rows for rules deleted in Grafana, so emitted series do not grow without bound. Narrowed: override rows for deleted nodes/services are handled by the removal API hooks (§4.8), not here | Eng | GA | +| 5 | **Multi-scope precedence signed off** (`node` → `service` → `cluster`; cluster legal only for service-level join labels) *before* the proto freezes | Product | API freeze | +| 5b | **API style chosen** — node-centric vs generic routes (§10 Q1). Routes are additive-only, so deciding late means carrying both families forever | Product + Eng | API freeze | +| 6 | **Shared resolver** (§4.10) is the single implementation of precedence, table-tested across scope combinations | Eng | service/cluster increment | +| 7 | **Fan-out re-measured at representative scale and under HA** | Eng | GA | +| 8 | **Every overridable param emits its `T_` step at creation** — no conditional omission | Eng | merge | + +--- + +## 10. Open questions + +Everything that gates GA or the API freeze has moved to §9, **except the first item below, which is +open but time-critical**: it must be settled before the proto is frozen. + +### 1. Which API style — node-centric routes, or generic scope/target routes? + +**Undecided, and urgent.** HTTP routes are additive-only, exactly like proto fields: whichever style +ships first can never be removed. So if node-centric routes ship now and generic routes are added when +service and cluster scope land, PMM carries **both families indefinitely**. Choosing later is not +neutral — it is choosing duplication. + +Note this is a *different* question from the one already settled. The decision to "generalise the schema +and API now, with node-only behaviour" covers the **message fields** (`scope`/`target` alongside +`node_id`). It does not cover the **paths**, and the paths are the half that cannot be changed afterwards. + +**Style A — node-centric** (what the feature branch implements): + +``` +GET /v1/alerting/nodes/{node_id}/thresholds +POST /v1/alerting/nodes/{node_id}/thresholds +DELETE /v1/alerting/nodes/{node_id}/thresholds/{rule_id}/{param_name} +``` + +**Style B — generic:** + +``` +GET /v1/alerting/thresholds?scope=node&target=[&rule_id=] # all filters optional +POST /v1/alerting/thresholds body: {scope, target, rule_id, param_name, value} +DELETE /v1/alerting/thresholds?scope=&target=&rule_id=¶m_name= +POST /v1/alerting/thresholds:batchUpdate body: {updates: [...]} # optional, see below +``` + +| | Style A — node-centric | Style B — generic | +|---|---|---| +| Adding service/cluster scope later | needs a second route family, kept forever | no new routes | +| Discoverability / REST shape | ✅ clearer resource hierarchy | ⚠️ filters are less self-describing | +| Serves admin and debug reads ("all overrides for rule R", "everything") | ✅ needs extra endpoints | ✅ one route, optional filters | +| UI must send `scope` explicitly | no | yes (trivial) | +| Matches the branch, so no UI rework | ✅ | ⚠️ small client change | + +**Three facts that constrain the choice regardless of which style wins:** + +1. **`target` cannot be a path segment.** `node_id` and `service_id` are opaque IDs, but a `cluster` target is an **arbitrary label value** — `prod/us-east` breaks grpc-gateway path matching outright (path params do not match `/` without `{target=**}`), and spaces or unicode need encoding. PMM's own convention agrees: every existing path segment in `api/` is an opaque ID, never a free-form value. So `/thresholds/{scope}/{target}` is not an option in either style. +2. **A batch method is idiomatic here.** PMM already uses AIP-style custom verbs — `/v1/accesscontrol/roles:assign`, `/v1/actions:startServiceAction`, `/v1/advisors/checks:batchChange`, `/v1/inventory/services:getTypes`. This matters because the UI diffs a whole modal of rows on submit and currently fires **N separate set/delete calls**, so a partial failure leaves the modal half-applied with no way to report which rows landed. `checks:batchChange` is the in-repo precedent for making that one transactional call. Worth deciding alongside the style, since it is the same proto change. +3. **`rule_id` is not a unique key in the response.** Duplicated rules in Grafana share a `pmm_rule_id` (§4.4), so a list can legitimately return two entries with the same `rule_id` *and* `param_name`, differing only in rule title. Nothing breaks — the field is `repeated` — but a client keying a map on `(rule_id, param_name)` would silently collapse them. State it in the proto comments rather than leaving it to be discovered. + +**Recommendation if a tie-break is needed:** Style B, on the strength of the additive-only argument +alone — the cost of being wrong is permanent route duplication, versus a one-off loss of REST +readability. But this is an API-review call, and it belongs with gate 5's proto freeze in §9. + +### Genuinely open, not blocking + +2. **Audit history.** Nothing records "the threshold when this fired" beyond VM retention (~30 days). If + that is required, a small append-only table beats shaping the primary design around it. +3. **Do any planned overridable templates resist the `$X [[ .param ]]` shape?** The AST split + (§5.2) rejects a threshold used inside arithmetic, in a subquery, in a nested comparison, or with + vector matching on the comparison itself. Worth sketching one candidate before step 6 lands, so the + rejection set is validated against real intent rather than assumed. +4. **Whether an immediate delete hook should accompany the reconciler GC** (§4.8) for faster UI feedback. + It cannot replace the reconciler pass — removals can happen while pmm-managed is down, and in HA only + the leader may write — so this is a UX question, not a correctness one. +5. **Dynamic defaults.** The default is a literal in the rule, so changing it for everyone is a rule + edit. If defaults ever need to change without one — schedules, computed baselines — that is the + trigger to revisit the thresholds-datasource alternative in §7, where the whole computation is in Go. diff --git a/dynamic-thresholds-performance-testing.md b/dynamic-thresholds-performance-testing.md new file mode 100644 index 0000000000..e5c71a8d40 --- /dev/null +++ b/dynamic-thresholds-performance-testing.md @@ -0,0 +1,356 @@ +# Dynamic Alert Thresholds — performance testing session (2026-08-21) + +Companion to [`dynamic-thresholds-main-decision.md`](dynamic-thresholds-main-decision.md). That document's +§8 "Still to verify" item 2 flagged two open scale questions: **the override-only collector's cost has +never been measured (only emit-everything was measured, in §4.12)**, and **service/cluster scope has no +implementation to measure at all**. This session builds just enough of both to get real numbers, then +measures them live on the same kind of dev server §8's numbers came from. It does not attempt to finish +the feature — see [What was deliberately not done](#what-was-deliberately-not-done). + +> A chart-based visual companion to this document — the same measurements, plus the head-to-head +> comparison in [Scenario D](#scenario-d--override-only-vs-emit-everything-head-to-head) and the fixed +> real-world parameter sweep in [Scenario E](#scenario-e--fixed-real-world-parameter-sweep-1000-nodes-250-overrides-120-rules) +> — was published as an Artifact ("Threshold collector benchmarks"). + +## Environment + +Same dev container (`pmm-server`, `perconalab/pmm-server:3-dev-latest`) used for the live measurements +already in §8 of the decision doc, on this branch (`PMM-14912-dynamic-thresholds`, `73b7032db`). + +| | | +|---|---| +| PostgreSQL | 14.24 (Percona Distribution), `max_connections=2000` | +| VictoriaMetrics | scraped every `MR = 10 s` | +| Grafana | 12.4.5 | +| Real inventory | 11 nodes, 10 services (MySQL/PostgreSQL/Valkey/MongoDB test instances) | +| ClickHouse / qan-api2 | **down for this whole session** — ClickHouse is crash-looping on an unrelated `AccessControl` config error, pre-existing in this container, nothing to do with this feature. QAN has no dependency on alert thresholds, so this doesn't affect anything below; noted for completeness only. | + +Baseline `/debug/metrics` scrape before any change: **25 ms**, 499 lines, 58 KB, 0 threshold series. + +### Baseline infra snapshot (before synthetic load) + +| Store | Size | Detail | +|---|---|---| +| PostgreSQL | 10 MB total | Largest tables: `agents` 312 KB, `nodes` 280 KB, `alert_rules` 64 KB, `alert_rule_threshold_overrides` 32 KB (empty), `settings` 48 KB. 20 active connections. | +| VictoriaMetrics | ~59 MB on disk (`indexdb` 18.5 MB + `storage/small` 46.6 MB) | 42,323 active series, 467,147 label-value pairs, 59 scrape targets, ingesting ~5,745 rows/s, 892 MB resident. | +| ClickHouse | 116 KB | Crash-looping (see above); no QAN data this session. | +| pmm-managed process | 628 MB RSS, 162 goroutines | Sampled via `process_resident_memory_bytes{job="pmm-managed"}` / `go_goroutines{job="pmm-managed"}` through VictoriaMetrics. | +| Host | load1 ≈ 3.8, memory ≈ 31.6% used | Sampled via `node_load1` / `node_memory_*` for the `pmm-server` node — the same series PMM's own Node Overview / health dashboards render. The in-app Browser tool couldn't reach `https://localhost` (sandboxed-network policy blocks local-network navigation), so these were pulled directly from VictoriaMetrics' query API instead of a dashboard screenshot — same underlying data. | + +## What changed on this branch for this session + +The branch's existing collector (`threshold_metrics.go`) implemented **emit-everything** — the design +§7 of the decision doc rejects, and §4.12 measured. To benchmark the **recommended** design (override-only +emission, tombstones, multi-scope resolver) there was nothing to run yet, so this session built a real, +working version of the two pieces actually in scope (see the earlier scoping discussion in this +conversation): + +| File | Change | +|---|---| +| [`managed/models/database.go`](managed/models/database.go) | Migration 119's `alert_rule_threshold_overrides` rewritten to the polymorphic `scope`/`target` schema with a `cleared_at` tombstone column and the NaN/±Inf `CHECK`, per §4.4/§4.5 of the decision doc. | +| [`managed/models/alert_rule_threshold_override_model.go`](managed/models/alert_rule_threshold_override_model.go) | `ThresholdScope` type + `node`/`service`/`cluster` constants; struct fields `Scope`/`Target`/`ClearedAt` replacing `NodeID`. Regenerated via `go generate` (reform), not hand-edited. | +| [`managed/models/alert_rule_helpers.go`](managed/models/alert_rule_helpers.go) | `FindThresholdOverridesByTarget`, tombstone-aware `UpsertThresholdOverride`/`ClearThresholdOverride`, hard-delete `DeleteThresholdOverridesForTarget` (refuses `cluster` scope, per §4.8). | +| [`managed/models/threshold_resolver.go`](managed/models/threshold_resolver.go) *(new)* | `ResolveThresholds` — the shared precedence resolver from §4.10: node > service > cluster, tombstones contribute no candidate for their own scope, unresolvable targets are skipped. Table-tested in [`threshold_resolver_test.go`](managed/models/threshold_resolver_test.go) (7 cases) plus a microbenchmark. | +| [`managed/models/service_helpers.go`](managed/models/service_helpers.go) | `FindServicesByClusters` — bounded by the clusters actually queried, for cluster-scope expansion. | +| [`managed/services/alerting/threshold_metrics.go`](managed/services/alerting/threshold_metrics.go) | Full rewrite: override/tombstone-only emission (`pmm_alert_threshold_override{rule_id,param,target}`), one query for overrides + bounded ID→name/cluster→services resolution, 3 s `Collect` timeout, `Describe` sends the descriptor directly instead of `prom.DescribeByCollect` — the three defects §4.12/§6 step 3 called out, fixed in the same pass. **Both emission modes now live side by side** behind a `ThresholdEmitMode` parameter: `ThresholdEmitOverridesOnly` (default, recommended) and `ThresholdEmitEveryTarget` (the §7-rejected alternative, generalised from its original node-only shape to also emit for every service, so cluster/service scope can be A/B'd too). Selected via `PMM_DEV_THRESHOLD_EMIT_MODE=all-targets` — a dev-only toggle (`PMM_DEV_` prefix, never a GA knob), read once at startup in `managed/cmd/pmm-managed/main.go`. | +| [`managed/services/alerting/threshold_overrides.go`](managed/services/alerting/threshold_overrides.go) | Adapted to the new schema; `DeleteNodeThreshold` now calls `ClearThresholdOverride` (tombstone) instead of a hard delete, and `ListNodeThresholds` skips tombstoned rows — both required by §4.4. | + +`go build ./managed/...` and `go vet ./managed/...` are clean; `gofmt -l` reports nothing. Existing +`rule_builder_dynamic_test.go` tests (unaffected — they don't touch this schema) still pass. Two pre-existing, +unrelated failures were left alone: a host-only `mkdir /srv: read-only file system` failure in +`service_test.go` (macOS host has no `/srv`) and a flaky timing assertion in +`software_version_helpers_test.go`; neither touches thresholds. + +## Methodology + +Same technique as the decision doc's §4.12 emit-everything table: seed real Postgres rows in the dev +container, hot-swap the rebuilt `pmm-managed` binary (`make env-root TARGET=run-managed-ci`), and time +repeated `GET /debug/metrics` scrapes. Where the shared endpoint's total time was dominated by other, +unrelated collectors (see below), `EXPLAIN ANALYZE` isolates this collector's own two queries, and a Go +microbenchmark isolates `ResolveThresholds` in memory. All synthetic rows were prefixed `bench-` and +deleted at the end of the session; the container was left in its original state (11 nodes / 10 services / +1 pre-existing rule / 0 overrides). + +## Results + +### A — override count, node scope (2,000 synthetic nodes held constant) + +| Overrides | median | max | emitted series | scrape total lines | +|---|---|---|---|---| +| 0 | 108 ms | 133 ms | 0 | 2,530 | +| 100 | 119 ms | 175 ms | 100 | 2,643 | +| 1,000 | 124 ms | 141 ms | 1,000 | 3,549 | +| 2,000 (all nodes overridden) | 143 ms | 192 ms | 2,000 | 4,549 | + +Collector's own added cost going from 0 → 2,000 override series: **~35 ms**, not the multiplicative blowup +the old design showed. The 108 ms floor here is the pre-existing, unrelated inventory collector iterating +2,011 nodes on every scrape (see below) — not this feature. + +### A4 — same total overrides, fragmented across many rules/params + +The old design's real exposure (§4.12) was **rules × params**, not raw override count: 1011×42×20 blew the +9 s timeout at 26.9 s. This is the one number from that table worth reproducing under the new design: + +| Scenario | median | max | emitted series | +|---|---|---|---| +| 1,000 overrides, 1 rule × 1 param | 124 ms | 141 ms | 1,000 | +| 840 overrides, **42 rules × 20 params** | 121 ms | 136 ms | 840 | + +Statistically indistinguishable. Fragmenting the same override volume across 840 distinct `(rule, param)` +groups costs nothing extra — the collector still issues exactly one overrides query and one bounded +node lookup, then partitions in memory. **This is the direct answer to §4.12's worst row**: the +multi-param exposure was a property of emit-everything's node-multiplication, and it is structurally +gone under override-only emission, not just empirically smaller. + +### B — total inventory size, override count held constant at 50 + +| Total nodes | median | max | emitted series (this collector) | +|---|---|---|---| +| 2,000 | 108 ms | 118 ms | 50 | +| 6,000 | 324 ms | 382 ms | 50 | +| 10,000 | 461 ms | 508 ms | 50 | + +Emitted series from **this** collector stayed at 50 throughout — the ~350 ms of growth is entirely the +pre-existing `pmm_managed_inventory_nodes` collector (unrelated to this feature) still emitting one line +per node. Isolated with `EXPLAIN ANALYZE` at the 10,000-node point: + +- `SELECT * FROM alert_rule_threshold_overrides` (all 50 rows): **1.7 ms** +- `SELECT * FROM nodes WHERE node_id IN (<50 ids>)`, index-bounded: **3.6 ms** + +Under 6 ms combined, flat regardless of total inventory — confirming the design's central claim +(§4.12/§9 gate 7) that this collector's cost is bounded by *targets ever tuned*, not fleet size. + +### C — cluster-scope expansion (new: not measured anywhere before this session) + +Synthetic services seeded per cluster, one cluster-scope override per cluster on top of the scenario-B +inventory (so total scrape time below still carries that unrelated ~460 ms inventory-collector floor): + +| Cluster overrides | Services in those clusters | emitted series | scrape median | scrape max | +|---|---|---|---|---| +| 1 (100 services) | 100 | 100 | 799 ms | 886 ms | +| 2 (+1,000 services) | 1,100 | 1,100 | 823 ms | 1,033 ms | +| 3 (+5,000 services) | 6,100 | 6,100 | 845 ms | 1,055 ms | +| **50 small clusters × 200 services** (§4.2's stated worst case) | 10,000 | 10,000 | 1,482 ms | 1,566 ms | + +Isolated: + +- `SELECT * FROM services WHERE cluster IN (<3 clusters>)` at 6,100 matched rows: **23 ms** +- Same query for the 50-cluster case, 10,000 matched rows out of 16,110 total synthetic services: **47 ms** +- `ResolveThresholds` in memory, 50 cluster overrides expanding to 10,000 candidates (Go microbenchmark, + `BenchmarkResolveThresholds`): **~1 ms** + +§4.2's cardinality warning — "a cluster override over 200 nodes would emit 200 series" — is real and the +collector handles it correctly at 50× that scale, cheaply. The cluster-expansion query +(`services.cluster IN (...)`) is a **sequential scan** — there is no index on `services.cluster`. At +16,110 synthetic rows it's still only 47 ms. + +> **Revised in Scenario E below.** This session first flagged that scan as an "index it before cluster +> scope ships" action item. A more careful re-test — a *selective* override set (10% of clusters, not +> the ~100% coverage the number above was measured against) at 10,010 services — still executes in +> **6.6 ms**. Postgres reads a table this size in one or two pages regardless of a `WHERE` clause; an +> index cannot beat that. **The recommendation is retracted**: don't add the index on the strength of +> this testing. Revisit only if a real deployment's `services` table is one to two orders of magnitude +> larger than anything measured here — the scan cost is linear in table size, so re-test at that scale +> rather than assuming today's numbers still hold. + +## Scenario D — override-only vs. emit-everything, head to head + +Everything above measured the recommended design in isolation. To compare it directly against the +rejected alternative on identical data, the collector now carries **both** emission modes side by side +(see [What changed](#what-changed-on-this-branch-for-this-session) above) — a live toggle, not a +re-implementation from memory of §4.12's numbers. `ThresholdEmitEveryTarget` is a deliberate +generalisation of that historical, node-only design: it now also emits for every **service**, so the +comparison covers cluster/service scope too, not just node scope. + +### D1 — node scope, same override sweep as scenario A + +2,000-node inventory; two registered rules with default params (the pre-existing real rule plus +`bench-rule-1`) — same DB state, same moment, mode flipped via `PMM_DEV_THRESHOLD_EMIT_MODE` and a +restart between passes: + +| Overrides | override-only median / max | override-only series | all-targets median / max | all-targets series | +|---|---|---|---|---| +| 0 | 118 / 203 ms | 0 | 149 / 199 ms | **4,042** | +| 100 | 111 / 112 ms | 100 | 163 / 174 ms | **4,042** | +| 1,000 | 119 / 138 ms | 1,000 | 199 / 221 ms | **4,042** | +| 2,000 | 143 / 150 ms | 2,000 | 209 / 243 ms | **4,042** | + +The all-targets column is already the whole point: **4,042 emitted series regardless of override +count — including at zero.** That number is `2 rules × 1 param × (2,011 nodes + 10 services)`; it has +nothing to do with how many overrides exist, because this mode was never about overrides, it's about +inventory. Override-only's series count tracks the override count exactly, and its scrape time is lower +at every point on the sweep. A second, smaller effect: all-targets' own time still drifts upward with +override count (149→209 ms) even though its emitted-series count doesn't move — `ResolveThresholds` has +more candidates to fold into its per-target map, a real but minor cost next to the ~150 ms fixed floor of +walking the full inventory twice per rule. + +### D2 — cluster scope, same DB state under both modes + +2,011 nodes + 6,100 synthetic services across 3 clusters, one cluster-scope override per cluster: + +| Mode | median | max | emitted series | +|---|---|---|---| +| override-only | 550 ms | 925 ms | 6,100 | +| all-targets | 681 ms | 692 ms | **16,242** | + +All-targets pays for the full `2 rules × 1 param × (2,011 nodes + 6,100 services)` universe — 2.7× the +series override-only emits for the exact same three cluster overrides, because it was always going to emit +for every service whether or not a cluster override existed. + +### The extreme point, not a controlled pair + +Pushing all-targets to `2,011 nodes + 16,100 services` (50 small clusters × 200 services, matching §4.12's +worst-case shape but now over nodes *and* services): **36,242 emitted series, 1,790 ms median / 1,885 ms +max** — still under the 9 s budget for one param on one rule pair, but this is exactly the shape that hit +26.9 s at 42 rules × 20 params in §4.12's original, node-only measurement. The nearest override-only +comparison is scenario C's 50-cluster point (§C above): **10,000 series, 1,482 ms**, measured over a +different total inventory (10,000 nodes instead of 2,011). The two runs aren't a controlled pair — don't +read the exact ms gap as precise — but the series counts are exact and the qualitative result matches D1 +and D2 at every controlled point: override-only's cost is anchored to what was actually overridden, +all-targets' is anchored to total inventory size, full stop. + +*(Operational side note, not a thresholds finding: seeding 16,100 synthetic services made the unrelated +Advisor **checks** service log an error per service — "no available pmm agents" — on its periodic pass, +which briefly slowed `pmm-managed`'s graceful shutdown between mode-toggle restarts. Nothing to do with +the threshold collector; mentioned only because it's what made a restart briefly look stuck.)* + +## Scenario E — fixed real-world parameter sweep (1,000 nodes, 250 overrides, 120 rules) + +Requested directly: a single fixed, realistic parameter set — **1,000 nodes, 250 overrides, 100 rules with +1 param + 20 rules with 2 params** (140 `(rule, param)` groups total) — run through node, service, and +cluster scope, each under both emission modes. Unlike scenarios A–D, this inventory is **left seeded on +the dev container** for further exploration (see the note at the end of this section), rather than cleaned +up after measuring. + +### Setup + +| | | +|---|---| +| Nodes | 1,000 synthetic + 11 real = 1,011 | +| Services | 1,000 synthetic (100 clusters × 10) + 10 real = 1,010, later expanded to 10,010 (1,000 clusters × 10) for the cluster-selectivity re-test below | +| Alert rules | 100 with 1 param (`threshold`) + 20 with 2 params (`param_a`, `param_b`) = **140 `(rule, param)` groups** | +| Overrides | 250 rows, spread across all 140 groups (~1.8 per group on average), scope varied per pass | + +The 250 overrides were distributed with a fixed, reproducible mapping (`idx % 140` → group, `idx` → target) +so the same 250-row shape is tested identically at each scope, and it self-verified: no two rows landed on +the same `(rule, param, scope, target)` key, so all 250 inserted cleanly under the real +`UNIQUE (rule_id, param_name, scope, target)` constraint every time. + +### Results + +| Scope | Mode | Emitted series | Scrape median | Scrape max | Isolated DB cost | +|---|---|---|---|---|---| +| Node | overrides-only | **250** | 108 ms | 114 ms | overrides scan 1.6 ms + node lookup 2.1 ms | +| Node | all-targets | **284,961** | **6,907 ms** | 6,907 ms | dominated by the emission loop, not the query | +| Service | overrides-only | **250** | 109 ms | 111 ms | service lookup 2.3 ms | +| Service | all-targets | **284,961** | **7,394 ms** | 7,394 ms | same order as node scope — see below | +| Cluster | overrides-only, 100% of 100 clusters selected (1,010 services) | **2,500** | 156 ms | 164 ms | seq scan, matches ~100% of table: 5.9 ms | +| Cluster | overrides-only, 10% of 1,000 clusters selected (10,010 services) | **2,500** | 620 ms | 818 ms | seq scan, 10% selective: 6.6 ms | +| Cluster | all-targets | *not run — see below* | — | — | — | + +Two clean, exact formulas fall out of these six rows: + +- **override-only emits exactly `resolved distinct targets`** — 250 override rows resolve to 250 distinct + node/service names at node/service scope, and to **2,500** at cluster scope, because each of the 250 + cluster-scope rows expands onto its cluster's 10 services (250 × 10 = 2,500, exactly what was measured — + not an estimate). +- **all-targets emits exactly `groups × (nodes + services)`, independent of override count and scope** — + **141** groups (the 140 bench groups plus one pre-existing real alert rule left over from earlier in the + session, confirmed by `SELECT sum(...jsonb_object_keys(default_params)...) FROM alert_rules` = 141) × + `(1,011 + 1,010) = 2,021` = **284,961** — exact, not approximate; there is no rounding gap. Node scope and + service scope produced the **same** all-targets series count and the same order-of-magnitude scrape time + (6,907 ms vs 7,394 ms) — confirming empirically, not just architecturally, that all-targets' cost has + nothing to do with which scope is being overridden. + +**The headline number: 6.9–7.4 seconds of a 9-second scrape budget**, for just 140 `(rule, param)` groups — +the exact shape §4.12 warned about (there, 42 rules × 20 params on nodes alone hit 26.9 s and blew the +budget outright). This run didn't blow the budget, but it used **77–82% of it**, with only 120 rules in +play — a fraction of what a real PMM deployment (42+ shipped templates, more once overridable params +grow) would register once several are made overridable under this design. + +**A collector-implementation note surfaced by this run, not by earlier ones:** the 3 s `Collect` timeout +this session added (§4.12/§6 step 3) bounds the *DB query* phase, but `collectEveryTarget`'s emission loop +— the part that actually costs 6–7 s here — runs entirely in Go after the queries return, with no +`ctx.Done()` check. **The timeout does not actually cap all-targets' worst case.** This is a real gap in +the current implementation of the (already-rejected) alternative, not a design-doc gap — worth noting +precisely because it means the measured 6.9–7.4 s undersells the actual risk: without a check inside the +loop, a slightly larger rule count would run past 9 s with nothing stopping it. + +**Cluster-scope all-targets was deliberately not run at the 10,010-service inventory.** Extrapolating the +formula above: `141 × (1,011 + 10,010) = 1,553,961` series — over 5× the 284,961-series run that already +took ~7 s and ~425% host CPU. Running it risked destabilizing the shared dev container (other test +databases run alongside it) for a number whose conclusion is already obvious from the formula. Treat +1.5M as an estimate, not a measurement, and note it as exactly that if it's cited elsewhere. + +### The index question, revisited — retracted + +Scenario C flagged the missing index on `services.cluster` as an action item, measured against an override +set covering essentially the whole services table (100%-selective). Re-tested here deliberately +*selectively* — 100 of 1,000 clusters overridden, 10,010 total services — the sequential scan still runs in +**6.6 ms**. Postgres reads a table this size in a handful of pages regardless of the `WHERE` clause; there +is no query for an index to speed up yet. **The recommendation is retracted.** Postgres' sequential-scan +cost is linear in table size, so this conclusion should be re-tested rather than assumed if a real +deployment's `services` table is one to two orders of magnitude larger than anything measured here — but +nothing in this session's data supports adding the index now. + +### Left in place for further exploration + +Unlike scenarios A–D, this session's seed data was **not** cleaned up. As of this section, the dev +container carries: 1,011 nodes, 10,010 services (1,000 clusters × 10 + real), 121 alert rules, and 250 +cluster-scope overrides (the last configuration measured) resolving to 2,500 effective thresholds. The +collector is back on `overrides-only` (the recommended default) — `all-targets` was never left active. + +## PMM health during the load + +Sampled while the 10,000-node / 16,000-service synthetic inventory was in place (the heaviest point of +this session, scenario C's worst row): host load1 stayed the same order of magnitude, `pmm-managed` RSS +and goroutine count did not visibly spike attributable to this collector specifically — the dominant +cost throughout was the pre-existing inventory collectors serializing tens of thousands of unrelated +lines into the same `/debug/metrics` response, not anything this feature adds. No scrape exceeded the +`0.9 × MR = 9 s` budget at any point tested here. + +## What was deliberately not done + +Scoped out at the start of this session, not discovered as blockers partway through: + +- **`rule_builder.go` / PromQL generation is untouched.** The `T_` / `label_replace` / fan-out-over- + observed-expression query shape from §2/§4.3 was not wired up; the existing direct-join rule builder + and its tests are unaffected. Query-side cost for node scope was already measured live in §8 of the + decision doc and remains valid. There is **no** end-to-end (Grafana rule → collector → firing) test of + service/cluster scope from this session — only the collector and resolver were exercised directly + against Postgres. +- **No proto/API changes.** `scope`/`target` are not exposed externally; the existing node-centric + endpoints now route through `ThresholdScopeNode` internally, unchanged from the outside. +- **Removal hooks not wired.** `DeleteThresholdOverridesForTarget` exists but isn't called from + `RemoveNode`/`RemoveService` yet (§4.8). +- **Reconciler untouched**, **HA not tested** — this session used one `pmm-managed` instance; §9 gate 7's + "under HA" half is still open. +- Not a shippable increment by itself — this is benchmarking-oriented code sized to answer the two + questions asked (override-only collector cost, service/cluster resolver cost), not to complete the + ten-step plan in §6 of the decision doc. + +## Bottom line + +Both open scale questions from §8/§9 now have real numbers instead of none: + +1. **The override-only collector costs what the design assumed it would** — single-digit milliseconds of + DB time, flat with inventory size, flat with how many rules/params the same override volume is spread + across. The multi-param blowup that killed emit-everything (§4.12, 42×20 → 26.9 s) does not reproduce + under this design at any scale tried here. +2. **Cluster-scope expansion works and is cheap up to 10,000 expanded series.** No index needed on + `services.cluster` — retracted in Scenario E after a more careful re-test at realistic selectivity. +3. **Head to head, on identical data, override-only wins at every point tested** (scenario D) — and the + gap is structural, not incidental: emit-everything's cost is a function of *inventory size*, so it pays + the same ~4,042-series price whether zero or two thousand overrides exist, while override-only's cost + is a function of *what was actually tuned*. Both modes now live in the same binary + (`PMM_DEV_THRESHOLD_EMIT_MODE=all-targets`), so this isn't a claim resting on the branch's older, + now-replaced code — it's the same code path, same DB, same endpoint, mode flipped by one env var. +4. **At a fixed, realistic parameter set (1,000 nodes, 250 overrides, 140 rule×param groups — Scenario E), + all-targets consumed 77–82% of the 9 s scrape budget** regardless of scope, while override-only used + 1–7%. This is with only 120 rules; more overridable templates only makes the gap worse, and the + `Collect` timeout added this session does not actually bound this cost — a real implementation gap in + the (rejected) alternative, not a design gap. + +HA and the full query-side integration with service/cluster join labels remain open per +[§9](dynamic-thresholds-main-decision.md#9-decision-gates) — this session narrows, but does not close, +those gates. From 33f7e72b04035acdd6f47c9e47042ebb84235c62 Mon Sep 17 00:00:00 2001 From: Matej Kubinec Date: Thu, 27 Aug 2026 12:54:04 +0200 Subject: [PATCH 02/15] PMM-14912 Add threshold overrides and collector Introduce the storage and metric layers for per-target alert threshold overrides, so tuning a threshold for one target is a data change rather than a rule change. Editing a rule resets alert state for every instance on it, which under `for: 5m` blinds every other target for five minutes when one is tuned. Migration 119 adds alert_rules, a registry of PMM-created rules holding the parameter snapshot Grafana cannot supply, and alert_rule_threshold_overrides. Its target column is polymorphic - a node id, a service id, or a cluster label value - so it carries no foreign key: cluster is a label value with no referent table. A CHECK rejects NaN and both infinities, this being the schema's first float column. Clearing an override tombstones the row instead of deleting it. A deleted row stops being emitted, and a series that stops being emitted keeps resolving for a full VictoriaMetrics lookbehind, so the clear takes minutes to land; changing the value of a live series takes one scrape. Measured end to end through a Grafana rule: 309s by absence, 14-21s by value change. ResolveThresholds is the single implementation of scope precedence, shared by the collector and later the API. Precedence cannot be expressed in PromQL - reducing operands to a common label set is what makes `or` prefer the left, and that reduction destroys the scope information needed to rank by - so nothing in the query backstops it. The order is service, node, cluster: a service runs on exactly one node and belongs to at most one cluster, so only that half is derivable; node over cluster is a convention, since the two cross-cut. The collector emits only overridden and tombstoned targets. Emitting a series per target instead scales with inventory rather than with what was actually tuned: measured at 1,000 nodes and 140 rule/parameter groups it consumed 77-82% of the 9s scrape budget, against 1-7% for this shape. Collect is bounded by a 3s timeout re-checked inside the emission loop, not only around the queries. The metric name and label set are shared constants rather than restated in the rule builder, so the generated query and the emitted series cannot drift - a rule pointing at a metric nobody emits fails silently, it simply never fires. Only node scope is reachable so far; service and cluster exist in the schema and resolver only. Templates gain an overridable flag, rejected on single-expression templates, which have no injection point yet. Co-Authored-By: Claude Opus 5 (1M context) --- dynamic-thresholds-main-decision.md | 10 +- managed/cmd/pmm-managed/main.go | 3 + managed/models/alert_rule_helpers.go | 315 ++++++++++++++++ managed/models/alert_rule_helpers_test.go | 352 ++++++++++++++++++ managed/models/alert_rule_model.go | 94 +++++ managed/models/alert_rule_model_reform.go | 151 ++++++++ .../alert_rule_threshold_override_model.go | 118 ++++++ ...rt_rule_threshold_override_model_reform.go | 171 +++++++++ managed/models/database.go | 40 ++ managed/models/template_helpers.go | 13 +- managed/models/template_model.go | 6 + managed/models/threshold_resolver.go | 130 +++++++ managed/models/threshold_resolver_test.go | 262 +++++++++++++ managed/pi/alert/overridable.go | 78 ++++ managed/pi/alert/overridable_test.go | 191 ++++++++++ managed/pi/alert/parameter.go | 65 +++- managed/pi/alert/template.go | 5 + .../services/alerting/threshold_metrics.go | 252 +++++++++++++ .../alerting/threshold_metrics_test.go | 218 +++++++++++ 19 files changed, 2460 insertions(+), 14 deletions(-) create mode 100644 managed/models/alert_rule_helpers.go create mode 100644 managed/models/alert_rule_helpers_test.go create mode 100644 managed/models/alert_rule_model.go create mode 100644 managed/models/alert_rule_model_reform.go create mode 100644 managed/models/alert_rule_threshold_override_model.go create mode 100644 managed/models/alert_rule_threshold_override_model_reform.go create mode 100644 managed/models/threshold_resolver.go create mode 100644 managed/models/threshold_resolver_test.go create mode 100644 managed/pi/alert/overridable.go create mode 100644 managed/pi/alert/overridable_test.go create mode 100644 managed/services/alerting/threshold_metrics.go create mode 100644 managed/services/alerting/threshold_metrics_test.go diff --git a/dynamic-thresholds-main-decision.md b/dynamic-thresholds-main-decision.md index 857d459edd..2514f9fa34 100644 --- a/dynamic-thresholds-main-decision.md +++ b/dynamic-thresholds-main-decision.md @@ -524,7 +524,7 @@ Three measurements settle this: Reducing to a common label set is required for `or` to mean "prefer the left", but that reduction is exactly what destroys the scope information needed to rank by. So the collector emits only the effective value per target, and **`ListNodeThresholds` and the collector must share one precedence function** — -PromQL provides no backstop if they diverge. Proposed order: `node` → `service` → `cluster`. +PromQL provides no backstop if they diverge. Order: `service` → `node` → `cluster` (§4.10). ### 4.7 Semantics that fall out for free @@ -665,7 +665,11 @@ func ResolveEffective(rule *models.AlertRule, param string, Invariants to assert and table-test: - **Exactly one emitted series per `(rule, param, target)`.** This is not a tie-break preference: if the resolver ever emits two series with identical labels, the Prometheus gatherer errors and the **entire** `/metrics` response fails. `node_name` and `service_name` are both `UNIQUE` (`database.go:117`, `:139`), so live targets cannot collide — the risk is a resolver bug, so assert it. -- **Precedence `node` → `service` → `cluster`**, most specific first. A param legal at both service and cluster scope with both present resolves to **service**. +- **Precedence `service` → `node` → `cluster`**, most specific first. A param legal at both service and cluster scope with both present resolves to **service**. + + Only the first relation is derivable: **a service runs on exactly one node and belongs to at most one cluster**, so a service override is strictly narrower than either and must win. `node` over `cluster` is a *convention*, not a containment — the two cross-cut, since a cluster spans several nodes while a node hosts services from several clusters (the same fact that constrains cluster scope in the next bullet). One machine is the narrower intent, so node wins, but nothing derives it. + + In practice the two rarely meet: node scope resolves to `node_name` while service and cluster scope resolve to `service_name`, and a rule joins on one label. They collide only when a node and a service share a name string — which nothing prevents, since each is unique within its own table but not across them. The ranking decides that case, so it must be right even though it is currently unreachable through the API. - **Cluster scope is only legal for params whose join label is service-level.** A node can host services from several clusters, so "the cluster override for this node" has no unique answer. This is a validation rule on `override_scopes`, not a runtime tie-break. - **Unresolvable targets are skipped**, never defaulted to something else. @@ -1056,7 +1060,7 @@ unknown; each needs a decision or a test result. | 3 | **Desugaring semantics verified live** — firing, pending under `for:`, recovery, NoData, annotation rendering — *before* step 6 is implemented | Eng | step 6 | | 4 | **Cross-store creation semantics** implemented per §4.9 (ordering, idempotency key, K-cycle absence rule) | Eng | merge | | 4b | **Registry-row orphan GC** — rows for rules deleted in Grafana, so emitted series do not grow without bound. Narrowed: override rows for deleted nodes/services are handled by the removal API hooks (§4.8), not here | Eng | GA | -| 5 | **Multi-scope precedence signed off** (`node` → `service` → `cluster`; cluster legal only for service-level join labels) *before* the proto freezes | Product | API freeze | +| 5 | **Multi-scope precedence signed off** (`service` → `node` → `cluster`; cluster legal only for service-level join labels) *before* the proto freezes | Product | API freeze | | 5b | **API style chosen** — node-centric vs generic routes (§10 Q1). Routes are additive-only, so deciding late means carrying both families forever | Product + Eng | API freeze | | 6 | **Shared resolver** (§4.10) is the single implementation of precedence, table-tested across scope combinations | Eng | service/cluster increment | | 7 | **Fan-out re-measured at representative scale and under HA** | Eng | GA | diff --git a/managed/cmd/pmm-managed/main.go b/managed/cmd/pmm-managed/main.go index 227f86921c..895937003f 100644 --- a/managed/cmd/pmm-managed/main.go +++ b/managed/cmd/pmm-managed/main.go @@ -1029,6 +1029,9 @@ func main() { //nolint:gocognit,maintidx,cyclop } alertingService.CollectTemplates(ctx) + alertThresholdMetricsCollector := alerting.NewAlertThresholdMetricsCollector(db) + prom.MustRegister(alertThresholdMetricsCollector) + agentService := agents.NewAgentService(agentsRegistry) versioner := agents.NewVersionerService(agentsRegistry) diff --git a/managed/models/alert_rule_helpers.go b/managed/models/alert_rule_helpers.go new file mode 100644 index 0000000000..90b57d9eeb --- /dev/null +++ b/managed/models/alert_rule_helpers.go @@ -0,0 +1,315 @@ +// Copyright (C) 2023 Percona LLC +// +// This program is free software: you can redistribute it and/or modify +// it under the terms of the GNU Affero General Public License as published by +// the Free Software Foundation, either version 3 of the License, or +// (at your option) any later version. +// +// This program is distributed in the hope that it will be useful, +// but WITHOUT ANY WARRANTY; without even the implied warranty of +// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +// GNU Affero General Public License for more details. +// +// You should have received a copy of the GNU Affero General Public License +// along with this program. If not, see . + +package models + +import ( + "errors" + "fmt" + + "github.com/google/uuid" + "google.golang.org/grpc/codes" + "google.golang.org/grpc/status" + "gopkg.in/reform.v1" +) + +func checkThresholdOverrideKey(ruleID, paramName string, scope ThresholdScope, target string) error { + if ruleID == "" { + return status.Error(codes.InvalidArgument, "Empty rule ID.") + } + + if paramName == "" { + return status.Error(codes.InvalidArgument, "Empty parameter name.") + } + + err := scope.Validate() + if err != nil { + return err + } + + if target == "" { + return status.Error(codes.InvalidArgument, "Empty target.") + } + + return nil +} + +// FindAlertRules returns all alert rules registered by PMM. +func FindAlertRules(q *reform.Querier) ([]*AlertRule, error) { + structs, err := q.SelectAllFrom(AlertRuleTable, "") + if err != nil { + return nil, fmt.Errorf("failed to select alert rules: %w", err) + } + + rules := make([]*AlertRule, len(structs)) + for i, s := range structs { + rules[i] = s.(*AlertRule) //nolint:forcetypeassert + } + + return rules, nil +} + +// FindAlertRuleByID returns an alert rule by its PMM-minted ID. +func FindAlertRuleByID(q *reform.Querier, ruleID string) (*AlertRule, error) { + if ruleID == "" { + return nil, status.Error(codes.InvalidArgument, "Empty rule ID.") + } + + rule := &AlertRule{RuleID: ruleID} + err := q.Reload(rule) + if err != nil { + if errors.Is(err, reform.ErrNoRows) { + return nil, status.Errorf(codes.NotFound, "Alert rule with ID %q not found.", ruleID) + } + + return nil, err + } + + return rule, nil +} + +// FindAllThresholdOverrides returns every threshold override row, tombstones included. +// The collector needs the tombstones: they are what keeps a cleared target's series +// alive at the rule's default. +func FindAllThresholdOverrides(q *reform.Querier) ([]*AlertRuleThresholdOverride, error) { + return selectThresholdOverrides(q, "") +} + +// FindThresholdOverridesByRule returns every override row for one rule, tombstones included. +func FindThresholdOverridesByRule(q *reform.Querier, ruleID string) ([]*AlertRuleThresholdOverride, error) { + if ruleID == "" { + return nil, status.Error(codes.InvalidArgument, "Empty rule ID.") + } + + return selectThresholdOverrides(q, "WHERE rule_id = "+q.Placeholder(1), ruleID) +} + +// FindThresholdOverridesByTarget returns every override row for one target, tombstones included. +func FindThresholdOverridesByTarget(q *reform.Querier, scope ThresholdScope, target string) ([]*AlertRuleThresholdOverride, error) { + err := scope.Validate() + if err != nil { + return nil, err + } + + if target == "" { + return nil, status.Error(codes.InvalidArgument, "Empty target.") + } + + tail := fmt.Sprintf("WHERE scope = %s AND target = %s", q.Placeholder(1), q.Placeholder(2)) + + return selectThresholdOverrides(q, tail, string(scope), target) +} + +func selectThresholdOverrides(q *reform.Querier, tail string, args ...any) ([]*AlertRuleThresholdOverride, error) { + structs, err := q.SelectAllFrom(AlertRuleThresholdOverrideTable, tail, args...) + if err != nil { + return nil, fmt.Errorf("failed to select threshold overrides: %w", err) + } + + overrides := make([]*AlertRuleThresholdOverride, len(structs)) + for i, s := range structs { + overrides[i] = s.(*AlertRuleThresholdOverride) //nolint:forcetypeassert + } + + return overrides, nil +} + +func findThresholdOverride(q *reform.Querier, ruleID, paramName string, scope ThresholdScope, target string) (*AlertRuleThresholdOverride, error) { + tail := fmt.Sprintf("WHERE rule_id = %s AND param_name = %s AND scope = %s AND target = %s", + q.Placeholder(1), q.Placeholder(2), q.Placeholder(3), q.Placeholder(4)) + + override := &AlertRuleThresholdOverride{} + err := q.SelectOneTo(override, tail, ruleID, paramName, string(scope), target) + if err != nil { + return nil, err + } + + return override, nil +} + +// CreateAlertRuleParams are params for creating a new alert rule registry row. +type CreateAlertRuleParams struct { + RuleID string + Params AlertRuleParams +} + +// CreateAlertRule registers an alert rule created by PMM. +func CreateAlertRule(q *reform.Querier, params *CreateAlertRuleParams) (*AlertRule, error) { + if params.RuleID == "" { + return nil, status.Error(codes.InvalidArgument, "Empty rule ID.") + } + + rule := &AlertRule{ + RuleID: params.RuleID, + Params: params.Params, + } + if rule.Params == nil { + rule.Params = AlertRuleParams{} + } + + err := q.Insert(rule) + if err != nil { + return nil, fmt.Errorf("failed to create alert rule: %w", err) + } + + return rule, nil +} + +// ChangeAlertRuleGrafanaUID stores the Grafana rule UID for an already-registered rule. +// The UID is a cached handle, not the identity, so it is set after Grafana has accepted +// the rule rather than being required up front. +func ChangeAlertRuleGrafanaUID(q *reform.Querier, ruleID, grafanaRuleUID string) (*AlertRule, error) { + rule, err := FindAlertRuleByID(q, ruleID) + if err != nil { + return nil, err + } + + if grafanaRuleUID == "" { + return nil, status.Error(codes.InvalidArgument, "Empty Grafana rule UID.") + } + + rule.GrafanaRuleUID = &grafanaRuleUID + err = q.Update(rule) + if err != nil { + return nil, fmt.Errorf("failed to update alert rule: %w", err) + } + + return rule, nil +} + +// UpsertThresholdOverride sets the override for one parameter of one rule at one target, +// creating the row if it does not exist. Writing to a tombstoned row revives it. +func UpsertThresholdOverride( + q *reform.Querier, + ruleID, paramName string, + scope ThresholdScope, + target string, + value float64, +) (*AlertRuleThresholdOverride, error) { + err := checkThresholdOverrideKey(ruleID, paramName, scope, target) + if err != nil { + return nil, err + } + + override, err := findThresholdOverride(q, ruleID, paramName, scope, target) + switch { + case err == nil: + override.Value = value + override.ClearedAt = nil + err = q.Update(override) + if err != nil { + return nil, fmt.Errorf("failed to update threshold override: %w", err) + } + + return override, nil + + case errors.Is(err, reform.ErrNoRows): + override = &AlertRuleThresholdOverride{ + ID: uuid.New().String(), + RuleID: ruleID, + ParamName: paramName, + Scope: scope, + Target: target, + Value: value, + } + err = q.Insert(override) + if err != nil { + return nil, fmt.Errorf("failed to create threshold override: %w", err) + } + + return override, nil + + default: + return nil, fmt.Errorf("failed to look up threshold override: %w", err) + } +} + +// ClearThresholdOverride tombstones an override instead of deleting it, so the emitted +// series keeps existing and merely changes value. Deleting the row would signal the +// clear by absence, which takes a full VictoriaMetrics lookbehind to become visible. +func ClearThresholdOverride(q *reform.Querier, ruleID, paramName string, scope ThresholdScope, target string) error { + err := checkThresholdOverrideKey(ruleID, paramName, scope, target) + if err != nil { + return err + } + + override, err := findThresholdOverride(q, ruleID, paramName, scope, target) + if err != nil { + if errors.Is(err, reform.ErrNoRows) { + return status.Errorf(codes.NotFound, "Threshold override for rule %q parameter %q not found.", ruleID, paramName) + } + + return fmt.Errorf("failed to look up threshold override: %w", err) + } + + if override.IsCleared() { + return nil + } + + override.ClearedAt = new(Now()) + err = q.Update(override) + if err != nil { + return fmt.Errorf("failed to clear threshold override: %w", err) + } + + return nil +} + +// DeleteThresholdOverridesForTarget hard-deletes every override for a target, and is for +// entity removal only. A user clearing an override tombstones it (the target still +// exists and its series must keep resolving); a removed node or service has no target +// left to emit for, so a tombstone there would be pure residue. +// +// Cluster scope is rejected: there is no "delete a cluster" operation to hook, and a +// cluster override with no matching services is dormant rather than stale - services may +// be added to that cluster later, and the override should apply again when they are. +func DeleteThresholdOverridesForTarget(q *reform.Querier, scope ThresholdScope, target string) error { + err := scope.Validate() + if err != nil { + return err + } + + if scope == ThresholdScopeCluster { + return status.Error(codes.InvalidArgument, "Cluster-scoped threshold overrides are not deleted by target removal.") + } + + if target == "" { + return status.Error(codes.InvalidArgument, "Empty target.") + } + + tail := fmt.Sprintf("WHERE scope = %s AND target = %s", q.Placeholder(1), q.Placeholder(2)) + _, err = q.DeleteFrom(AlertRuleThresholdOverrideTable, tail, string(scope), target) + if err != nil { + return fmt.Errorf("failed to delete threshold overrides: %w", err) + } + + return nil +} + +// DeleteAlertRule removes a rule registry row. Its overrides go with it through the +// foreign key's ON DELETE CASCADE. +func DeleteAlertRule(q *reform.Querier, ruleID string) error { + _, err := FindAlertRuleByID(q, ruleID) + if err != nil { + return err + } + + err = q.Delete(&AlertRule{RuleID: ruleID}) + if err != nil { + return fmt.Errorf("failed to delete alert rule: %w", err) + } + + return nil +} diff --git a/managed/models/alert_rule_helpers_test.go b/managed/models/alert_rule_helpers_test.go new file mode 100644 index 0000000000..63438e8089 --- /dev/null +++ b/managed/models/alert_rule_helpers_test.go @@ -0,0 +1,352 @@ +// Copyright (C) 2023 Percona LLC +// +// This program is free software: you can redistribute it and/or modify +// it under the terms of the GNU Affero General Public License as published by +// the Free Software Foundation, either version 3 of the License, or +// (at your option) any later version. +// +// This program is distributed in the hope that it will be useful, +// but WITHOUT ANY WARRANTY; without even the implied warranty of +// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +// GNU Affero General Public License for more details. +// +// You should have received a copy of the GNU Affero General Public License +// along with this program. If not, see . + +package models_test + +import ( + "math" + "testing" + + "github.com/google/uuid" + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + "google.golang.org/grpc/codes" + "google.golang.org/grpc/status" + "gopkg.in/reform.v1" + "gopkg.in/reform.v1/dialects/postgresql" + + "github.com/percona/pmm/managed/models" + "github.com/percona/pmm/managed/utils/testdb" +) + +func createTestAlertRule(t *testing.T, q *reform.Querier) *models.AlertRule { + t.Helper() + + rule, err := models.CreateAlertRule(q, &models.CreateAlertRuleParams{ + RuleID: uuid.New().String(), + Params: models.AlertRuleParams{ + "threshold": { + Default: 80, + JoinLabel: "node_name", + Scopes: []string{string(models.ThresholdScopeNode)}, + }, + }, + }) + require.NoError(t, err) + + return rule +} + +func TestAlertRuleRegistry(t *testing.T) { + sqlDB := testdb.Open(t, models.SkipFixtures, nil) + t.Cleanup(func() { + require.NoError(t, sqlDB.Close()) + }) + db := reform.NewDB(sqlDB, postgresql.Dialect, reform.NewPrintfLogger(t.Logf)) + + t.Run("create and find round-trips the params snapshot", func(t *testing.T) { + tx, err := db.Begin() + require.NoError(t, err) + defer func() { + require.NoError(t, tx.Rollback()) + }() + q := tx.Querier + + rule := createTestAlertRule(t, q) + assert.Nil(t, rule.GrafanaRuleUID) + + found, err := models.FindAlertRuleByID(q, rule.RuleID) + require.NoError(t, err) + assert.InDelta(t, 80.0, found.Params["threshold"].Default, 0.0001) + assert.Equal(t, "node_name", found.Params["threshold"].JoinLabel) + assert.Equal(t, []string{"node"}, found.Params["threshold"].Scopes) + }) + + t.Run("missing rule is NotFound", func(t *testing.T) { + tx, err := db.Begin() + require.NoError(t, err) + defer func() { + require.NoError(t, tx.Rollback()) + }() + + _, err = models.FindAlertRuleByID(tx.Querier, uuid.New().String()) + require.Error(t, err) + assert.Equal(t, codes.NotFound, status.Code(err)) + }) + + t.Run("grafana rule uid is set after the fact", func(t *testing.T) { + tx, err := db.Begin() + require.NoError(t, err) + defer func() { + require.NoError(t, tx.Rollback()) + }() + q := tx.Querier + + rule := createTestAlertRule(t, q) + updated, err := models.ChangeAlertRuleGrafanaUID(q, rule.RuleID, "grafana-uid-1") + require.NoError(t, err) + require.NotNil(t, updated.GrafanaRuleUID) + assert.Equal(t, "grafana-uid-1", *updated.GrafanaRuleUID) + }) +} + +func TestThresholdOverrides(t *testing.T) { + sqlDB := testdb.Open(t, models.SkipFixtures, nil) + t.Cleanup(func() { + require.NoError(t, sqlDB.Close()) + }) + db := reform.NewDB(sqlDB, postgresql.Dialect, reform.NewPrintfLogger(t.Logf)) + + t.Run("upsert creates then updates in place", func(t *testing.T) { + tx, err := db.Begin() + require.NoError(t, err) + defer func() { + require.NoError(t, tx.Rollback()) + }() + q := tx.Querier + + rule := createTestAlertRule(t, q) + + created, err := models.UpsertThresholdOverride(q, rule.RuleID, "threshold", models.ThresholdScopeNode, "node-id-1", 90) + require.NoError(t, err) + assert.InDelta(t, 90.0, created.Value, 0.0001) + assert.False(t, created.IsCleared()) + + updated, err := models.UpsertThresholdOverride(q, rule.RuleID, "threshold", models.ThresholdScopeNode, "node-id-1", 95) + require.NoError(t, err) + assert.Equal(t, created.ID, updated.ID, "upsert must reuse the row, not insert a second one") + assert.InDelta(t, 95.0, updated.Value, 0.0001) + + all, err := models.FindThresholdOverridesByRule(q, rule.RuleID) + require.NoError(t, err) + require.Len(t, all, 1) + }) + + t.Run("clear tombstones the row rather than deleting it", func(t *testing.T) { + tx, err := db.Begin() + require.NoError(t, err) + defer func() { + require.NoError(t, tx.Rollback()) + }() + q := tx.Querier + + rule := createTestAlertRule(t, q) + _, err = models.UpsertThresholdOverride(q, rule.RuleID, "threshold", models.ThresholdScopeNode, "node-id-1", 90) + require.NoError(t, err) + + require.NoError(t, models.ClearThresholdOverride(q, rule.RuleID, "threshold", models.ThresholdScopeNode, "node-id-1")) + + all, err := models.FindThresholdOverridesByRule(q, rule.RuleID) + require.NoError(t, err) + require.Len(t, all, 1, "the row must survive so the emitted series keeps existing") + assert.True(t, all[0].IsCleared()) + assert.InDelta(t, 90.0, all[0].Value, 0.0001, "the stale value is kept for audit") + }) + + t.Run("clear is idempotent", func(t *testing.T) { + tx, err := db.Begin() + require.NoError(t, err) + defer func() { + require.NoError(t, tx.Rollback()) + }() + q := tx.Querier + + rule := createTestAlertRule(t, q) + _, err = models.UpsertThresholdOverride(q, rule.RuleID, "threshold", models.ThresholdScopeNode, "node-id-1", 90) + require.NoError(t, err) + + require.NoError(t, models.ClearThresholdOverride(q, rule.RuleID, "threshold", models.ThresholdScopeNode, "node-id-1")) + require.NoError(t, models.ClearThresholdOverride(q, rule.RuleID, "threshold", models.ThresholdScopeNode, "node-id-1")) + }) + + t.Run("clearing an override that was never set is NotFound", func(t *testing.T) { + tx, err := db.Begin() + require.NoError(t, err) + defer func() { + require.NoError(t, tx.Rollback()) + }() + q := tx.Querier + + rule := createTestAlertRule(t, q) + err = models.ClearThresholdOverride(q, rule.RuleID, "threshold", models.ThresholdScopeNode, "node-id-1") + require.Error(t, err) + assert.Equal(t, codes.NotFound, status.Code(err)) + }) + + t.Run("upsert revives a tombstone", func(t *testing.T) { + tx, err := db.Begin() + require.NoError(t, err) + defer func() { + require.NoError(t, tx.Rollback()) + }() + q := tx.Querier + + rule := createTestAlertRule(t, q) + created, err := models.UpsertThresholdOverride(q, rule.RuleID, "threshold", models.ThresholdScopeNode, "node-id-1", 90) + require.NoError(t, err) + require.NoError(t, models.ClearThresholdOverride(q, rule.RuleID, "threshold", models.ThresholdScopeNode, "node-id-1")) + + revived, err := models.UpsertThresholdOverride(q, rule.RuleID, "threshold", models.ThresholdScopeNode, "node-id-1", 75) + require.NoError(t, err) + assert.Equal(t, created.ID, revived.ID) + assert.False(t, revived.IsCleared(), "writing a value must clear the tombstone") + assert.InDelta(t, 75.0, revived.Value, 0.0001) + }) + + t.Run("the unique key is rule, param, scope and target", func(t *testing.T) { + tx, err := db.Begin() + require.NoError(t, err) + defer func() { + require.NoError(t, tx.Rollback()) + }() + q := tx.Querier + + rule := createTestAlertRule(t, q) + + // Same target at three scopes, plus a second param, are four distinct rows. + for _, scope := range []models.ThresholdScope{ + models.ThresholdScopeNode, + models.ThresholdScopeService, + models.ThresholdScopeCluster, + } { + _, err = models.UpsertThresholdOverride(q, rule.RuleID, "threshold", scope, "same-target", 90) + require.NoError(t, err) + } + _, err = models.UpsertThresholdOverride(q, rule.RuleID, "other", models.ThresholdScopeNode, "same-target", 90) + require.NoError(t, err) + + all, err := models.FindThresholdOverridesByRule(q, rule.RuleID) + require.NoError(t, err) + assert.Len(t, all, 4) + }) + + t.Run("find by target is scoped", func(t *testing.T) { + tx, err := db.Begin() + require.NoError(t, err) + defer func() { + require.NoError(t, tx.Rollback()) + }() + q := tx.Querier + + rule := createTestAlertRule(t, q) + _, err = models.UpsertThresholdOverride(q, rule.RuleID, "threshold", models.ThresholdScopeNode, "shared", 90) + require.NoError(t, err) + _, err = models.UpsertThresholdOverride(q, rule.RuleID, "threshold", models.ThresholdScopeService, "shared", 70) + require.NoError(t, err) + + found, err := models.FindThresholdOverridesByTarget(q, models.ThresholdScopeNode, "shared") + require.NoError(t, err) + require.Len(t, found, 1) + assert.InDelta(t, 90.0, found[0].Value, 0.0001) + }) + + t.Run("delete for target hard-deletes, unlike clear", func(t *testing.T) { + tx, err := db.Begin() + require.NoError(t, err) + defer func() { + require.NoError(t, tx.Rollback()) + }() + q := tx.Querier + + rule := createTestAlertRule(t, q) + _, err = models.UpsertThresholdOverride(q, rule.RuleID, "threshold", models.ThresholdScopeNode, "node-id-1", 90) + require.NoError(t, err) + _, err = models.UpsertThresholdOverride(q, rule.RuleID, "threshold", models.ThresholdScopeNode, "node-id-2", 91) + require.NoError(t, err) + + require.NoError(t, models.DeleteThresholdOverridesForTarget(q, models.ThresholdScopeNode, "node-id-1")) + + all, err := models.FindThresholdOverridesByRule(q, rule.RuleID) + require.NoError(t, err) + require.Len(t, all, 1) + assert.Equal(t, "node-id-2", all[0].Target) + }) + + t.Run("delete for target refuses cluster scope", func(t *testing.T) { + tx, err := db.Begin() + require.NoError(t, err) + defer func() { + require.NoError(t, tx.Rollback()) + }() + + // A cluster override with no matching services is dormant, not stale. + err = models.DeleteThresholdOverridesForTarget(tx.Querier, models.ThresholdScopeCluster, "prod") + require.Error(t, err) + assert.Equal(t, codes.InvalidArgument, status.Code(err)) + }) + + t.Run("deleting the rule cascades to its overrides", func(t *testing.T) { + tx, err := db.Begin() + require.NoError(t, err) + defer func() { + require.NoError(t, tx.Rollback()) + }() + q := tx.Querier + + rule := createTestAlertRule(t, q) + _, err = models.UpsertThresholdOverride(q, rule.RuleID, "threshold", models.ThresholdScopeNode, "node-id-1", 90) + require.NoError(t, err) + + require.NoError(t, models.DeleteAlertRule(q, rule.RuleID)) + + all, err := models.FindThresholdOverridesByRule(q, rule.RuleID) + require.NoError(t, err) + assert.Empty(t, all) + }) + + t.Run("an unknown scope is rejected before touching the database", func(t *testing.T) { + tx, err := db.Begin() + require.NoError(t, err) + defer func() { + require.NoError(t, tx.Rollback()) + }() + q := tx.Querier + + rule := createTestAlertRule(t, q) + _, err = models.UpsertThresholdOverride(q, rule.RuleID, "threshold", models.ThresholdScope("rack"), "r1", 90) + require.Error(t, err) + assert.Equal(t, codes.InvalidArgument, status.Code(err)) + }) +} + +// TestThresholdOverrideRejectsNonFiniteValues exercises migration 119's CHECK. This +// would be main's first float column, so there is no existing precedent to inherit and +// the guard has to be verified rather than assumed. +func TestThresholdOverrideRejectsNonFiniteValues(t *testing.T) { + sqlDB := testdb.Open(t, models.SkipFixtures, nil) + t.Cleanup(func() { + require.NoError(t, sqlDB.Close()) + }) + db := reform.NewDB(sqlDB, postgresql.Dialect, reform.NewPrintfLogger(t.Logf)) + + for name, value := range map[string]float64{ + "NaN": math.NaN(), + "positive infinity": math.Inf(1), + "negative infinity": math.Inf(-1), + } { + t.Run(name, func(t *testing.T) { + tx, err := db.Begin() + require.NoError(t, err) + defer func() { + require.NoError(t, tx.Rollback()) + }() + q := tx.Querier + + rule := createTestAlertRule(t, q) + _, err = models.UpsertThresholdOverride(q, rule.RuleID, "threshold", models.ThresholdScopeNode, "node-id-1", value) + require.Error(t, err, "the database must reject %s", name) + }) + } +} diff --git a/managed/models/alert_rule_model.go b/managed/models/alert_rule_model.go new file mode 100644 index 0000000000..a18b70f04a --- /dev/null +++ b/managed/models/alert_rule_model.go @@ -0,0 +1,94 @@ +// Copyright (C) 2023 Percona LLC +// +// This program is free software: you can redistribute it and/or modify +// it under the terms of the GNU Affero General Public License as published by +// the Free Software Foundation, either version 3 of the License, or +// (at your option) any later version. +// +// This program is distributed in the hope that it will be useful, +// but WITHOUT ANY WARRANTY; without even the implied warranty of +// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +// GNU Affero General Public License for more details. +// +// You should have received a copy of the GNU Affero General Public License +// along with this program. If not, see . + +package models + +import ( + "database/sql/driver" + "time" + + "gopkg.in/reform.v1" +) + +//go:generate go tool reform + +// AlertRuleParam is the snapshot of one overridable parameter, taken when the rule was +// created. The template it came from can be edited or deleted afterwards, so this is the +// only durable record of what the rule actually evaluates against, and the only place the +// effective default can be read back from. +type AlertRuleParam struct { + Default float64 `json:"default"` + JoinLabel string `json:"join_label"` + Scopes []string `json:"scopes"` + Unit string `json:"unit,omitempty"` + Summary string `json:"summary,omitempty"` + Min *float64 `json:"min,omitempty"` + Max *float64 `json:"max,omitempty"` +} + +// AlertRuleParams maps a parameter name to its snapshot. +type AlertRuleParams map[string]AlertRuleParam + +// Value implements database/sql/driver.Valuer interface. Should be defined on the value. +func (p AlertRuleParams) Value() (driver.Value, error) { return jsonValue(p) } + +// Scan implements database/sql.Scanner interface. Should be defined on the pointer. +func (p *AlertRuleParams) Scan(src any) error { return jsonScan(p, src) } + +// AlertRule represents a PMM-created Grafana alert rule that carries overridable +// thresholds. The row is a registry entry, not the rule itself: Grafana remains the +// authority for the rule definition, and PMM keeps only what Grafana cannot supply. +// +//reform:alert_rules +type AlertRule struct { + RuleID string `reform:"rule_id,pk"` + // GrafanaRuleUID is a cached handle for the rule in Grafana, never the identity. + // It is nil until the rule has been created there. + GrafanaRuleUID *string `reform:"grafana_rule_uid"` + Params AlertRuleParams `reform:"params"` + CreatedAt time.Time `reform:"created_at"` + UpdatedAt time.Time `reform:"updated_at"` +} + +// BeforeInsert implements reform.BeforeInserter interface. +func (r *AlertRule) BeforeInsert() error { + now := Now() + r.CreatedAt = now + r.UpdatedAt = now + + return nil +} + +// BeforeUpdate implements reform.BeforeUpdater interface. +func (r *AlertRule) BeforeUpdate() error { + r.UpdatedAt = Now() + + return nil +} + +// AfterFind implements reform.AfterFinder interface. +func (r *AlertRule) AfterFind() error { + r.CreatedAt = r.CreatedAt.UTC() + r.UpdatedAt = r.UpdatedAt.UTC() + + return nil +} + +// check interfaces. +var ( + _ reform.BeforeInserter = (*AlertRule)(nil) + _ reform.BeforeUpdater = (*AlertRule)(nil) + _ reform.AfterFinder = (*AlertRule)(nil) +) diff --git a/managed/models/alert_rule_model_reform.go b/managed/models/alert_rule_model_reform.go new file mode 100644 index 0000000000..107c21593b --- /dev/null +++ b/managed/models/alert_rule_model_reform.go @@ -0,0 +1,151 @@ +// Code generated by gopkg.in/reform.v1. DO NOT EDIT. + +package models + +import ( + "fmt" + "strings" + + "gopkg.in/reform.v1" + "gopkg.in/reform.v1/parse" +) + +type alertRuleTableType struct { + s parse.StructInfo + z []interface{} +} + +// Schema returns a schema name in SQL database (""). +func (v *alertRuleTableType) Schema() string { + return v.s.SQLSchema +} + +// Name returns a view or table name in SQL database ("alert_rules"). +func (v *alertRuleTableType) Name() string { + return v.s.SQLName +} + +// Columns returns a new slice of column names for that view or table in SQL database. +func (v *alertRuleTableType) Columns() []string { + return []string{ + "rule_id", + "grafana_rule_uid", + "params", + "created_at", + "updated_at", + } +} + +// NewStruct makes a new struct for that view or table. +func (v *alertRuleTableType) NewStruct() reform.Struct { + return new(AlertRule) +} + +// NewRecord makes a new record for that table. +func (v *alertRuleTableType) NewRecord() reform.Record { + return new(AlertRule) +} + +// PKColumnIndex returns an index of primary key column for that table in SQL database. +func (v *alertRuleTableType) PKColumnIndex() uint { + return uint(v.s.PKFieldIndex) +} + +// AlertRuleTable represents alert_rules view or table in SQL database. +var AlertRuleTable = &alertRuleTableType{ + s: parse.StructInfo{ + Type: "AlertRule", + SQLName: "alert_rules", + Fields: []parse.FieldInfo{ + {Name: "RuleID", Type: "string", Column: "rule_id"}, + {Name: "GrafanaRuleUID", Type: "*string", Column: "grafana_rule_uid"}, + {Name: "Params", Type: "AlertRuleParams", Column: "params"}, + {Name: "CreatedAt", Type: "time.Time", Column: "created_at"}, + {Name: "UpdatedAt", Type: "time.Time", Column: "updated_at"}, + }, + PKFieldIndex: 0, + }, + z: new(AlertRule).Values(), +} + +// String returns a string representation of this struct or record. +func (s AlertRule) String() string { + res := make([]string, 5) + res[0] = "RuleID: " + reform.Inspect(s.RuleID, true) + res[1] = "GrafanaRuleUID: " + reform.Inspect(s.GrafanaRuleUID, true) + res[2] = "Params: " + reform.Inspect(s.Params, true) + res[3] = "CreatedAt: " + reform.Inspect(s.CreatedAt, true) + res[4] = "UpdatedAt: " + reform.Inspect(s.UpdatedAt, true) + return strings.Join(res, ", ") +} + +// Values returns a slice of struct or record field values. +// Returned interface{} values are never untyped nils. +func (s *AlertRule) Values() []interface{} { + return []interface{}{ + s.RuleID, + s.GrafanaRuleUID, + s.Params, + s.CreatedAt, + s.UpdatedAt, + } +} + +// Pointers returns a slice of pointers to struct or record fields. +// Returned interface{} values are never untyped nils. +func (s *AlertRule) Pointers() []interface{} { + return []interface{}{ + &s.RuleID, + &s.GrafanaRuleUID, + &s.Params, + &s.CreatedAt, + &s.UpdatedAt, + } +} + +// View returns View object for that struct. +func (s *AlertRule) View() reform.View { + return AlertRuleTable +} + +// Table returns Table object for that record. +func (s *AlertRule) Table() reform.Table { + return AlertRuleTable +} + +// PKValue returns a value of primary key for that record. +// Returned interface{} value is never untyped nil. +func (s *AlertRule) PKValue() interface{} { + return s.RuleID +} + +// PKPointer returns a pointer to primary key field for that record. +// Returned interface{} value is never untyped nil. +func (s *AlertRule) PKPointer() interface{} { + return &s.RuleID +} + +// HasPK returns true if record has non-zero primary key set, false otherwise. +func (s *AlertRule) HasPK() bool { + return s.RuleID != AlertRuleTable.z[AlertRuleTable.s.PKFieldIndex] +} + +// SetPK sets record primary key, if possible. +// +// Deprecated: prefer direct field assignment where possible: s.RuleID = pk. +func (s *AlertRule) SetPK(pk interface{}) { + reform.SetPK(s, pk) +} + +// check interfaces +var ( + _ reform.View = AlertRuleTable + _ reform.Struct = (*AlertRule)(nil) + _ reform.Table = AlertRuleTable + _ reform.Record = (*AlertRule)(nil) + _ fmt.Stringer = (*AlertRule)(nil) +) + +func init() { + parse.AssertUpToDate(&AlertRuleTable.s, new(AlertRule)) +} diff --git a/managed/models/alert_rule_threshold_override_model.go b/managed/models/alert_rule_threshold_override_model.go new file mode 100644 index 0000000000..ba8604c3de --- /dev/null +++ b/managed/models/alert_rule_threshold_override_model.go @@ -0,0 +1,118 @@ +// Copyright (C) 2023 Percona LLC +// +// This program is free software: you can redistribute it and/or modify +// it under the terms of the GNU Affero General Public License as published by +// the Free Software Foundation, either version 3 of the License, or +// (at your option) any later version. +// +// This program is distributed in the hope that it will be useful, +// but WITHOUT ANY WARRANTY; without even the implied warranty of +// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +// GNU Affero General Public License for more details. +// +// You should have received a copy of the GNU Affero General Public License +// along with this program. If not, see . + +package models + +import ( + "time" + + "google.golang.org/grpc/codes" + "google.golang.org/grpc/status" + "gopkg.in/reform.v1" +) + +//go:generate go tool reform + +// ThresholdScope says what an override's target refers to. +type ThresholdScope string + +// Threshold override scopes. Declaration order carries no meaning: precedence lives in +// thresholdScopeSpecificity, which ranks service above node above cluster. +const ( + ThresholdScopeNode = ThresholdScope("node") + ThresholdScopeService = ThresholdScope("service") + ThresholdScopeCluster = ThresholdScope("cluster") +) + +// Validate validates the threshold override scope. +// +// This returns a gRPC status error rather than an InvalidArgumentError, matching +// template_helpers.go in the same feature area, so that every validation failure the +// threshold helpers can produce surfaces as the same code without the service layer +// having to convert two different error shapes. +func (s ThresholdScope) Validate() error { + switch s { + case ThresholdScopeNode: + case ThresholdScopeService: + case ThresholdScopeCluster: + default: + return status.Errorf(codes.InvalidArgument, "Invalid threshold scope %q.", string(s)) + } + + return nil +} + +// AlertRuleThresholdOverride is a per-target threshold for one parameter of one alert +// rule. Clearing an override tombstones the row rather than deleting it: a deleted row +// stops being emitted, and a series that stops being emitted keeps resolving for the +// whole of VictoriaMetrics' lookbehind, so the clear would take minutes to take effect +// instead of one scrape. +// +//reform:alert_rule_threshold_overrides +type AlertRuleThresholdOverride struct { + ID string `reform:"id,pk"` + RuleID string `reform:"rule_id"` + ParamName string `reform:"param_name"` + Scope ThresholdScope `reform:"scope"` + // Target is a node_id, a service_id, or a cluster label value, depending on Scope. + Target string `reform:"target"` + Value float64 `reform:"value"` + // ClearedAt marks the row as a tombstone. The stale Value is kept for audit and + // must never be emitted: a cleared override resolves through the remaining scopes, + // falling back to the rule's default only when none apply. + ClearedAt *time.Time `reform:"cleared_at"` + CreatedAt time.Time `reform:"created_at"` + UpdatedAt time.Time `reform:"updated_at"` +} + +// IsCleared reports whether the override has been cleared and is therefore a tombstone. +func (o *AlertRuleThresholdOverride) IsCleared() bool { + return o.ClearedAt != nil +} + +// BeforeInsert implements reform.BeforeInserter interface. +func (o *AlertRuleThresholdOverride) BeforeInsert() error { + now := Now() + o.CreatedAt = now + o.UpdatedAt = now + + return nil +} + +// BeforeUpdate implements reform.BeforeUpdater interface. +func (o *AlertRuleThresholdOverride) BeforeUpdate() error { + o.UpdatedAt = Now() + + return nil +} + +// AfterFind implements reform.AfterFinder interface. +func (o *AlertRuleThresholdOverride) AfterFind() error { + o.CreatedAt = o.CreatedAt.UTC() + o.UpdatedAt = o.UpdatedAt.UTC() + if o.ClearedAt != nil { + cleared := o.ClearedAt.UTC() + o.ClearedAt = &cleared + } + + return nil +} + +// check interfaces. +var ( + _ reform.BeforeInserter = (*AlertRuleThresholdOverride)(nil) + _ reform.BeforeUpdater = (*AlertRuleThresholdOverride)(nil) + _ reform.AfterFinder = (*AlertRuleThresholdOverride)(nil) +) diff --git a/managed/models/alert_rule_threshold_override_model_reform.go b/managed/models/alert_rule_threshold_override_model_reform.go new file mode 100644 index 0000000000..63bba32c28 --- /dev/null +++ b/managed/models/alert_rule_threshold_override_model_reform.go @@ -0,0 +1,171 @@ +// Code generated by gopkg.in/reform.v1. DO NOT EDIT. + +package models + +import ( + "fmt" + "strings" + + "gopkg.in/reform.v1" + "gopkg.in/reform.v1/parse" +) + +type alertRuleThresholdOverrideTableType struct { + s parse.StructInfo + z []interface{} +} + +// Schema returns a schema name in SQL database (""). +func (v *alertRuleThresholdOverrideTableType) Schema() string { + return v.s.SQLSchema +} + +// Name returns a view or table name in SQL database ("alert_rule_threshold_overrides"). +func (v *alertRuleThresholdOverrideTableType) Name() string { + return v.s.SQLName +} + +// Columns returns a new slice of column names for that view or table in SQL database. +func (v *alertRuleThresholdOverrideTableType) Columns() []string { + return []string{ + "id", + "rule_id", + "param_name", + "scope", + "target", + "value", + "cleared_at", + "created_at", + "updated_at", + } +} + +// NewStruct makes a new struct for that view or table. +func (v *alertRuleThresholdOverrideTableType) NewStruct() reform.Struct { + return new(AlertRuleThresholdOverride) +} + +// NewRecord makes a new record for that table. +func (v *alertRuleThresholdOverrideTableType) NewRecord() reform.Record { + return new(AlertRuleThresholdOverride) +} + +// PKColumnIndex returns an index of primary key column for that table in SQL database. +func (v *alertRuleThresholdOverrideTableType) PKColumnIndex() uint { + return uint(v.s.PKFieldIndex) +} + +// AlertRuleThresholdOverrideTable represents alert_rule_threshold_overrides view or table in SQL database. +var AlertRuleThresholdOverrideTable = &alertRuleThresholdOverrideTableType{ + s: parse.StructInfo{ + Type: "AlertRuleThresholdOverride", + SQLName: "alert_rule_threshold_overrides", + Fields: []parse.FieldInfo{ + {Name: "ID", Type: "string", Column: "id"}, + {Name: "RuleID", Type: "string", Column: "rule_id"}, + {Name: "ParamName", Type: "string", Column: "param_name"}, + {Name: "Scope", Type: "ThresholdScope", Column: "scope"}, + {Name: "Target", Type: "string", Column: "target"}, + {Name: "Value", Type: "float64", Column: "value"}, + {Name: "ClearedAt", Type: "*time.Time", Column: "cleared_at"}, + {Name: "CreatedAt", Type: "time.Time", Column: "created_at"}, + {Name: "UpdatedAt", Type: "time.Time", Column: "updated_at"}, + }, + PKFieldIndex: 0, + }, + z: new(AlertRuleThresholdOverride).Values(), +} + +// String returns a string representation of this struct or record. +func (s AlertRuleThresholdOverride) String() string { + res := make([]string, 9) + res[0] = "ID: " + reform.Inspect(s.ID, true) + res[1] = "RuleID: " + reform.Inspect(s.RuleID, true) + res[2] = "ParamName: " + reform.Inspect(s.ParamName, true) + res[3] = "Scope: " + reform.Inspect(s.Scope, true) + res[4] = "Target: " + reform.Inspect(s.Target, true) + res[5] = "Value: " + reform.Inspect(s.Value, true) + res[6] = "ClearedAt: " + reform.Inspect(s.ClearedAt, true) + res[7] = "CreatedAt: " + reform.Inspect(s.CreatedAt, true) + res[8] = "UpdatedAt: " + reform.Inspect(s.UpdatedAt, true) + return strings.Join(res, ", ") +} + +// Values returns a slice of struct or record field values. +// Returned interface{} values are never untyped nils. +func (s *AlertRuleThresholdOverride) Values() []interface{} { + return []interface{}{ + s.ID, + s.RuleID, + s.ParamName, + s.Scope, + s.Target, + s.Value, + s.ClearedAt, + s.CreatedAt, + s.UpdatedAt, + } +} + +// Pointers returns a slice of pointers to struct or record fields. +// Returned interface{} values are never untyped nils. +func (s *AlertRuleThresholdOverride) Pointers() []interface{} { + return []interface{}{ + &s.ID, + &s.RuleID, + &s.ParamName, + &s.Scope, + &s.Target, + &s.Value, + &s.ClearedAt, + &s.CreatedAt, + &s.UpdatedAt, + } +} + +// View returns View object for that struct. +func (s *AlertRuleThresholdOverride) View() reform.View { + return AlertRuleThresholdOverrideTable +} + +// Table returns Table object for that record. +func (s *AlertRuleThresholdOverride) Table() reform.Table { + return AlertRuleThresholdOverrideTable +} + +// PKValue returns a value of primary key for that record. +// Returned interface{} value is never untyped nil. +func (s *AlertRuleThresholdOverride) PKValue() interface{} { + return s.ID +} + +// PKPointer returns a pointer to primary key field for that record. +// Returned interface{} value is never untyped nil. +func (s *AlertRuleThresholdOverride) PKPointer() interface{} { + return &s.ID +} + +// HasPK returns true if record has non-zero primary key set, false otherwise. +func (s *AlertRuleThresholdOverride) HasPK() bool { + return s.ID != AlertRuleThresholdOverrideTable.z[AlertRuleThresholdOverrideTable.s.PKFieldIndex] +} + +// SetPK sets record primary key, if possible. +// +// Deprecated: prefer direct field assignment where possible: s.ID = pk. +func (s *AlertRuleThresholdOverride) SetPK(pk interface{}) { + reform.SetPK(s, pk) +} + +// check interfaces +var ( + _ reform.View = AlertRuleThresholdOverrideTable + _ reform.Struct = (*AlertRuleThresholdOverride)(nil) + _ reform.Table = AlertRuleThresholdOverrideTable + _ reform.Record = (*AlertRuleThresholdOverride)(nil) + _ fmt.Stringer = (*AlertRuleThresholdOverride)(nil) +) + +func init() { + parse.AssertUpToDate(&AlertRuleThresholdOverrideTable.s, new(AlertRuleThresholdOverride)) +} diff --git a/managed/models/database.go b/managed/models/database.go index 431e480fe3..7e708e1ac0 100644 --- a/managed/models/database.go +++ b/managed/models/database.go @@ -1185,6 +1185,46 @@ var databaseSchema = [][]string{ `ALTER TABLE dumps ADD COLUMN encrypted boolean NOT NULL DEFAULT false`, `UPDATE dumps SET encrypted = false`, }, + 119: { + `CREATE TABLE alert_rules ( + rule_id VARCHAR NOT NULL, + grafana_rule_uid VARCHAR CHECK (grafana_rule_uid <> ''), + params JSONB NOT NULL, + created_at TIMESTAMP NOT NULL, + updated_at TIMESTAMP NOT NULL, + + PRIMARY KEY (rule_id), + UNIQUE (grafana_rule_uid) + )`, + + // target is polymorphic - a node_id, a service_id, or a cluster label value - + // so it carries no foreign key: cluster is a label value with no referent table. + // Rows for a deleted node or service are removed by the removal API instead. + `CREATE TABLE alert_rule_threshold_overrides ( + id VARCHAR NOT NULL, + rule_id VARCHAR NOT NULL, + param_name VARCHAR NOT NULL CHECK (param_name <> ''), + scope VARCHAR NOT NULL CHECK (scope <> ''), + target VARCHAR NOT NULL CHECK (target <> ''), + value DOUBLE PRECISION NOT NULL + CHECK (value = value AND value > '-Infinity'::float8 AND value < 'Infinity'::float8), + cleared_at TIMESTAMP, + created_at TIMESTAMP NOT NULL, + updated_at TIMESTAMP NOT NULL, + + PRIMARY KEY (id), + UNIQUE (rule_id, param_name, scope, target), + FOREIGN KEY (rule_id) REFERENCES alert_rules (rule_id) ON DELETE CASCADE + )`, + + `CREATE INDEX alert_rule_threshold_overrides_target_idx + ON alert_rule_threshold_overrides (scope, target)`, + + // The foreign key above does not imply an index in PostgreSQL, and the + // collector reads by rule_id on every scrape. + `CREATE INDEX alert_rule_threshold_overrides_rule_idx + ON alert_rule_threshold_overrides (rule_id)`, + }, } // ^^^ Avoid default values in schema definition. ^^^ diff --git a/managed/models/template_helpers.go b/managed/models/template_helpers.go index 471a805e1e..2085574f0d 100644 --- a/managed/models/template_helpers.go +++ b/managed/models/template_helpers.go @@ -225,10 +225,15 @@ func ConvertParamsDefinitions(params []alert.Parameter) (AlertExprParamsDefiniti res := make(AlertExprParamsDefinitions, 0, len(params)) for _, param := range params { p := AlertExprParamDefinition{ - Name: param.Name, - Summary: param.Summary, - Unit: ParamUnit(param.Unit), - Type: ParamType(param.Type), + Name: param.Name, + Summary: param.Summary, + Unit: ParamUnit(param.Unit), + Type: ParamType(param.Type), + Overridable: param.Overridable, + } + + if param.Overridable { + p.OverrideScopes = param.GetOverrideScopes() } switch param.Type { diff --git a/managed/models/template_model.go b/managed/models/template_model.go index 60cc77a2f8..3d715ad4b1 100644 --- a/managed/models/template_model.go +++ b/managed/models/template_model.go @@ -110,6 +110,12 @@ type AlertExprParamDefinition struct { FloatParam *FloatParam `json:"float_param"` // BoolParam *BoolParam `json:"bool_param"` // StringParam *StringParam `json:"string_param"` + + // Overridable reports whether a per-target threshold override may be set for this + // parameter without rewriting the alert rule. + Overridable bool `json:"overridable,omitempty"` + // OverrideScopes lists the scopes an override may be set at. Empty means node. + OverrideScopes []string `json:"override_scopes,omitempty"` } // ParamType represents parameter type. diff --git a/managed/models/threshold_resolver.go b/managed/models/threshold_resolver.go new file mode 100644 index 0000000000..22024ef1b7 --- /dev/null +++ b/managed/models/threshold_resolver.go @@ -0,0 +1,130 @@ +// Copyright (C) 2023 Percona LLC +// +// This program is free software: you can redistribute it and/or modify +// it under the terms of the GNU Affero General Public License as published by +// the Free Software Foundation, either version 3 of the License, or +// (at your option) any later version. +// +// This program is distributed in the hope that it will be useful, +// but WITHOUT ANY WARRANTY; without even the implied warranty of +// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +// GNU Affero General Public License for more details. +// +// You should have received a copy of the GNU Affero General Public License +// along with this program. If not, see . + +package models + +// thresholdScopeSpecificity ranks scopes most-specific-first, so a narrower override +// wins over a broader one covering the same target. +// +// Service ranks highest because it is the only relation that actually holds: a service +// runs on exactly one node and belongs to at most one cluster, so a service override is +// strictly narrower than either. Node over cluster is a convention rather than a +// containment - the two cross-cut, since a cluster spans several nodes while a node +// hosts services from several clusters - but one machine is the narrower intent. +// +// Precedence cannot be expressed in PromQL: reducing both sides of an `or` to a common +// label set is what makes `or` prefer the left operand, but that reduction is exactly +// what destroys the scope information needed to rank by. So precedence is resolved here, +// in Go, and there is no backstop in the query if this function is wrong. +var thresholdScopeSpecificity = map[ThresholdScope]int{ + ThresholdScopeService: 3, + ThresholdScopeNode: 2, + ThresholdScopeCluster: 1, +} + +// ThresholdInventory maps override targets onto the join-label values an alert rule +// matches on. Targets missing from it no longer exist and are skipped. +type ThresholdInventory struct { + // NodeNames maps node_id to node_name. + NodeNames map[string]string + // ServiceNames maps service_id to service_name. + ServiceNames map[string]string + // ServicesByCluster maps a cluster label value to the names of its services. + ServicesByCluster map[string][]string +} + +// targetNames returns the join-label values an override applies to. A node or service +// override yields at most one; a cluster override fans out onto every service in that +// cluster. An unresolvable target yields none, so a row left behind by a deleted entity +// is inert rather than wrong. +func (inv ThresholdInventory) targetNames(override *AlertRuleThresholdOverride) []string { + switch override.Scope { + case ThresholdScopeNode: + name, ok := inv.NodeNames[override.Target] + if !ok { + return nil + } + + return []string{name} + + case ThresholdScopeService: + name, ok := inv.ServiceNames[override.Target] + if !ok { + return nil + } + + return []string{name} + + case ThresholdScopeCluster: + return inv.ServicesByCluster[override.Target] + } + + // do not add `default:` to make exhaustive linter do its job + + return nil +} + +// ResolveThresholds returns the effective threshold for every target covered by an +// override or a tombstone, keyed by the join-label value the rule matches on. +// +// This is the single implementation of precedence. Both the metrics collector and the +// API must call it: if they resolved separately and drifted, the value the API reports +// and the value the rule evaluates against would silently disagree. +// +// Tombstoned rows contribute no candidate for their own scope, so clearing a service +// override correctly falls through to a covering cluster override rather than jumping +// straight to the default. A tombstoned target with no surviving override at any scope +// resolves to defaultValue - which is what makes clearing an override a value change on +// an existing series rather than the series disappearing. +func ResolveThresholds(overrides []*AlertRuleThresholdOverride, defaultValue float64, inv ThresholdInventory) map[string]float64 { + resolved := make(map[string]float64, len(overrides)) + specificity := make(map[string]int, len(overrides)) + + var cleared []string + + for _, override := range overrides { + names := inv.targetNames(override) + if len(names) == 0 { + continue + } + + if override.IsCleared() { + cleared = append(cleared, names...) + continue + } + + rank := thresholdScopeSpecificity[override.Scope] + for _, name := range names { + existing, ok := specificity[name] + if ok && existing >= rank { + continue + } + + resolved[name] = override.Value + specificity[name] = rank + } + } + + // A cleared target keeps its series alive at the rule's default, unless a coarser + // override still applies to it. + for _, name := range cleared { + _, ok := resolved[name] + if !ok { + resolved[name] = defaultValue + } + } + + return resolved +} diff --git a/managed/models/threshold_resolver_test.go b/managed/models/threshold_resolver_test.go new file mode 100644 index 0000000000..aef2038fb0 --- /dev/null +++ b/managed/models/threshold_resolver_test.go @@ -0,0 +1,262 @@ +// Copyright (C) 2023 Percona LLC +// +// This program is free software: you can redistribute it and/or modify +// it under the terms of the GNU Affero General Public License as published by +// the Free Software Foundation, either version 3 of the License, or +// (at your option) any later version. +// +// This program is distributed in the hope that it will be useful, +// but WITHOUT ANY WARRANTY; without even the implied warranty of +// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +// GNU Affero General Public License for more details. +// +// You should have received a copy of the GNU Affero General Public License +// along with this program. If not, see . + +package models + +import ( + "fmt" + "testing" + "time" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" +) + +const testDefault = 80.0 + +func testInventory() ThresholdInventory { + return ThresholdInventory{ + NodeNames: map[string]string{ + "node-id-1": "node-1", + "node-id-2": "node-2", + }, + ServiceNames: map[string]string{ + "svc-id-1": "svc-1", + "svc-id-2": "svc-2", + }, + ServicesByCluster: map[string][]string{ + "prod": {"svc-1", "svc-2"}, + }, + } +} + +func override(scope ThresholdScope, target string, value float64) *AlertRuleThresholdOverride { + return &AlertRuleThresholdOverride{ + ID: fmt.Sprintf("%s-%s", scope, target), + RuleID: "rule-1", + ParamName: "threshold", + Scope: scope, + Target: target, + Value: value, + } +} + +func tombstone(scope ThresholdScope, target string, value float64) *AlertRuleThresholdOverride { + o := override(scope, target, value) + cleared := time.Now() + o.ClearedAt = &cleared + + return o +} + +func TestResolveThresholds(t *testing.T) { + t.Parallel() + + for _, tc := range []struct { + name string + overrides []*AlertRuleThresholdOverride + expected map[string]float64 + }{ + { + name: "no overrides emits nothing", + overrides: nil, + expected: map[string]float64{}, + }, + { + name: "node override resolves to the node name", + overrides: []*AlertRuleThresholdOverride{override(ThresholdScopeNode, "node-id-1", 90)}, + expected: map[string]float64{"node-1": 90}, + }, + { + name: "service override resolves to the service name", + overrides: []*AlertRuleThresholdOverride{override(ThresholdScopeService, "svc-id-1", 91)}, + expected: map[string]float64{"svc-1": 91}, + }, + { + name: "cluster override fans out onto every service in the cluster", + overrides: []*AlertRuleThresholdOverride{override(ThresholdScopeCluster, "prod", 70)}, + expected: map[string]float64{"svc-1": 70, "svc-2": 70}, + }, + { + name: "service beats cluster regardless of value", + overrides: []*AlertRuleThresholdOverride{ + override(ThresholdScopeCluster, "prod", 99), + override(ThresholdScopeService, "svc-id-1", 50), + }, + // svc-1 takes the more specific 50 even though the cluster value is larger: + // precedence is by scope, not by magnitude. + expected: map[string]float64{"svc-1": 50, "svc-2": 99}, + }, + { + name: "unresolvable target is skipped, never defaulted", + overrides: []*AlertRuleThresholdOverride{ + override(ThresholdScopeNode, "deleted-node-id", 90), + }, + expected: map[string]float64{}, + }, + { + name: "unknown cluster expands to nothing", + overrides: []*AlertRuleThresholdOverride{ + override(ThresholdScopeCluster, "staging", 90), + }, + expected: map[string]float64{}, + }, + { + name: "tombstone with no surviving override resolves to the default", + overrides: []*AlertRuleThresholdOverride{ + tombstone(ThresholdScopeNode, "node-id-1", 90), + }, + expected: map[string]float64{"node-1": testDefault}, + }, + { + name: "tombstone falls through to a covering cluster override, not the default", + overrides: []*AlertRuleThresholdOverride{ + override(ThresholdScopeCluster, "prod", 70), + tombstone(ThresholdScopeService, "svc-id-1", 50), + }, + expected: map[string]float64{"svc-1": 70, "svc-2": 70}, + }, + { + name: "tombstoned cluster override clears every service it covered", + overrides: []*AlertRuleThresholdOverride{ + tombstone(ThresholdScopeCluster, "prod", 70), + }, + expected: map[string]float64{"svc-1": testDefault, "svc-2": testDefault}, + }, + { + name: "a tombstone never contributes its stale value", + overrides: []*AlertRuleThresholdOverride{ + tombstone(ThresholdScopeNode, "node-id-1", 12345), + }, + expected: map[string]float64{"node-1": testDefault}, + }, + } { + t.Run(tc.name, func(t *testing.T) { + t.Parallel() + + actual := ResolveThresholds(tc.overrides, testDefault, testInventory()) + assert.Equal(t, tc.expected, actual) + }) + } +} + +// TestResolveThresholdsPrecedenceAcrossAllScopes pins the order that is derivable rather +// than conventional: a service runs on exactly one node and belongs to at most one +// cluster, so a service override is strictly narrower than either and must win. +func TestResolveThresholdsPrecedenceAcrossAllScopes(t *testing.T) { + t.Parallel() + + inv := testInventory() + // A node whose name collides with a service name is the only way node and service + // scope can reach the same target, since they otherwise resolve into separate label + // namespaces. Nothing in the schema prevents it: node_name and service_name are + // unique within their own tables, not across them. + inv.NodeNames["node-id-3"] = "svc-1" + + overrides := []*AlertRuleThresholdOverride{ + override(ThresholdScopeCluster, "prod", 10), + override(ThresholdScopeService, "svc-id-1", 20), + override(ThresholdScopeNode, "node-id-3", 30), + } + + resolved := ResolveThresholds(overrides, testDefault, inv) + assert.InDelta(t, 20.0, resolved["svc-1"], 0.0001, "service scope must win over node and cluster") +} + +// TestResolveThresholdsNodeBeatsCluster pins the conventional half of the order. Node and +// cluster cross-cut rather than nest - a cluster spans several nodes, a node hosts +// services from several clusters - so this is a chosen tie-break, not a containment. +func TestResolveThresholdsNodeBeatsCluster(t *testing.T) { + t.Parallel() + + inv := testInventory() + inv.NodeNames["node-id-3"] = "svc-1" + + overrides := []*AlertRuleThresholdOverride{ + override(ThresholdScopeCluster, "prod", 10), + override(ThresholdScopeNode, "node-id-3", 30), + } + + resolved := ResolveThresholds(overrides, testDefault, inv) + assert.InDelta(t, 30.0, resolved["svc-1"], 0.0001) +} + +func TestResolveThresholdsIsOrderIndependent(t *testing.T) { + t.Parallel() + + forward := []*AlertRuleThresholdOverride{ + override(ThresholdScopeCluster, "prod", 99), + override(ThresholdScopeService, "svc-id-1", 50), + } + reversed := []*AlertRuleThresholdOverride{forward[1], forward[0]} + + inv := testInventory() + assert.Equal(t, + ResolveThresholds(forward, testDefault, inv), + ResolveThresholds(reversed, testDefault, inv), + "precedence must not depend on row order returned by the database") +} + +// TestResolveThresholdsEmitsOneValuePerTarget guards the invariant that matters most +// operationally: two series with identical labels make the Prometheus gatherer fail the +// entire /metrics response, taking every other collector down with it. +func TestResolveThresholdsEmitsOneValuePerTarget(t *testing.T) { + t.Parallel() + + inv := testInventory() + inv.ServicesByCluster["prod"] = []string{"svc-1", "svc-1", "svc-2"} + + overrides := []*AlertRuleThresholdOverride{ + override(ThresholdScopeCluster, "prod", 70), + override(ThresholdScopeNode, "node-id-1", 90), + tombstone(ThresholdScopeNode, "node-id-2", 60), + } + + resolved := ResolveThresholds(overrides, testDefault, inv) + require.Len(t, resolved, 4) + assert.InDelta(t, 70.0, resolved["svc-1"], 0.0001) + assert.InDelta(t, 70.0, resolved["svc-2"], 0.0001) + assert.InDelta(t, 90.0, resolved["node-1"], 0.0001) + assert.InDelta(t, testDefault, resolved["node-2"], 0.0001) +} + +func BenchmarkResolveThresholds(b *testing.B) { + inv := ThresholdInventory{ + NodeNames: make(map[string]string, 1000), + ServiceNames: map[string]string{}, + ServicesByCluster: make(map[string][]string, 50), + } + + var overrides []*AlertRuleThresholdOverride + for i := range 1000 { + id := fmt.Sprintf("node-id-%d", i) + inv.NodeNames[id] = fmt.Sprintf("node-%d", i) + overrides = append(overrides, override(ThresholdScopeNode, id, float64(i%100))) + } + + for i := range 50 { + cluster := fmt.Sprintf("cluster-%d", i) + services := make([]string, 0, 200) + for j := range 200 { + services = append(services, fmt.Sprintf("svc-%d-%d", i, j)) + } + inv.ServicesByCluster[cluster] = services + overrides = append(overrides, override(ThresholdScopeCluster, cluster, 55)) + } + + for b.Loop() { + ResolveThresholds(overrides, testDefault, inv) + } +} diff --git a/managed/pi/alert/overridable.go b/managed/pi/alert/overridable.go new file mode 100644 index 0000000000..9fd10e352c --- /dev/null +++ b/managed/pi/alert/overridable.go @@ -0,0 +1,78 @@ +// Copyright (C) 2023 Percona LLC +// +// This program is free software: you can redistribute it and/or modify +// it under the terms of the GNU Affero General Public License as published by +// the Free Software Foundation, either version 3 of the License, or +// (at your option) any later version. +// +// This program is distributed in the hope that it will be useful, +// but WITHOUT ANY WARRANTY; without even the implied warranty of +// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +// GNU Affero General Public License for more details. +// +// You should have received a copy of the GNU Affero General Public License +// along with this program. If not, see . + +package alert + +import ( + "fmt" + "regexp" +) + +// ParamTokenRegexp returns a regexp matching a parameter's placeholder token, tolerating +// the optional whitespace the template syntax allows, e.g. both `[[ .threshold ]]` and +// `[[.threshold]]`. The name is quoted, so any parameter name is safe to pass. +func ParamTokenRegexp(name string) *regexp.Regexp { + return regexp.MustCompile(`\[\[\s*\.` + regexp.QuoteMeta(name) + `\s*\]\]`) +} + +// ParamReferencedInExpressions reports whether any expression step references the parameter. +func (r *Template) ParamReferencedInExpressions(name string) bool { + re := ParamTokenRegexp(name) + for _, expression := range r.Expressions { + if re.MatchString(expression.Expression) { + return true + } + } + + return false +} + +// OverridableParams returns the template's overridable parameters, in declaration order. +func (r *Template) OverridableParams() []Parameter { + var params []Parameter + for _, param := range r.Params { + if param.Overridable { + params = append(params, param) + } + } + + return params +} + +// validateOverridableParams checks the constraints that depend on the template's shape, +// rather than on the parameter alone. +func (r *Template) validateOverridableParams() error { + for _, param := range r.Params { + if !param.Overridable { + continue + } + + // A single-expression template bakes its threshold into the RHS of a PromQL + // comparison, which has to be split apart before a threshold step can be + // injected. That is deliberately not supported yet, so reject it here instead + // of silently ignoring the flag and shipping a rule that never overrides. + if !r.UsesMultipleExpressions() { + return fmt.Errorf("parameter %q cannot be overridable: only multi-expression templates support overridable parameters", param.Name) + } + + // The threshold is injected as a separate query step and referenced from the + // expression, so a parameter that no expression mentions has nothing to override. + if !r.ParamReferencedInExpressions(param.Name) { + return fmt.Errorf("overridable parameter %q must be referenced by an expression step", param.Name) + } + } + + return nil +} diff --git a/managed/pi/alert/overridable_test.go b/managed/pi/alert/overridable_test.go new file mode 100644 index 0000000000..c91f5760a2 --- /dev/null +++ b/managed/pi/alert/overridable_test.go @@ -0,0 +1,191 @@ +// Copyright (C) 2023 Percona LLC +// +// This program is free software: you can redistribute it and/or modify +// it under the terms of the GNU Affero General Public License as published by +// the Free Software Foundation, either version 3 of the License, or +// (at your option) any later version. +// +// This program is distributed in the hope that it will be useful, +// but WITHOUT ANY WARRANTY; without even the implied warranty of +// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +// GNU Affero General Public License for more details. +// +// You should have received a copy of the GNU Affero General Public License +// along with this program. If not, see . + +package alert + +import ( + "testing" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + + "github.com/percona/pmm/managed/pi/common" +) + +// overridableTemplate returns a valid multi-expression template whose single param is +// overridable, so each test can vary exactly the one field it cares about. +func overridableTemplate() Template { + return Template{ + Name: "test_template", + Version: 1, + Summary: "summary", + For: 300, + Severity: common.Warning, + Queries: []TemplateQuery{{ + RefID: "A", + Expr: "up", + }}, + Expressions: []TemplateExpression{{ + RefID: "C", + Type: "math", + Expression: "$A > [[ .threshold ]]", + }}, + Condition: "C", + Params: []Parameter{{ + Name: "threshold", + Summary: "threshold", + Type: Float, + Value: 80, + Overridable: true, + }}, + } +} + +func TestParamTokenRegexp(t *testing.T) { + t.Parallel() + + re := ParamTokenRegexp("threshold") + + assert.True(t, re.MatchString("$A > [[ .threshold ]]")) + assert.True(t, re.MatchString("$A > [[.threshold]]")) + assert.True(t, re.MatchString("$A > [[ .threshold ]]")) + + assert.False(t, re.MatchString("$A > [[ .other ]]")) + assert.False(t, re.MatchString("$A > 80")) + + // A prefix must not match a longer parameter name. + assert.False(t, ParamTokenRegexp("thresh").MatchString("$A > [[ .threshold ]]")) +} + +func TestParamTokenRegexpQuotesName(t *testing.T) { + t.Parallel() + + // A name containing regexp metacharacters must be matched literally, not as a pattern. + re := ParamTokenRegexp("a.b") + assert.True(t, re.MatchString("[[ .a.b ]]")) + assert.False(t, re.MatchString("[[ .axb ]]")) +} + +func TestParamReferencedInExpressions(t *testing.T) { + t.Parallel() + + template := overridableTemplate() + + assert.True(t, template.ParamReferencedInExpressions("threshold")) + assert.False(t, template.ParamReferencedInExpressions("missing")) +} + +func TestGetOverrideScopesDefaultsToNode(t *testing.T) { + t.Parallel() + + param := Parameter{Name: "threshold", Type: Float, Overridable: true} + assert.Equal(t, []string{OverrideScopeNode}, param.GetOverrideScopes()) + + param.OverrideScopes = []string{OverrideScopeService, OverrideScopeCluster} + assert.Equal(t, []string{OverrideScopeService, OverrideScopeCluster}, param.GetOverrideScopes()) +} + +func TestOverridableParams(t *testing.T) { + t.Parallel() + + template := overridableTemplate() + template.Params = append(template.Params, Parameter{ + Name: "other", + Summary: "other", + Type: Float, + Value: 1, + }) + + params := template.OverridableParams() + require.Len(t, params, 1) + assert.Equal(t, "threshold", params[0].Name) +} + +func TestValidateOverridableTemplate(t *testing.T) { + t.Parallel() + + template := overridableTemplate() + require.NoError(t, template.Validate()) +} + +func TestValidateOverridableRejectsSingleExpression(t *testing.T) { + t.Parallel() + + template := overridableTemplate() + template.Queries = nil + template.Expressions = nil + template.Condition = "" + template.Expr = "up > [[ .threshold ]]" + + err := template.Validate() + require.Error(t, err) + assert.Contains(t, err.Error(), "only multi-expression templates") +} + +func TestValidateOverridableRejectsUnreferencedParam(t *testing.T) { + t.Parallel() + + template := overridableTemplate() + template.Expressions[0].Expression = "$A > 80" + + err := template.Validate() + require.Error(t, err) + assert.Contains(t, err.Error(), "must be referenced by an expression step") +} + +func TestValidateOverridableRejectsNonFloat(t *testing.T) { + t.Parallel() + + template := overridableTemplate() + template.Params[0].Type = String + template.Params[0].Value = "80" + + err := template.Validate() + require.Error(t, err) + assert.Contains(t, err.Error(), "must be of type float") +} + +func TestValidateOverridableRejectsUnknownScope(t *testing.T) { + t.Parallel() + + template := overridableTemplate() + template.Params[0].OverrideScopes = []string{"rack"} + + err := template.Validate() + require.Error(t, err) + assert.Contains(t, err.Error(), "unknown override scope") +} + +func TestValidateRejectsScopesWithoutOverridable(t *testing.T) { + t.Parallel() + + template := overridableTemplate() + template.Params[0].Overridable = false + template.Params[0].OverrideScopes = []string{OverrideScopeNode} + + err := template.Validate() + require.Error(t, err) + assert.Contains(t, err.Error(), "not overridable") +} + +func TestValidateAcceptsNonOverridableTemplates(t *testing.T) { + t.Parallel() + + template := overridableTemplate() + template.Params[0].Overridable = false + template.Expressions[0].Expression = "$A > [[ .threshold ]]" + + require.NoError(t, template.Validate()) +} diff --git a/managed/pi/alert/parameter.go b/managed/pi/alert/parameter.go index bb2a6a5486..57492f50a6 100644 --- a/managed/pi/alert/parameter.go +++ b/managed/pi/alert/parameter.go @@ -21,14 +21,33 @@ import ( "strconv" ) +// Override scopes an overridable parameter may be tuned at. +const ( + OverrideScopeNode = "node" + OverrideScopeService = "service" + OverrideScopeCluster = "cluster" +) + // Parameter represents alerting template or rule parameter. type Parameter struct { - Name string `yaml:"name"` // required - Summary string `yaml:"summary"` // required - Unit Unit `yaml:"unit,omitempty"` // optional - Type Type `yaml:"type"` // required - Range []any `yaml:"range,flow,omitempty"` - Value any `yaml:"value,omitempty"` + Name string `yaml:"name"` // required + Summary string `yaml:"summary"` // required + Unit Unit `yaml:"unit,omitempty"` // optional + Type Type `yaml:"type"` // required + Range []any `yaml:"range,flow,omitempty"` // optional + Value any `yaml:"value,omitempty"` // optional + Overridable bool `yaml:"overridable,omitempty"` // optional + OverrideScopes []string `yaml:"override_scopes,flow,omitempty"` // optional +} + +// GetOverrideScopes returns the scopes an override may be set at, defaulting to node +// when the template does not declare any. +func (p *Parameter) GetOverrideScopes() []string { + if len(p.OverrideScopes) == 0 { + return []string{OverrideScopeNode} + } + + return p.OverrideScopes } // GetValueForBool casts parameter value to the bool. @@ -132,7 +151,39 @@ func (p *Parameter) Validate() error { return err } - return p.validateRange() + err = p.validateRange() + if err != nil { + return err + } + + return p.validateOverride() +} + +// validateOverride checks the constraints an overridable parameter carries on its own. +// Constraints that depend on the template's shape are checked in Template.Validate. +func (p *Parameter) validateOverride() error { + if !p.Overridable { + if len(p.OverrideScopes) != 0 { + return errors.New("override_scopes is set but the parameter is not overridable") + } + + return nil + } + + // The threshold travels as a float64 gauge sample, so no other type can carry it. + if p.Type != Float { + return fmt.Errorf("an overridable parameter must be of type float, got %s", p.Type) + } + + for _, scope := range p.OverrideScopes { + switch scope { + case OverrideScopeNode, OverrideScopeService, OverrideScopeCluster: + default: + return fmt.Errorf("unknown override scope %q", scope) + } + } + + return nil } func (p *Parameter) validateValue() error { diff --git a/managed/pi/alert/template.go b/managed/pi/alert/template.go index 696fb9159f..c5a111709c 100644 --- a/managed/pi/alert/template.go +++ b/managed/pi/alert/template.go @@ -146,6 +146,11 @@ func (r *Template) Validate() error { return err } + err = r.validateOverridableParams() + if err != nil { + return err + } + return r.Severity.Validate() } diff --git a/managed/services/alerting/threshold_metrics.go b/managed/services/alerting/threshold_metrics.go new file mode 100644 index 0000000000..925ead357d --- /dev/null +++ b/managed/services/alerting/threshold_metrics.go @@ -0,0 +1,252 @@ +// Copyright (C) 2023 Percona LLC +// +// This program is free software: you can redistribute it and/or modify +// it under the terms of the GNU Affero General Public License as published by +// the Free Software Foundation, either version 3 of the License, or +// (at your option) any later version. +// +// This program is distributed in the hope that it will be useful, +// but WITHOUT ANY WARRANTY; without even the implied warranty of +// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +// GNU Affero General Public License for more details. +// +// You should have received a copy of the GNU Affero General Public License +// along with this program. If not, see . + +package alerting + +import ( + "context" + "time" + + prom "github.com/prometheus/client_golang/prometheus" + "github.com/sirupsen/logrus" + "gopkg.in/reform.v1" + + "github.com/percona/pmm/managed/models" +) + +const ( + // thresholdCollectTimeout bounds one scrape. The threshold collector shares + // /debug/metrics, and its 0.9 * MR budget, with the inventory and HA collectors, so + // overrunning here would take those down too. + thresholdCollectTimeout = 3 * time.Second + + // thresholdCtxCheckInterval is how often the emission loop re-checks the deadline. + // The queries are bounded by the context, but the loop that follows them is not, + // so without this a large enough result set could run past the scrape budget with + // nothing stopping it. + thresholdCtxCheckInterval = 1000 + + // thresholdMetricName is the gauge the injected threshold query reads. It is shared + // with rule_builder.go on purpose: the metric name and its label set are a contract + // between the collector and the generated PromQL, and a rule pointing at a metric + // nobody emits fails silently - it simply never fires. + thresholdMetricName = "pmm_alert_threshold_override" + + thresholdRuleIDLabel = "rule_id" + thresholdParamLabel = "param" + // thresholdTargetLabel is generic rather than node_name/service_name because one + // fixed descriptor has to serve every scope; the rule query maps it onto whichever + // label it joins on with label_replace. + thresholdTargetLabel = "target" +) + +// AlertThresholdMetricsCollector exposes the effective threshold for every target that +// carries an override. +// +// Only overridden targets are emitted. Emitting a series for every target instead would +// scale with inventory rather than with what was actually tuned: measured at 1,000 nodes +// and 140 rule/parameter groups it consumed 77-82% of the 9s scrape budget, against 1-7% +// for this shape. Targets with no override get their threshold from the default clause +// of the rule query instead. +type AlertThresholdMetricsCollector struct { + db *reform.DB + l *logrus.Entry + + desc *prom.Desc +} + +// NewAlertThresholdMetricsCollector creates a new instance of AlertThresholdMetricsCollector. +func NewAlertThresholdMetricsCollector(db *reform.DB) *AlertThresholdMetricsCollector { + return &AlertThresholdMetricsCollector{ + db: db, + l: logrus.WithField("component", "alerting/threshold-metrics"), + desc: prom.NewDesc( + thresholdMetricName, + "Effective alert threshold for a rule parameter and target. Emitted only where an "+ + "override or a tombstone exists; targets without either fall back to the rule's "+ + "default, which the rule query materialises from its own observed expression.", + []string{thresholdRuleIDLabel, thresholdParamLabel, thresholdTargetLabel}, + nil, + ), + } +} + +// Describe sends the metric description to the provided channel. +// +// This deliberately does not use prom.DescribeByCollect, which would run a full Collect, +// and therefore a database query, merely to describe the collector. +func (c *AlertThresholdMetricsCollector) Describe(ch chan<- *prom.Desc) { + ch <- c.desc +} + +// thresholdGroup is the set of override rows sharing one rule and parameter, which is +// the granularity precedence is resolved at. +type thresholdGroup struct { + ruleID string + paramName string + overrides []*models.AlertRuleThresholdOverride +} + +// Collect sends the collected metrics to the provided channel. A failure is logged and +// yields no threshold metrics for that scrape rather than failing the whole response. +func (c *AlertThresholdMetricsCollector) Collect(ch chan<- prom.Metric) { + ctx, cancelCtx := context.WithTimeout(context.Background(), thresholdCollectTimeout) + defer cancelCtx() + + var ( + groups []thresholdGroup + rules map[string]*models.AlertRule + inv models.ThresholdInventory + ) + + errTx := c.db.InTransactionContext(ctx, nil, func(tx *reform.TX) error { + overrides, err := models.FindAllThresholdOverrides(tx.Querier) + if err != nil { + return err + } + + // The common case is no overrides at all, and it costs nothing. + if len(overrides) == 0 { + return nil + } + + groups = groupThresholdOverrides(overrides) + + allRules, err := models.FindAlertRules(tx.Querier) + if err != nil { + return err + } + + rules = make(map[string]*models.AlertRule, len(allRules)) + for _, rule := range allRules { + rules[rule.RuleID] = rule + } + + inv, err = loadThresholdInventory(tx.Querier, overrides) + + return err + }) + if errTx != nil { + c.l.Warnf("Failed to collect alert thresholds: %v", errTx) + + return + } + + emitted := 0 + for _, group := range groups { + rule, ok := rules[group.ruleID] + if !ok { + continue + } + + param, ok := rule.Params[group.paramName] + if !ok { + continue + } + + for target, value := range models.ResolveThresholds(group.overrides, param.Default, inv) { + if emitted%thresholdCtxCheckInterval == 0 && ctx.Err() != nil { + c.l.Warnf("Alert threshold collection timed out after %d series", emitted) + + return + } + emitted++ + + ch <- prom.MustNewConstMetric(c.desc, prom.GaugeValue, value, group.ruleID, group.paramName, target) + } + } +} + +// groupThresholdOverrides partitions rows by rule and parameter, which is the unit +// precedence applies within: an override on one parameter says nothing about another. +func groupThresholdOverrides(overrides []*models.AlertRuleThresholdOverride) []thresholdGroup { + type key struct { + ruleID string + paramName string + } + + index := make(map[key]int) + + var groups []thresholdGroup + + for _, override := range overrides { + k := key{ruleID: override.RuleID, paramName: override.ParamName} + + i, ok := index[k] + if !ok { + groups = append(groups, thresholdGroup{ruleID: k.ruleID, paramName: k.paramName}) + i = len(groups) - 1 + index[k] = i + } + + groups[i].overrides = append(groups[i].overrides, override) + } + + return groups +} + +// loadThresholdInventory resolves the IDs the overrides actually reference, rather than +// reading the whole inventory. That is what keeps this collector's cost a function of +// how many targets were tuned instead of how large the fleet is. +func loadThresholdInventory(q *reform.Querier, overrides []*models.AlertRuleThresholdOverride) (models.ThresholdInventory, error) { + var nodeIDs, serviceIDs []string + + for _, override := range overrides { + switch override.Scope { + case models.ThresholdScopeNode: + nodeIDs = append(nodeIDs, override.Target) + case models.ThresholdScopeService: + serviceIDs = append(serviceIDs, override.Target) + case models.ThresholdScopeCluster: + // Cluster scope needs services looked up by cluster label, which arrives + // with the service/cluster increment. Until then such a row cannot be + // created through the API, and one inserted directly resolves to nothing + // and is simply inert. + } + + // do not add `default:` to make exhaustive linter do its job + } + + inv := models.ThresholdInventory{} + + if len(nodeIDs) != 0 { + nodes, err := models.FindNodesByIDs(q, nodeIDs) + if err != nil { + return inv, err + } + + inv.NodeNames = make(map[string]string, len(nodes)) + for _, node := range nodes { + inv.NodeNames[node.NodeID] = node.NodeName + } + } + + if len(serviceIDs) != 0 { + services, err := models.FindServicesByIDs(q, serviceIDs) + if err != nil { + return inv, err + } + + inv.ServiceNames = make(map[string]string, len(services)) + for id, service := range services { + inv.ServiceNames[id] = service.ServiceName + } + } + + return inv, nil +} + +// check interfaces. +var _ prom.Collector = (*AlertThresholdMetricsCollector)(nil) diff --git a/managed/services/alerting/threshold_metrics_test.go b/managed/services/alerting/threshold_metrics_test.go new file mode 100644 index 0000000000..a197ff762d --- /dev/null +++ b/managed/services/alerting/threshold_metrics_test.go @@ -0,0 +1,218 @@ +// Copyright (C) 2023 Percona LLC +// +// This program is free software: you can redistribute it and/or modify +// it under the terms of the GNU Affero General Public License as published by +// the Free Software Foundation, either version 3 of the License, or +// (at your option) any later version. +// +// This program is distributed in the hope that it will be useful, +// but WITHOUT ANY WARRANTY; without even the implied warranty of +// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +// GNU Affero General Public License for more details. +// +// You should have received a copy of the GNU Affero General Public License +// along with this program. If not, see . + +package alerting + +import ( + "strings" + "testing" + + prom "github.com/prometheus/client_golang/prometheus" + "github.com/prometheus/client_golang/prometheus/testutil" + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + "gopkg.in/reform.v1" + "gopkg.in/reform.v1/dialects/postgresql" + + "github.com/percona/pmm/managed/models" + "github.com/percona/pmm/managed/utils/testdb" +) + +const testRuleID = "rule-fixed-for-tests" + +// thresholdExpositionHeader is the HELP/TYPE preamble CollectAndCompare requires. It is +// derived from the live descriptor rather than restated, so a help-text edit does not +// break these tests. +func thresholdExposition(t *testing.T, c *AlertThresholdMetricsCollector, samples ...string) *strings.Reader { + t.Helper() + + desc := c.desc.String() + start := strings.Index(desc, `help: "`) + require.GreaterOrEqual(t, start, 0) + help := desc[start+len(`help: "`):] + help = help[:strings.Index(help, `"`)] + + body := "\n# HELP " + thresholdMetricName + " " + help + + "\n# TYPE " + thresholdMetricName + " gauge\n" + + strings.Join(samples, "\n") + "\n" + + return strings.NewReader(body) +} + +// TestThresholdCollectorDescribeDoesNotQuery passes a nil database on purpose: if +// Describe ever reverts to prom.DescribeByCollect it would run a full Collect, and +// therefore a query, and this test would panic instead of passing. +func TestThresholdCollectorDescribeDoesNotQuery(t *testing.T) { + t.Parallel() + + c := NewAlertThresholdMetricsCollector(nil) + + ch := make(chan *prom.Desc, 1) + c.Describe(ch) + close(ch) + + require.Len(t, ch, 1) + assert.Contains(t, (<-ch).String(), thresholdMetricName) +} + +func TestGroupThresholdOverrides(t *testing.T) { + t.Parallel() + + overrides := []*models.AlertRuleThresholdOverride{ + {RuleID: "r1", ParamName: "a", Target: "t1"}, + {RuleID: "r1", ParamName: "b", Target: "t1"}, + {RuleID: "r1", ParamName: "a", Target: "t2"}, + {RuleID: "r2", ParamName: "a", Target: "t1"}, + } + + groups := groupThresholdOverrides(overrides) + require.Len(t, groups, 3, "one group per (rule, param), not per row") + + // Order follows first appearance, so grouping is deterministic. + assert.Equal(t, "r1", groups[0].ruleID) + assert.Equal(t, "a", groups[0].paramName) + assert.Len(t, groups[0].overrides, 2) + + assert.Equal(t, "b", groups[1].paramName) + assert.Len(t, groups[1].overrides, 1) + + assert.Equal(t, "r2", groups[2].ruleID) +} + +func setupThresholdCollector(t *testing.T) (*AlertThresholdMetricsCollector, *reform.DB) { + t.Helper() + + sqlDB := testdb.Open(t, models.SkipFixtures, nil) + t.Cleanup(func() { + require.NoError(t, sqlDB.Close()) + }) + db := reform.NewDB(sqlDB, postgresql.Dialect, reform.NewPrintfLogger(t.Logf)) + + return NewAlertThresholdMetricsCollector(db), db +} + +func createThresholdRule(t *testing.T, db *reform.DB) { + t.Helper() + + _, err := models.CreateAlertRule(db.Querier, &models.CreateAlertRuleParams{ + RuleID: testRuleID, + Params: models.AlertRuleParams{ + "threshold": { + Default: 80, + JoinLabel: "node_name", + Scopes: []string{string(models.ThresholdScopeNode)}, + }, + }, + }) + require.NoError(t, err) +} + +func createThresholdNode(t *testing.T, db *reform.DB, name string) *models.Node { + t.Helper() + + node, err := models.CreateNode(db.Querier, models.GenericNodeType, &models.CreateNodeParams{ + NodeName: name, + Address: name + ".example.com", + }) + require.NoError(t, err) + + return node +} + +func TestThresholdCollectorEmitsNothingWithoutOverrides(t *testing.T) { + c, db := setupThresholdCollector(t) + createThresholdRule(t, db) + + assert.Equal(t, 0, testutil.CollectAndCount(c, thresholdMetricName), + "a rule with no overrides must emit no series at all") +} + +func TestThresholdCollectorEmitsOverride(t *testing.T) { + c, db := setupThresholdCollector(t) + createThresholdRule(t, db) + node := createThresholdNode(t, db, "node-1") + + _, err := models.UpsertThresholdOverride(db.Querier, testRuleID, "threshold", models.ThresholdScopeNode, node.NodeID, 90) + require.NoError(t, err) + + expected := thresholdExposition(t, c, + `pmm_alert_threshold_override{param="threshold",rule_id="rule-fixed-for-tests",target="node-1"} 90`) + require.NoError(t, testutil.CollectAndCompare(c, expected, thresholdMetricName)) +} + +// TestThresholdCollectorEmitsDefaultForTombstone is the behaviour that makes clearing an +// override fast: the series keeps being emitted and merely changes value. If a cleared +// override stopped being emitted instead, the clear would take a full VictoriaMetrics +// lookbehind to become visible - measured at 309s, against 14-21s for a value change. +func TestThresholdCollectorEmitsDefaultForTombstone(t *testing.T) { + c, db := setupThresholdCollector(t) + createThresholdRule(t, db) + node := createThresholdNode(t, db, "node-1") + + _, err := models.UpsertThresholdOverride(db.Querier, testRuleID, "threshold", models.ThresholdScopeNode, node.NodeID, 90) + require.NoError(t, err) + require.NoError(t, models.ClearThresholdOverride(db.Querier, testRuleID, "threshold", models.ThresholdScopeNode, node.NodeID)) + + expected := thresholdExposition(t, c, + `pmm_alert_threshold_override{param="threshold",rule_id="rule-fixed-for-tests",target="node-1"} 80`) + require.NoError(t, testutil.CollectAndCompare(c, expected, thresholdMetricName)) +} + +// TestThresholdCollectorSkipsDeletedTarget covers the backstop that keeps a row left +// behind by a deleted node inert rather than wrong. +func TestThresholdCollectorSkipsDeletedTarget(t *testing.T) { + c, db := setupThresholdCollector(t) + createThresholdRule(t, db) + + _, err := models.UpsertThresholdOverride(db.Querier, testRuleID, "threshold", models.ThresholdScopeNode, "no-such-node", 90) + require.NoError(t, err) + + assert.Equal(t, 0, testutil.CollectAndCount(c, thresholdMetricName)) +} + +// TestThresholdCollectorSkipsUnknownParam guards against emitting a series for a +// parameter the rule no longer declares, which would have no default to fall back to. +func TestThresholdCollectorSkipsUnknownParam(t *testing.T) { + c, db := setupThresholdCollector(t) + createThresholdRule(t, db) + node := createThresholdNode(t, db, "node-1") + + _, err := models.UpsertThresholdOverride(db.Querier, testRuleID, "gone", models.ThresholdScopeNode, node.NodeID, 90) + require.NoError(t, err) + + assert.Equal(t, 0, testutil.CollectAndCount(c, thresholdMetricName)) +} + +func TestThresholdCollectorEmitsOnePerTargetAcrossParams(t *testing.T) { + c, db := setupThresholdCollector(t) + + _, err := models.CreateAlertRule(db.Querier, &models.CreateAlertRuleParams{ + RuleID: testRuleID, + Params: models.AlertRuleParams{ + "threshold": {Default: 80, JoinLabel: "node_name"}, + "second": {Default: 10, JoinLabel: "node_name"}, + }, + }) + require.NoError(t, err) + + node := createThresholdNode(t, db, "node-1") + for _, param := range []string{"threshold", "second"} { + _, err = models.UpsertThresholdOverride(db.Querier, testRuleID, param, models.ThresholdScopeNode, node.NodeID, 42) + require.NoError(t, err) + } + + // Two params on one target are two distinct series, not a duplicate-label collision. + assert.Equal(t, 2, testutil.CollectAndCount(c, thresholdMetricName)) +} From 342e7abf2068ec958ba06e29c6e48a40e34abdfa Mon Sep 17 00:00:00 2001 From: Matej Kubinec Date: Thu, 27 Aug 2026 13:31:03 +0200 Subject: [PATCH 03/15] PMM-14912 Inject threshold query into alert rules Generate a threshold query step for every overridable parameter and point the rule's expression at it, so the effective threshold is read from data at evaluation time instead of being baked into the rule definition. Tuning a threshold then never rewrites the rule, and so never resets alert state for the other targets evaluated by it. The injected step is: max by () (label_replace(pmm_alert_threshold_override{...}, "", "$1", "target", "(.*)")) or (max by () () * 0 + ) Each clause earns its place. label_replace maps the collector's generic target label onto whichever label this rule joins on; without it the two operands carry different label sets and `or` returns both instead of preferring the left. max by strips instance and job - the threshold is scraped from pmm-managed, which can never match an observed series - reduces both operands to one label set, and collapses the duplicate series an HA cluster emits. The second clause manufactures the default for every target the observed query reports, by reusing that query and discarding its value, so the threshold shares the observed data's fate and cannot go missing while the data it guards is still arriving. The join label is derived from the parameter's scopes rather than declared separately: a node override identifies its target by node name, while service and cluster overrides both identify theirs by service name. Mixing node with either of the other two is therefore incoherent, since a rule joins on one label, and is now rejected when the template is parsed. A parameter is paired with the query it is actually compared against - the nearest query reference to its left - so a template comparing two queries in one expression fans each default out over the right one. Alert filters narrow the observed query but not the threshold, since a filtered threshold would leave the targets the filter excludes with no threshold at all. The generated selector is asserted against the collector's own descriptor. That pairing is the one failure mode with no runtime signal: a rule selecting a metric nobody emits matches nothing and silently never fires. Rules with no PMM-minted ID generate exactly what they did before, so none of this takes effect until the rule registry supplies one. Co-Authored-By: Claude Opus 5 (1M context) --- managed/pi/alert/overridable_test.go | 20 + managed/pi/alert/parameter.go | 15 +- managed/services/alerting/rule_builder.go | 235 +++++++++++- .../alerting/rule_builder_dynamic_test.go | 341 ++++++++++++++++++ .../services/alerting/rule_builder_test.go | 12 +- managed/services/alerting/service.go | 7 +- 6 files changed, 619 insertions(+), 11 deletions(-) create mode 100644 managed/services/alerting/rule_builder_dynamic_test.go diff --git a/managed/pi/alert/overridable_test.go b/managed/pi/alert/overridable_test.go index c91f5760a2..34f48f18d7 100644 --- a/managed/pi/alert/overridable_test.go +++ b/managed/pi/alert/overridable_test.go @@ -189,3 +189,23 @@ func TestValidateAcceptsNonOverridableTemplates(t *testing.T) { require.NoError(t, template.Validate()) } + +func TestValidateOverridableRejectsMixedScopeFamilies(t *testing.T) { + t.Parallel() + + template := overridableTemplate() + template.Params[0].OverrideScopes = []string{OverrideScopeNode, OverrideScopeCluster} + + err := template.Validate() + require.Error(t, err) + assert.Contains(t, err.Error(), "join on different labels") +} + +func TestValidateOverridableAcceptsServiceAndCluster(t *testing.T) { + t.Parallel() + + template := overridableTemplate() + template.Params[0].OverrideScopes = []string{OverrideScopeService, OverrideScopeCluster} + + require.NoError(t, template.Validate()) +} diff --git a/managed/pi/alert/parameter.go b/managed/pi/alert/parameter.go index 57492f50a6..9ef8328386 100644 --- a/managed/pi/alert/parameter.go +++ b/managed/pi/alert/parameter.go @@ -175,14 +175,27 @@ func (p *Parameter) validateOverride() error { return fmt.Errorf("an overridable parameter must be of type float, got %s", p.Type) } + var node, service bool + for _, scope := range p.OverrideScopes { switch scope { - case OverrideScopeNode, OverrideScopeService, OverrideScopeCluster: + case OverrideScopeNode: + node = true + case OverrideScopeService, OverrideScopeCluster: + service = true default: return fmt.Errorf("unknown override scope %q", scope) } } + // A node override identifies its target by node name, while service and cluster + // overrides both identify theirs by service name. A rule joins its threshold on one + // label, so a parameter offering both would silently ignore overrides set at the + // scope that does not match. + if node && service { + return errors.New("override scopes cannot mix node with service or cluster, which join on different labels") + } + return nil } diff --git a/managed/services/alerting/rule_builder.go b/managed/services/alerting/rule_builder.go index dec0db32a2..e9a5479333 100644 --- a/managed/services/alerting/rule_builder.go +++ b/managed/services/alerting/rule_builder.go @@ -18,6 +18,7 @@ package alerting import ( "encoding/json" "fmt" + "regexp" "strings" alertingv1 "github.com/percona/pmm/api/alerting/v1" @@ -34,8 +35,21 @@ const ( expressionTypeMath = "math" queryIntervalMs = 1000 maxDataPoints = 43200 + + // thresholdRefIDPrefix prefixes the ref ID of each injected threshold query. + thresholdRefIDPrefix = "T_" + + // The label the injected threshold query joins the observed query on. It follows + // from the scope: an override targets a node by node_name, and a service - whether + // named directly or reached through its cluster - by service_name. + nodeJoinLabel = "node_name" + serviceJoinLabel = "service_name" ) +// thresholdRefIDSanitizer strips anything a Grafana ref ID cannot carry, so a parameter +// name with punctuation still yields a usable ref ID. +var thresholdRefIDSanitizer = regexp.MustCompile(`[^A-Za-z0-9_]`) + type promQueryModel struct { Expr string `json:"expr"` RefID string `json:"refId"` @@ -58,13 +72,17 @@ type mathExpressionModel struct { func buildGrafanaRuleData( template *alert.Template, metricsDatasourceUID string, + ruleID string, params map[string]string, filters []*alertingv1.Filter, ) ([]services.Data, string, error) { if template.UsesMultipleExpressions() { - return buildMultiExpressionRuleData(template, metricsDatasourceUID, params, filters) + return buildMultiExpressionRuleData(template, metricsDatasourceUID, ruleID, params, filters) } + // Overridable parameters are rejected on single-expression templates at parse time, + // so nothing reaches here needing a threshold step. + expr, err := fillAndFilterExpr(template.Expr, params, filters) if err != nil { return nil, "", err @@ -81,10 +99,16 @@ func buildGrafanaRuleData( func buildMultiExpressionRuleData( template *alert.Template, metricsDatasourceUID string, + ruleID string, params map[string]string, filters []*alertingv1.Filter, ) ([]services.Data, string, error) { - data := make([]services.Data, 0, len(template.Queries)+len(template.Expressions)) + injections, err := planThresholdInjections(template, ruleID, params) + if err != nil { + return nil, "", err + } + + data := make([]services.Data, 0, len(template.Queries)+len(template.Expressions)+len(injections)) for _, query := range template.Queries { expr, err := fillAndFilterExpr(query.Expr, params, filters) @@ -100,8 +124,21 @@ func buildMultiExpressionRuleData( data = append(data, item) } + for _, injection := range injections { + item, err := newPromQueryData(metricsDatasourceUID, injection.refID, injection.expr) + if err != nil { + return nil, "", err + } + + data = append(data, item) + } + for _, expression := range template.Expressions { - expr, err := fillExprWithParams(expression.Expression, params) + // Swap the parameter tokens for their threshold ref IDs before filling, so the + // default is never baked into the rule. + body := swapOverridableTokens(expression.Expression, injections) + + expr, err := fillExprWithParams(body, params) if err != nil { return nil, "", fmt.Errorf("failed to fill expression %s: %w", expression.RefID, err) } @@ -207,3 +244,195 @@ func parseAlertTemplate(yamlContent string) (*alert.Template, error) { return &templates[0], nil } + +// thresholdInjection is one generated threshold query step: the ref ID the expression +// will reference, and the PromQL that resolves the effective threshold per target. +type thresholdInjection struct { + paramName string + refID string + expr string +} + +// planThresholdInjections builds one threshold query per overridable parameter. It +// returns nothing when the rule has no PMM-minted ID, which is how rules created before +// this feature - and rules with no overridable parameters - keep their previous shape. +func planThresholdInjections(template *alert.Template, ruleID string, params map[string]string) ([]thresholdInjection, error) { + overridable := template.OverridableParams() + if ruleID == "" || len(overridable) == 0 { + return nil, nil + } + + taken := make(map[string]struct{}, len(template.Queries)+len(template.Expressions)) + for _, query := range template.Queries { + taken[query.RefID] = struct{}{} + } + for _, expression := range template.Expressions { + taken[expression.RefID] = struct{}{} + } + + injections := make([]thresholdInjection, 0, len(overridable)) + for _, param := range overridable { + joinLabel, err := joinLabelForScopes(param.GetOverrideScopes()) + if err != nil { + return nil, fmt.Errorf("parameter %q: %w", param.Name, err) + } + + observed, err := observedQueryForParam(template, param.Name) + if err != nil { + return nil, err + } + + // The fan-out reuses the observed query with its parameters filled but its + // filters left off: a filtered threshold would leave the targets the filter + // excludes with no threshold at all. + observedExpr, err := fillExprWithParams(observed.Expr, params) + if err != nil { + return nil, fmt.Errorf("failed to fill query %s for parameter %q: %w", observed.RefID, param.Name, err) + } + + defaultValue, ok := params[param.Name] + if !ok { + return nil, fmt.Errorf("no value supplied for overridable parameter %q", param.Name) + } + + refID := allocateThresholdRefID(param.Name, taken) + injections = append(injections, thresholdInjection{ + paramName: param.Name, + refID: refID, + expr: thresholdQueryExpr(ruleID, param.Name, joinLabel, observedExpr, defaultValue), + }) + } + + return injections, nil +} + +// thresholdQueryExpr renders the injected threshold step. +// +// The first clause carries the overrides, with label_replace mapping the collector's +// generic target label onto whichever label this rule joins on - without it the two +// operands of the `or` have different label sets and `or` returns both instead of +// preferring the left. +// +// The second clause manufactures the default for every target the observed query +// reports, by reusing that query and discarding its value with `* 0`. Fanning out over +// the observed query rather than over an inventory metric is what makes the threshold +// share the observed data's fate: it cannot go missing while the data it guards is still +// arriving, so a rule cannot silently stop evaluating. +// +// `max by` is load-bearing in both clauses. It strips instance and job - the threshold is +// scraped from pmm-managed, which can never match an observed series - reduces both +// operands to identical label sets so `or` prefers the left, and collapses the duplicate +// series an HA cluster emits. +func thresholdQueryExpr(ruleID, paramName, joinLabel, observedExpr, defaultValue string) string { + return fmt.Sprintf( + `max by (%s) (label_replace(%s{%s=%q, %s=%q}, %q, "$1", %q, "(.*)")) or (max by (%s) (%s) * 0 + %s)`, + joinLabel, + thresholdMetricName, thresholdRuleIDLabel, ruleID, thresholdParamLabel, paramName, + joinLabel, thresholdTargetLabel, + joinLabel, observedExpr, defaultValue, + ) +} + +// joinLabelForScopes derives the join label from the scopes a parameter may be overridden +// at. Node overrides resolve to a node_name while service and cluster overrides both +// resolve to a service_name, so a parameter cannot mix node with the other two: a rule +// joins on one label, and overrides landing in the other namespace would never match. +func joinLabelForScopes(scopes []string) (string, error) { + var node, service bool + for _, scope := range scopes { + switch scope { + case alert.OverrideScopeNode: + node = true + case alert.OverrideScopeService, alert.OverrideScopeCluster: + service = true + default: + return "", fmt.Errorf("unknown override scope %q", scope) + } + } + + if node && service { + return "", fmt.Errorf("override scopes %v mix node with service or cluster, which join on different labels", scopes) + } + + if node { + return nodeJoinLabel, nil + } + + return serviceJoinLabel, nil +} + +// observedQueryForParam returns the query a parameter is compared against, which is the +// one the default clause fans out over. It is the nearest query reference to the left of +// the parameter's token, so a template comparing several queries in one expression - +// `$A > [[ .a ]] && $B > [[ .b ]]` - pairs each parameter with its own query. +func observedQueryForParam(template *alert.Template, paramName string) (alert.TemplateQuery, error) { + token := alert.ParamTokenRegexp(paramName) + + for _, expression := range template.Expressions { + loc := token.FindStringIndex(expression.Expression) + if loc == nil { + continue + } + + preceding := expression.Expression[:loc[0]] + + var ( + found alert.TemplateQuery + at = -1 + ) + + for _, query := range template.Queries { + ref := regexp.MustCompile(`\$` + regexp.QuoteMeta(query.RefID) + `\b`) + + matches := ref.FindAllStringIndex(preceding, -1) + if len(matches) == 0 { + continue + } + + last := matches[len(matches)-1][0] + if last > at { + at, found = last, query + } + } + + if at < 0 { + return alert.TemplateQuery{}, fmt.Errorf( + "overridable parameter %q is not compared against any query in expression %s", paramName, expression.RefID) + } + + return found, nil + } + + return alert.TemplateQuery{}, fmt.Errorf("overridable parameter %q is not referenced by any expression", paramName) +} + +// allocateThresholdRefID derives a ref ID for a parameter's threshold query, suffixing it +// if the template already uses that ref ID. +func allocateThresholdRefID(paramName string, taken map[string]struct{}) string { + base := thresholdRefIDPrefix + thresholdRefIDSanitizer.ReplaceAllString(paramName, "_") + + refID := base + for i := 1; ; i++ { + _, clash := taken[refID] + if !clash { + break + } + + refID = fmt.Sprintf("%s_%d", base, i) + } + + taken[refID] = struct{}{} + + return refID +} + +// swapOverridableTokens rewrites each overridable parameter's token to its threshold ref +// ID. Replacement is literal so that a `$` in the ref ID is never treated as an expansion. +func swapOverridableTokens(expression string, injections []thresholdInjection) string { + for _, injection := range injections { + expression = alert.ParamTokenRegexp(injection.paramName). + ReplaceAllLiteralString(expression, "$"+injection.refID) + } + + return expression +} diff --git a/managed/services/alerting/rule_builder_dynamic_test.go b/managed/services/alerting/rule_builder_dynamic_test.go new file mode 100644 index 0000000000..8351e841d3 --- /dev/null +++ b/managed/services/alerting/rule_builder_dynamic_test.go @@ -0,0 +1,341 @@ +// Copyright (C) 2023 Percona LLC +// +// This program is free software: you can redistribute it and/or modify +// it under the terms of the GNU Affero General Public License as published by +// the Free Software Foundation, either version 3 of the License, or +// (at your option) any later version. +// +// This program is distributed in the hope that it will be useful, +// but WITHOUT ANY WARRANTY; without even the implied warranty of +// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +// GNU Affero General Public License for more details. +// +// You should have received a copy of the GNU Affero General Public License +// along with this program. If not, see . + +package alerting + +import ( + "encoding/json" + "regexp" + "strings" + "testing" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + + alertingv1 "github.com/percona/pmm/api/alerting/v1" + "github.com/percona/pmm/managed/pi/alert" + "github.com/percona/pmm/managed/services" +) + +const testObservedExpr = `avg by(node_name) (rate(node_cpu_seconds_total[5m]))` + +func overridableRuleTemplate() *alert.Template { + return &alert.Template{ + Name: "test_template", + Version: 1, + Summary: "summary", + Queries: []alert.TemplateQuery{{ + RefID: "A", + Expr: testObservedExpr, + }}, + Expressions: []alert.TemplateExpression{{ + RefID: "C", + Type: "math", + Expression: "$A > [[ .threshold ]]", + }}, + Condition: "C", + Params: []alert.Parameter{{ + Name: "threshold", + Summary: "threshold", + Type: alert.Float, + Value: 80, + Overridable: true, + }}, + } +} + +// dataByRefID indexes generated steps so assertions can address one by name. +func dataByRefID(t *testing.T, data []services.Data) map[string]services.Data { + t.Helper() + + byRef := make(map[string]services.Data, len(data)) + for _, item := range data { + byRef[item.RefID] = item + } + + return byRef +} + +// exprOf pulls the PromQL back out of a generated prom query step. +func exprOf(t *testing.T, item services.Data) string { + t.Helper() + + var model promQueryModel + require.NoError(t, json.Unmarshal(item.Model, &model)) + + return model.Expr +} + +// expressionOf pulls the body back out of a generated math expression step. +func expressionOf(t *testing.T, item services.Data) string { + t.Helper() + + var model mathExpressionModel + require.NoError(t, json.Unmarshal(item.Model, &model)) + + return model.Expression +} + +func TestBuildRuleDataInjectsThresholdQuery(t *testing.T) { + t.Parallel() + + data, condition, err := buildGrafanaRuleData( + overridableRuleTemplate(), "metrics-uid", "rule-1", + map[string]string{"threshold": "80"}, nil) + require.NoError(t, err) + assert.Equal(t, "C", condition) + + byRef := dataByRefID(t, data) + require.Len(t, data, 3, "observed query, injected threshold, math expression") + require.Contains(t, byRef, "T_threshold") + + expr := exprOf(t, byRef["T_threshold"]) + + // The override clause, mapping the collector's generic target label onto this + // rule's join label. + assert.Contains(t, expr, `pmm_alert_threshold_override{rule_id="rule-1", param="threshold"}`) + assert.Contains(t, expr, `label_replace(`) + assert.Contains(t, expr, `"node_name", "$1", "target", "(.*)"`) + + // The default clause, fanned out over the rule's own observed query. + assert.Contains(t, expr, `or (max by (node_name) (`+testObservedExpr+`) * 0 + 80)`) + + // The threshold query is a metrics query, not an expression. + assert.Equal(t, "metrics-uid", byRef["T_threshold"].DatasourceUID) +} + +func TestBuildRuleDataSwapsTokenForThresholdRef(t *testing.T) { + t.Parallel() + + data, _, err := buildGrafanaRuleData( + overridableRuleTemplate(), "metrics-uid", "rule-1", + map[string]string{"threshold": "80"}, nil) + require.NoError(t, err) + + body := expressionOf(t, dataByRefID(t, data)["C"]) + + assert.Equal(t, "$A > $T_threshold", body) + assert.NotContains(t, body, "80", "the default must never be baked into the expression") +} + +// TestBuildRuleDataWithoutRuleIDIsUnchanged pins the compatibility path: a rule with no +// PMM-minted ID generates exactly what it did before this feature existed. +func TestBuildRuleDataWithoutRuleIDIsUnchanged(t *testing.T) { + t.Parallel() + + data, _, err := buildGrafanaRuleData( + overridableRuleTemplate(), "metrics-uid", "", + map[string]string{"threshold": "80"}, nil) + require.NoError(t, err) + + require.Len(t, data, 2) + assert.NotContains(t, dataByRefID(t, data), "T_threshold") + assert.Equal(t, "$A > 80", expressionOf(t, dataByRefID(t, data)["C"])) +} + +func TestBuildRuleDataWithoutOverridableParams(t *testing.T) { + t.Parallel() + + template := overridableRuleTemplate() + template.Params[0].Overridable = false + + data, _, err := buildGrafanaRuleData( + template, "metrics-uid", "rule-1", + map[string]string{"threshold": "80"}, nil) + require.NoError(t, err) + + require.Len(t, data, 2) + assert.Equal(t, "$A > 80", expressionOf(t, dataByRefID(t, data)["C"])) +} + +// TestThresholdQueryMatchesCollectorDescriptor is the contract test between the generated +// PromQL and the emitted series. The proof-of-concept this replaces shipped a builder +// querying pmm_alert_threshold while the collector emitted pmm_alert_threshold_override, +// so every rule matched nothing and silently never fired. Nothing about that failure is +// visible at runtime, which is why it is asserted here. +func TestThresholdQueryMatchesCollectorDescriptor(t *testing.T) { + t.Parallel() + + desc := NewAlertThresholdMetricsCollector(nil).desc.String() + + fqName := regexp.MustCompile(`fqName: "([^"]+)"`).FindStringSubmatch(desc) + require.Len(t, fqName, 2, "could not read fqName from %s", desc) + + labels := regexp.MustCompile(`variableLabels: \{([^}]*)\}`).FindStringSubmatch(desc) + require.Len(t, labels, 2, "could not read variableLabels from %s", desc) + + data, _, err := buildGrafanaRuleData( + overridableRuleTemplate(), "metrics-uid", "rule-1", + map[string]string{"threshold": "80"}, nil) + require.NoError(t, err) + + expr := exprOf(t, dataByRefID(t, data)["T_threshold"]) + + assert.Contains(t, expr, fqName[1]+"{", "the query must select the metric the collector registers") + + for _, label := range strings.Split(labels[1], ",") { + label = strings.TrimSpace(label) + require.NotEmpty(t, label) + assert.Contains(t, expr, label, "the query must reference every label the collector emits") + } +} + +func TestThresholdRefIDAvoidsTemplateCollision(t *testing.T) { + t.Parallel() + + template := overridableRuleTemplate() + // The template already uses the ref ID the parameter would otherwise claim. + template.Queries = append(template.Queries, alert.TemplateQuery{ + RefID: "T_threshold", + Expr: "up", + }) + + data, _, err := buildGrafanaRuleData( + template, "metrics-uid", "rule-1", + map[string]string{"threshold": "80"}, nil) + require.NoError(t, err) + + byRef := dataByRefID(t, data) + assert.Contains(t, byRef, "T_threshold_1") + assert.Equal(t, "$A > $T_threshold_1", expressionOf(t, byRef["C"])) +} + +// TestThresholdPairsEachParamWithItsOwnQuery covers a template comparing two queries in +// one expression: each parameter's default must fan out over the query it is actually +// compared against, not over whichever query happens to come first. +func TestThresholdPairsEachParamWithItsOwnQuery(t *testing.T) { + t.Parallel() + + template := &alert.Template{ + Name: "dual", + Version: 1, + Summary: "summary", + Queries: []alert.TemplateQuery{ + {RefID: "A", Expr: "query_a"}, + {RefID: "B", Expr: "query_b"}, + }, + Expressions: []alert.TemplateExpression{{ + RefID: "C", + Type: "math", + Expression: "$A > [[ .first ]] && $B > [[ .second ]]", + }}, + Condition: "C", + Params: []alert.Parameter{ + {Name: "first", Summary: "first", Type: alert.Float, Value: 1, Overridable: true}, + {Name: "second", Summary: "second", Type: alert.Float, Value: 2, Overridable: true}, + }, + } + + data, _, err := buildGrafanaRuleData( + template, "metrics-uid", "rule-1", + map[string]string{"first": "1", "second": "2"}, nil) + require.NoError(t, err) + + byRef := dataByRefID(t, data) + + assert.Contains(t, exprOf(t, byRef["T_first"]), `(query_a) * 0 + 1)`) + assert.Contains(t, exprOf(t, byRef["T_second"]), `(query_b) * 0 + 2)`) + assert.Equal(t, "$A > $T_first && $B > $T_second", expressionOf(t, byRef["C"])) +} + +// TestThresholdQueryIsNotFiltered pins that alert filters narrow the observed query but +// not the threshold. A filtered threshold would leave the targets the filter excludes +// with no threshold at all. +func TestThresholdQueryIsNotFiltered(t *testing.T) { + t.Parallel() + + data, _, err := buildGrafanaRuleData( + overridableRuleTemplate(), "metrics-uid", "rule-1", + map[string]string{"threshold": "80"}, + []*alertingv1.Filter{{ + Type: alertingv1.FilterType_FILTER_TYPE_MATCH, + Label: "node_name", + Regexp: "prod-.*", + }}) + require.NoError(t, err) + + byRef := dataByRefID(t, data) + + assert.Contains(t, exprOf(t, byRef["A"]), "label_match(", "the observed query is filtered") + assert.NotContains(t, exprOf(t, byRef["T_threshold"]), "label_match(", "the threshold query is not") +} + +func TestJoinLabelForScopes(t *testing.T) { + t.Parallel() + + for _, tc := range []struct { + name string + scopes []string + want string + wantErr string + }{ + {name: "node", scopes: []string{alert.OverrideScopeNode}, want: nodeJoinLabel}, + {name: "service", scopes: []string{alert.OverrideScopeService}, want: serviceJoinLabel}, + {name: "cluster", scopes: []string{alert.OverrideScopeCluster}, want: serviceJoinLabel}, + { + name: "service and cluster share a join label", + scopes: []string{alert.OverrideScopeService, alert.OverrideScopeCluster}, + want: serviceJoinLabel, + }, + { + name: "node cannot be mixed with cluster", + scopes: []string{alert.OverrideScopeNode, alert.OverrideScopeCluster}, + wantErr: "join on different labels", + }, + { + name: "unknown scope", + scopes: []string{"rack"}, + wantErr: "unknown override scope", + }, + } { + t.Run(tc.name, func(t *testing.T) { + t.Parallel() + + got, err := joinLabelForScopes(tc.scopes) + if tc.wantErr != "" { + require.Error(t, err) + assert.Contains(t, err.Error(), tc.wantErr) + + return + } + + require.NoError(t, err) + assert.Equal(t, tc.want, got) + }) + } +} + +func TestObservedQueryForParamErrors(t *testing.T) { + t.Parallel() + + t.Run("parameter compared against no query", func(t *testing.T) { + t.Parallel() + + template := overridableRuleTemplate() + template.Expressions[0].Expression = "[[ .threshold ]] > 1" + + _, err := observedQueryForParam(template, "threshold") + require.Error(t, err) + assert.Contains(t, err.Error(), "not compared against any query") + }) + + t.Run("parameter referenced by no expression", func(t *testing.T) { + t.Parallel() + + _, err := observedQueryForParam(overridableRuleTemplate(), "missing") + require.Error(t, err) + assert.Contains(t, err.Error(), "not referenced by any expression") + }) +} diff --git a/managed/services/alerting/rule_builder_test.go b/managed/services/alerting/rule_builder_test.go index f4d1ebe575..f5ae779ccf 100644 --- a/managed/services/alerting/rule_builder_test.go +++ b/managed/services/alerting/rule_builder_test.go @@ -31,7 +31,7 @@ func TestBuildGrafanaRuleDataSingleExpression(t *testing.T) { data, condition, err := buildGrafanaRuleData(&alert.Template{ Expr: "up == 1", - }, "metrics-uid", nil, nil) + }, "metrics-uid", "", nil, nil) require.NoError(t, err) assert.Equal(t, "A", condition) require.Len(t, data, 1) @@ -53,7 +53,7 @@ func TestBuildGrafanaRuleDataMultiExpression(t *testing.T) { Expression: "$A > $B", }}, Condition: "C", - }, "metrics-uid", map[string]string{}, nil) + }, "metrics-uid", "", map[string]string{}, nil) require.NoError(t, err) assert.Equal(t, "C", condition) require.Len(t, data, 3) @@ -85,7 +85,7 @@ func TestBuildGrafanaRuleDataMultiExpressionWithParamsAndFilters(t *testing.T) { Expression: "$A < $B", }}, Condition: "C", - }, "metrics-uid", map[string]string{ + }, "metrics-uid", "", map[string]string{ "window": "[5m]", "threshold": "80", }, []*alertingv1.Filter{{ @@ -120,7 +120,7 @@ func TestBuildGrafanaRuleDataModelContract(t *testing.T) { }, Expressions: []alert.TemplateExpression{{RefID: "C", Type: "math", Expression: "$A > $B"}}, Condition: "C", - }, "metrics-uid", map[string]string{}, nil) + }, "metrics-uid", "", map[string]string{}, nil) require.NoError(t, err) require.Len(t, data, 3) @@ -152,7 +152,7 @@ func TestBuildGrafanaRuleDataMismatchFilter(t *testing.T) { Queries: []alert.TemplateQuery{{RefID: "A", Expr: "up"}}, Expressions: []alert.TemplateExpression{{RefID: "C", Type: "math", Expression: "$A > 0"}}, Condition: "C", - }, "metrics-uid", map[string]string{}, []*alertingv1.Filter{{ + }, "metrics-uid", "", map[string]string{}, []*alertingv1.Filter{{ Type: alertingv1.FilterType_FILTER_TYPE_MISMATCH, Label: "node_name", Regexp: "staging.*", @@ -205,7 +205,7 @@ func TestBuildGrafanaRuleDataMultiExpressionErrors(t *testing.T) { t.Run(tc.name, func(t *testing.T) { t.Parallel() - _, _, err := buildGrafanaRuleData(tc.tmpl, "metrics-uid", map[string]string{}, tc.filters) + _, _, err := buildGrafanaRuleData(tc.tmpl, "metrics-uid", "", map[string]string{}, tc.filters) require.Error(t, err) assert.Contains(t, err.Error(), tc.wantErr) }) diff --git a/managed/services/alerting/service.go b/managed/services/alerting/service.go index 0ff33df1b2..43cb321bc0 100644 --- a/managed/services/alerting/service.go +++ b/managed/services/alerting/service.go @@ -729,7 +729,12 @@ func (s *Service) CreateRule(ctx context.Context, req *alerting.CreateRuleReques return nil, status.Errorf(codes.Internal, "Invalid template %s: %v.", req.TemplateName, err) } - ruleData, condition, err := buildGrafanaRuleData(alertTemplate, metricsDatasourceUID, paramsValues.AsStringMap(), req.Filters) + // A rule only carries threshold steps once it has a PMM-minted ID to key its + // overrides on. Minting and persisting that ID is the registry work; until then an + // empty ID leaves the generated rule exactly as it was before this feature. + var ruleID string + + ruleData, condition, err := buildGrafanaRuleData(alertTemplate, metricsDatasourceUID, ruleID, paramsValues.AsStringMap(), req.Filters) if err != nil { return nil, fmt.Errorf("failed to build alert rule data: %w", err) } From 75804e666fe874c92973ba1e668867e141ee65ce Mon Sep 17 00:00:00 2001 From: Matej Kubinec Date: Thu, 27 Aug 2026 14:47:21 +0200 Subject: [PATCH 04/15] PMM-14912 Register rules with overridable thresholds Mint a PMM identity for every rule created from a template with an overridable parameter, and persist what the threshold resolves against. This is what switches the feature on: until a rule has an ID to key overrides against, the builder emits no threshold step and the rule is generated exactly as before. CreateRule now mints the ID first, because it is the idempotency key for everything that follows, then stamps it on the Grafana rule as pmm_rule_id and stores a registry row keyed on it. The identity travels on the rule itself rather than being its Grafana UID, so a rule copied or renamed in Grafana can still be matched back to its overrides; the UID is only a cache of where the rule currently lives. The row carries a snapshot of each overridable parameter - default, join label, scopes, unit and range. The template it came from can be edited or deleted afterwards, so this is the only durable record of what the rule actually evaluates against, and of the range an override is validated within. The default stored is the value supplied at creation, not the template's, so a rule created with 42 has 42 as its baseline. The row is written before the rule reaches Grafana. The other order would leave a live rule whose thresholds can never be overridden and whose row may never arrive; this order can only leave an orphaned row, which the reconciler reaps, and a Grafana failure deletes it immediately. CreateRuleResponse gains rule_id so callers can address the rule's thresholds without looking it up. Regenerating for that field also picked up two formatting fixes in code from the previous commit that make format would otherwise have flagged. Co-Authored-By: Claude Opus 5 (1M context) --- api/alerting/v1/alerting.pb.go | 19 +- api/alerting/v1/alerting.pb.validate.go | 2 + api/alerting/v1/alerting.proto | 8 +- .../alerting_service/create_rule_responses.go | 48 ++- api/alerting/v1/json/v1.json | 9 +- api/swagger/swagger-dev.json | 9 +- api/swagger/swagger.json | 9 +- managed/services/alerting/rule_builder.go | 3 +- .../alerting/rule_builder_dynamic_test.go | 24 +- managed/services/alerting/service.go | 116 ++++++- .../alerting/service_threshold_test.go | 298 ++++++++++++++++++ 11 files changed, 522 insertions(+), 23 deletions(-) create mode 100644 managed/services/alerting/service_threshold_test.go diff --git a/api/alerting/v1/alerting.pb.go b/api/alerting/v1/alerting.pb.go index 2b62792497..739f03f177 100644 --- a/api/alerting/v1/alerting.pb.go +++ b/api/alerting/v1/alerting.pb.go @@ -1402,7 +1402,12 @@ func (x *CreateRuleRequest) GetInterval() *durationpb.Duration { } type CreateRuleResponse struct { - state protoimpl.MessageState `protogen:"open.v1"` + state protoimpl.MessageState `protogen:"open.v1"` + // Identifier PMM assigns to a rule whose thresholds can be overridden per target. + // Empty when the rule has no overridable parameters, since nothing can be keyed on it. + // This is the rule's identity for threshold purposes rather than its Grafana UID: + // copying or renaming the rule in Grafana preserves it. + RuleId string `protobuf:"bytes,1,opt,name=rule_id,json=ruleId,proto3" json:"rule_id,omitempty"` unknownFields protoimpl.UnknownFields sizeCache protoimpl.SizeCache } @@ -1437,6 +1442,13 @@ func (*CreateRuleResponse) Descriptor() ([]byte, []int) { return file_alerting_v1_alerting_proto_rawDescGZIP(), []int{18} } +func (x *CreateRuleResponse) GetRuleId() string { + if x != nil { + return x.RuleId + } + return "" +} + var File_alerting_v1_alerting_proto protoreflect.FileDescriptor const file_alerting_v1_alerting_proto_rawDesc = "" + @@ -1550,8 +1562,9 @@ const file_alerting_v1_alerting_proto_rawDesc = "" + " \x01(\v2\x19.google.protobuf.DurationR\binterval\x1a?\n" + "\x11CustomLabelsEntry\x12\x10\n" + "\x03key\x18\x01 \x01(\tR\x03key\x12\x14\n" + - "\x05value\x18\x02 \x01(\tR\x05value:\x028\x01\"\x14\n" + - "\x12CreateRuleResponse*\xa6\x01\n" + + "\x05value\x18\x02 \x01(\tR\x05value:\x028\x01\"-\n" + + "\x12CreateRuleResponse\x12\x17\n" + + "\arule_id\x18\x01 \x01(\tR\x06ruleId*\xa6\x01\n" + "\x0eTemplateSource\x12\x1f\n" + "\x1bTEMPLATE_SOURCE_UNSPECIFIED\x10\x00\x12\x1c\n" + "\x18TEMPLATE_SOURCE_BUILT_IN\x10\x01\x12\x18\n" + diff --git a/api/alerting/v1/alerting.pb.validate.go b/api/alerting/v1/alerting.pb.validate.go index 978008ce59..6ab74dee29 100644 --- a/api/alerting/v1/alerting.pb.validate.go +++ b/api/alerting/v1/alerting.pb.validate.go @@ -2571,6 +2571,8 @@ func (m *CreateRuleResponse) validate(all bool) error { var errors []error + // no validation rules for RuleId + if len(errors) > 0 { return CreateRuleResponseMultiError(errors) } diff --git a/api/alerting/v1/alerting.proto b/api/alerting/v1/alerting.proto index 1b7dd727ac..60876174af 100644 --- a/api/alerting/v1/alerting.proto +++ b/api/alerting/v1/alerting.proto @@ -208,7 +208,13 @@ message CreateRuleRequest { google.protobuf.Duration interval = 10; } -message CreateRuleResponse {} +message CreateRuleResponse { + // Identifier PMM assigns to a rule whose thresholds can be overridden per target. + // Empty when the rule has no overridable parameters, since nothing can be keyed on it. + // This is the rule's identity for threshold purposes rather than its Grafana UID: + // copying or renaming the rule in Grafana preserves it. + string rule_id = 1; +} // Alerting service lets to manage alerting templates and create alerting rules from them. service AlertingService { diff --git a/api/alerting/v1/json/client/alerting_service/create_rule_responses.go b/api/alerting/v1/json/client/alerting_service/create_rule_responses.go index 908bc88e1d..e86e37804a 100644 --- a/api/alerting/v1/json/client/alerting_service/create_rule_responses.go +++ b/api/alerting/v1/json/client/alerting_service/create_rule_responses.go @@ -54,7 +54,7 @@ CreateRuleOK describes a response with status code 200, with default header valu A successful response. */ type CreateRuleOK struct { - Payload any + Payload *CreateRuleOKBody } // IsSuccess returns true when this create rule Ok response has a 2xx status code @@ -97,13 +97,15 @@ func (o *CreateRuleOK) String() string { return fmt.Sprintf("[POST /v1/alerting/rules][%d] createRuleOk %s", 200, payload) } -func (o *CreateRuleOK) GetPayload() any { +func (o *CreateRuleOK) GetPayload() *CreateRuleOKBody { return o.Payload } func (o *CreateRuleOK) readResponse(response runtime.ClientResponse, consumer runtime.Consumer, formats strfmt.Registry) error { + o.Payload = new(CreateRuleOKBody) + // response payload - if err := consumer.Consume(response.Body(), &o.Payload); err != nil && !stderrors.Is(err, io.EOF) { + if err := consumer.Consume(response.Body(), o.Payload); err != nil && !stderrors.Is(err, io.EOF) { return err } @@ -681,6 +683,46 @@ func (o *CreateRuleDefaultBodyDetailsItems0) UnmarshalBinary(b []byte) error { return nil } +/* +CreateRuleOKBody create rule OK body +swagger:model CreateRuleOKBody +*/ +type CreateRuleOKBody struct { + // Identifier PMM assigns to a rule whose thresholds can be overridden per target. + // Empty when the rule has no overridable parameters, since nothing can be keyed on it. + // This is the rule's identity for threshold purposes rather than its Grafana UID: + // copying or renaming the rule in Grafana preserves it. + RuleID string `json:"rule_id,omitempty"` +} + +// Validate validates this create rule OK body +func (o *CreateRuleOKBody) Validate(formats strfmt.Registry) error { + return nil +} + +// ContextValidate validates this create rule OK body based on context it is used +func (o *CreateRuleOKBody) ContextValidate(ctx context.Context, formats strfmt.Registry) error { + return nil +} + +// MarshalBinary interface implementation +func (o *CreateRuleOKBody) MarshalBinary() ([]byte, error) { + if o == nil { + return nil, nil + } + return swag.WriteJSON(o) +} + +// UnmarshalBinary interface implementation +func (o *CreateRuleOKBody) UnmarshalBinary(b []byte) error { + var res CreateRuleOKBody + if err := swag.ReadJSON(b, &res); err != nil { + return err + } + *o = res + return nil +} + /* CreateRuleParamsBodyFiltersItems0 Filter represents a single filter condition. swagger:model CreateRuleParamsBodyFiltersItems0 diff --git a/api/alerting/v1/json/v1.json b/api/alerting/v1/json/v1.json index 2223c2443d..0aca3c38ae 100644 --- a/api/alerting/v1/json/v1.json +++ b/api/alerting/v1/json/v1.json @@ -167,7 +167,14 @@ "200": { "description": "A successful response.", "schema": { - "type": "object" + "type": "object", + "properties": { + "rule_id": { + "description": "Identifier PMM assigns to a rule whose thresholds can be overridden per target.\nEmpty when the rule has no overridable parameters, since nothing can be keyed on it.\nThis is the rule's identity for threshold purposes rather than its Grafana UID:\ncopying or renaming the rule in Grafana preserves it.", + "type": "string", + "x-order": 0 + } + } } }, "default": { diff --git a/api/swagger/swagger-dev.json b/api/swagger/swagger-dev.json index 2e47884738..286fbb5c12 100644 --- a/api/swagger/swagger-dev.json +++ b/api/swagger/swagger-dev.json @@ -2155,7 +2155,14 @@ "200": { "description": "A successful response.", "schema": { - "type": "object" + "type": "object", + "properties": { + "rule_id": { + "description": "Identifier PMM assigns to a rule whose thresholds can be overridden per target.\nEmpty when the rule has no overridable parameters, since nothing can be keyed on it.\nThis is the rule's identity for threshold purposes rather than its Grafana UID:\ncopying or renaming the rule in Grafana preserves it.", + "type": "string", + "x-order": 0 + } + } } }, "default": { diff --git a/api/swagger/swagger.json b/api/swagger/swagger.json index cb4c916b74..8287656f36 100644 --- a/api/swagger/swagger.json +++ b/api/swagger/swagger.json @@ -1638,7 +1638,14 @@ "200": { "description": "A successful response.", "schema": { - "type": "object" + "type": "object", + "properties": { + "rule_id": { + "description": "Identifier PMM assigns to a rule whose thresholds can be overridden per target.\nEmpty when the rule has no overridable parameters, since nothing can be keyed on it.\nThis is the rule's identity for threshold purposes rather than its Grafana UID:\ncopying or renaming the rule in Grafana preserves it.", + "type": "string", + "x-order": 0 + } + } } }, "default": { diff --git a/managed/services/alerting/rule_builder.go b/managed/services/alerting/rule_builder.go index e9a5479333..519438c609 100644 --- a/managed/services/alerting/rule_builder.go +++ b/managed/services/alerting/rule_builder.go @@ -397,7 +397,8 @@ func observedQueryForParam(template *alert.Template, paramName string) (alert.Te if at < 0 { return alert.TemplateQuery{}, fmt.Errorf( - "overridable parameter %q is not compared against any query in expression %s", paramName, expression.RefID) + "overridable parameter %q is not compared against any query in expression %s", paramName, expression.RefID, + ) } return found, nil diff --git a/managed/services/alerting/rule_builder_dynamic_test.go b/managed/services/alerting/rule_builder_dynamic_test.go index 8351e841d3..07f29961b0 100644 --- a/managed/services/alerting/rule_builder_dynamic_test.go +++ b/managed/services/alerting/rule_builder_dynamic_test.go @@ -93,7 +93,8 @@ func TestBuildRuleDataInjectsThresholdQuery(t *testing.T) { data, condition, err := buildGrafanaRuleData( overridableRuleTemplate(), "metrics-uid", "rule-1", - map[string]string{"threshold": "80"}, nil) + map[string]string{"threshold": "80"}, nil, + ) require.NoError(t, err) assert.Equal(t, "C", condition) @@ -121,7 +122,8 @@ func TestBuildRuleDataSwapsTokenForThresholdRef(t *testing.T) { data, _, err := buildGrafanaRuleData( overridableRuleTemplate(), "metrics-uid", "rule-1", - map[string]string{"threshold": "80"}, nil) + map[string]string{"threshold": "80"}, nil, + ) require.NoError(t, err) body := expressionOf(t, dataByRefID(t, data)["C"]) @@ -137,7 +139,8 @@ func TestBuildRuleDataWithoutRuleIDIsUnchanged(t *testing.T) { data, _, err := buildGrafanaRuleData( overridableRuleTemplate(), "metrics-uid", "", - map[string]string{"threshold": "80"}, nil) + map[string]string{"threshold": "80"}, nil, + ) require.NoError(t, err) require.Len(t, data, 2) @@ -153,7 +156,8 @@ func TestBuildRuleDataWithoutOverridableParams(t *testing.T) { data, _, err := buildGrafanaRuleData( template, "metrics-uid", "rule-1", - map[string]string{"threshold": "80"}, nil) + map[string]string{"threshold": "80"}, nil, + ) require.NoError(t, err) require.Len(t, data, 2) @@ -178,7 +182,8 @@ func TestThresholdQueryMatchesCollectorDescriptor(t *testing.T) { data, _, err := buildGrafanaRuleData( overridableRuleTemplate(), "metrics-uid", "rule-1", - map[string]string{"threshold": "80"}, nil) + map[string]string{"threshold": "80"}, nil, + ) require.NoError(t, err) expr := exprOf(t, dataByRefID(t, data)["T_threshold"]) @@ -204,7 +209,8 @@ func TestThresholdRefIDAvoidsTemplateCollision(t *testing.T) { data, _, err := buildGrafanaRuleData( template, "metrics-uid", "rule-1", - map[string]string{"threshold": "80"}, nil) + map[string]string{"threshold": "80"}, nil, + ) require.NoError(t, err) byRef := dataByRefID(t, data) @@ -240,7 +246,8 @@ func TestThresholdPairsEachParamWithItsOwnQuery(t *testing.T) { data, _, err := buildGrafanaRuleData( template, "metrics-uid", "rule-1", - map[string]string{"first": "1", "second": "2"}, nil) + map[string]string{"first": "1", "second": "2"}, nil, + ) require.NoError(t, err) byRef := dataByRefID(t, data) @@ -263,7 +270,8 @@ func TestThresholdQueryIsNotFiltered(t *testing.T) { Type: alertingv1.FilterType_FILTER_TYPE_MATCH, Label: "node_name", Regexp: "prod-.*", - }}) + }}, + ) require.NoError(t, err) byRef := dataByRefID(t, data) diff --git a/managed/services/alerting/service.go b/managed/services/alerting/service.go index 43cb321bc0..1a8581b93a 100644 --- a/managed/services/alerting/service.go +++ b/managed/services/alerting/service.go @@ -32,6 +32,7 @@ import ( "time" "github.com/AlekSi/pointer" + "github.com/google/uuid" "github.com/sirupsen/logrus" "google.golang.org/grpc/codes" "google.golang.org/grpc/status" @@ -730,9 +731,22 @@ func (s *Service) CreateRule(ctx context.Context, req *alerting.CreateRuleReques } // A rule only carries threshold steps once it has a PMM-minted ID to key its - // overrides on. Minting and persisting that ID is the registry work; until then an - // empty ID leaves the generated rule exactly as it was before this feature. - var ruleID string + // overrides on. The ID is minted first because it is the idempotency key for every + // step that follows; a rule with no overridable parameters never gets one, and is + // generated exactly as it was before this feature. + var ( + ruleID string + ruleParams models.AlertRuleParams + ) + + if len(alertTemplate.OverridableParams()) != 0 { + ruleID = uuid.New().String() + + ruleParams, err = collectOverridableParams(alertTemplate, paramsValues) + if err != nil { + return nil, status.Errorf(codes.InvalidArgument, "Invalid overridable parameters: %v.", err) + } + } ruleData, condition, err := buildGrafanaRuleData(alertTemplate, metricsDatasourceUID, ruleID, paramsValues.AsStringMap(), req.Filters) if err != nil { @@ -773,6 +787,14 @@ func (s *Service) CreateRule(ctx context.Context, req *alerting.CreateRuleReques labels["percona_alerting"] = "1" // TODO: do we actually need it? labels["severity"] = common.Severity(req.Severity).String() labels["template_name"] = req.TemplateName + + // The rule's identity for threshold purposes travels on the rule itself, so a rule + // can still be matched back to its registry row after being copied or renamed in + // Grafana. The stored Grafana UID is only a cache of where it currently lives. + if ruleID != "" { + labels["pmm_rule_id"] = ruleID + } + labelSourceRefID := queryRefForRuleLabels(alertTemplate) ensureRuleLabel(labels, "node_name", buildRuleLabelTemplate("node_name", labelSourceRefID)) ensureRuleLabel(labels, "service_name", buildRuleLabelTemplate("service_name", labelSourceRefID)) @@ -796,12 +818,98 @@ func (s *Service) CreateRule(ctx context.Context, req *alerting.CreateRuleReques interval = req.Interval.AsDuration().String() } + // The registry row is written before the rule exists in Grafana. The other order + // would leave a rule whose thresholds cannot be overridden and whose row may never + // arrive; this order can only leave an orphaned row, which the reconciler reaps. + if ruleID != "" { + err = s.db.InTransaction(func(tx *reform.TX) error { + _, err := models.CreateAlertRule(tx.Querier, &models.CreateAlertRuleParams{ + RuleID: ruleID, + Params: ruleParams, + }) + + return err + }) + if err != nil { + return nil, fmt.Errorf("failed to register alert rule: %w", err) + } + } + err = s.grafanaClient.CreateAlertRule(ctx, req.FolderUid, req.Group, interval, &rule) if err != nil { + s.deleteRuleRegistration(ruleID) + return nil, err } - return &alerting.CreateRuleResponse{}, nil + return &alerting.CreateRuleResponse{RuleId: ruleID}, nil +} + +// deleteRuleRegistration removes a registry row whose Grafana rule was never created. +// Failure is logged rather than returned: the caller is already reporting the original +// error, and a row left behind is reaped by the reconciler. +func (s *Service) deleteRuleRegistration(ruleID string) { + if ruleID == "" { + return + } + + err := s.db.InTransaction(func(tx *reform.TX) error { + return models.DeleteAlertRule(tx.Querier, ruleID) + }) + if err != nil { + s.l.WithError(err).WithField("rule_id", ruleID).Warn("Failed to roll back alert rule registration") + } +} + +// collectOverridableParams snapshots what an overridable parameter needs in order to be +// resolved later. The template it came from can be edited or deleted afterwards, so this +// is the only durable record of the default the rule actually evaluates against, and of +// the range an override is validated within. +func collectOverridableParams(template *alert.Template, values AlertExprParamsValues) (models.AlertRuleParams, error) { + overridable := template.OverridableParams() + if len(overridable) == 0 { + return nil, nil + } + + byName := make(map[string]AlertExprParamValue, len(values)) + for _, value := range values { + byName[value.Name] = value + } + + params := make(models.AlertRuleParams, len(overridable)) + + for _, param := range overridable { + joinLabel, err := joinLabelForScopes(param.GetOverrideScopes()) + if err != nil { + return nil, fmt.Errorf("parameter %q: %w", param.Name, err) + } + + supplied, ok := byName[param.Name] + if !ok { + return nil, fmt.Errorf("no value supplied for overridable parameter %q", param.Name) + } + + snapshot := models.AlertRuleParam{ + Default: supplied.FloatValue, + JoinLabel: joinLabel, + Scopes: param.GetOverrideScopes(), + Unit: string(param.Unit), + Summary: param.Summary, + } + + if len(param.Range) != 0 { + pMin, pMax, err := param.GetRangeForFloat() + if err != nil { + return nil, fmt.Errorf("parameter %q: failed to parse range: %w", param.Name, err) + } + + snapshot.Min, snapshot.Max = new(pMin), new(pMax) + } + + params[param.Name] = snapshot + } + + return params, nil } func convertParamsValuesToModel(params []*alerting.ParamValue) (AlertExprParamsValues, error) { diff --git a/managed/services/alerting/service_threshold_test.go b/managed/services/alerting/service_threshold_test.go new file mode 100644 index 0000000000..964946992d --- /dev/null +++ b/managed/services/alerting/service_threshold_test.go @@ -0,0 +1,298 @@ +// Copyright (C) 2023 Percona LLC +// +// This program is free software: you can redistribute it and/or modify +// it under the terms of the GNU Affero General Public License as published by +// the Free Software Foundation, either version 3 of the License, or +// (at your option) any later version. +// +// This program is distributed in the hope that it will be useful, +// but WITHOUT ANY WARRANTY; without even the implied warranty of +// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +// GNU Affero General Public License for more details. +// +// You should have received a copy of the GNU Affero General Public License +// along with this program. If not, see . + +package alerting + +import ( + "errors" + "strings" + "testing" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/mock" + "github.com/stretchr/testify/require" + "gopkg.in/reform.v1" + "gopkg.in/reform.v1/dialects/postgresql" + + alerting "github.com/percona/pmm/api/alerting/v1" + managementv1 "github.com/percona/pmm/api/management/v1" + "github.com/percona/pmm/managed/models" + "github.com/percona/pmm/managed/pi/alert" + "github.com/percona/pmm/managed/services" + "github.com/percona/pmm/managed/utils/testdb" +) + +const overridableWiringYAML = `templates: + - name: test_overridable_wiring + version: 1 + summary: Overridable threshold wiring + queries: + - ref_id: A + expr: |- + (1 - avg by(node_name) (rate(node_cpu_seconds_total{mode="idle"}[5m]))) * 100 + expressions: + - ref_id: C + type: math + expression: "$A > [[ .threshold ]]" + condition: C + params: + - name: threshold + summary: A percentage from configured maximum + unit: "%" + type: float + range: [0, 100] + value: 80 + overridable: true + for: 5m + severity: warning + annotations: + summary: Node high CPU load ({{ $labels.node_name }}) +` + +func templateFromYAML(t *testing.T, yaml string) *models.Template { + t.Helper() + + parsed, err := alert.Parse(strings.NewReader(yaml), &alert.ParseParams{ + DisallowUnknownFields: true, + DisallowInvalidTemplates: true, + }) + require.NoError(t, err) + require.Len(t, parsed, 1) + + tm, err := models.ConvertTemplate(&parsed[0], models.UserAPISource) + require.NoError(t, err) + + return tm +} + +// TestCreateRuleRegistersOverridableRule covers the registry lifecycle: a rule with an +// overridable parameter gets a PMM-minted ID, that ID reaches Grafana as a label and the +// database as a row, and the snapshot it stores is what later resolves the threshold. +func TestCreateRuleRegistersOverridableRule(t *testing.T) { + ctx := t.Context() + sqlDB := testdb.Open(t, models.SkipFixtures, nil) + t.Cleanup(func() { require.NoError(t, sqlDB.Close()) }) + db := reform.NewDB(sqlDB, postgresql.Dialect, reform.NewPrintfLogger(t.Logf)) + + tm := templateFromYAML(t, overridableWiringYAML) + plain := templateFromYAML(t, multiExpressionWiringYAML) + + setup := func(t *testing.T) (*Service, *mockGrafanaClient) { + t.Helper() + + m := newMockGrafanaClient(t) + svc, err := NewService(db, m) + require.NoError(t, err) + svc.templates = map[string]models.Template{ + tm.Name: *tm, + plain.Name: *plain, + } + + return svc, m + } + + thresholdParam := []*alerting.ParamValue{{ + Name: "threshold", + Type: alerting.ParamType_PARAM_TYPE_FLOAT, + Value: &alerting.ParamValue_Float{Float: 80}, + }} + + createRule := func(t *testing.T, svc *Service, templateName string) error { + t.Helper() + + _, err := svc.CreateRule(ctx, &alerting.CreateRuleRequest{ + TemplateName: templateName, + Name: "test-rule", + FolderUid: "folder-uid", + Group: "test-group", + Severity: managementv1.Severity_SEVERITY_WARNING, + Params: thresholdParam, + }) + + return err + } + + t.Run("stamps the identity label and injects the threshold step", func(t *testing.T) { + svc, m := setup(t) + m.On("GetDatasourceUIDByName", mock.Anything, "Metrics").Return("metrics-uid", nil) + + var captured *services.Rule + m.On("CreateAlertRule", mock.Anything, "folder-uid", "test-group", mock.Anything, mock.Anything). + Run(func(args mock.Arguments) { captured = args.Get(4).(*services.Rule) }). //nolint:forcetypeassert + Return(nil) + + res, err := svc.CreateRule(ctx, &alerting.CreateRuleRequest{ + TemplateName: tm.Name, + Name: "test-rule", + FolderUid: "folder-uid", + Group: "test-group", + Severity: managementv1.Severity_SEVERITY_WARNING, + Params: thresholdParam, + }) + require.NoError(t, err) + require.NotNil(t, captured) + + ruleID := captured.Labels["pmm_rule_id"] + require.NotEmpty(t, ruleID, "an overridable rule must carry its PMM identity") + assert.Equal(t, ruleID, res.RuleId, "the response must return the same identity the rule carries") + + byRef := dataByRefID(t, captured.GrafanaAlert.Data) + require.Contains(t, byRef, "T_threshold") + assert.Contains(t, exprOf(t, byRef["T_threshold"]), `rule_id="`+ruleID+`"`, + "the injected query must select the same rule ID the label carries") + + // The registry row exists and holds the snapshot the resolver will need. + rule, err := models.FindAlertRuleByID(db.Querier, ruleID) + require.NoError(t, err) + + param, ok := rule.Params["threshold"] + require.True(t, ok) + assert.InDelta(t, 80.0, param.Default, 0.0001) + assert.Equal(t, "node_name", param.JoinLabel) + assert.Equal(t, []string{"node"}, param.Scopes) + assert.Equal(t, "%", param.Unit) + require.NotNil(t, param.Min) + require.NotNil(t, param.Max) + assert.InDelta(t, 0.0, *param.Min, 0.0001) + assert.InDelta(t, 100.0, *param.Max, 0.0001) + }) + + t.Run("the stored default is the value supplied, not the template's", func(t *testing.T) { + svc, m := setup(t) + m.On("GetDatasourceUIDByName", mock.Anything, "Metrics").Return("metrics-uid", nil) + + var captured *services.Rule + m.On("CreateAlertRule", mock.Anything, "folder-uid", "test-group", mock.Anything, mock.Anything). + Run(func(args mock.Arguments) { captured = args.Get(4).(*services.Rule) }). //nolint:forcetypeassert + Return(nil) + + _, err := svc.CreateRule(ctx, &alerting.CreateRuleRequest{ + TemplateName: tm.Name, + Name: "test-rule", + FolderUid: "folder-uid", + Group: "test-group", + Severity: managementv1.Severity_SEVERITY_WARNING, + Params: []*alerting.ParamValue{{ + Name: "threshold", + Type: alerting.ParamType_PARAM_TYPE_FLOAT, + Value: &alerting.ParamValue_Float{Float: 42}, + }}, + }) + require.NoError(t, err) + + rule, err := models.FindAlertRuleByID(db.Querier, captured.Labels["pmm_rule_id"]) + require.NoError(t, err) + assert.InDelta(t, 42.0, rule.Params["threshold"].Default, 0.0001) + }) + + t.Run("a rule with no overridable parameters is not registered", func(t *testing.T) { + svc, m := setup(t) + m.On("GetDatasourceUIDByName", mock.Anything, "Metrics").Return("metrics-uid", nil) + + var captured *services.Rule + m.On("CreateAlertRule", mock.Anything, "folder-uid", "test-group", mock.Anything, mock.Anything). + Run(func(args mock.Arguments) { captured = args.Get(4).(*services.Rule) }). //nolint:forcetypeassert + Return(nil) + + before, err := models.FindAlertRules(db.Querier) + require.NoError(t, err) + + require.NoError(t, createRule(t, svc, plain.Name)) + require.NotNil(t, captured) + + assert.Empty(t, captured.Labels["pmm_rule_id"]) + assert.NotContains(t, dataByRefID(t, captured.GrafanaAlert.Data), "T_threshold") + + after, err := models.FindAlertRules(db.Querier) + require.NoError(t, err) + assert.Len(t, after, len(before), "no registry row should have been written") + }) + + // TestCreateRule writes the registry row before creating the rule in Grafana, so a + // Grafana failure must not leave the row behind. The other ordering would be worse - + // a rule whose thresholds can never be overridden - but this one still has to clean up. + t.Run("a Grafana failure rolls the registry row back", func(t *testing.T) { + svc, m := setup(t) + m.On("GetDatasourceUIDByName", mock.Anything, "Metrics").Return("metrics-uid", nil) + m.On("CreateAlertRule", mock.Anything, "folder-uid", "test-group", mock.Anything, mock.Anything). + Return(errors.New("grafana rejected the rule")) + + before, err := models.FindAlertRules(db.Querier) + require.NoError(t, err) + + err = createRule(t, svc, tm.Name) + require.Error(t, err) + + after, err := models.FindAlertRules(db.Querier) + require.NoError(t, err) + assert.Len(t, after, len(before), "the registry row must not outlive the failed creation") + }) +} + +func TestCollectOverridableParams(t *testing.T) { + t.Parallel() + + template := overridableRuleTemplate() + + t.Run("snapshots the supplied value and derived join label", func(t *testing.T) { + t.Parallel() + + params, err := collectOverridableParams(template, AlertExprParamsValues{{ + Name: "threshold", + Type: models.Float, + FloatValue: 55, + }}) + require.NoError(t, err) + + require.Contains(t, params, "threshold") + assert.InDelta(t, 55.0, params["threshold"].Default, 0.0001) + assert.Equal(t, "node_name", params["threshold"].JoinLabel) + assert.Equal(t, []string{alert.OverrideScopeNode}, params["threshold"].Scopes) + }) + + t.Run("service and cluster scopes join on service_name", func(t *testing.T) { + t.Parallel() + + scoped := overridableRuleTemplate() + scoped.Params[0].OverrideScopes = []string{alert.OverrideScopeService, alert.OverrideScopeCluster} + + params, err := collectOverridableParams(scoped, AlertExprParamsValues{{ + Name: "threshold", + Type: models.Float, + FloatValue: 55, + }}) + require.NoError(t, err) + assert.Equal(t, "service_name", params["threshold"].JoinLabel) + }) + + t.Run("a missing value is rejected", func(t *testing.T) { + t.Parallel() + + _, err := collectOverridableParams(template, AlertExprParamsValues{}) + require.Error(t, err) + assert.Contains(t, err.Error(), "no value supplied") + }) + + t.Run("a template with no overridable parameters snapshots nothing", func(t *testing.T) { + t.Parallel() + + plain := overridableRuleTemplate() + plain.Params[0].Overridable = false + + params, err := collectOverridableParams(plain, AlertExprParamsValues{}) + require.NoError(t, err) + assert.Nil(t, params) + }) +} From 26931947fb921df3605aaf4035b0e78d42523de9 Mon Sep 17 00:00:00 2001 From: Matej Kubinec Date: Thu, 27 Aug 2026 17:10:21 +0200 Subject: [PATCH 05/15] PMM-14912 Add threshold override API Expose per-target threshold overrides so a threshold can be retuned without touching the rule. Four RPCs under /v1/alerting/thresholds: list, set, clear, and a batch verb applying several changes at once. The routes are generic rather than node-centric - scope and target travel as fields, not path segments. HTTP routes are additive-only, so shipping node-centric paths now would mean carrying a second route family forever once service and cluster scope land. A target cannot be a path segment in any case: a cluster target is an arbitrary label value, and one containing a slash would break gateway path matching outright. Threshold APIs require admin, matching Inventory where they are managed. Without an explicit rule they would inherit viewer from the "/v1/alerting" prefix, which would both let a read-only user retune every alert and expose thresholds outside the screen that gates them. This is a path rule rather than a method rule, so reads, writes and the batch verb are covered alike - prefix resolution stops at the colon, which the tests assert rather than assume. Batch updates apply in one transaction. A client editing several rows at once otherwise has no way to report which ones took effect after a partial failure, which is the whole reason the endpoint exists. The API needs to know which override is winning, not only its value, and deriving that in the service would have made precedence exist in two places. ResolveThresholds now delegates to ResolveThresholdsDetailed, which reports the winning row alongside the value, so what this API says and what the collector emits cannot drift. Two behaviours worth knowing rather than discovering: - Service and cluster scope return Unimplemented, not InvalidArgument. Both are already carried by the schema, the resolver and the proto, so enabling them is a validation change rather than an API change. - ListThresholds reports every overridable parameter when given a target, including untouched ones, and only actual overrides when not. Without a target there is no bounded set of targets to enumerate. Values are validated at the API as well as by the column's CHECK: a non-finite or out-of-range threshold reaching the database would surface as an opaque internal error rather than a bad request. Co-Authored-By: Claude Opus 5 (1M context) --- api/alerting/v1/alerting.pb.go | 912 +++++++++++- api/alerting/v1/alerting.pb.gw.go | 300 +++- api/alerting/v1/alerting.pb.validate.go | 1304 +++++++++++++++++ api/alerting/v1/alerting.proto | 116 ++ api/alerting/v1/alerting_grpc.pb.go | 176 ++- .../alerting_service_client.go | 176 +++ .../batch_update_thresholds_parameters.go | 141 ++ .../batch_update_thresholds_responses.go | 929 ++++++++++++ .../clear_threshold_parameters.go | 261 ++++ .../clear_threshold_responses.go | 411 ++++++ .../list_thresholds_parameters.go | 241 +++ .../list_thresholds_responses.go | 705 +++++++++ .../set_threshold_parameters.go | 141 ++ .../set_threshold_responses.go | 808 ++++++++++ api/alerting/v1/json/v1.json | 555 +++++++ api/swagger/swagger-dev.json | 555 +++++++ api/swagger/swagger.json | 555 +++++++ managed/models/threshold_resolver.go | 37 +- .../services/alerting/threshold_overrides.go | 511 +++++++ .../alerting/threshold_overrides_test.go | 317 ++++ managed/services/grafana/auth_server.go | 1 + managed/services/grafana/auth_server_test.go | 14 +- 22 files changed, 9071 insertions(+), 95 deletions(-) create mode 100644 api/alerting/v1/json/client/alerting_service/batch_update_thresholds_parameters.go create mode 100644 api/alerting/v1/json/client/alerting_service/batch_update_thresholds_responses.go create mode 100644 api/alerting/v1/json/client/alerting_service/clear_threshold_parameters.go create mode 100644 api/alerting/v1/json/client/alerting_service/clear_threshold_responses.go create mode 100644 api/alerting/v1/json/client/alerting_service/list_thresholds_parameters.go create mode 100644 api/alerting/v1/json/client/alerting_service/list_thresholds_responses.go create mode 100644 api/alerting/v1/json/client/alerting_service/set_threshold_parameters.go create mode 100644 api/alerting/v1/json/client/alerting_service/set_threshold_responses.go create mode 100644 managed/services/alerting/threshold_overrides.go create mode 100644 managed/services/alerting/threshold_overrides_test.go diff --git a/api/alerting/v1/alerting.pb.go b/api/alerting/v1/alerting.pb.go index 739f03f177..c4b9a149dc 100644 --- a/api/alerting/v1/alerting.pb.go +++ b/api/alerting/v1/alerting.pb.go @@ -138,6 +138,63 @@ func (FilterType) EnumDescriptor() ([]byte, []int) { return file_alerting_v1_alerting_proto_rawDescGZIP(), []int{1} } +// ThresholdScope says what a threshold override's target refers to. +type ThresholdScope int32 + +const ( + ThresholdScope_THRESHOLD_SCOPE_UNSPECIFIED ThresholdScope = 0 + // Target is a Node ID. + ThresholdScope_THRESHOLD_SCOPE_NODE ThresholdScope = 1 + // Target is a Service ID. + ThresholdScope_THRESHOLD_SCOPE_SERVICE ThresholdScope = 2 + // Target is a cluster label value. Unlike the others it names no inventory entity, + // so it cannot be validated for existence and is never removed by entity deletion. + ThresholdScope_THRESHOLD_SCOPE_CLUSTER ThresholdScope = 3 +) + +// Enum value maps for ThresholdScope. +var ( + ThresholdScope_name = map[int32]string{ + 0: "THRESHOLD_SCOPE_UNSPECIFIED", + 1: "THRESHOLD_SCOPE_NODE", + 2: "THRESHOLD_SCOPE_SERVICE", + 3: "THRESHOLD_SCOPE_CLUSTER", + } + ThresholdScope_value = map[string]int32{ + "THRESHOLD_SCOPE_UNSPECIFIED": 0, + "THRESHOLD_SCOPE_NODE": 1, + "THRESHOLD_SCOPE_SERVICE": 2, + "THRESHOLD_SCOPE_CLUSTER": 3, + } +) + +func (x ThresholdScope) Enum() *ThresholdScope { + p := new(ThresholdScope) + *p = x + return p +} + +func (x ThresholdScope) String() string { + return protoimpl.X.EnumStringOf(x.Descriptor(), protoreflect.EnumNumber(x)) +} + +func (ThresholdScope) Descriptor() protoreflect.EnumDescriptor { + return file_alerting_v1_alerting_proto_enumTypes[2].Descriptor() +} + +func (ThresholdScope) Type() protoreflect.EnumType { + return &file_alerting_v1_alerting_proto_enumTypes[2] +} + +func (x ThresholdScope) Number() protoreflect.EnumNumber { + return protoreflect.EnumNumber(x) +} + +// Deprecated: Use ThresholdScope.Descriptor instead. +func (ThresholdScope) EnumDescriptor() ([]byte, []int) { + return file_alerting_v1_alerting_proto_rawDescGZIP(), []int{2} +} + // BoolParamDefinition represents boolean parameter's default value. type BoolParamDefinition struct { state protoimpl.MessageState `protogen:"open.v1"` @@ -1449,6 +1506,629 @@ func (x *CreateRuleResponse) GetRuleId() string { return "" } +// Threshold is one overridable parameter of one rule, as it applies to one target. +type Threshold struct { + state protoimpl.MessageState `protogen:"open.v1"` + // Identifier PMM assigned to the rule. Not unique within a response: rules duplicated + // in Grafana share it, so two entries can carry the same rule_id and param_name and + // differ only in which rule they came from. Do not key a map on it. + RuleId string `protobuf:"bytes,1,opt,name=rule_id,json=ruleId,proto3" json:"rule_id,omitempty"` + // Machine-readable name of the overridable parameter. + ParamName string `protobuf:"bytes,2,opt,name=param_name,json=paramName,proto3" json:"param_name,omitempty"` + // Short human-readable parameter summary, as it was when the rule was created. + Summary string `protobuf:"bytes,3,opt,name=summary,proto3" json:"summary,omitempty"` + // Parameter unit. + Unit ParamUnit `protobuf:"varint,4,opt,name=unit,proto3,enum=alerting.v1.ParamUnit" json:"unit,omitempty"` + // Value the rule falls back to when no override applies. + DefaultValue float64 `protobuf:"fixed64,5,opt,name=default_value,json=defaultValue,proto3" json:"default_value,omitempty"` + // Value the rule currently evaluates this target against. + EffectiveValue float64 `protobuf:"fixed64,6,opt,name=effective_value,json=effectiveValue,proto3" json:"effective_value,omitempty"` + // Whether effective_value comes from an override rather than the default. + IsOverridden bool `protobuf:"varint,7,opt,name=is_overridden,json=isOverridden,proto3" json:"is_overridden,omitempty"` + // Scope the effective override was set at. Unspecified when not overridden. + Scope ThresholdScope `protobuf:"varint,8,opt,name=scope,proto3,enum=alerting.v1.ThresholdScope" json:"scope,omitempty"` + // Target the effective override was set on. Empty when not overridden. + Target string `protobuf:"bytes,9,opt,name=target,proto3" json:"target,omitempty"` + unknownFields protoimpl.UnknownFields + sizeCache protoimpl.SizeCache +} + +func (x *Threshold) Reset() { + *x = Threshold{} + mi := &file_alerting_v1_alerting_proto_msgTypes[19] + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + ms.StoreMessageInfo(mi) +} + +func (x *Threshold) String() string { + return protoimpl.X.MessageStringOf(x) +} + +func (*Threshold) ProtoMessage() {} + +func (x *Threshold) ProtoReflect() protoreflect.Message { + mi := &file_alerting_v1_alerting_proto_msgTypes[19] + if x != nil { + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + if ms.LoadMessageInfo() == nil { + ms.StoreMessageInfo(mi) + } + return ms + } + return mi.MessageOf(x) +} + +// Deprecated: Use Threshold.ProtoReflect.Descriptor instead. +func (*Threshold) Descriptor() ([]byte, []int) { + return file_alerting_v1_alerting_proto_rawDescGZIP(), []int{19} +} + +func (x *Threshold) GetRuleId() string { + if x != nil { + return x.RuleId + } + return "" +} + +func (x *Threshold) GetParamName() string { + if x != nil { + return x.ParamName + } + return "" +} + +func (x *Threshold) GetSummary() string { + if x != nil { + return x.Summary + } + return "" +} + +func (x *Threshold) GetUnit() ParamUnit { + if x != nil { + return x.Unit + } + return ParamUnit_PARAM_UNIT_UNSPECIFIED +} + +func (x *Threshold) GetDefaultValue() float64 { + if x != nil { + return x.DefaultValue + } + return 0 +} + +func (x *Threshold) GetEffectiveValue() float64 { + if x != nil { + return x.EffectiveValue + } + return 0 +} + +func (x *Threshold) GetIsOverridden() bool { + if x != nil { + return x.IsOverridden + } + return false +} + +func (x *Threshold) GetScope() ThresholdScope { + if x != nil { + return x.Scope + } + return ThresholdScope_THRESHOLD_SCOPE_UNSPECIFIED +} + +func (x *Threshold) GetTarget() string { + if x != nil { + return x.Target + } + return "" +} + +type ListThresholdsRequest struct { + state protoimpl.MessageState `protogen:"open.v1"` + // Scope of the target to report thresholds for. Must be set together with target. + Scope ThresholdScope `protobuf:"varint,1,opt,name=scope,proto3,enum=alerting.v1.ThresholdScope" json:"scope,omitempty"` + // Target to report thresholds for. When set, every overridable parameter is returned + // for that target, overridden or not. When empty, only existing overrides are + // returned, since there is otherwise no bounded set to enumerate. + Target string `protobuf:"bytes,2,opt,name=target,proto3" json:"target,omitempty"` + // Return only thresholds of this rule. + RuleId string `protobuf:"bytes,3,opt,name=rule_id,json=ruleId,proto3" json:"rule_id,omitempty"` + unknownFields protoimpl.UnknownFields + sizeCache protoimpl.SizeCache +} + +func (x *ListThresholdsRequest) Reset() { + *x = ListThresholdsRequest{} + mi := &file_alerting_v1_alerting_proto_msgTypes[20] + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + ms.StoreMessageInfo(mi) +} + +func (x *ListThresholdsRequest) String() string { + return protoimpl.X.MessageStringOf(x) +} + +func (*ListThresholdsRequest) ProtoMessage() {} + +func (x *ListThresholdsRequest) ProtoReflect() protoreflect.Message { + mi := &file_alerting_v1_alerting_proto_msgTypes[20] + if x != nil { + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + if ms.LoadMessageInfo() == nil { + ms.StoreMessageInfo(mi) + } + return ms + } + return mi.MessageOf(x) +} + +// Deprecated: Use ListThresholdsRequest.ProtoReflect.Descriptor instead. +func (*ListThresholdsRequest) Descriptor() ([]byte, []int) { + return file_alerting_v1_alerting_proto_rawDescGZIP(), []int{20} +} + +func (x *ListThresholdsRequest) GetScope() ThresholdScope { + if x != nil { + return x.Scope + } + return ThresholdScope_THRESHOLD_SCOPE_UNSPECIFIED +} + +func (x *ListThresholdsRequest) GetTarget() string { + if x != nil { + return x.Target + } + return "" +} + +func (x *ListThresholdsRequest) GetRuleId() string { + if x != nil { + return x.RuleId + } + return "" +} + +type ListThresholdsResponse struct { + state protoimpl.MessageState `protogen:"open.v1"` + Thresholds []*Threshold `protobuf:"bytes,1,rep,name=thresholds,proto3" json:"thresholds,omitempty"` + unknownFields protoimpl.UnknownFields + sizeCache protoimpl.SizeCache +} + +func (x *ListThresholdsResponse) Reset() { + *x = ListThresholdsResponse{} + mi := &file_alerting_v1_alerting_proto_msgTypes[21] + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + ms.StoreMessageInfo(mi) +} + +func (x *ListThresholdsResponse) String() string { + return protoimpl.X.MessageStringOf(x) +} + +func (*ListThresholdsResponse) ProtoMessage() {} + +func (x *ListThresholdsResponse) ProtoReflect() protoreflect.Message { + mi := &file_alerting_v1_alerting_proto_msgTypes[21] + if x != nil { + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + if ms.LoadMessageInfo() == nil { + ms.StoreMessageInfo(mi) + } + return ms + } + return mi.MessageOf(x) +} + +// Deprecated: Use ListThresholdsResponse.ProtoReflect.Descriptor instead. +func (*ListThresholdsResponse) Descriptor() ([]byte, []int) { + return file_alerting_v1_alerting_proto_rawDescGZIP(), []int{21} +} + +func (x *ListThresholdsResponse) GetThresholds() []*Threshold { + if x != nil { + return x.Thresholds + } + return nil +} + +type SetThresholdRequest struct { + state protoimpl.MessageState `protogen:"open.v1"` + Scope ThresholdScope `protobuf:"varint,1,opt,name=scope,proto3,enum=alerting.v1.ThresholdScope" json:"scope,omitempty"` + Target string `protobuf:"bytes,2,opt,name=target,proto3" json:"target,omitempty"` + RuleId string `protobuf:"bytes,3,opt,name=rule_id,json=ruleId,proto3" json:"rule_id,omitempty"` + ParamName string `protobuf:"bytes,4,opt,name=param_name,json=paramName,proto3" json:"param_name,omitempty"` + // Must be finite and within the parameter's declared range. + Value float64 `protobuf:"fixed64,5,opt,name=value,proto3" json:"value,omitempty"` + unknownFields protoimpl.UnknownFields + sizeCache protoimpl.SizeCache +} + +func (x *SetThresholdRequest) Reset() { + *x = SetThresholdRequest{} + mi := &file_alerting_v1_alerting_proto_msgTypes[22] + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + ms.StoreMessageInfo(mi) +} + +func (x *SetThresholdRequest) String() string { + return protoimpl.X.MessageStringOf(x) +} + +func (*SetThresholdRequest) ProtoMessage() {} + +func (x *SetThresholdRequest) ProtoReflect() protoreflect.Message { + mi := &file_alerting_v1_alerting_proto_msgTypes[22] + if x != nil { + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + if ms.LoadMessageInfo() == nil { + ms.StoreMessageInfo(mi) + } + return ms + } + return mi.MessageOf(x) +} + +// Deprecated: Use SetThresholdRequest.ProtoReflect.Descriptor instead. +func (*SetThresholdRequest) Descriptor() ([]byte, []int) { + return file_alerting_v1_alerting_proto_rawDescGZIP(), []int{22} +} + +func (x *SetThresholdRequest) GetScope() ThresholdScope { + if x != nil { + return x.Scope + } + return ThresholdScope_THRESHOLD_SCOPE_UNSPECIFIED +} + +func (x *SetThresholdRequest) GetTarget() string { + if x != nil { + return x.Target + } + return "" +} + +func (x *SetThresholdRequest) GetRuleId() string { + if x != nil { + return x.RuleId + } + return "" +} + +func (x *SetThresholdRequest) GetParamName() string { + if x != nil { + return x.ParamName + } + return "" +} + +func (x *SetThresholdRequest) GetValue() float64 { + if x != nil { + return x.Value + } + return 0 +} + +type SetThresholdResponse struct { + state protoimpl.MessageState `protogen:"open.v1"` + Threshold *Threshold `protobuf:"bytes,1,opt,name=threshold,proto3" json:"threshold,omitempty"` + unknownFields protoimpl.UnknownFields + sizeCache protoimpl.SizeCache +} + +func (x *SetThresholdResponse) Reset() { + *x = SetThresholdResponse{} + mi := &file_alerting_v1_alerting_proto_msgTypes[23] + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + ms.StoreMessageInfo(mi) +} + +func (x *SetThresholdResponse) String() string { + return protoimpl.X.MessageStringOf(x) +} + +func (*SetThresholdResponse) ProtoMessage() {} + +func (x *SetThresholdResponse) ProtoReflect() protoreflect.Message { + mi := &file_alerting_v1_alerting_proto_msgTypes[23] + if x != nil { + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + if ms.LoadMessageInfo() == nil { + ms.StoreMessageInfo(mi) + } + return ms + } + return mi.MessageOf(x) +} + +// Deprecated: Use SetThresholdResponse.ProtoReflect.Descriptor instead. +func (*SetThresholdResponse) Descriptor() ([]byte, []int) { + return file_alerting_v1_alerting_proto_rawDescGZIP(), []int{23} +} + +func (x *SetThresholdResponse) GetThreshold() *Threshold { + if x != nil { + return x.Threshold + } + return nil +} + +type ClearThresholdRequest struct { + state protoimpl.MessageState `protogen:"open.v1"` + Scope ThresholdScope `protobuf:"varint,1,opt,name=scope,proto3,enum=alerting.v1.ThresholdScope" json:"scope,omitempty"` + Target string `protobuf:"bytes,2,opt,name=target,proto3" json:"target,omitempty"` + RuleId string `protobuf:"bytes,3,opt,name=rule_id,json=ruleId,proto3" json:"rule_id,omitempty"` + ParamName string `protobuf:"bytes,4,opt,name=param_name,json=paramName,proto3" json:"param_name,omitempty"` + unknownFields protoimpl.UnknownFields + sizeCache protoimpl.SizeCache +} + +func (x *ClearThresholdRequest) Reset() { + *x = ClearThresholdRequest{} + mi := &file_alerting_v1_alerting_proto_msgTypes[24] + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + ms.StoreMessageInfo(mi) +} + +func (x *ClearThresholdRequest) String() string { + return protoimpl.X.MessageStringOf(x) +} + +func (*ClearThresholdRequest) ProtoMessage() {} + +func (x *ClearThresholdRequest) ProtoReflect() protoreflect.Message { + mi := &file_alerting_v1_alerting_proto_msgTypes[24] + if x != nil { + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + if ms.LoadMessageInfo() == nil { + ms.StoreMessageInfo(mi) + } + return ms + } + return mi.MessageOf(x) +} + +// Deprecated: Use ClearThresholdRequest.ProtoReflect.Descriptor instead. +func (*ClearThresholdRequest) Descriptor() ([]byte, []int) { + return file_alerting_v1_alerting_proto_rawDescGZIP(), []int{24} +} + +func (x *ClearThresholdRequest) GetScope() ThresholdScope { + if x != nil { + return x.Scope + } + return ThresholdScope_THRESHOLD_SCOPE_UNSPECIFIED +} + +func (x *ClearThresholdRequest) GetTarget() string { + if x != nil { + return x.Target + } + return "" +} + +func (x *ClearThresholdRequest) GetRuleId() string { + if x != nil { + return x.RuleId + } + return "" +} + +func (x *ClearThresholdRequest) GetParamName() string { + if x != nil { + return x.ParamName + } + return "" +} + +type ClearThresholdResponse struct { + state protoimpl.MessageState `protogen:"open.v1"` + unknownFields protoimpl.UnknownFields + sizeCache protoimpl.SizeCache +} + +func (x *ClearThresholdResponse) Reset() { + *x = ClearThresholdResponse{} + mi := &file_alerting_v1_alerting_proto_msgTypes[25] + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + ms.StoreMessageInfo(mi) +} + +func (x *ClearThresholdResponse) String() string { + return protoimpl.X.MessageStringOf(x) +} + +func (*ClearThresholdResponse) ProtoMessage() {} + +func (x *ClearThresholdResponse) ProtoReflect() protoreflect.Message { + mi := &file_alerting_v1_alerting_proto_msgTypes[25] + if x != nil { + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + if ms.LoadMessageInfo() == nil { + ms.StoreMessageInfo(mi) + } + return ms + } + return mi.MessageOf(x) +} + +// Deprecated: Use ClearThresholdResponse.ProtoReflect.Descriptor instead. +func (*ClearThresholdResponse) Descriptor() ([]byte, []int) { + return file_alerting_v1_alerting_proto_rawDescGZIP(), []int{25} +} + +// ThresholdUpdate sets or clears one override. +type ThresholdUpdate struct { + state protoimpl.MessageState `protogen:"open.v1"` + Scope ThresholdScope `protobuf:"varint,1,opt,name=scope,proto3,enum=alerting.v1.ThresholdScope" json:"scope,omitempty"` + Target string `protobuf:"bytes,2,opt,name=target,proto3" json:"target,omitempty"` + RuleId string `protobuf:"bytes,3,opt,name=rule_id,json=ruleId,proto3" json:"rule_id,omitempty"` + ParamName string `protobuf:"bytes,4,opt,name=param_name,json=paramName,proto3" json:"param_name,omitempty"` + // Omit to clear the override rather than set it. + Value *float64 `protobuf:"fixed64,5,opt,name=value,proto3,oneof" json:"value,omitempty"` + unknownFields protoimpl.UnknownFields + sizeCache protoimpl.SizeCache +} + +func (x *ThresholdUpdate) Reset() { + *x = ThresholdUpdate{} + mi := &file_alerting_v1_alerting_proto_msgTypes[26] + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + ms.StoreMessageInfo(mi) +} + +func (x *ThresholdUpdate) String() string { + return protoimpl.X.MessageStringOf(x) +} + +func (*ThresholdUpdate) ProtoMessage() {} + +func (x *ThresholdUpdate) ProtoReflect() protoreflect.Message { + mi := &file_alerting_v1_alerting_proto_msgTypes[26] + if x != nil { + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + if ms.LoadMessageInfo() == nil { + ms.StoreMessageInfo(mi) + } + return ms + } + return mi.MessageOf(x) +} + +// Deprecated: Use ThresholdUpdate.ProtoReflect.Descriptor instead. +func (*ThresholdUpdate) Descriptor() ([]byte, []int) { + return file_alerting_v1_alerting_proto_rawDescGZIP(), []int{26} +} + +func (x *ThresholdUpdate) GetScope() ThresholdScope { + if x != nil { + return x.Scope + } + return ThresholdScope_THRESHOLD_SCOPE_UNSPECIFIED +} + +func (x *ThresholdUpdate) GetTarget() string { + if x != nil { + return x.Target + } + return "" +} + +func (x *ThresholdUpdate) GetRuleId() string { + if x != nil { + return x.RuleId + } + return "" +} + +func (x *ThresholdUpdate) GetParamName() string { + if x != nil { + return x.ParamName + } + return "" +} + +func (x *ThresholdUpdate) GetValue() float64 { + if x != nil && x.Value != nil { + return *x.Value + } + return 0 +} + +type BatchUpdateThresholdsRequest struct { + state protoimpl.MessageState `protogen:"open.v1"` + // Applied in one transaction: either every update lands or none does. A client + // editing several rows at once cannot otherwise report which ones took effect. + Updates []*ThresholdUpdate `protobuf:"bytes,1,rep,name=updates,proto3" json:"updates,omitempty"` + unknownFields protoimpl.UnknownFields + sizeCache protoimpl.SizeCache +} + +func (x *BatchUpdateThresholdsRequest) Reset() { + *x = BatchUpdateThresholdsRequest{} + mi := &file_alerting_v1_alerting_proto_msgTypes[27] + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + ms.StoreMessageInfo(mi) +} + +func (x *BatchUpdateThresholdsRequest) String() string { + return protoimpl.X.MessageStringOf(x) +} + +func (*BatchUpdateThresholdsRequest) ProtoMessage() {} + +func (x *BatchUpdateThresholdsRequest) ProtoReflect() protoreflect.Message { + mi := &file_alerting_v1_alerting_proto_msgTypes[27] + if x != nil { + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + if ms.LoadMessageInfo() == nil { + ms.StoreMessageInfo(mi) + } + return ms + } + return mi.MessageOf(x) +} + +// Deprecated: Use BatchUpdateThresholdsRequest.ProtoReflect.Descriptor instead. +func (*BatchUpdateThresholdsRequest) Descriptor() ([]byte, []int) { + return file_alerting_v1_alerting_proto_rawDescGZIP(), []int{27} +} + +func (x *BatchUpdateThresholdsRequest) GetUpdates() []*ThresholdUpdate { + if x != nil { + return x.Updates + } + return nil +} + +type BatchUpdateThresholdsResponse struct { + state protoimpl.MessageState `protogen:"open.v1"` + // Thresholds that were set, in request order. Cleared ones are omitted. + Thresholds []*Threshold `protobuf:"bytes,1,rep,name=thresholds,proto3" json:"thresholds,omitempty"` + unknownFields protoimpl.UnknownFields + sizeCache protoimpl.SizeCache +} + +func (x *BatchUpdateThresholdsResponse) Reset() { + *x = BatchUpdateThresholdsResponse{} + mi := &file_alerting_v1_alerting_proto_msgTypes[28] + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + ms.StoreMessageInfo(mi) +} + +func (x *BatchUpdateThresholdsResponse) String() string { + return protoimpl.X.MessageStringOf(x) +} + +func (*BatchUpdateThresholdsResponse) ProtoMessage() {} + +func (x *BatchUpdateThresholdsResponse) ProtoReflect() protoreflect.Message { + mi := &file_alerting_v1_alerting_proto_msgTypes[28] + if x != nil { + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + if ms.LoadMessageInfo() == nil { + ms.StoreMessageInfo(mi) + } + return ms + } + return mi.MessageOf(x) +} + +// Deprecated: Use BatchUpdateThresholdsResponse.ProtoReflect.Descriptor instead. +func (*BatchUpdateThresholdsResponse) Descriptor() ([]byte, []int) { + return file_alerting_v1_alerting_proto_rawDescGZIP(), []int{28} +} + +func (x *BatchUpdateThresholdsResponse) GetThresholds() []*Threshold { + if x != nil { + return x.Thresholds + } + return nil +} + var File_alerting_v1_alerting_proto protoreflect.FileDescriptor const file_alerting_v1_alerting_proto_rawDesc = "" + @@ -1564,7 +2244,56 @@ const file_alerting_v1_alerting_proto_rawDesc = "" + "\x03key\x18\x01 \x01(\tR\x03key\x12\x14\n" + "\x05value\x18\x02 \x01(\tR\x05value:\x028\x01\"-\n" + "\x12CreateRuleResponse\x12\x17\n" + - "\arule_id\x18\x01 \x01(\tR\x06ruleId*\xa6\x01\n" + + "\arule_id\x18\x01 \x01(\tR\x06ruleId\"\xc7\x02\n" + + "\tThreshold\x12\x17\n" + + "\arule_id\x18\x01 \x01(\tR\x06ruleId\x12\x1d\n" + + "\n" + + "param_name\x18\x02 \x01(\tR\tparamName\x12\x18\n" + + "\asummary\x18\x03 \x01(\tR\asummary\x12*\n" + + "\x04unit\x18\x04 \x01(\x0e2\x16.alerting.v1.ParamUnitR\x04unit\x12#\n" + + "\rdefault_value\x18\x05 \x01(\x01R\fdefaultValue\x12'\n" + + "\x0feffective_value\x18\x06 \x01(\x01R\x0eeffectiveValue\x12#\n" + + "\ris_overridden\x18\a \x01(\bR\fisOverridden\x121\n" + + "\x05scope\x18\b \x01(\x0e2\x1b.alerting.v1.ThresholdScopeR\x05scope\x12\x16\n" + + "\x06target\x18\t \x01(\tR\x06target\"{\n" + + "\x15ListThresholdsRequest\x121\n" + + "\x05scope\x18\x01 \x01(\x0e2\x1b.alerting.v1.ThresholdScopeR\x05scope\x12\x16\n" + + "\x06target\x18\x02 \x01(\tR\x06target\x12\x17\n" + + "\arule_id\x18\x03 \x01(\tR\x06ruleId\"P\n" + + "\x16ListThresholdsResponse\x126\n" + + "\n" + + "thresholds\x18\x01 \x03(\v2\x16.alerting.v1.ThresholdR\n" + + "thresholds\"\xc9\x01\n" + + "\x13SetThresholdRequest\x121\n" + + "\x05scope\x18\x01 \x01(\x0e2\x1b.alerting.v1.ThresholdScopeR\x05scope\x12\x1f\n" + + "\x06target\x18\x02 \x01(\tB\a\xfaB\x04r\x02\x10\x01R\x06target\x12 \n" + + "\arule_id\x18\x03 \x01(\tB\a\xfaB\x04r\x02\x10\x01R\x06ruleId\x12&\n" + + "\n" + + "param_name\x18\x04 \x01(\tB\a\xfaB\x04r\x02\x10\x01R\tparamName\x12\x14\n" + + "\x05value\x18\x05 \x01(\x01R\x05value\"L\n" + + "\x14SetThresholdResponse\x124\n" + + "\tthreshold\x18\x01 \x01(\v2\x16.alerting.v1.ThresholdR\tthreshold\"\xb5\x01\n" + + "\x15ClearThresholdRequest\x121\n" + + "\x05scope\x18\x01 \x01(\x0e2\x1b.alerting.v1.ThresholdScopeR\x05scope\x12\x1f\n" + + "\x06target\x18\x02 \x01(\tB\a\xfaB\x04r\x02\x10\x01R\x06target\x12 \n" + + "\arule_id\x18\x03 \x01(\tB\a\xfaB\x04r\x02\x10\x01R\x06ruleId\x12&\n" + + "\n" + + "param_name\x18\x04 \x01(\tB\a\xfaB\x04r\x02\x10\x01R\tparamName\"\x18\n" + + "\x16ClearThresholdResponse\"\xd4\x01\n" + + "\x0fThresholdUpdate\x121\n" + + "\x05scope\x18\x01 \x01(\x0e2\x1b.alerting.v1.ThresholdScopeR\x05scope\x12\x1f\n" + + "\x06target\x18\x02 \x01(\tB\a\xfaB\x04r\x02\x10\x01R\x06target\x12 \n" + + "\arule_id\x18\x03 \x01(\tB\a\xfaB\x04r\x02\x10\x01R\x06ruleId\x12&\n" + + "\n" + + "param_name\x18\x04 \x01(\tB\a\xfaB\x04r\x02\x10\x01R\tparamName\x12\x19\n" + + "\x05value\x18\x05 \x01(\x01H\x00R\x05value\x88\x01\x01B\b\n" + + "\x06_value\"`\n" + + "\x1cBatchUpdateThresholdsRequest\x12@\n" + + "\aupdates\x18\x01 \x03(\v2\x1c.alerting.v1.ThresholdUpdateB\b\xfaB\x05\x92\x01\x02\b\x01R\aupdates\"W\n" + + "\x1dBatchUpdateThresholdsResponse\x126\n" + + "\n" + + "thresholds\x18\x01 \x03(\v2\x16.alerting.v1.ThresholdR\n" + + "thresholds*\xa6\x01\n" + "\x0eTemplateSource\x12\x1f\n" + "\x1bTEMPLATE_SOURCE_UNSPECIFIED\x10\x00\x12\x1c\n" + "\x18TEMPLATE_SOURCE_BUILT_IN\x10\x01\x12\x18\n" + @@ -1575,14 +2304,23 @@ const file_alerting_v1_alerting_proto_rawDesc = "" + "FilterType\x12\x1b\n" + "\x17FILTER_TYPE_UNSPECIFIED\x10\x00\x12\x15\n" + "\x11FILTER_TYPE_MATCH\x10\x01\x12\x18\n" + - "\x14FILTER_TYPE_MISMATCH\x10\x022\xfe\x04\n" + + "\x14FILTER_TYPE_MISMATCH\x10\x02*\x85\x01\n" + + "\x0eThresholdScope\x12\x1f\n" + + "\x1bTHRESHOLD_SCOPE_UNSPECIFIED\x10\x00\x12\x18\n" + + "\x14THRESHOLD_SCOPE_NODE\x10\x01\x12\x1b\n" + + "\x17THRESHOLD_SCOPE_SERVICE\x10\x02\x12\x1b\n" + + "\x17THRESHOLD_SCOPE_CLUSTER\x10\x032\x90\t\n" + "\x0fAlertingService\x12v\n" + "\rListTemplates\x12!.alerting.v1.ListTemplatesRequest\x1a\".alerting.v1.ListTemplatesResponse\"\x1e\x82\xd3\xe4\x93\x02\x18\x12\x16/v1/alerting/templates\x12|\n" + "\x0eCreateTemplate\x12\".alerting.v1.CreateTemplateRequest\x1a#.alerting.v1.CreateTemplateResponse\"!\x82\xd3\xe4\x93\x02\x1b:\x01*\"\x16/v1/alerting/templates\x12\x83\x01\n" + "\x0eUpdateTemplate\x12\".alerting.v1.UpdateTemplateRequest\x1a#.alerting.v1.UpdateTemplateResponse\"(\x82\xd3\xe4\x93\x02\":\x01*\x1a\x1d/v1/alerting/templates/{name}\x12\x80\x01\n" + "\x0eDeleteTemplate\x12\".alerting.v1.DeleteTemplateRequest\x1a#.alerting.v1.DeleteTemplateResponse\"%\x82\xd3\xe4\x93\x02\x1f*\x1d/v1/alerting/templates/{name}\x12l\n" + "\n" + - "CreateRule\x12\x1e.alerting.v1.CreateRuleRequest\x1a\x1f.alerting.v1.CreateRuleResponse\"\x1d\x82\xd3\xe4\x93\x02\x17:\x01*\"\x12/v1/alerting/rulesB\xa0\x01\n" + + "CreateRule\x12\x1e.alerting.v1.CreateRuleRequest\x1a\x1f.alerting.v1.CreateRuleResponse\"\x1d\x82\xd3\xe4\x93\x02\x17:\x01*\"\x12/v1/alerting/rules\x12z\n" + + "\x0eListThresholds\x12\".alerting.v1.ListThresholdsRequest\x1a#.alerting.v1.ListThresholdsResponse\"\x1f\x82\xd3\xe4\x93\x02\x19\x12\x17/v1/alerting/thresholds\x12w\n" + + "\fSetThreshold\x12 .alerting.v1.SetThresholdRequest\x1a!.alerting.v1.SetThresholdResponse\"\"\x82\xd3\xe4\x93\x02\x1c:\x01*\"\x17/v1/alerting/thresholds\x12z\n" + + "\x0eClearThreshold\x12\".alerting.v1.ClearThresholdRequest\x1a#.alerting.v1.ClearThresholdResponse\"\x1f\x82\xd3\xe4\x93\x02\x19*\x17/v1/alerting/thresholds\x12\x9e\x01\n" + + "\x15BatchUpdateThresholds\x12).alerting.v1.BatchUpdateThresholdsRequest\x1a*.alerting.v1.BatchUpdateThresholdsResponse\".\x82\xd3\xe4\x93\x02(:\x01*\"#/v1/alerting/thresholds:batchUpdateB\xa0\x01\n" + "\x0fcom.alerting.v1B\rAlertingProtoP\x01Z1github.com/percona/pmm/api/alerting/v1;alertingv1\xa2\x02\x03AXX\xaa\x02\vAlerting.V1\xca\x02\vAlerting\\V1\xe2\x02\x17Alerting\\V1\\GPBMetadata\xea\x02\fAlerting::V1b\x06proto3" var ( @@ -1598,80 +2336,109 @@ func file_alerting_v1_alerting_proto_rawDescGZIP() []byte { } var ( - file_alerting_v1_alerting_proto_enumTypes = make([]protoimpl.EnumInfo, 2) - file_alerting_v1_alerting_proto_msgTypes = make([]protoimpl.MessageInfo, 22) + file_alerting_v1_alerting_proto_enumTypes = make([]protoimpl.EnumInfo, 3) + file_alerting_v1_alerting_proto_msgTypes = make([]protoimpl.MessageInfo, 32) file_alerting_v1_alerting_proto_goTypes = []any{ - TemplateSource(0), // 0: alerting.v1.TemplateSource - FilterType(0), // 1: alerting.v1.FilterType - (*BoolParamDefinition)(nil), // 2: alerting.v1.BoolParamDefinition - (*FloatParamDefinition)(nil), // 3: alerting.v1.FloatParamDefinition - (*StringParamDefinition)(nil), // 4: alerting.v1.StringParamDefinition - (*ParamDefinition)(nil), // 5: alerting.v1.ParamDefinition - (*TemplateQuery)(nil), // 6: alerting.v1.TemplateQuery - (*TemplateExpression)(nil), // 7: alerting.v1.TemplateExpression - (*Template)(nil), // 8: alerting.v1.Template - (*ListTemplatesRequest)(nil), // 9: alerting.v1.ListTemplatesRequest - (*ListTemplatesResponse)(nil), // 10: alerting.v1.ListTemplatesResponse - (*CreateTemplateRequest)(nil), // 11: alerting.v1.CreateTemplateRequest - (*CreateTemplateResponse)(nil), // 12: alerting.v1.CreateTemplateResponse - (*UpdateTemplateRequest)(nil), // 13: alerting.v1.UpdateTemplateRequest - (*UpdateTemplateResponse)(nil), // 14: alerting.v1.UpdateTemplateResponse - (*DeleteTemplateRequest)(nil), // 15: alerting.v1.DeleteTemplateRequest - (*DeleteTemplateResponse)(nil), // 16: alerting.v1.DeleteTemplateResponse - (*Filter)(nil), // 17: alerting.v1.Filter - (*ParamValue)(nil), // 18: alerting.v1.ParamValue - (*CreateRuleRequest)(nil), // 19: alerting.v1.CreateRuleRequest - (*CreateRuleResponse)(nil), // 20: alerting.v1.CreateRuleResponse - nil, // 21: alerting.v1.Template.LabelsEntry - nil, // 22: alerting.v1.Template.AnnotationsEntry - nil, // 23: alerting.v1.CreateRuleRequest.CustomLabelsEntry - ParamUnit(0), // 24: alerting.v1.ParamUnit - ParamType(0), // 25: alerting.v1.ParamType - (*durationpb.Duration)(nil), // 26: google.protobuf.Duration - v1.Severity(0), // 27: management.v1.Severity - (*timestamppb.Timestamp)(nil), // 28: google.protobuf.Timestamp + TemplateSource(0), // 0: alerting.v1.TemplateSource + FilterType(0), // 1: alerting.v1.FilterType + ThresholdScope(0), // 2: alerting.v1.ThresholdScope + (*BoolParamDefinition)(nil), // 3: alerting.v1.BoolParamDefinition + (*FloatParamDefinition)(nil), // 4: alerting.v1.FloatParamDefinition + (*StringParamDefinition)(nil), // 5: alerting.v1.StringParamDefinition + (*ParamDefinition)(nil), // 6: alerting.v1.ParamDefinition + (*TemplateQuery)(nil), // 7: alerting.v1.TemplateQuery + (*TemplateExpression)(nil), // 8: alerting.v1.TemplateExpression + (*Template)(nil), // 9: alerting.v1.Template + (*ListTemplatesRequest)(nil), // 10: alerting.v1.ListTemplatesRequest + (*ListTemplatesResponse)(nil), // 11: alerting.v1.ListTemplatesResponse + (*CreateTemplateRequest)(nil), // 12: alerting.v1.CreateTemplateRequest + (*CreateTemplateResponse)(nil), // 13: alerting.v1.CreateTemplateResponse + (*UpdateTemplateRequest)(nil), // 14: alerting.v1.UpdateTemplateRequest + (*UpdateTemplateResponse)(nil), // 15: alerting.v1.UpdateTemplateResponse + (*DeleteTemplateRequest)(nil), // 16: alerting.v1.DeleteTemplateRequest + (*DeleteTemplateResponse)(nil), // 17: alerting.v1.DeleteTemplateResponse + (*Filter)(nil), // 18: alerting.v1.Filter + (*ParamValue)(nil), // 19: alerting.v1.ParamValue + (*CreateRuleRequest)(nil), // 20: alerting.v1.CreateRuleRequest + (*CreateRuleResponse)(nil), // 21: alerting.v1.CreateRuleResponse + (*Threshold)(nil), // 22: alerting.v1.Threshold + (*ListThresholdsRequest)(nil), // 23: alerting.v1.ListThresholdsRequest + (*ListThresholdsResponse)(nil), // 24: alerting.v1.ListThresholdsResponse + (*SetThresholdRequest)(nil), // 25: alerting.v1.SetThresholdRequest + (*SetThresholdResponse)(nil), // 26: alerting.v1.SetThresholdResponse + (*ClearThresholdRequest)(nil), // 27: alerting.v1.ClearThresholdRequest + (*ClearThresholdResponse)(nil), // 28: alerting.v1.ClearThresholdResponse + (*ThresholdUpdate)(nil), // 29: alerting.v1.ThresholdUpdate + (*BatchUpdateThresholdsRequest)(nil), // 30: alerting.v1.BatchUpdateThresholdsRequest + (*BatchUpdateThresholdsResponse)(nil), // 31: alerting.v1.BatchUpdateThresholdsResponse + nil, // 32: alerting.v1.Template.LabelsEntry + nil, // 33: alerting.v1.Template.AnnotationsEntry + nil, // 34: alerting.v1.CreateRuleRequest.CustomLabelsEntry + ParamUnit(0), // 35: alerting.v1.ParamUnit + ParamType(0), // 36: alerting.v1.ParamType + (*durationpb.Duration)(nil), // 37: google.protobuf.Duration + v1.Severity(0), // 38: management.v1.Severity + (*timestamppb.Timestamp)(nil), // 39: google.protobuf.Timestamp } ) var file_alerting_v1_alerting_proto_depIdxs = []int32{ - 24, // 0: alerting.v1.ParamDefinition.unit:type_name -> alerting.v1.ParamUnit - 25, // 1: alerting.v1.ParamDefinition.type:type_name -> alerting.v1.ParamType - 2, // 2: alerting.v1.ParamDefinition.bool:type_name -> alerting.v1.BoolParamDefinition - 3, // 3: alerting.v1.ParamDefinition.float:type_name -> alerting.v1.FloatParamDefinition - 4, // 4: alerting.v1.ParamDefinition.string:type_name -> alerting.v1.StringParamDefinition - 5, // 5: alerting.v1.Template.params:type_name -> alerting.v1.ParamDefinition - 26, // 6: alerting.v1.Template.for:type_name -> google.protobuf.Duration - 27, // 7: alerting.v1.Template.severity:type_name -> management.v1.Severity - 21, // 8: alerting.v1.Template.labels:type_name -> alerting.v1.Template.LabelsEntry - 22, // 9: alerting.v1.Template.annotations:type_name -> alerting.v1.Template.AnnotationsEntry + 35, // 0: alerting.v1.ParamDefinition.unit:type_name -> alerting.v1.ParamUnit + 36, // 1: alerting.v1.ParamDefinition.type:type_name -> alerting.v1.ParamType + 3, // 2: alerting.v1.ParamDefinition.bool:type_name -> alerting.v1.BoolParamDefinition + 4, // 3: alerting.v1.ParamDefinition.float:type_name -> alerting.v1.FloatParamDefinition + 5, // 4: alerting.v1.ParamDefinition.string:type_name -> alerting.v1.StringParamDefinition + 6, // 5: alerting.v1.Template.params:type_name -> alerting.v1.ParamDefinition + 37, // 6: alerting.v1.Template.for:type_name -> google.protobuf.Duration + 38, // 7: alerting.v1.Template.severity:type_name -> management.v1.Severity + 32, // 8: alerting.v1.Template.labels:type_name -> alerting.v1.Template.LabelsEntry + 33, // 9: alerting.v1.Template.annotations:type_name -> alerting.v1.Template.AnnotationsEntry 0, // 10: alerting.v1.Template.source:type_name -> alerting.v1.TemplateSource - 28, // 11: alerting.v1.Template.created_at:type_name -> google.protobuf.Timestamp - 6, // 12: alerting.v1.Template.queries:type_name -> alerting.v1.TemplateQuery - 7, // 13: alerting.v1.Template.expressions:type_name -> alerting.v1.TemplateExpression - 8, // 14: alerting.v1.ListTemplatesResponse.templates:type_name -> alerting.v1.Template + 39, // 11: alerting.v1.Template.created_at:type_name -> google.protobuf.Timestamp + 7, // 12: alerting.v1.Template.queries:type_name -> alerting.v1.TemplateQuery + 8, // 13: alerting.v1.Template.expressions:type_name -> alerting.v1.TemplateExpression + 9, // 14: alerting.v1.ListTemplatesResponse.templates:type_name -> alerting.v1.Template 1, // 15: alerting.v1.Filter.type:type_name -> alerting.v1.FilterType - 25, // 16: alerting.v1.ParamValue.type:type_name -> alerting.v1.ParamType - 18, // 17: alerting.v1.CreateRuleRequest.params:type_name -> alerting.v1.ParamValue - 26, // 18: alerting.v1.CreateRuleRequest.for:type_name -> google.protobuf.Duration - 27, // 19: alerting.v1.CreateRuleRequest.severity:type_name -> management.v1.Severity - 23, // 20: alerting.v1.CreateRuleRequest.custom_labels:type_name -> alerting.v1.CreateRuleRequest.CustomLabelsEntry - 17, // 21: alerting.v1.CreateRuleRequest.filters:type_name -> alerting.v1.Filter - 26, // 22: alerting.v1.CreateRuleRequest.interval:type_name -> google.protobuf.Duration - 9, // 23: alerting.v1.AlertingService.ListTemplates:input_type -> alerting.v1.ListTemplatesRequest - 11, // 24: alerting.v1.AlertingService.CreateTemplate:input_type -> alerting.v1.CreateTemplateRequest - 13, // 25: alerting.v1.AlertingService.UpdateTemplate:input_type -> alerting.v1.UpdateTemplateRequest - 15, // 26: alerting.v1.AlertingService.DeleteTemplate:input_type -> alerting.v1.DeleteTemplateRequest - 19, // 27: alerting.v1.AlertingService.CreateRule:input_type -> alerting.v1.CreateRuleRequest - 10, // 28: alerting.v1.AlertingService.ListTemplates:output_type -> alerting.v1.ListTemplatesResponse - 12, // 29: alerting.v1.AlertingService.CreateTemplate:output_type -> alerting.v1.CreateTemplateResponse - 14, // 30: alerting.v1.AlertingService.UpdateTemplate:output_type -> alerting.v1.UpdateTemplateResponse - 16, // 31: alerting.v1.AlertingService.DeleteTemplate:output_type -> alerting.v1.DeleteTemplateResponse - 20, // 32: alerting.v1.AlertingService.CreateRule:output_type -> alerting.v1.CreateRuleResponse - 28, // [28:33] is the sub-list for method output_type - 23, // [23:28] is the sub-list for method input_type - 23, // [23:23] is the sub-list for extension type_name - 23, // [23:23] is the sub-list for extension extendee - 0, // [0:23] is the sub-list for field type_name + 36, // 16: alerting.v1.ParamValue.type:type_name -> alerting.v1.ParamType + 19, // 17: alerting.v1.CreateRuleRequest.params:type_name -> alerting.v1.ParamValue + 37, // 18: alerting.v1.CreateRuleRequest.for:type_name -> google.protobuf.Duration + 38, // 19: alerting.v1.CreateRuleRequest.severity:type_name -> management.v1.Severity + 34, // 20: alerting.v1.CreateRuleRequest.custom_labels:type_name -> alerting.v1.CreateRuleRequest.CustomLabelsEntry + 18, // 21: alerting.v1.CreateRuleRequest.filters:type_name -> alerting.v1.Filter + 37, // 22: alerting.v1.CreateRuleRequest.interval:type_name -> google.protobuf.Duration + 35, // 23: alerting.v1.Threshold.unit:type_name -> alerting.v1.ParamUnit + 2, // 24: alerting.v1.Threshold.scope:type_name -> alerting.v1.ThresholdScope + 2, // 25: alerting.v1.ListThresholdsRequest.scope:type_name -> alerting.v1.ThresholdScope + 22, // 26: alerting.v1.ListThresholdsResponse.thresholds:type_name -> alerting.v1.Threshold + 2, // 27: alerting.v1.SetThresholdRequest.scope:type_name -> alerting.v1.ThresholdScope + 22, // 28: alerting.v1.SetThresholdResponse.threshold:type_name -> alerting.v1.Threshold + 2, // 29: alerting.v1.ClearThresholdRequest.scope:type_name -> alerting.v1.ThresholdScope + 2, // 30: alerting.v1.ThresholdUpdate.scope:type_name -> alerting.v1.ThresholdScope + 29, // 31: alerting.v1.BatchUpdateThresholdsRequest.updates:type_name -> alerting.v1.ThresholdUpdate + 22, // 32: alerting.v1.BatchUpdateThresholdsResponse.thresholds:type_name -> alerting.v1.Threshold + 10, // 33: alerting.v1.AlertingService.ListTemplates:input_type -> alerting.v1.ListTemplatesRequest + 12, // 34: alerting.v1.AlertingService.CreateTemplate:input_type -> alerting.v1.CreateTemplateRequest + 14, // 35: alerting.v1.AlertingService.UpdateTemplate:input_type -> alerting.v1.UpdateTemplateRequest + 16, // 36: alerting.v1.AlertingService.DeleteTemplate:input_type -> alerting.v1.DeleteTemplateRequest + 20, // 37: alerting.v1.AlertingService.CreateRule:input_type -> alerting.v1.CreateRuleRequest + 23, // 38: alerting.v1.AlertingService.ListThresholds:input_type -> alerting.v1.ListThresholdsRequest + 25, // 39: alerting.v1.AlertingService.SetThreshold:input_type -> alerting.v1.SetThresholdRequest + 27, // 40: alerting.v1.AlertingService.ClearThreshold:input_type -> alerting.v1.ClearThresholdRequest + 30, // 41: alerting.v1.AlertingService.BatchUpdateThresholds:input_type -> alerting.v1.BatchUpdateThresholdsRequest + 11, // 42: alerting.v1.AlertingService.ListTemplates:output_type -> alerting.v1.ListTemplatesResponse + 13, // 43: alerting.v1.AlertingService.CreateTemplate:output_type -> alerting.v1.CreateTemplateResponse + 15, // 44: alerting.v1.AlertingService.UpdateTemplate:output_type -> alerting.v1.UpdateTemplateResponse + 17, // 45: alerting.v1.AlertingService.DeleteTemplate:output_type -> alerting.v1.DeleteTemplateResponse + 21, // 46: alerting.v1.AlertingService.CreateRule:output_type -> alerting.v1.CreateRuleResponse + 24, // 47: alerting.v1.AlertingService.ListThresholds:output_type -> alerting.v1.ListThresholdsResponse + 26, // 48: alerting.v1.AlertingService.SetThreshold:output_type -> alerting.v1.SetThresholdResponse + 28, // 49: alerting.v1.AlertingService.ClearThreshold:output_type -> alerting.v1.ClearThresholdResponse + 31, // 50: alerting.v1.AlertingService.BatchUpdateThresholds:output_type -> alerting.v1.BatchUpdateThresholdsResponse + 42, // [42:51] is the sub-list for method output_type + 33, // [33:42] is the sub-list for method input_type + 33, // [33:33] is the sub-list for extension type_name + 33, // [33:33] is the sub-list for extension extendee + 0, // [0:33] is the sub-list for field type_name } func init() { file_alerting_v1_alerting_proto_init() } @@ -1694,13 +2461,14 @@ func file_alerting_v1_alerting_proto_init() { (*ParamValue_Float)(nil), (*ParamValue_String_)(nil), } + file_alerting_v1_alerting_proto_msgTypes[26].OneofWrappers = []any{} type x struct{} out := protoimpl.TypeBuilder{ File: protoimpl.DescBuilder{ GoPackagePath: reflect.TypeOf(x{}).PkgPath(), RawDescriptor: unsafe.Slice(unsafe.StringData(file_alerting_v1_alerting_proto_rawDesc), len(file_alerting_v1_alerting_proto_rawDesc)), - NumEnums: 2, - NumMessages: 22, + NumEnums: 3, + NumMessages: 32, NumExtensions: 0, NumServices: 1, }, diff --git a/api/alerting/v1/alerting.pb.gw.go b/api/alerting/v1/alerting.pb.gw.go index 20f0459d21..0ce754a502 100644 --- a/api/alerting/v1/alerting.pb.gw.go +++ b/api/alerting/v1/alerting.pb.gw.go @@ -208,6 +208,130 @@ func local_request_AlertingService_CreateRule_0(ctx context.Context, marshaler r return msg, metadata, err } +var filter_AlertingService_ListThresholds_0 = &utilities.DoubleArray{Encoding: map[string]int{}, Base: []int(nil), Check: []int(nil)} + +func request_AlertingService_ListThresholds_0(ctx context.Context, marshaler runtime.Marshaler, client AlertingServiceClient, req *http.Request, pathParams map[string]string) (proto.Message, runtime.ServerMetadata, error) { + var ( + protoReq ListThresholdsRequest + metadata runtime.ServerMetadata + ) + if err := req.ParseForm(); err != nil { + return nil, metadata, status.Errorf(codes.InvalidArgument, "%v", err) + } + if err := runtime.PopulateQueryParameters(&protoReq, req.Form, filter_AlertingService_ListThresholds_0); err != nil { + return nil, metadata, status.Errorf(codes.InvalidArgument, "%v", err) + } + if req.Body != nil { + _, _ = io.Copy(io.Discard, req.Body) + } + msg, err := client.ListThresholds(ctx, &protoReq, grpc.Header(&metadata.HeaderMD), grpc.Trailer(&metadata.TrailerMD)) + return msg, metadata, err +} + +func local_request_AlertingService_ListThresholds_0(ctx context.Context, marshaler runtime.Marshaler, server AlertingServiceServer, req *http.Request, pathParams map[string]string) (proto.Message, runtime.ServerMetadata, error) { + var ( + protoReq ListThresholdsRequest + metadata runtime.ServerMetadata + ) + if err := req.ParseForm(); err != nil { + return nil, metadata, status.Errorf(codes.InvalidArgument, "%v", err) + } + if err := runtime.PopulateQueryParameters(&protoReq, req.Form, filter_AlertingService_ListThresholds_0); err != nil { + return nil, metadata, status.Errorf(codes.InvalidArgument, "%v", err) + } + msg, err := server.ListThresholds(ctx, &protoReq) + return msg, metadata, err +} + +func request_AlertingService_SetThreshold_0(ctx context.Context, marshaler runtime.Marshaler, client AlertingServiceClient, req *http.Request, pathParams map[string]string) (proto.Message, runtime.ServerMetadata, error) { + var ( + protoReq SetThresholdRequest + metadata runtime.ServerMetadata + ) + if err := marshaler.NewDecoder(req.Body).Decode(&protoReq); err != nil && !errors.Is(err, io.EOF) { + return nil, metadata, status.Errorf(codes.InvalidArgument, "%v", err) + } + if req.Body != nil { + _, _ = io.Copy(io.Discard, req.Body) + } + msg, err := client.SetThreshold(ctx, &protoReq, grpc.Header(&metadata.HeaderMD), grpc.Trailer(&metadata.TrailerMD)) + return msg, metadata, err +} + +func local_request_AlertingService_SetThreshold_0(ctx context.Context, marshaler runtime.Marshaler, server AlertingServiceServer, req *http.Request, pathParams map[string]string) (proto.Message, runtime.ServerMetadata, error) { + var ( + protoReq SetThresholdRequest + metadata runtime.ServerMetadata + ) + if err := marshaler.NewDecoder(req.Body).Decode(&protoReq); err != nil && !errors.Is(err, io.EOF) { + return nil, metadata, status.Errorf(codes.InvalidArgument, "%v", err) + } + msg, err := server.SetThreshold(ctx, &protoReq) + return msg, metadata, err +} + +var filter_AlertingService_ClearThreshold_0 = &utilities.DoubleArray{Encoding: map[string]int{}, Base: []int(nil), Check: []int(nil)} + +func request_AlertingService_ClearThreshold_0(ctx context.Context, marshaler runtime.Marshaler, client AlertingServiceClient, req *http.Request, pathParams map[string]string) (proto.Message, runtime.ServerMetadata, error) { + var ( + protoReq ClearThresholdRequest + metadata runtime.ServerMetadata + ) + if err := req.ParseForm(); err != nil { + return nil, metadata, status.Errorf(codes.InvalidArgument, "%v", err) + } + if err := runtime.PopulateQueryParameters(&protoReq, req.Form, filter_AlertingService_ClearThreshold_0); err != nil { + return nil, metadata, status.Errorf(codes.InvalidArgument, "%v", err) + } + if req.Body != nil { + _, _ = io.Copy(io.Discard, req.Body) + } + msg, err := client.ClearThreshold(ctx, &protoReq, grpc.Header(&metadata.HeaderMD), grpc.Trailer(&metadata.TrailerMD)) + return msg, metadata, err +} + +func local_request_AlertingService_ClearThreshold_0(ctx context.Context, marshaler runtime.Marshaler, server AlertingServiceServer, req *http.Request, pathParams map[string]string) (proto.Message, runtime.ServerMetadata, error) { + var ( + protoReq ClearThresholdRequest + metadata runtime.ServerMetadata + ) + if err := req.ParseForm(); err != nil { + return nil, metadata, status.Errorf(codes.InvalidArgument, "%v", err) + } + if err := runtime.PopulateQueryParameters(&protoReq, req.Form, filter_AlertingService_ClearThreshold_0); err != nil { + return nil, metadata, status.Errorf(codes.InvalidArgument, "%v", err) + } + msg, err := server.ClearThreshold(ctx, &protoReq) + return msg, metadata, err +} + +func request_AlertingService_BatchUpdateThresholds_0(ctx context.Context, marshaler runtime.Marshaler, client AlertingServiceClient, req *http.Request, pathParams map[string]string) (proto.Message, runtime.ServerMetadata, error) { + var ( + protoReq BatchUpdateThresholdsRequest + metadata runtime.ServerMetadata + ) + if err := marshaler.NewDecoder(req.Body).Decode(&protoReq); err != nil && !errors.Is(err, io.EOF) { + return nil, metadata, status.Errorf(codes.InvalidArgument, "%v", err) + } + if req.Body != nil { + _, _ = io.Copy(io.Discard, req.Body) + } + msg, err := client.BatchUpdateThresholds(ctx, &protoReq, grpc.Header(&metadata.HeaderMD), grpc.Trailer(&metadata.TrailerMD)) + return msg, metadata, err +} + +func local_request_AlertingService_BatchUpdateThresholds_0(ctx context.Context, marshaler runtime.Marshaler, server AlertingServiceServer, req *http.Request, pathParams map[string]string) (proto.Message, runtime.ServerMetadata, error) { + var ( + protoReq BatchUpdateThresholdsRequest + metadata runtime.ServerMetadata + ) + if err := marshaler.NewDecoder(req.Body).Decode(&protoReq); err != nil && !errors.Is(err, io.EOF) { + return nil, metadata, status.Errorf(codes.InvalidArgument, "%v", err) + } + msg, err := server.BatchUpdateThresholds(ctx, &protoReq) + return msg, metadata, err +} + // RegisterAlertingServiceHandlerServer registers the http handlers for service AlertingService to "mux". // UnaryRPC :call AlertingServiceServer directly. // StreamingRPC :currently unsupported pending https://github.com/grpc/grpc-go/issues/906. @@ -314,6 +438,86 @@ func RegisterAlertingServiceHandlerServer(ctx context.Context, mux *runtime.Serv } forward_AlertingService_CreateRule_0(annotatedContext, mux, outboundMarshaler, w, req, resp, mux.GetForwardResponseOptions()...) }) + mux.Handle(http.MethodGet, pattern_AlertingService_ListThresholds_0, func(w http.ResponseWriter, req *http.Request, pathParams map[string]string) { + ctx, cancel := context.WithCancel(req.Context()) + defer cancel() + var stream runtime.ServerTransportStream + ctx = grpc.NewContextWithServerTransportStream(ctx, &stream) + inboundMarshaler, outboundMarshaler := runtime.MarshalerForRequest(mux, req) + annotatedContext, err := runtime.AnnotateIncomingContext(ctx, mux, req, "/alerting.v1.AlertingService/ListThresholds", runtime.WithHTTPPathPattern("/v1/alerting/thresholds")) + if err != nil { + runtime.HTTPError(ctx, mux, outboundMarshaler, w, req, err) + return + } + resp, md, err := local_request_AlertingService_ListThresholds_0(annotatedContext, inboundMarshaler, server, req, pathParams) + md.HeaderMD, md.TrailerMD = metadata.Join(md.HeaderMD, stream.Header()), metadata.Join(md.TrailerMD, stream.Trailer()) + annotatedContext = runtime.NewServerMetadataContext(annotatedContext, md) + if err != nil { + runtime.HTTPError(annotatedContext, mux, outboundMarshaler, w, req, err) + return + } + forward_AlertingService_ListThresholds_0(annotatedContext, mux, outboundMarshaler, w, req, resp, mux.GetForwardResponseOptions()...) + }) + mux.Handle(http.MethodPost, pattern_AlertingService_SetThreshold_0, func(w http.ResponseWriter, req *http.Request, pathParams map[string]string) { + ctx, cancel := context.WithCancel(req.Context()) + defer cancel() + var stream runtime.ServerTransportStream + ctx = grpc.NewContextWithServerTransportStream(ctx, &stream) + inboundMarshaler, outboundMarshaler := runtime.MarshalerForRequest(mux, req) + annotatedContext, err := runtime.AnnotateIncomingContext(ctx, mux, req, "/alerting.v1.AlertingService/SetThreshold", runtime.WithHTTPPathPattern("/v1/alerting/thresholds")) + if err != nil { + runtime.HTTPError(ctx, mux, outboundMarshaler, w, req, err) + return + } + resp, md, err := local_request_AlertingService_SetThreshold_0(annotatedContext, inboundMarshaler, server, req, pathParams) + md.HeaderMD, md.TrailerMD = metadata.Join(md.HeaderMD, stream.Header()), metadata.Join(md.TrailerMD, stream.Trailer()) + annotatedContext = runtime.NewServerMetadataContext(annotatedContext, md) + if err != nil { + runtime.HTTPError(annotatedContext, mux, outboundMarshaler, w, req, err) + return + } + forward_AlertingService_SetThreshold_0(annotatedContext, mux, outboundMarshaler, w, req, resp, mux.GetForwardResponseOptions()...) + }) + mux.Handle(http.MethodDelete, pattern_AlertingService_ClearThreshold_0, func(w http.ResponseWriter, req *http.Request, pathParams map[string]string) { + ctx, cancel := context.WithCancel(req.Context()) + defer cancel() + var stream runtime.ServerTransportStream + ctx = grpc.NewContextWithServerTransportStream(ctx, &stream) + inboundMarshaler, outboundMarshaler := runtime.MarshalerForRequest(mux, req) + annotatedContext, err := runtime.AnnotateIncomingContext(ctx, mux, req, "/alerting.v1.AlertingService/ClearThreshold", runtime.WithHTTPPathPattern("/v1/alerting/thresholds")) + if err != nil { + runtime.HTTPError(ctx, mux, outboundMarshaler, w, req, err) + return + } + resp, md, err := local_request_AlertingService_ClearThreshold_0(annotatedContext, inboundMarshaler, server, req, pathParams) + md.HeaderMD, md.TrailerMD = metadata.Join(md.HeaderMD, stream.Header()), metadata.Join(md.TrailerMD, stream.Trailer()) + annotatedContext = runtime.NewServerMetadataContext(annotatedContext, md) + if err != nil { + runtime.HTTPError(annotatedContext, mux, outboundMarshaler, w, req, err) + return + } + forward_AlertingService_ClearThreshold_0(annotatedContext, mux, outboundMarshaler, w, req, resp, mux.GetForwardResponseOptions()...) + }) + mux.Handle(http.MethodPost, pattern_AlertingService_BatchUpdateThresholds_0, func(w http.ResponseWriter, req *http.Request, pathParams map[string]string) { + ctx, cancel := context.WithCancel(req.Context()) + defer cancel() + var stream runtime.ServerTransportStream + ctx = grpc.NewContextWithServerTransportStream(ctx, &stream) + inboundMarshaler, outboundMarshaler := runtime.MarshalerForRequest(mux, req) + annotatedContext, err := runtime.AnnotateIncomingContext(ctx, mux, req, "/alerting.v1.AlertingService/BatchUpdateThresholds", runtime.WithHTTPPathPattern("/v1/alerting/thresholds:batchUpdate")) + if err != nil { + runtime.HTTPError(ctx, mux, outboundMarshaler, w, req, err) + return + } + resp, md, err := local_request_AlertingService_BatchUpdateThresholds_0(annotatedContext, inboundMarshaler, server, req, pathParams) + md.HeaderMD, md.TrailerMD = metadata.Join(md.HeaderMD, stream.Header()), metadata.Join(md.TrailerMD, stream.Trailer()) + annotatedContext = runtime.NewServerMetadataContext(annotatedContext, md) + if err != nil { + runtime.HTTPError(annotatedContext, mux, outboundMarshaler, w, req, err) + return + } + forward_AlertingService_BatchUpdateThresholds_0(annotatedContext, mux, outboundMarshaler, w, req, resp, mux.GetForwardResponseOptions()...) + }) return nil } @@ -439,21 +643,97 @@ func RegisterAlertingServiceHandlerClient(ctx context.Context, mux *runtime.Serv } forward_AlertingService_CreateRule_0(annotatedContext, mux, outboundMarshaler, w, req, resp, mux.GetForwardResponseOptions()...) }) + mux.Handle(http.MethodGet, pattern_AlertingService_ListThresholds_0, func(w http.ResponseWriter, req *http.Request, pathParams map[string]string) { + ctx, cancel := context.WithCancel(req.Context()) + defer cancel() + inboundMarshaler, outboundMarshaler := runtime.MarshalerForRequest(mux, req) + annotatedContext, err := runtime.AnnotateContext(ctx, mux, req, "/alerting.v1.AlertingService/ListThresholds", runtime.WithHTTPPathPattern("/v1/alerting/thresholds")) + if err != nil { + runtime.HTTPError(ctx, mux, outboundMarshaler, w, req, err) + return + } + resp, md, err := request_AlertingService_ListThresholds_0(annotatedContext, inboundMarshaler, client, req, pathParams) + annotatedContext = runtime.NewServerMetadataContext(annotatedContext, md) + if err != nil { + runtime.HTTPError(annotatedContext, mux, outboundMarshaler, w, req, err) + return + } + forward_AlertingService_ListThresholds_0(annotatedContext, mux, outboundMarshaler, w, req, resp, mux.GetForwardResponseOptions()...) + }) + mux.Handle(http.MethodPost, pattern_AlertingService_SetThreshold_0, func(w http.ResponseWriter, req *http.Request, pathParams map[string]string) { + ctx, cancel := context.WithCancel(req.Context()) + defer cancel() + inboundMarshaler, outboundMarshaler := runtime.MarshalerForRequest(mux, req) + annotatedContext, err := runtime.AnnotateContext(ctx, mux, req, "/alerting.v1.AlertingService/SetThreshold", runtime.WithHTTPPathPattern("/v1/alerting/thresholds")) + if err != nil { + runtime.HTTPError(ctx, mux, outboundMarshaler, w, req, err) + return + } + resp, md, err := request_AlertingService_SetThreshold_0(annotatedContext, inboundMarshaler, client, req, pathParams) + annotatedContext = runtime.NewServerMetadataContext(annotatedContext, md) + if err != nil { + runtime.HTTPError(annotatedContext, mux, outboundMarshaler, w, req, err) + return + } + forward_AlertingService_SetThreshold_0(annotatedContext, mux, outboundMarshaler, w, req, resp, mux.GetForwardResponseOptions()...) + }) + mux.Handle(http.MethodDelete, pattern_AlertingService_ClearThreshold_0, func(w http.ResponseWriter, req *http.Request, pathParams map[string]string) { + ctx, cancel := context.WithCancel(req.Context()) + defer cancel() + inboundMarshaler, outboundMarshaler := runtime.MarshalerForRequest(mux, req) + annotatedContext, err := runtime.AnnotateContext(ctx, mux, req, "/alerting.v1.AlertingService/ClearThreshold", runtime.WithHTTPPathPattern("/v1/alerting/thresholds")) + if err != nil { + runtime.HTTPError(ctx, mux, outboundMarshaler, w, req, err) + return + } + resp, md, err := request_AlertingService_ClearThreshold_0(annotatedContext, inboundMarshaler, client, req, pathParams) + annotatedContext = runtime.NewServerMetadataContext(annotatedContext, md) + if err != nil { + runtime.HTTPError(annotatedContext, mux, outboundMarshaler, w, req, err) + return + } + forward_AlertingService_ClearThreshold_0(annotatedContext, mux, outboundMarshaler, w, req, resp, mux.GetForwardResponseOptions()...) + }) + mux.Handle(http.MethodPost, pattern_AlertingService_BatchUpdateThresholds_0, func(w http.ResponseWriter, req *http.Request, pathParams map[string]string) { + ctx, cancel := context.WithCancel(req.Context()) + defer cancel() + inboundMarshaler, outboundMarshaler := runtime.MarshalerForRequest(mux, req) + annotatedContext, err := runtime.AnnotateContext(ctx, mux, req, "/alerting.v1.AlertingService/BatchUpdateThresholds", runtime.WithHTTPPathPattern("/v1/alerting/thresholds:batchUpdate")) + if err != nil { + runtime.HTTPError(ctx, mux, outboundMarshaler, w, req, err) + return + } + resp, md, err := request_AlertingService_BatchUpdateThresholds_0(annotatedContext, inboundMarshaler, client, req, pathParams) + annotatedContext = runtime.NewServerMetadataContext(annotatedContext, md) + if err != nil { + runtime.HTTPError(annotatedContext, mux, outboundMarshaler, w, req, err) + return + } + forward_AlertingService_BatchUpdateThresholds_0(annotatedContext, mux, outboundMarshaler, w, req, resp, mux.GetForwardResponseOptions()...) + }) return nil } var ( - pattern_AlertingService_ListTemplates_0 = runtime.MustPattern(runtime.NewPattern(1, []int{2, 0, 2, 1, 2, 2}, []string{"v1", "alerting", "templates"}, "")) - pattern_AlertingService_CreateTemplate_0 = runtime.MustPattern(runtime.NewPattern(1, []int{2, 0, 2, 1, 2, 2}, []string{"v1", "alerting", "templates"}, "")) - pattern_AlertingService_UpdateTemplate_0 = runtime.MustPattern(runtime.NewPattern(1, []int{2, 0, 2, 1, 2, 2, 1, 0, 4, 1, 5, 3}, []string{"v1", "alerting", "templates", "name"}, "")) - pattern_AlertingService_DeleteTemplate_0 = runtime.MustPattern(runtime.NewPattern(1, []int{2, 0, 2, 1, 2, 2, 1, 0, 4, 1, 5, 3}, []string{"v1", "alerting", "templates", "name"}, "")) - pattern_AlertingService_CreateRule_0 = runtime.MustPattern(runtime.NewPattern(1, []int{2, 0, 2, 1, 2, 2}, []string{"v1", "alerting", "rules"}, "")) + pattern_AlertingService_ListTemplates_0 = runtime.MustPattern(runtime.NewPattern(1, []int{2, 0, 2, 1, 2, 2}, []string{"v1", "alerting", "templates"}, "")) + pattern_AlertingService_CreateTemplate_0 = runtime.MustPattern(runtime.NewPattern(1, []int{2, 0, 2, 1, 2, 2}, []string{"v1", "alerting", "templates"}, "")) + pattern_AlertingService_UpdateTemplate_0 = runtime.MustPattern(runtime.NewPattern(1, []int{2, 0, 2, 1, 2, 2, 1, 0, 4, 1, 5, 3}, []string{"v1", "alerting", "templates", "name"}, "")) + pattern_AlertingService_DeleteTemplate_0 = runtime.MustPattern(runtime.NewPattern(1, []int{2, 0, 2, 1, 2, 2, 1, 0, 4, 1, 5, 3}, []string{"v1", "alerting", "templates", "name"}, "")) + pattern_AlertingService_CreateRule_0 = runtime.MustPattern(runtime.NewPattern(1, []int{2, 0, 2, 1, 2, 2}, []string{"v1", "alerting", "rules"}, "")) + pattern_AlertingService_ListThresholds_0 = runtime.MustPattern(runtime.NewPattern(1, []int{2, 0, 2, 1, 2, 2}, []string{"v1", "alerting", "thresholds"}, "")) + pattern_AlertingService_SetThreshold_0 = runtime.MustPattern(runtime.NewPattern(1, []int{2, 0, 2, 1, 2, 2}, []string{"v1", "alerting", "thresholds"}, "")) + pattern_AlertingService_ClearThreshold_0 = runtime.MustPattern(runtime.NewPattern(1, []int{2, 0, 2, 1, 2, 2}, []string{"v1", "alerting", "thresholds"}, "")) + pattern_AlertingService_BatchUpdateThresholds_0 = runtime.MustPattern(runtime.NewPattern(1, []int{2, 0, 2, 1, 2, 2}, []string{"v1", "alerting", "thresholds"}, "batchUpdate")) ) var ( - forward_AlertingService_ListTemplates_0 = runtime.ForwardResponseMessage - forward_AlertingService_CreateTemplate_0 = runtime.ForwardResponseMessage - forward_AlertingService_UpdateTemplate_0 = runtime.ForwardResponseMessage - forward_AlertingService_DeleteTemplate_0 = runtime.ForwardResponseMessage - forward_AlertingService_CreateRule_0 = runtime.ForwardResponseMessage + forward_AlertingService_ListTemplates_0 = runtime.ForwardResponseMessage + forward_AlertingService_CreateTemplate_0 = runtime.ForwardResponseMessage + forward_AlertingService_UpdateTemplate_0 = runtime.ForwardResponseMessage + forward_AlertingService_DeleteTemplate_0 = runtime.ForwardResponseMessage + forward_AlertingService_CreateRule_0 = runtime.ForwardResponseMessage + forward_AlertingService_ListThresholds_0 = runtime.ForwardResponseMessage + forward_AlertingService_SetThreshold_0 = runtime.ForwardResponseMessage + forward_AlertingService_ClearThreshold_0 = runtime.ForwardResponseMessage + forward_AlertingService_BatchUpdateThresholds_0 = runtime.ForwardResponseMessage ) diff --git a/api/alerting/v1/alerting.pb.validate.go b/api/alerting/v1/alerting.pb.validate.go index 6ab74dee29..0ce9bb4233 100644 --- a/api/alerting/v1/alerting.pb.validate.go +++ b/api/alerting/v1/alerting.pb.validate.go @@ -2653,3 +2653,1307 @@ var _ interface { Cause() error ErrorName() string } = CreateRuleResponseValidationError{} + +// Validate checks the field values on Threshold with the rules defined in the +// proto definition for this message. If any rules are violated, the first +// error encountered is returned, or nil if there are no violations. +func (m *Threshold) Validate() error { + return m.validate(false) +} + +// ValidateAll checks the field values on Threshold with the rules defined in +// the proto definition for this message. If any rules are violated, the +// result is a list of violation errors wrapped in ThresholdMultiError, or nil +// if none found. +func (m *Threshold) ValidateAll() error { + return m.validate(true) +} + +func (m *Threshold) validate(all bool) error { + if m == nil { + return nil + } + + var errors []error + + // no validation rules for RuleId + + // no validation rules for ParamName + + // no validation rules for Summary + + // no validation rules for Unit + + // no validation rules for DefaultValue + + // no validation rules for EffectiveValue + + // no validation rules for IsOverridden + + // no validation rules for Scope + + // no validation rules for Target + + if len(errors) > 0 { + return ThresholdMultiError(errors) + } + + return nil +} + +// ThresholdMultiError is an error wrapping multiple validation errors returned +// by Threshold.ValidateAll() if the designated constraints aren't met. +type ThresholdMultiError []error + +// Error returns a concatenation of all the error messages it wraps. +func (m ThresholdMultiError) Error() string { + msgs := make([]string, 0, len(m)) + for _, err := range m { + msgs = append(msgs, err.Error()) + } + return strings.Join(msgs, "; ") +} + +// AllErrors returns a list of validation violation errors. +func (m ThresholdMultiError) AllErrors() []error { return m } + +// ThresholdValidationError is the validation error returned by +// Threshold.Validate if the designated constraints aren't met. +type ThresholdValidationError struct { + field string + reason string + cause error + key bool +} + +// Field function returns field value. +func (e ThresholdValidationError) Field() string { return e.field } + +// Reason function returns reason value. +func (e ThresholdValidationError) Reason() string { return e.reason } + +// Cause function returns cause value. +func (e ThresholdValidationError) Cause() error { return e.cause } + +// Key function returns key value. +func (e ThresholdValidationError) Key() bool { return e.key } + +// ErrorName returns error name. +func (e ThresholdValidationError) ErrorName() string { return "ThresholdValidationError" } + +// Error satisfies the builtin error interface +func (e ThresholdValidationError) Error() string { + cause := "" + if e.cause != nil { + cause = fmt.Sprintf(" | caused by: %v", e.cause) + } + + key := "" + if e.key { + key = "key for " + } + + return fmt.Sprintf( + "invalid %sThreshold.%s: %s%s", + key, + e.field, + e.reason, + cause, + ) +} + +var _ error = ThresholdValidationError{} + +var _ interface { + Field() string + Reason() string + Key() bool + Cause() error + ErrorName() string +} = ThresholdValidationError{} + +// Validate checks the field values on ListThresholdsRequest with the rules +// defined in the proto definition for this message. If any rules are +// violated, the first error encountered is returned, or nil if there are no violations. +func (m *ListThresholdsRequest) Validate() error { + return m.validate(false) +} + +// ValidateAll checks the field values on ListThresholdsRequest with the rules +// defined in the proto definition for this message. If any rules are +// violated, the result is a list of violation errors wrapped in +// ListThresholdsRequestMultiError, or nil if none found. +func (m *ListThresholdsRequest) ValidateAll() error { + return m.validate(true) +} + +func (m *ListThresholdsRequest) validate(all bool) error { + if m == nil { + return nil + } + + var errors []error + + // no validation rules for Scope + + // no validation rules for Target + + // no validation rules for RuleId + + if len(errors) > 0 { + return ListThresholdsRequestMultiError(errors) + } + + return nil +} + +// ListThresholdsRequestMultiError is an error wrapping multiple validation +// errors returned by ListThresholdsRequest.ValidateAll() if the designated +// constraints aren't met. +type ListThresholdsRequestMultiError []error + +// Error returns a concatenation of all the error messages it wraps. +func (m ListThresholdsRequestMultiError) Error() string { + msgs := make([]string, 0, len(m)) + for _, err := range m { + msgs = append(msgs, err.Error()) + } + return strings.Join(msgs, "; ") +} + +// AllErrors returns a list of validation violation errors. +func (m ListThresholdsRequestMultiError) AllErrors() []error { return m } + +// ListThresholdsRequestValidationError is the validation error returned by +// ListThresholdsRequest.Validate if the designated constraints aren't met. +type ListThresholdsRequestValidationError struct { + field string + reason string + cause error + key bool +} + +// Field function returns field value. +func (e ListThresholdsRequestValidationError) Field() string { return e.field } + +// Reason function returns reason value. +func (e ListThresholdsRequestValidationError) Reason() string { return e.reason } + +// Cause function returns cause value. +func (e ListThresholdsRequestValidationError) Cause() error { return e.cause } + +// Key function returns key value. +func (e ListThresholdsRequestValidationError) Key() bool { return e.key } + +// ErrorName returns error name. +func (e ListThresholdsRequestValidationError) ErrorName() string { + return "ListThresholdsRequestValidationError" +} + +// Error satisfies the builtin error interface +func (e ListThresholdsRequestValidationError) Error() string { + cause := "" + if e.cause != nil { + cause = fmt.Sprintf(" | caused by: %v", e.cause) + } + + key := "" + if e.key { + key = "key for " + } + + return fmt.Sprintf( + "invalid %sListThresholdsRequest.%s: %s%s", + key, + e.field, + e.reason, + cause, + ) +} + +var _ error = ListThresholdsRequestValidationError{} + +var _ interface { + Field() string + Reason() string + Key() bool + Cause() error + ErrorName() string +} = ListThresholdsRequestValidationError{} + +// Validate checks the field values on ListThresholdsResponse with the rules +// defined in the proto definition for this message. If any rules are +// violated, the first error encountered is returned, or nil if there are no violations. +func (m *ListThresholdsResponse) Validate() error { + return m.validate(false) +} + +// ValidateAll checks the field values on ListThresholdsResponse with the rules +// defined in the proto definition for this message. If any rules are +// violated, the result is a list of violation errors wrapped in +// ListThresholdsResponseMultiError, or nil if none found. +func (m *ListThresholdsResponse) ValidateAll() error { + return m.validate(true) +} + +func (m *ListThresholdsResponse) validate(all bool) error { + if m == nil { + return nil + } + + var errors []error + + for idx, item := range m.GetThresholds() { + _, _ = idx, item + + if all { + switch v := interface{}(item).(type) { + case interface{ ValidateAll() error }: + if err := v.ValidateAll(); err != nil { + errors = append(errors, ListThresholdsResponseValidationError{ + field: fmt.Sprintf("Thresholds[%v]", idx), + reason: "embedded message failed validation", + cause: err, + }) + } + case interface{ Validate() error }: + if err := v.Validate(); err != nil { + errors = append(errors, ListThresholdsResponseValidationError{ + field: fmt.Sprintf("Thresholds[%v]", idx), + reason: "embedded message failed validation", + cause: err, + }) + } + } + } else if v, ok := interface{}(item).(interface{ Validate() error }); ok { + if err := v.Validate(); err != nil { + return ListThresholdsResponseValidationError{ + field: fmt.Sprintf("Thresholds[%v]", idx), + reason: "embedded message failed validation", + cause: err, + } + } + } + + } + + if len(errors) > 0 { + return ListThresholdsResponseMultiError(errors) + } + + return nil +} + +// ListThresholdsResponseMultiError is an error wrapping multiple validation +// errors returned by ListThresholdsResponse.ValidateAll() if the designated +// constraints aren't met. +type ListThresholdsResponseMultiError []error + +// Error returns a concatenation of all the error messages it wraps. +func (m ListThresholdsResponseMultiError) Error() string { + msgs := make([]string, 0, len(m)) + for _, err := range m { + msgs = append(msgs, err.Error()) + } + return strings.Join(msgs, "; ") +} + +// AllErrors returns a list of validation violation errors. +func (m ListThresholdsResponseMultiError) AllErrors() []error { return m } + +// ListThresholdsResponseValidationError is the validation error returned by +// ListThresholdsResponse.Validate if the designated constraints aren't met. +type ListThresholdsResponseValidationError struct { + field string + reason string + cause error + key bool +} + +// Field function returns field value. +func (e ListThresholdsResponseValidationError) Field() string { return e.field } + +// Reason function returns reason value. +func (e ListThresholdsResponseValidationError) Reason() string { return e.reason } + +// Cause function returns cause value. +func (e ListThresholdsResponseValidationError) Cause() error { return e.cause } + +// Key function returns key value. +func (e ListThresholdsResponseValidationError) Key() bool { return e.key } + +// ErrorName returns error name. +func (e ListThresholdsResponseValidationError) ErrorName() string { + return "ListThresholdsResponseValidationError" +} + +// Error satisfies the builtin error interface +func (e ListThresholdsResponseValidationError) Error() string { + cause := "" + if e.cause != nil { + cause = fmt.Sprintf(" | caused by: %v", e.cause) + } + + key := "" + if e.key { + key = "key for " + } + + return fmt.Sprintf( + "invalid %sListThresholdsResponse.%s: %s%s", + key, + e.field, + e.reason, + cause, + ) +} + +var _ error = ListThresholdsResponseValidationError{} + +var _ interface { + Field() string + Reason() string + Key() bool + Cause() error + ErrorName() string +} = ListThresholdsResponseValidationError{} + +// Validate checks the field values on SetThresholdRequest with the rules +// defined in the proto definition for this message. If any rules are +// violated, the first error encountered is returned, or nil if there are no violations. +func (m *SetThresholdRequest) Validate() error { + return m.validate(false) +} + +// ValidateAll checks the field values on SetThresholdRequest with the rules +// defined in the proto definition for this message. If any rules are +// violated, the result is a list of violation errors wrapped in +// SetThresholdRequestMultiError, or nil if none found. +func (m *SetThresholdRequest) ValidateAll() error { + return m.validate(true) +} + +func (m *SetThresholdRequest) validate(all bool) error { + if m == nil { + return nil + } + + var errors []error + + // no validation rules for Scope + + if utf8.RuneCountInString(m.GetTarget()) < 1 { + err := SetThresholdRequestValidationError{ + field: "Target", + reason: "value length must be at least 1 runes", + } + if !all { + return err + } + errors = append(errors, err) + } + + if utf8.RuneCountInString(m.GetRuleId()) < 1 { + err := SetThresholdRequestValidationError{ + field: "RuleId", + reason: "value length must be at least 1 runes", + } + if !all { + return err + } + errors = append(errors, err) + } + + if utf8.RuneCountInString(m.GetParamName()) < 1 { + err := SetThresholdRequestValidationError{ + field: "ParamName", + reason: "value length must be at least 1 runes", + } + if !all { + return err + } + errors = append(errors, err) + } + + // no validation rules for Value + + if len(errors) > 0 { + return SetThresholdRequestMultiError(errors) + } + + return nil +} + +// SetThresholdRequestMultiError is an error wrapping multiple validation +// errors returned by SetThresholdRequest.ValidateAll() if the designated +// constraints aren't met. +type SetThresholdRequestMultiError []error + +// Error returns a concatenation of all the error messages it wraps. +func (m SetThresholdRequestMultiError) Error() string { + msgs := make([]string, 0, len(m)) + for _, err := range m { + msgs = append(msgs, err.Error()) + } + return strings.Join(msgs, "; ") +} + +// AllErrors returns a list of validation violation errors. +func (m SetThresholdRequestMultiError) AllErrors() []error { return m } + +// SetThresholdRequestValidationError is the validation error returned by +// SetThresholdRequest.Validate if the designated constraints aren't met. +type SetThresholdRequestValidationError struct { + field string + reason string + cause error + key bool +} + +// Field function returns field value. +func (e SetThresholdRequestValidationError) Field() string { return e.field } + +// Reason function returns reason value. +func (e SetThresholdRequestValidationError) Reason() string { return e.reason } + +// Cause function returns cause value. +func (e SetThresholdRequestValidationError) Cause() error { return e.cause } + +// Key function returns key value. +func (e SetThresholdRequestValidationError) Key() bool { return e.key } + +// ErrorName returns error name. +func (e SetThresholdRequestValidationError) ErrorName() string { + return "SetThresholdRequestValidationError" +} + +// Error satisfies the builtin error interface +func (e SetThresholdRequestValidationError) Error() string { + cause := "" + if e.cause != nil { + cause = fmt.Sprintf(" | caused by: %v", e.cause) + } + + key := "" + if e.key { + key = "key for " + } + + return fmt.Sprintf( + "invalid %sSetThresholdRequest.%s: %s%s", + key, + e.field, + e.reason, + cause, + ) +} + +var _ error = SetThresholdRequestValidationError{} + +var _ interface { + Field() string + Reason() string + Key() bool + Cause() error + ErrorName() string +} = SetThresholdRequestValidationError{} + +// Validate checks the field values on SetThresholdResponse with the rules +// defined in the proto definition for this message. If any rules are +// violated, the first error encountered is returned, or nil if there are no violations. +func (m *SetThresholdResponse) Validate() error { + return m.validate(false) +} + +// ValidateAll checks the field values on SetThresholdResponse with the rules +// defined in the proto definition for this message. If any rules are +// violated, the result is a list of violation errors wrapped in +// SetThresholdResponseMultiError, or nil if none found. +func (m *SetThresholdResponse) ValidateAll() error { + return m.validate(true) +} + +func (m *SetThresholdResponse) validate(all bool) error { + if m == nil { + return nil + } + + var errors []error + + if all { + switch v := interface{}(m.GetThreshold()).(type) { + case interface{ ValidateAll() error }: + if err := v.ValidateAll(); err != nil { + errors = append(errors, SetThresholdResponseValidationError{ + field: "Threshold", + reason: "embedded message failed validation", + cause: err, + }) + } + case interface{ Validate() error }: + if err := v.Validate(); err != nil { + errors = append(errors, SetThresholdResponseValidationError{ + field: "Threshold", + reason: "embedded message failed validation", + cause: err, + }) + } + } + } else if v, ok := interface{}(m.GetThreshold()).(interface{ Validate() error }); ok { + if err := v.Validate(); err != nil { + return SetThresholdResponseValidationError{ + field: "Threshold", + reason: "embedded message failed validation", + cause: err, + } + } + } + + if len(errors) > 0 { + return SetThresholdResponseMultiError(errors) + } + + return nil +} + +// SetThresholdResponseMultiError is an error wrapping multiple validation +// errors returned by SetThresholdResponse.ValidateAll() if the designated +// constraints aren't met. +type SetThresholdResponseMultiError []error + +// Error returns a concatenation of all the error messages it wraps. +func (m SetThresholdResponseMultiError) Error() string { + msgs := make([]string, 0, len(m)) + for _, err := range m { + msgs = append(msgs, err.Error()) + } + return strings.Join(msgs, "; ") +} + +// AllErrors returns a list of validation violation errors. +func (m SetThresholdResponseMultiError) AllErrors() []error { return m } + +// SetThresholdResponseValidationError is the validation error returned by +// SetThresholdResponse.Validate if the designated constraints aren't met. +type SetThresholdResponseValidationError struct { + field string + reason string + cause error + key bool +} + +// Field function returns field value. +func (e SetThresholdResponseValidationError) Field() string { return e.field } + +// Reason function returns reason value. +func (e SetThresholdResponseValidationError) Reason() string { return e.reason } + +// Cause function returns cause value. +func (e SetThresholdResponseValidationError) Cause() error { return e.cause } + +// Key function returns key value. +func (e SetThresholdResponseValidationError) Key() bool { return e.key } + +// ErrorName returns error name. +func (e SetThresholdResponseValidationError) ErrorName() string { + return "SetThresholdResponseValidationError" +} + +// Error satisfies the builtin error interface +func (e SetThresholdResponseValidationError) Error() string { + cause := "" + if e.cause != nil { + cause = fmt.Sprintf(" | caused by: %v", e.cause) + } + + key := "" + if e.key { + key = "key for " + } + + return fmt.Sprintf( + "invalid %sSetThresholdResponse.%s: %s%s", + key, + e.field, + e.reason, + cause, + ) +} + +var _ error = SetThresholdResponseValidationError{} + +var _ interface { + Field() string + Reason() string + Key() bool + Cause() error + ErrorName() string +} = SetThresholdResponseValidationError{} + +// Validate checks the field values on ClearThresholdRequest with the rules +// defined in the proto definition for this message. If any rules are +// violated, the first error encountered is returned, or nil if there are no violations. +func (m *ClearThresholdRequest) Validate() error { + return m.validate(false) +} + +// ValidateAll checks the field values on ClearThresholdRequest with the rules +// defined in the proto definition for this message. If any rules are +// violated, the result is a list of violation errors wrapped in +// ClearThresholdRequestMultiError, or nil if none found. +func (m *ClearThresholdRequest) ValidateAll() error { + return m.validate(true) +} + +func (m *ClearThresholdRequest) validate(all bool) error { + if m == nil { + return nil + } + + var errors []error + + // no validation rules for Scope + + if utf8.RuneCountInString(m.GetTarget()) < 1 { + err := ClearThresholdRequestValidationError{ + field: "Target", + reason: "value length must be at least 1 runes", + } + if !all { + return err + } + errors = append(errors, err) + } + + if utf8.RuneCountInString(m.GetRuleId()) < 1 { + err := ClearThresholdRequestValidationError{ + field: "RuleId", + reason: "value length must be at least 1 runes", + } + if !all { + return err + } + errors = append(errors, err) + } + + if utf8.RuneCountInString(m.GetParamName()) < 1 { + err := ClearThresholdRequestValidationError{ + field: "ParamName", + reason: "value length must be at least 1 runes", + } + if !all { + return err + } + errors = append(errors, err) + } + + if len(errors) > 0 { + return ClearThresholdRequestMultiError(errors) + } + + return nil +} + +// ClearThresholdRequestMultiError is an error wrapping multiple validation +// errors returned by ClearThresholdRequest.ValidateAll() if the designated +// constraints aren't met. +type ClearThresholdRequestMultiError []error + +// Error returns a concatenation of all the error messages it wraps. +func (m ClearThresholdRequestMultiError) Error() string { + msgs := make([]string, 0, len(m)) + for _, err := range m { + msgs = append(msgs, err.Error()) + } + return strings.Join(msgs, "; ") +} + +// AllErrors returns a list of validation violation errors. +func (m ClearThresholdRequestMultiError) AllErrors() []error { return m } + +// ClearThresholdRequestValidationError is the validation error returned by +// ClearThresholdRequest.Validate if the designated constraints aren't met. +type ClearThresholdRequestValidationError struct { + field string + reason string + cause error + key bool +} + +// Field function returns field value. +func (e ClearThresholdRequestValidationError) Field() string { return e.field } + +// Reason function returns reason value. +func (e ClearThresholdRequestValidationError) Reason() string { return e.reason } + +// Cause function returns cause value. +func (e ClearThresholdRequestValidationError) Cause() error { return e.cause } + +// Key function returns key value. +func (e ClearThresholdRequestValidationError) Key() bool { return e.key } + +// ErrorName returns error name. +func (e ClearThresholdRequestValidationError) ErrorName() string { + return "ClearThresholdRequestValidationError" +} + +// Error satisfies the builtin error interface +func (e ClearThresholdRequestValidationError) Error() string { + cause := "" + if e.cause != nil { + cause = fmt.Sprintf(" | caused by: %v", e.cause) + } + + key := "" + if e.key { + key = "key for " + } + + return fmt.Sprintf( + "invalid %sClearThresholdRequest.%s: %s%s", + key, + e.field, + e.reason, + cause, + ) +} + +var _ error = ClearThresholdRequestValidationError{} + +var _ interface { + Field() string + Reason() string + Key() bool + Cause() error + ErrorName() string +} = ClearThresholdRequestValidationError{} + +// Validate checks the field values on ClearThresholdResponse with the rules +// defined in the proto definition for this message. If any rules are +// violated, the first error encountered is returned, or nil if there are no violations. +func (m *ClearThresholdResponse) Validate() error { + return m.validate(false) +} + +// ValidateAll checks the field values on ClearThresholdResponse with the rules +// defined in the proto definition for this message. If any rules are +// violated, the result is a list of violation errors wrapped in +// ClearThresholdResponseMultiError, or nil if none found. +func (m *ClearThresholdResponse) ValidateAll() error { + return m.validate(true) +} + +func (m *ClearThresholdResponse) validate(all bool) error { + if m == nil { + return nil + } + + var errors []error + + if len(errors) > 0 { + return ClearThresholdResponseMultiError(errors) + } + + return nil +} + +// ClearThresholdResponseMultiError is an error wrapping multiple validation +// errors returned by ClearThresholdResponse.ValidateAll() if the designated +// constraints aren't met. +type ClearThresholdResponseMultiError []error + +// Error returns a concatenation of all the error messages it wraps. +func (m ClearThresholdResponseMultiError) Error() string { + msgs := make([]string, 0, len(m)) + for _, err := range m { + msgs = append(msgs, err.Error()) + } + return strings.Join(msgs, "; ") +} + +// AllErrors returns a list of validation violation errors. +func (m ClearThresholdResponseMultiError) AllErrors() []error { return m } + +// ClearThresholdResponseValidationError is the validation error returned by +// ClearThresholdResponse.Validate if the designated constraints aren't met. +type ClearThresholdResponseValidationError struct { + field string + reason string + cause error + key bool +} + +// Field function returns field value. +func (e ClearThresholdResponseValidationError) Field() string { return e.field } + +// Reason function returns reason value. +func (e ClearThresholdResponseValidationError) Reason() string { return e.reason } + +// Cause function returns cause value. +func (e ClearThresholdResponseValidationError) Cause() error { return e.cause } + +// Key function returns key value. +func (e ClearThresholdResponseValidationError) Key() bool { return e.key } + +// ErrorName returns error name. +func (e ClearThresholdResponseValidationError) ErrorName() string { + return "ClearThresholdResponseValidationError" +} + +// Error satisfies the builtin error interface +func (e ClearThresholdResponseValidationError) Error() string { + cause := "" + if e.cause != nil { + cause = fmt.Sprintf(" | caused by: %v", e.cause) + } + + key := "" + if e.key { + key = "key for " + } + + return fmt.Sprintf( + "invalid %sClearThresholdResponse.%s: %s%s", + key, + e.field, + e.reason, + cause, + ) +} + +var _ error = ClearThresholdResponseValidationError{} + +var _ interface { + Field() string + Reason() string + Key() bool + Cause() error + ErrorName() string +} = ClearThresholdResponseValidationError{} + +// Validate checks the field values on ThresholdUpdate with the rules defined +// in the proto definition for this message. If any rules are violated, the +// first error encountered is returned, or nil if there are no violations. +func (m *ThresholdUpdate) Validate() error { + return m.validate(false) +} + +// ValidateAll checks the field values on ThresholdUpdate with the rules +// defined in the proto definition for this message. If any rules are +// violated, the result is a list of violation errors wrapped in +// ThresholdUpdateMultiError, or nil if none found. +func (m *ThresholdUpdate) ValidateAll() error { + return m.validate(true) +} + +func (m *ThresholdUpdate) validate(all bool) error { + if m == nil { + return nil + } + + var errors []error + + // no validation rules for Scope + + if utf8.RuneCountInString(m.GetTarget()) < 1 { + err := ThresholdUpdateValidationError{ + field: "Target", + reason: "value length must be at least 1 runes", + } + if !all { + return err + } + errors = append(errors, err) + } + + if utf8.RuneCountInString(m.GetRuleId()) < 1 { + err := ThresholdUpdateValidationError{ + field: "RuleId", + reason: "value length must be at least 1 runes", + } + if !all { + return err + } + errors = append(errors, err) + } + + if utf8.RuneCountInString(m.GetParamName()) < 1 { + err := ThresholdUpdateValidationError{ + field: "ParamName", + reason: "value length must be at least 1 runes", + } + if !all { + return err + } + errors = append(errors, err) + } + + if m.Value != nil { + // no validation rules for Value + } + + if len(errors) > 0 { + return ThresholdUpdateMultiError(errors) + } + + return nil +} + +// ThresholdUpdateMultiError is an error wrapping multiple validation errors +// returned by ThresholdUpdate.ValidateAll() if the designated constraints +// aren't met. +type ThresholdUpdateMultiError []error + +// Error returns a concatenation of all the error messages it wraps. +func (m ThresholdUpdateMultiError) Error() string { + msgs := make([]string, 0, len(m)) + for _, err := range m { + msgs = append(msgs, err.Error()) + } + return strings.Join(msgs, "; ") +} + +// AllErrors returns a list of validation violation errors. +func (m ThresholdUpdateMultiError) AllErrors() []error { return m } + +// ThresholdUpdateValidationError is the validation error returned by +// ThresholdUpdate.Validate if the designated constraints aren't met. +type ThresholdUpdateValidationError struct { + field string + reason string + cause error + key bool +} + +// Field function returns field value. +func (e ThresholdUpdateValidationError) Field() string { return e.field } + +// Reason function returns reason value. +func (e ThresholdUpdateValidationError) Reason() string { return e.reason } + +// Cause function returns cause value. +func (e ThresholdUpdateValidationError) Cause() error { return e.cause } + +// Key function returns key value. +func (e ThresholdUpdateValidationError) Key() bool { return e.key } + +// ErrorName returns error name. +func (e ThresholdUpdateValidationError) ErrorName() string { return "ThresholdUpdateValidationError" } + +// Error satisfies the builtin error interface +func (e ThresholdUpdateValidationError) Error() string { + cause := "" + if e.cause != nil { + cause = fmt.Sprintf(" | caused by: %v", e.cause) + } + + key := "" + if e.key { + key = "key for " + } + + return fmt.Sprintf( + "invalid %sThresholdUpdate.%s: %s%s", + key, + e.field, + e.reason, + cause, + ) +} + +var _ error = ThresholdUpdateValidationError{} + +var _ interface { + Field() string + Reason() string + Key() bool + Cause() error + ErrorName() string +} = ThresholdUpdateValidationError{} + +// Validate checks the field values on BatchUpdateThresholdsRequest with the +// rules defined in the proto definition for this message. If any rules are +// violated, the first error encountered is returned, or nil if there are no violations. +func (m *BatchUpdateThresholdsRequest) Validate() error { + return m.validate(false) +} + +// ValidateAll checks the field values on BatchUpdateThresholdsRequest with the +// rules defined in the proto definition for this message. If any rules are +// violated, the result is a list of violation errors wrapped in +// BatchUpdateThresholdsRequestMultiError, or nil if none found. +func (m *BatchUpdateThresholdsRequest) ValidateAll() error { + return m.validate(true) +} + +func (m *BatchUpdateThresholdsRequest) validate(all bool) error { + if m == nil { + return nil + } + + var errors []error + + if len(m.GetUpdates()) < 1 { + err := BatchUpdateThresholdsRequestValidationError{ + field: "Updates", + reason: "value must contain at least 1 item(s)", + } + if !all { + return err + } + errors = append(errors, err) + } + + for idx, item := range m.GetUpdates() { + _, _ = idx, item + + if all { + switch v := interface{}(item).(type) { + case interface{ ValidateAll() error }: + if err := v.ValidateAll(); err != nil { + errors = append(errors, BatchUpdateThresholdsRequestValidationError{ + field: fmt.Sprintf("Updates[%v]", idx), + reason: "embedded message failed validation", + cause: err, + }) + } + case interface{ Validate() error }: + if err := v.Validate(); err != nil { + errors = append(errors, BatchUpdateThresholdsRequestValidationError{ + field: fmt.Sprintf("Updates[%v]", idx), + reason: "embedded message failed validation", + cause: err, + }) + } + } + } else if v, ok := interface{}(item).(interface{ Validate() error }); ok { + if err := v.Validate(); err != nil { + return BatchUpdateThresholdsRequestValidationError{ + field: fmt.Sprintf("Updates[%v]", idx), + reason: "embedded message failed validation", + cause: err, + } + } + } + + } + + if len(errors) > 0 { + return BatchUpdateThresholdsRequestMultiError(errors) + } + + return nil +} + +// BatchUpdateThresholdsRequestMultiError is an error wrapping multiple +// validation errors returned by BatchUpdateThresholdsRequest.ValidateAll() if +// the designated constraints aren't met. +type BatchUpdateThresholdsRequestMultiError []error + +// Error returns a concatenation of all the error messages it wraps. +func (m BatchUpdateThresholdsRequestMultiError) Error() string { + msgs := make([]string, 0, len(m)) + for _, err := range m { + msgs = append(msgs, err.Error()) + } + return strings.Join(msgs, "; ") +} + +// AllErrors returns a list of validation violation errors. +func (m BatchUpdateThresholdsRequestMultiError) AllErrors() []error { return m } + +// BatchUpdateThresholdsRequestValidationError is the validation error returned +// by BatchUpdateThresholdsRequest.Validate if the designated constraints +// aren't met. +type BatchUpdateThresholdsRequestValidationError struct { + field string + reason string + cause error + key bool +} + +// Field function returns field value. +func (e BatchUpdateThresholdsRequestValidationError) Field() string { return e.field } + +// Reason function returns reason value. +func (e BatchUpdateThresholdsRequestValidationError) Reason() string { return e.reason } + +// Cause function returns cause value. +func (e BatchUpdateThresholdsRequestValidationError) Cause() error { return e.cause } + +// Key function returns key value. +func (e BatchUpdateThresholdsRequestValidationError) Key() bool { return e.key } + +// ErrorName returns error name. +func (e BatchUpdateThresholdsRequestValidationError) ErrorName() string { + return "BatchUpdateThresholdsRequestValidationError" +} + +// Error satisfies the builtin error interface +func (e BatchUpdateThresholdsRequestValidationError) Error() string { + cause := "" + if e.cause != nil { + cause = fmt.Sprintf(" | caused by: %v", e.cause) + } + + key := "" + if e.key { + key = "key for " + } + + return fmt.Sprintf( + "invalid %sBatchUpdateThresholdsRequest.%s: %s%s", + key, + e.field, + e.reason, + cause, + ) +} + +var _ error = BatchUpdateThresholdsRequestValidationError{} + +var _ interface { + Field() string + Reason() string + Key() bool + Cause() error + ErrorName() string +} = BatchUpdateThresholdsRequestValidationError{} + +// Validate checks the field values on BatchUpdateThresholdsResponse with the +// rules defined in the proto definition for this message. If any rules are +// violated, the first error encountered is returned, or nil if there are no violations. +func (m *BatchUpdateThresholdsResponse) Validate() error { + return m.validate(false) +} + +// ValidateAll checks the field values on BatchUpdateThresholdsResponse with +// the rules defined in the proto definition for this message. If any rules +// are violated, the result is a list of violation errors wrapped in +// BatchUpdateThresholdsResponseMultiError, or nil if none found. +func (m *BatchUpdateThresholdsResponse) ValidateAll() error { + return m.validate(true) +} + +func (m *BatchUpdateThresholdsResponse) validate(all bool) error { + if m == nil { + return nil + } + + var errors []error + + for idx, item := range m.GetThresholds() { + _, _ = idx, item + + if all { + switch v := interface{}(item).(type) { + case interface{ ValidateAll() error }: + if err := v.ValidateAll(); err != nil { + errors = append(errors, BatchUpdateThresholdsResponseValidationError{ + field: fmt.Sprintf("Thresholds[%v]", idx), + reason: "embedded message failed validation", + cause: err, + }) + } + case interface{ Validate() error }: + if err := v.Validate(); err != nil { + errors = append(errors, BatchUpdateThresholdsResponseValidationError{ + field: fmt.Sprintf("Thresholds[%v]", idx), + reason: "embedded message failed validation", + cause: err, + }) + } + } + } else if v, ok := interface{}(item).(interface{ Validate() error }); ok { + if err := v.Validate(); err != nil { + return BatchUpdateThresholdsResponseValidationError{ + field: fmt.Sprintf("Thresholds[%v]", idx), + reason: "embedded message failed validation", + cause: err, + } + } + } + + } + + if len(errors) > 0 { + return BatchUpdateThresholdsResponseMultiError(errors) + } + + return nil +} + +// BatchUpdateThresholdsResponseMultiError is an error wrapping multiple +// validation errors returned by BatchUpdateThresholdsResponse.ValidateAll() +// if the designated constraints aren't met. +type BatchUpdateThresholdsResponseMultiError []error + +// Error returns a concatenation of all the error messages it wraps. +func (m BatchUpdateThresholdsResponseMultiError) Error() string { + msgs := make([]string, 0, len(m)) + for _, err := range m { + msgs = append(msgs, err.Error()) + } + return strings.Join(msgs, "; ") +} + +// AllErrors returns a list of validation violation errors. +func (m BatchUpdateThresholdsResponseMultiError) AllErrors() []error { return m } + +// BatchUpdateThresholdsResponseValidationError is the validation error +// returned by BatchUpdateThresholdsResponse.Validate if the designated +// constraints aren't met. +type BatchUpdateThresholdsResponseValidationError struct { + field string + reason string + cause error + key bool +} + +// Field function returns field value. +func (e BatchUpdateThresholdsResponseValidationError) Field() string { return e.field } + +// Reason function returns reason value. +func (e BatchUpdateThresholdsResponseValidationError) Reason() string { return e.reason } + +// Cause function returns cause value. +func (e BatchUpdateThresholdsResponseValidationError) Cause() error { return e.cause } + +// Key function returns key value. +func (e BatchUpdateThresholdsResponseValidationError) Key() bool { return e.key } + +// ErrorName returns error name. +func (e BatchUpdateThresholdsResponseValidationError) ErrorName() string { + return "BatchUpdateThresholdsResponseValidationError" +} + +// Error satisfies the builtin error interface +func (e BatchUpdateThresholdsResponseValidationError) Error() string { + cause := "" + if e.cause != nil { + cause = fmt.Sprintf(" | caused by: %v", e.cause) + } + + key := "" + if e.key { + key = "key for " + } + + return fmt.Sprintf( + "invalid %sBatchUpdateThresholdsResponse.%s: %s%s", + key, + e.field, + e.reason, + cause, + ) +} + +var _ error = BatchUpdateThresholdsResponseValidationError{} + +var _ interface { + Field() string + Reason() string + Key() bool + Cause() error + ErrorName() string +} = BatchUpdateThresholdsResponseValidationError{} diff --git a/api/alerting/v1/alerting.proto b/api/alerting/v1/alerting.proto index 60876174af..e905bc56e6 100644 --- a/api/alerting/v1/alerting.proto +++ b/api/alerting/v1/alerting.proto @@ -216,6 +216,99 @@ message CreateRuleResponse { string rule_id = 1; } +// ThresholdScope says what a threshold override's target refers to. +enum ThresholdScope { + THRESHOLD_SCOPE_UNSPECIFIED = 0; + // Target is a Node ID. + THRESHOLD_SCOPE_NODE = 1; + // Target is a Service ID. + THRESHOLD_SCOPE_SERVICE = 2; + // Target is a cluster label value. + THRESHOLD_SCOPE_CLUSTER = 3; +} + +// Threshold is one overridable parameter of one rule, as it applies to one target. +message Threshold { + // Identifier PMM assigned to the rule. Not unique within a response: rules duplicated + // in Grafana share it, so two entries can carry the same rule_id and param_name and + // differ only in which rule they came from. Do not key a map on it. + string rule_id = 1; + // Machine-readable name of the overridable parameter. + string param_name = 2; + // Short human-readable parameter summary, as it was when the rule was created. + string summary = 3; + // Parameter unit. + ParamUnit unit = 4; + // Value the rule falls back to when no override applies. + double default_value = 5; + // Value the rule currently evaluates this target against. + double effective_value = 6; + // Whether effective_value comes from an override rather than the default. + bool is_overridden = 7; + // Scope the effective override was set at. Unspecified when not overridden. + ThresholdScope scope = 8; + // Target the effective override was set on. Empty when not overridden. + string target = 9; +} + +message ListThresholdsRequest { + // Scope of the target to report thresholds for. Must be set together with target. + ThresholdScope scope = 1; + // Target to report thresholds for. When set, every overridable parameter is returned + // for that target, overridden or not. When empty, only existing overrides are + // returned, since there is otherwise no bounded set to enumerate. + string target = 2; + // Return only thresholds of this rule. + string rule_id = 3; +} + +message ListThresholdsResponse { + repeated Threshold thresholds = 1; +} + +message SetThresholdRequest { + ThresholdScope scope = 1; + string target = 2 [(validate.rules).string.min_len = 1]; + string rule_id = 3 [(validate.rules).string.min_len = 1]; + string param_name = 4 [(validate.rules).string.min_len = 1]; + // Must be finite and within the parameter's declared range. + double value = 5; +} + +message SetThresholdResponse { + Threshold threshold = 1; +} + +message ClearThresholdRequest { + ThresholdScope scope = 1; + string target = 2 [(validate.rules).string.min_len = 1]; + string rule_id = 3 [(validate.rules).string.min_len = 1]; + string param_name = 4 [(validate.rules).string.min_len = 1]; +} + +message ClearThresholdResponse {} + +// ThresholdUpdate sets or clears one override. +message ThresholdUpdate { + ThresholdScope scope = 1; + string target = 2 [(validate.rules).string.min_len = 1]; + string rule_id = 3 [(validate.rules).string.min_len = 1]; + string param_name = 4 [(validate.rules).string.min_len = 1]; + // Omit to clear the override rather than set it. + optional double value = 5; +} + +message BatchUpdateThresholdsRequest { + // Applied in one transaction: either every update lands or none does. A client + // editing several rows at once cannot otherwise report which ones took effect. + repeated ThresholdUpdate updates = 1 [(validate.rules).repeated.min_items = 1]; +} + +message BatchUpdateThresholdsResponse { + // Thresholds that were set, in request order. Cleared ones are omitted. + repeated Threshold thresholds = 1; +} + // Alerting service lets to manage alerting templates and create alerting rules from them. service AlertingService { // ListTemplates returns a list of all collected alert rule templates. @@ -247,4 +340,27 @@ service AlertingService { body: "*" }; } + // ListThresholds returns per-target threshold overrides. + rpc ListThresholds(ListThresholdsRequest) returns (ListThresholdsResponse) { + option (google.api.http) = {get: "/v1/alerting/thresholds"}; + } + // SetThreshold overrides one rule parameter for one target. + rpc SetThreshold(SetThresholdRequest) returns (SetThresholdResponse) { + option (google.api.http) = { + post: "/v1/alerting/thresholds" + body: "*" + }; + } + // ClearThreshold removes an override so the target falls back to the rule's default, + // or to a broader override still covering it. + rpc ClearThreshold(ClearThresholdRequest) returns (ClearThresholdResponse) { + option (google.api.http) = {delete: "/v1/alerting/thresholds"}; + } + // BatchUpdateThresholds applies several set and clear operations in one transaction. + rpc BatchUpdateThresholds(BatchUpdateThresholdsRequest) returns (BatchUpdateThresholdsResponse) { + option (google.api.http) = { + post: "/v1/alerting/thresholds:batchUpdate" + body: "*" + }; + } } diff --git a/api/alerting/v1/alerting_grpc.pb.go b/api/alerting/v1/alerting_grpc.pb.go index d559dc19bf..18c1f79db2 100644 --- a/api/alerting/v1/alerting_grpc.pb.go +++ b/api/alerting/v1/alerting_grpc.pb.go @@ -20,11 +20,15 @@ import ( const _ = grpc.SupportPackageIsVersion9 const ( - AlertingService_ListTemplates_FullMethodName = "/alerting.v1.AlertingService/ListTemplates" - AlertingService_CreateTemplate_FullMethodName = "/alerting.v1.AlertingService/CreateTemplate" - AlertingService_UpdateTemplate_FullMethodName = "/alerting.v1.AlertingService/UpdateTemplate" - AlertingService_DeleteTemplate_FullMethodName = "/alerting.v1.AlertingService/DeleteTemplate" - AlertingService_CreateRule_FullMethodName = "/alerting.v1.AlertingService/CreateRule" + AlertingService_ListTemplates_FullMethodName = "/alerting.v1.AlertingService/ListTemplates" + AlertingService_CreateTemplate_FullMethodName = "/alerting.v1.AlertingService/CreateTemplate" + AlertingService_UpdateTemplate_FullMethodName = "/alerting.v1.AlertingService/UpdateTemplate" + AlertingService_DeleteTemplate_FullMethodName = "/alerting.v1.AlertingService/DeleteTemplate" + AlertingService_CreateRule_FullMethodName = "/alerting.v1.AlertingService/CreateRule" + AlertingService_ListThresholds_FullMethodName = "/alerting.v1.AlertingService/ListThresholds" + AlertingService_SetThreshold_FullMethodName = "/alerting.v1.AlertingService/SetThreshold" + AlertingService_ClearThreshold_FullMethodName = "/alerting.v1.AlertingService/ClearThreshold" + AlertingService_BatchUpdateThresholds_FullMethodName = "/alerting.v1.AlertingService/BatchUpdateThresholds" ) // AlertingServiceClient is the client API for AlertingService service. @@ -43,6 +47,15 @@ type AlertingServiceClient interface { DeleteTemplate(ctx context.Context, in *DeleteTemplateRequest, opts ...grpc.CallOption) (*DeleteTemplateResponse, error) // CreateRule creates alerting rule from the given template. CreateRule(ctx context.Context, in *CreateRuleRequest, opts ...grpc.CallOption) (*CreateRuleResponse, error) + // ListThresholds returns per-target threshold overrides. + ListThresholds(ctx context.Context, in *ListThresholdsRequest, opts ...grpc.CallOption) (*ListThresholdsResponse, error) + // SetThreshold overrides one rule parameter for one target. + SetThreshold(ctx context.Context, in *SetThresholdRequest, opts ...grpc.CallOption) (*SetThresholdResponse, error) + // ClearThreshold removes an override so the target falls back to the rule's default, + // or to a broader override still covering it. + ClearThreshold(ctx context.Context, in *ClearThresholdRequest, opts ...grpc.CallOption) (*ClearThresholdResponse, error) + // BatchUpdateThresholds applies several set and clear operations in one transaction. + BatchUpdateThresholds(ctx context.Context, in *BatchUpdateThresholdsRequest, opts ...grpc.CallOption) (*BatchUpdateThresholdsResponse, error) } type alertingServiceClient struct { @@ -103,6 +116,46 @@ func (c *alertingServiceClient) CreateRule(ctx context.Context, in *CreateRuleRe return out, nil } +func (c *alertingServiceClient) ListThresholds(ctx context.Context, in *ListThresholdsRequest, opts ...grpc.CallOption) (*ListThresholdsResponse, error) { + cOpts := append([]grpc.CallOption{grpc.StaticMethod()}, opts...) + out := new(ListThresholdsResponse) + err := c.cc.Invoke(ctx, AlertingService_ListThresholds_FullMethodName, in, out, cOpts...) + if err != nil { + return nil, err + } + return out, nil +} + +func (c *alertingServiceClient) SetThreshold(ctx context.Context, in *SetThresholdRequest, opts ...grpc.CallOption) (*SetThresholdResponse, error) { + cOpts := append([]grpc.CallOption{grpc.StaticMethod()}, opts...) + out := new(SetThresholdResponse) + err := c.cc.Invoke(ctx, AlertingService_SetThreshold_FullMethodName, in, out, cOpts...) + if err != nil { + return nil, err + } + return out, nil +} + +func (c *alertingServiceClient) ClearThreshold(ctx context.Context, in *ClearThresholdRequest, opts ...grpc.CallOption) (*ClearThresholdResponse, error) { + cOpts := append([]grpc.CallOption{grpc.StaticMethod()}, opts...) + out := new(ClearThresholdResponse) + err := c.cc.Invoke(ctx, AlertingService_ClearThreshold_FullMethodName, in, out, cOpts...) + if err != nil { + return nil, err + } + return out, nil +} + +func (c *alertingServiceClient) BatchUpdateThresholds(ctx context.Context, in *BatchUpdateThresholdsRequest, opts ...grpc.CallOption) (*BatchUpdateThresholdsResponse, error) { + cOpts := append([]grpc.CallOption{grpc.StaticMethod()}, opts...) + out := new(BatchUpdateThresholdsResponse) + err := c.cc.Invoke(ctx, AlertingService_BatchUpdateThresholds_FullMethodName, in, out, cOpts...) + if err != nil { + return nil, err + } + return out, nil +} + // AlertingServiceServer is the server API for AlertingService service. // All implementations must embed UnimplementedAlertingServiceServer // for forward compatibility. @@ -119,6 +172,15 @@ type AlertingServiceServer interface { DeleteTemplate(context.Context, *DeleteTemplateRequest) (*DeleteTemplateResponse, error) // CreateRule creates alerting rule from the given template. CreateRule(context.Context, *CreateRuleRequest) (*CreateRuleResponse, error) + // ListThresholds returns per-target threshold overrides. + ListThresholds(context.Context, *ListThresholdsRequest) (*ListThresholdsResponse, error) + // SetThreshold overrides one rule parameter for one target. + SetThreshold(context.Context, *SetThresholdRequest) (*SetThresholdResponse, error) + // ClearThreshold removes an override so the target falls back to the rule's default, + // or to a broader override still covering it. + ClearThreshold(context.Context, *ClearThresholdRequest) (*ClearThresholdResponse, error) + // BatchUpdateThresholds applies several set and clear operations in one transaction. + BatchUpdateThresholds(context.Context, *BatchUpdateThresholdsRequest) (*BatchUpdateThresholdsResponse, error) mustEmbedUnimplementedAlertingServiceServer() } @@ -148,6 +210,22 @@ func (UnimplementedAlertingServiceServer) DeleteTemplate(context.Context, *Delet func (UnimplementedAlertingServiceServer) CreateRule(context.Context, *CreateRuleRequest) (*CreateRuleResponse, error) { return nil, status.Error(codes.Unimplemented, "method CreateRule not implemented") } + +func (UnimplementedAlertingServiceServer) ListThresholds(context.Context, *ListThresholdsRequest) (*ListThresholdsResponse, error) { + return nil, status.Error(codes.Unimplemented, "method ListThresholds not implemented") +} + +func (UnimplementedAlertingServiceServer) SetThreshold(context.Context, *SetThresholdRequest) (*SetThresholdResponse, error) { + return nil, status.Error(codes.Unimplemented, "method SetThreshold not implemented") +} + +func (UnimplementedAlertingServiceServer) ClearThreshold(context.Context, *ClearThresholdRequest) (*ClearThresholdResponse, error) { + return nil, status.Error(codes.Unimplemented, "method ClearThreshold not implemented") +} + +func (UnimplementedAlertingServiceServer) BatchUpdateThresholds(context.Context, *BatchUpdateThresholdsRequest) (*BatchUpdateThresholdsResponse, error) { + return nil, status.Error(codes.Unimplemented, "method BatchUpdateThresholds not implemented") +} func (UnimplementedAlertingServiceServer) mustEmbedUnimplementedAlertingServiceServer() {} func (UnimplementedAlertingServiceServer) testEmbeddedByValue() {} @@ -259,6 +337,78 @@ func _AlertingService_CreateRule_Handler(srv interface{}, ctx context.Context, d return interceptor(ctx, in, info, handler) } +func _AlertingService_ListThresholds_Handler(srv interface{}, ctx context.Context, dec func(interface{}) error, interceptor grpc.UnaryServerInterceptor) (interface{}, error) { + in := new(ListThresholdsRequest) + if err := dec(in); err != nil { + return nil, err + } + if interceptor == nil { + return srv.(AlertingServiceServer).ListThresholds(ctx, in) + } + info := &grpc.UnaryServerInfo{ + Server: srv, + FullMethod: AlertingService_ListThresholds_FullMethodName, + } + handler := func(ctx context.Context, req interface{}) (interface{}, error) { + return srv.(AlertingServiceServer).ListThresholds(ctx, req.(*ListThresholdsRequest)) + } + return interceptor(ctx, in, info, handler) +} + +func _AlertingService_SetThreshold_Handler(srv interface{}, ctx context.Context, dec func(interface{}) error, interceptor grpc.UnaryServerInterceptor) (interface{}, error) { + in := new(SetThresholdRequest) + if err := dec(in); err != nil { + return nil, err + } + if interceptor == nil { + return srv.(AlertingServiceServer).SetThreshold(ctx, in) + } + info := &grpc.UnaryServerInfo{ + Server: srv, + FullMethod: AlertingService_SetThreshold_FullMethodName, + } + handler := func(ctx context.Context, req interface{}) (interface{}, error) { + return srv.(AlertingServiceServer).SetThreshold(ctx, req.(*SetThresholdRequest)) + } + return interceptor(ctx, in, info, handler) +} + +func _AlertingService_ClearThreshold_Handler(srv interface{}, ctx context.Context, dec func(interface{}) error, interceptor grpc.UnaryServerInterceptor) (interface{}, error) { + in := new(ClearThresholdRequest) + if err := dec(in); err != nil { + return nil, err + } + if interceptor == nil { + return srv.(AlertingServiceServer).ClearThreshold(ctx, in) + } + info := &grpc.UnaryServerInfo{ + Server: srv, + FullMethod: AlertingService_ClearThreshold_FullMethodName, + } + handler := func(ctx context.Context, req interface{}) (interface{}, error) { + return srv.(AlertingServiceServer).ClearThreshold(ctx, req.(*ClearThresholdRequest)) + } + return interceptor(ctx, in, info, handler) +} + +func _AlertingService_BatchUpdateThresholds_Handler(srv interface{}, ctx context.Context, dec func(interface{}) error, interceptor grpc.UnaryServerInterceptor) (interface{}, error) { + in := new(BatchUpdateThresholdsRequest) + if err := dec(in); err != nil { + return nil, err + } + if interceptor == nil { + return srv.(AlertingServiceServer).BatchUpdateThresholds(ctx, in) + } + info := &grpc.UnaryServerInfo{ + Server: srv, + FullMethod: AlertingService_BatchUpdateThresholds_FullMethodName, + } + handler := func(ctx context.Context, req interface{}) (interface{}, error) { + return srv.(AlertingServiceServer).BatchUpdateThresholds(ctx, req.(*BatchUpdateThresholdsRequest)) + } + return interceptor(ctx, in, info, handler) +} + // AlertingService_ServiceDesc is the grpc.ServiceDesc for AlertingService service. // It's only intended for direct use with grpc.RegisterService, // and not to be introspected or modified (even as a copy) @@ -286,6 +436,22 @@ var AlertingService_ServiceDesc = grpc.ServiceDesc{ MethodName: "CreateRule", Handler: _AlertingService_CreateRule_Handler, }, + { + MethodName: "ListThresholds", + Handler: _AlertingService_ListThresholds_Handler, + }, + { + MethodName: "SetThreshold", + Handler: _AlertingService_SetThreshold_Handler, + }, + { + MethodName: "ClearThreshold", + Handler: _AlertingService_ClearThreshold_Handler, + }, + { + MethodName: "BatchUpdateThresholds", + Handler: _AlertingService_BatchUpdateThresholds_Handler, + }, }, Streams: []grpc.StreamDesc{}, Metadata: "alerting/v1/alerting.proto", diff --git a/api/alerting/v1/json/client/alerting_service/alerting_service_client.go b/api/alerting/v1/json/client/alerting_service/alerting_service_client.go index 40690a8da6..9fca248488 100644 --- a/api/alerting/v1/json/client/alerting_service/alerting_service_client.go +++ b/api/alerting/v1/json/client/alerting_service/alerting_service_client.go @@ -51,6 +51,10 @@ type ClientOption func(*runtime.ClientOperation) // ClientService is the interface for Client methods type ClientService interface { + BatchUpdateThresholds(params *BatchUpdateThresholdsParams, opts ...ClientOption) (*BatchUpdateThresholdsOK, error) + + ClearThreshold(params *ClearThresholdParams, opts ...ClientOption) (*ClearThresholdOK, error) + CreateRule(params *CreateRuleParams, opts ...ClientOption) (*CreateRuleOK, error) CreateTemplate(params *CreateTemplateParams, opts ...ClientOption) (*CreateTemplateOK, error) @@ -59,11 +63,99 @@ type ClientService interface { ListTemplates(params *ListTemplatesParams, opts ...ClientOption) (*ListTemplatesOK, error) + ListThresholds(params *ListThresholdsParams, opts ...ClientOption) (*ListThresholdsOK, error) + + SetThreshold(params *SetThresholdParams, opts ...ClientOption) (*SetThresholdOK, error) + UpdateTemplate(params *UpdateTemplateParams, opts ...ClientOption) (*UpdateTemplateOK, error) SetTransport(transport runtime.ClientTransport) } +/* +BatchUpdateThresholds batches update thresholds applies several set and clear operations in one transaction +*/ +func (a *Client) BatchUpdateThresholds(params *BatchUpdateThresholdsParams, opts ...ClientOption) (*BatchUpdateThresholdsOK, error) { + // NOTE: parameters are not validated before sending + if params == nil { + params = NewBatchUpdateThresholdsParams() + } + op := &runtime.ClientOperation{ + ID: "BatchUpdateThresholds", + Method: "POST", + PathPattern: "/v1/alerting/thresholds:batchUpdate", + ProducesMediaTypes: []string{"application/json"}, + ConsumesMediaTypes: []string{"application/json"}, + Schemes: []string{"http", "https"}, + Params: params, + Reader: &BatchUpdateThresholdsReader{formats: a.formats}, + Context: params.Context, + Client: params.HTTPClient, + } + for _, opt := range opts { + opt(op) + } + result, err := a.transport.Submit(op) + if err != nil { + return nil, err + } + + // only one success response has to be checked + success, ok := result.(*BatchUpdateThresholdsOK) + if ok { + return success, nil + } + + // unexpected success response. + // + // a default response is provided: fill this and return an error + unexpectedSuccess := result.(*BatchUpdateThresholdsDefault) + + return nil, runtime.NewAPIError("unexpected success response: content available as default response in error", unexpectedSuccess, unexpectedSuccess.Code()) +} + +/* +ClearThreshold clears threshold removes an override so the target falls back to the rule s default or to a broader override still covering it +*/ +func (a *Client) ClearThreshold(params *ClearThresholdParams, opts ...ClientOption) (*ClearThresholdOK, error) { + // NOTE: parameters are not validated before sending + if params == nil { + params = NewClearThresholdParams() + } + op := &runtime.ClientOperation{ + ID: "ClearThreshold", + Method: "DELETE", + PathPattern: "/v1/alerting/thresholds", + ProducesMediaTypes: []string{"application/json"}, + ConsumesMediaTypes: []string{"application/json"}, + Schemes: []string{"http", "https"}, + Params: params, + Reader: &ClearThresholdReader{formats: a.formats}, + Context: params.Context, + Client: params.HTTPClient, + } + for _, opt := range opts { + opt(op) + } + result, err := a.transport.Submit(op) + if err != nil { + return nil, err + } + + // only one success response has to be checked + success, ok := result.(*ClearThresholdOK) + if ok { + return success, nil + } + + // unexpected success response. + // + // a default response is provided: fill this and return an error + unexpectedSuccess := result.(*ClearThresholdDefault) + + return nil, runtime.NewAPIError("unexpected success response: content available as default response in error", unexpectedSuccess, unexpectedSuccess.Code()) +} + /* CreateRule creates rule creates alerting rule from the given template */ @@ -232,6 +324,90 @@ func (a *Client) ListTemplates(params *ListTemplatesParams, opts ...ClientOption return nil, runtime.NewAPIError("unexpected success response: content available as default response in error", unexpectedSuccess, unexpectedSuccess.Code()) } +/* +ListThresholds lists thresholds returns per target threshold overrides +*/ +func (a *Client) ListThresholds(params *ListThresholdsParams, opts ...ClientOption) (*ListThresholdsOK, error) { + // NOTE: parameters are not validated before sending + if params == nil { + params = NewListThresholdsParams() + } + op := &runtime.ClientOperation{ + ID: "ListThresholds", + Method: "GET", + PathPattern: "/v1/alerting/thresholds", + ProducesMediaTypes: []string{"application/json"}, + ConsumesMediaTypes: []string{"application/json"}, + Schemes: []string{"http", "https"}, + Params: params, + Reader: &ListThresholdsReader{formats: a.formats}, + Context: params.Context, + Client: params.HTTPClient, + } + for _, opt := range opts { + opt(op) + } + result, err := a.transport.Submit(op) + if err != nil { + return nil, err + } + + // only one success response has to be checked + success, ok := result.(*ListThresholdsOK) + if ok { + return success, nil + } + + // unexpected success response. + // + // a default response is provided: fill this and return an error + unexpectedSuccess := result.(*ListThresholdsDefault) + + return nil, runtime.NewAPIError("unexpected success response: content available as default response in error", unexpectedSuccess, unexpectedSuccess.Code()) +} + +/* +SetThreshold sets threshold overrides one rule parameter for one target +*/ +func (a *Client) SetThreshold(params *SetThresholdParams, opts ...ClientOption) (*SetThresholdOK, error) { + // NOTE: parameters are not validated before sending + if params == nil { + params = NewSetThresholdParams() + } + op := &runtime.ClientOperation{ + ID: "SetThreshold", + Method: "POST", + PathPattern: "/v1/alerting/thresholds", + ProducesMediaTypes: []string{"application/json"}, + ConsumesMediaTypes: []string{"application/json"}, + Schemes: []string{"http", "https"}, + Params: params, + Reader: &SetThresholdReader{formats: a.formats}, + Context: params.Context, + Client: params.HTTPClient, + } + for _, opt := range opts { + opt(op) + } + result, err := a.transport.Submit(op) + if err != nil { + return nil, err + } + + // only one success response has to be checked + success, ok := result.(*SetThresholdOK) + if ok { + return success, nil + } + + // unexpected success response. + // + // a default response is provided: fill this and return an error + unexpectedSuccess := result.(*SetThresholdDefault) + + return nil, runtime.NewAPIError("unexpected success response: content available as default response in error", unexpectedSuccess, unexpectedSuccess.Code()) +} + /* UpdateTemplate updates template updates existing template previously created via API */ diff --git a/api/alerting/v1/json/client/alerting_service/batch_update_thresholds_parameters.go b/api/alerting/v1/json/client/alerting_service/batch_update_thresholds_parameters.go new file mode 100644 index 0000000000..805fd11da4 --- /dev/null +++ b/api/alerting/v1/json/client/alerting_service/batch_update_thresholds_parameters.go @@ -0,0 +1,141 @@ +// Code generated by go-swagger; DO NOT EDIT. + +package alerting_service + +import ( + "context" + "net/http" + "time" + + "github.com/go-openapi/errors" + "github.com/go-openapi/runtime" + cr "github.com/go-openapi/runtime/client" + "github.com/go-openapi/strfmt" +) + +// NewBatchUpdateThresholdsParams creates a new BatchUpdateThresholdsParams object, +// with the default timeout for this client. +// +// Default values are not hydrated, since defaults are normally applied by the API server side. +// +// To enforce default values in parameter, use SetDefaults or WithDefaults. +func NewBatchUpdateThresholdsParams() *BatchUpdateThresholdsParams { + return &BatchUpdateThresholdsParams{ + timeout: cr.DefaultTimeout, + } +} + +// NewBatchUpdateThresholdsParamsWithTimeout creates a new BatchUpdateThresholdsParams object +// with the ability to set a timeout on a request. +func NewBatchUpdateThresholdsParamsWithTimeout(timeout time.Duration) *BatchUpdateThresholdsParams { + return &BatchUpdateThresholdsParams{ + timeout: timeout, + } +} + +// NewBatchUpdateThresholdsParamsWithContext creates a new BatchUpdateThresholdsParams object +// with the ability to set a context for a request. +func NewBatchUpdateThresholdsParamsWithContext(ctx context.Context) *BatchUpdateThresholdsParams { + return &BatchUpdateThresholdsParams{ + Context: ctx, + } +} + +// NewBatchUpdateThresholdsParamsWithHTTPClient creates a new BatchUpdateThresholdsParams object +// with the ability to set a custom HTTPClient for a request. +func NewBatchUpdateThresholdsParamsWithHTTPClient(client *http.Client) *BatchUpdateThresholdsParams { + return &BatchUpdateThresholdsParams{ + HTTPClient: client, + } +} + +/* +BatchUpdateThresholdsParams contains all the parameters to send to the API endpoint + + for the batch update thresholds operation. + + Typically these are written to a http.Request. +*/ +type BatchUpdateThresholdsParams struct { + // Body. + Body BatchUpdateThresholdsBody + + timeout time.Duration + Context context.Context + HTTPClient *http.Client +} + +// WithDefaults hydrates default values in the batch update thresholds params (not the query body). +// +// All values with no default are reset to their zero value. +func (o *BatchUpdateThresholdsParams) WithDefaults() *BatchUpdateThresholdsParams { + o.SetDefaults() + return o +} + +// SetDefaults hydrates default values in the batch update thresholds params (not the query body). +// +// All values with no default are reset to their zero value. +func (o *BatchUpdateThresholdsParams) SetDefaults() { + // no default values defined for this parameter +} + +// WithTimeout adds the timeout to the batch update thresholds params +func (o *BatchUpdateThresholdsParams) WithTimeout(timeout time.Duration) *BatchUpdateThresholdsParams { + o.SetTimeout(timeout) + return o +} + +// SetTimeout adds the timeout to the batch update thresholds params +func (o *BatchUpdateThresholdsParams) SetTimeout(timeout time.Duration) { + o.timeout = timeout +} + +// WithContext adds the context to the batch update thresholds params +func (o *BatchUpdateThresholdsParams) WithContext(ctx context.Context) *BatchUpdateThresholdsParams { + o.SetContext(ctx) + return o +} + +// SetContext adds the context to the batch update thresholds params +func (o *BatchUpdateThresholdsParams) SetContext(ctx context.Context) { + o.Context = ctx +} + +// WithHTTPClient adds the HTTPClient to the batch update thresholds params +func (o *BatchUpdateThresholdsParams) WithHTTPClient(client *http.Client) *BatchUpdateThresholdsParams { + o.SetHTTPClient(client) + return o +} + +// SetHTTPClient adds the HTTPClient to the batch update thresholds params +func (o *BatchUpdateThresholdsParams) SetHTTPClient(client *http.Client) { + o.HTTPClient = client +} + +// WithBody adds the body to the batch update thresholds params +func (o *BatchUpdateThresholdsParams) WithBody(body BatchUpdateThresholdsBody) *BatchUpdateThresholdsParams { + o.SetBody(body) + return o +} + +// SetBody adds the body to the batch update thresholds params +func (o *BatchUpdateThresholdsParams) SetBody(body BatchUpdateThresholdsBody) { + o.Body = body +} + +// WriteToRequest writes these params to a swagger request +func (o *BatchUpdateThresholdsParams) WriteToRequest(r runtime.ClientRequest, reg strfmt.Registry) error { + if err := r.SetTimeout(o.timeout); err != nil { + return err + } + var res []error + if err := r.SetBodyParam(o.Body); err != nil { + return err + } + + if len(res) > 0 { + return errors.CompositeValidationError(res...) + } + return nil +} diff --git a/api/alerting/v1/json/client/alerting_service/batch_update_thresholds_responses.go b/api/alerting/v1/json/client/alerting_service/batch_update_thresholds_responses.go new file mode 100644 index 0000000000..70435b6f4c --- /dev/null +++ b/api/alerting/v1/json/client/alerting_service/batch_update_thresholds_responses.go @@ -0,0 +1,929 @@ +// Code generated by go-swagger; DO NOT EDIT. + +package alerting_service + +import ( + "context" + "encoding/json" + stderrors "errors" + "fmt" + "io" + "strconv" + + "github.com/go-openapi/errors" + "github.com/go-openapi/runtime" + "github.com/go-openapi/strfmt" + "github.com/go-openapi/swag" + "github.com/go-openapi/validate" +) + +// BatchUpdateThresholdsReader is a Reader for the BatchUpdateThresholds structure. +type BatchUpdateThresholdsReader struct { + formats strfmt.Registry +} + +// ReadResponse reads a server response into the received o. +func (o *BatchUpdateThresholdsReader) ReadResponse(response runtime.ClientResponse, consumer runtime.Consumer) (any, error) { + switch response.Code() { + case 200: + result := NewBatchUpdateThresholdsOK() + if err := result.readResponse(response, consumer, o.formats); err != nil { + return nil, err + } + return result, nil + default: + result := NewBatchUpdateThresholdsDefault(response.Code()) + if err := result.readResponse(response, consumer, o.formats); err != nil { + return nil, err + } + if response.Code()/100 == 2 { + return result, nil + } + return nil, result + } +} + +// NewBatchUpdateThresholdsOK creates a BatchUpdateThresholdsOK with default headers values +func NewBatchUpdateThresholdsOK() *BatchUpdateThresholdsOK { + return &BatchUpdateThresholdsOK{} +} + +/* +BatchUpdateThresholdsOK describes a response with status code 200, with default header values. + +A successful response. +*/ +type BatchUpdateThresholdsOK struct { + Payload *BatchUpdateThresholdsOKBody +} + +// IsSuccess returns true when this batch update thresholds Ok response has a 2xx status code +func (o *BatchUpdateThresholdsOK) IsSuccess() bool { + return true +} + +// IsRedirect returns true when this batch update thresholds Ok response has a 3xx status code +func (o *BatchUpdateThresholdsOK) IsRedirect() bool { + return false +} + +// IsClientError returns true when this batch update thresholds Ok response has a 4xx status code +func (o *BatchUpdateThresholdsOK) IsClientError() bool { + return false +} + +// IsServerError returns true when this batch update thresholds Ok response has a 5xx status code +func (o *BatchUpdateThresholdsOK) IsServerError() bool { + return false +} + +// IsCode returns true when this batch update thresholds Ok response a status code equal to that given +func (o *BatchUpdateThresholdsOK) IsCode(code int) bool { + return code == 200 +} + +// Code gets the status code for the batch update thresholds Ok response +func (o *BatchUpdateThresholdsOK) Code() int { + return 200 +} + +func (o *BatchUpdateThresholdsOK) Error() string { + payload, _ := json.Marshal(o.Payload) + return fmt.Sprintf("[POST /v1/alerting/thresholds:batchUpdate][%d] batchUpdateThresholdsOk %s", 200, payload) +} + +func (o *BatchUpdateThresholdsOK) String() string { + payload, _ := json.Marshal(o.Payload) + return fmt.Sprintf("[POST /v1/alerting/thresholds:batchUpdate][%d] batchUpdateThresholdsOk %s", 200, payload) +} + +func (o *BatchUpdateThresholdsOK) GetPayload() *BatchUpdateThresholdsOKBody { + return o.Payload +} + +func (o *BatchUpdateThresholdsOK) readResponse(response runtime.ClientResponse, consumer runtime.Consumer, formats strfmt.Registry) error { + o.Payload = new(BatchUpdateThresholdsOKBody) + + // response payload + if err := consumer.Consume(response.Body(), o.Payload); err != nil && !stderrors.Is(err, io.EOF) { + return err + } + + return nil +} + +// NewBatchUpdateThresholdsDefault creates a BatchUpdateThresholdsDefault with default headers values +func NewBatchUpdateThresholdsDefault(code int) *BatchUpdateThresholdsDefault { + return &BatchUpdateThresholdsDefault{ + _statusCode: code, + } +} + +/* +BatchUpdateThresholdsDefault describes a response with status code -1, with default header values. + +An unexpected error response. +*/ +type BatchUpdateThresholdsDefault struct { + _statusCode int + + Payload *BatchUpdateThresholdsDefaultBody +} + +// IsSuccess returns true when this batch update thresholds default response has a 2xx status code +func (o *BatchUpdateThresholdsDefault) IsSuccess() bool { + return o._statusCode/100 == 2 +} + +// IsRedirect returns true when this batch update thresholds default response has a 3xx status code +func (o *BatchUpdateThresholdsDefault) IsRedirect() bool { + return o._statusCode/100 == 3 +} + +// IsClientError returns true when this batch update thresholds default response has a 4xx status code +func (o *BatchUpdateThresholdsDefault) IsClientError() bool { + return o._statusCode/100 == 4 +} + +// IsServerError returns true when this batch update thresholds default response has a 5xx status code +func (o *BatchUpdateThresholdsDefault) IsServerError() bool { + return o._statusCode/100 == 5 +} + +// IsCode returns true when this batch update thresholds default response a status code equal to that given +func (o *BatchUpdateThresholdsDefault) IsCode(code int) bool { + return o._statusCode == code +} + +// Code gets the status code for the batch update thresholds default response +func (o *BatchUpdateThresholdsDefault) Code() int { + return o._statusCode +} + +func (o *BatchUpdateThresholdsDefault) Error() string { + payload, _ := json.Marshal(o.Payload) + return fmt.Sprintf("[POST /v1/alerting/thresholds:batchUpdate][%d] BatchUpdateThresholds default %s", o._statusCode, payload) +} + +func (o *BatchUpdateThresholdsDefault) String() string { + payload, _ := json.Marshal(o.Payload) + return fmt.Sprintf("[POST /v1/alerting/thresholds:batchUpdate][%d] BatchUpdateThresholds default %s", o._statusCode, payload) +} + +func (o *BatchUpdateThresholdsDefault) GetPayload() *BatchUpdateThresholdsDefaultBody { + return o.Payload +} + +func (o *BatchUpdateThresholdsDefault) readResponse(response runtime.ClientResponse, consumer runtime.Consumer, formats strfmt.Registry) error { + o.Payload = new(BatchUpdateThresholdsDefaultBody) + + // response payload + if err := consumer.Consume(response.Body(), o.Payload); err != nil && !stderrors.Is(err, io.EOF) { + return err + } + + return nil +} + +/* +BatchUpdateThresholdsBody batch update thresholds body +swagger:model BatchUpdateThresholdsBody +*/ +type BatchUpdateThresholdsBody struct { + // Applied in one transaction: either every update lands or none does. A client + // editing several rows at once cannot otherwise report which ones took effect. + Updates []*BatchUpdateThresholdsParamsBodyUpdatesItems0 `json:"updates"` +} + +// Validate validates this batch update thresholds body +func (o *BatchUpdateThresholdsBody) Validate(formats strfmt.Registry) error { + var res []error + + if err := o.validateUpdates(formats); err != nil { + res = append(res, err) + } + + if len(res) > 0 { + return errors.CompositeValidationError(res...) + } + return nil +} + +func (o *BatchUpdateThresholdsBody) validateUpdates(formats strfmt.Registry) error { + if swag.IsZero(o.Updates) { // not required + return nil + } + + for i := 0; i < len(o.Updates); i++ { + if swag.IsZero(o.Updates[i]) { // not required + continue + } + + if o.Updates[i] != nil { + if err := o.Updates[i].Validate(formats); err != nil { + ve := new(errors.Validation) + if stderrors.As(err, &ve) { + return ve.ValidateName("body" + "." + "updates" + "." + strconv.Itoa(i)) + } + ce := new(errors.CompositeError) + if stderrors.As(err, &ce) { + return ce.ValidateName("body" + "." + "updates" + "." + strconv.Itoa(i)) + } + + return err + } + } + + } + + return nil +} + +// ContextValidate validate this batch update thresholds body based on the context it is used +func (o *BatchUpdateThresholdsBody) ContextValidate(ctx context.Context, formats strfmt.Registry) error { + var res []error + + if err := o.contextValidateUpdates(ctx, formats); err != nil { + res = append(res, err) + } + + if len(res) > 0 { + return errors.CompositeValidationError(res...) + } + return nil +} + +func (o *BatchUpdateThresholdsBody) contextValidateUpdates(ctx context.Context, formats strfmt.Registry) error { + for i := 0; i < len(o.Updates); i++ { + if o.Updates[i] != nil { + + if swag.IsZero(o.Updates[i]) { // not required + return nil + } + + if err := o.Updates[i].ContextValidate(ctx, formats); err != nil { + ve := new(errors.Validation) + if stderrors.As(err, &ve) { + return ve.ValidateName("body" + "." + "updates" + "." + strconv.Itoa(i)) + } + ce := new(errors.CompositeError) + if stderrors.As(err, &ce) { + return ce.ValidateName("body" + "." + "updates" + "." + strconv.Itoa(i)) + } + + return err + } + } + } + + return nil +} + +// MarshalBinary interface implementation +func (o *BatchUpdateThresholdsBody) MarshalBinary() ([]byte, error) { + if o == nil { + return nil, nil + } + return swag.WriteJSON(o) +} + +// UnmarshalBinary interface implementation +func (o *BatchUpdateThresholdsBody) UnmarshalBinary(b []byte) error { + var res BatchUpdateThresholdsBody + if err := swag.ReadJSON(b, &res); err != nil { + return err + } + *o = res + return nil +} + +/* +BatchUpdateThresholdsDefaultBody batch update thresholds default body +swagger:model BatchUpdateThresholdsDefaultBody +*/ +type BatchUpdateThresholdsDefaultBody struct { + // code + Code int32 `json:"code,omitempty"` + + // message + Message string `json:"message,omitempty"` + + // details + Details []*BatchUpdateThresholdsDefaultBodyDetailsItems0 `json:"details"` +} + +// Validate validates this batch update thresholds default body +func (o *BatchUpdateThresholdsDefaultBody) Validate(formats strfmt.Registry) error { + var res []error + + if err := o.validateDetails(formats); err != nil { + res = append(res, err) + } + + if len(res) > 0 { + return errors.CompositeValidationError(res...) + } + return nil +} + +func (o *BatchUpdateThresholdsDefaultBody) validateDetails(formats strfmt.Registry) error { + if swag.IsZero(o.Details) { // not required + return nil + } + + for i := 0; i < len(o.Details); i++ { + if swag.IsZero(o.Details[i]) { // not required + continue + } + + if o.Details[i] != nil { + if err := o.Details[i].Validate(formats); err != nil { + ve := new(errors.Validation) + if stderrors.As(err, &ve) { + return ve.ValidateName("BatchUpdateThresholds default" + "." + "details" + "." + strconv.Itoa(i)) + } + ce := new(errors.CompositeError) + if stderrors.As(err, &ce) { + return ce.ValidateName("BatchUpdateThresholds default" + "." + "details" + "." + strconv.Itoa(i)) + } + + return err + } + } + + } + + return nil +} + +// ContextValidate validate this batch update thresholds default body based on the context it is used +func (o *BatchUpdateThresholdsDefaultBody) ContextValidate(ctx context.Context, formats strfmt.Registry) error { + var res []error + + if err := o.contextValidateDetails(ctx, formats); err != nil { + res = append(res, err) + } + + if len(res) > 0 { + return errors.CompositeValidationError(res...) + } + return nil +} + +func (o *BatchUpdateThresholdsDefaultBody) contextValidateDetails(ctx context.Context, formats strfmt.Registry) error { + for i := 0; i < len(o.Details); i++ { + if o.Details[i] != nil { + + if swag.IsZero(o.Details[i]) { // not required + return nil + } + + if err := o.Details[i].ContextValidate(ctx, formats); err != nil { + ve := new(errors.Validation) + if stderrors.As(err, &ve) { + return ve.ValidateName("BatchUpdateThresholds default" + "." + "details" + "." + strconv.Itoa(i)) + } + ce := new(errors.CompositeError) + if stderrors.As(err, &ce) { + return ce.ValidateName("BatchUpdateThresholds default" + "." + "details" + "." + strconv.Itoa(i)) + } + + return err + } + } + } + + return nil +} + +// MarshalBinary interface implementation +func (o *BatchUpdateThresholdsDefaultBody) MarshalBinary() ([]byte, error) { + if o == nil { + return nil, nil + } + return swag.WriteJSON(o) +} + +// UnmarshalBinary interface implementation +func (o *BatchUpdateThresholdsDefaultBody) UnmarshalBinary(b []byte) error { + var res BatchUpdateThresholdsDefaultBody + if err := swag.ReadJSON(b, &res); err != nil { + return err + } + *o = res + return nil +} + +/* +BatchUpdateThresholdsDefaultBodyDetailsItems0 batch update thresholds default body details items0 +swagger:model BatchUpdateThresholdsDefaultBodyDetailsItems0 +*/ +type BatchUpdateThresholdsDefaultBodyDetailsItems0 struct { + // at type + AtType string `json:"@type,omitempty"` + + // batch update thresholds default body details items0 + BatchUpdateThresholdsDefaultBodyDetailsItems0 map[string]any `json:"-"` +} + +// UnmarshalJSON unmarshals this object with additional properties from JSON +func (o *BatchUpdateThresholdsDefaultBodyDetailsItems0) UnmarshalJSON(data []byte) error { + // stage 1, bind the properties + var stage1 struct { + // at type + AtType string `json:"@type,omitempty"` + } + if err := json.Unmarshal(data, &stage1); err != nil { + return err + } + var rcv BatchUpdateThresholdsDefaultBodyDetailsItems0 + + rcv.AtType = stage1.AtType + *o = rcv + + // stage 2, remove properties and add to map + stage2 := make(map[string]json.RawMessage) + if err := json.Unmarshal(data, &stage2); err != nil { + return err + } + + delete(stage2, "@type") + // stage 3, add additional properties values + if len(stage2) > 0 { + result := make(map[string]any) + for k, v := range stage2 { + var toadd any + if err := json.Unmarshal(v, &toadd); err != nil { + return err + } + result[k] = toadd + } + o.BatchUpdateThresholdsDefaultBodyDetailsItems0 = result + } + + return nil +} + +// MarshalJSON marshals this object with additional properties into a JSON object +func (o BatchUpdateThresholdsDefaultBodyDetailsItems0) MarshalJSON() ([]byte, error) { + var stage1 struct { + // at type + AtType string `json:"@type,omitempty"` + } + + stage1.AtType = o.AtType + + // make JSON object for known properties + props, err := json.Marshal(stage1) + if err != nil { + return nil, err + } + + if len(o.BatchUpdateThresholdsDefaultBodyDetailsItems0) == 0 { // no additional properties + return props, nil + } + + // make JSON object for the additional properties + additional, err := json.Marshal(o.BatchUpdateThresholdsDefaultBodyDetailsItems0) + if err != nil { + return nil, err + } + + if len(props) < 3 { // "{}": only additional properties + return additional, nil + } + + // concatenate the 2 objects + return swag.ConcatJSON(props, additional), nil +} + +// Validate validates this batch update thresholds default body details items0 +func (o *BatchUpdateThresholdsDefaultBodyDetailsItems0) Validate(formats strfmt.Registry) error { + return nil +} + +// ContextValidate validates this batch update thresholds default body details items0 based on context it is used +func (o *BatchUpdateThresholdsDefaultBodyDetailsItems0) ContextValidate(ctx context.Context, formats strfmt.Registry) error { + return nil +} + +// MarshalBinary interface implementation +func (o *BatchUpdateThresholdsDefaultBodyDetailsItems0) MarshalBinary() ([]byte, error) { + if o == nil { + return nil, nil + } + return swag.WriteJSON(o) +} + +// UnmarshalBinary interface implementation +func (o *BatchUpdateThresholdsDefaultBodyDetailsItems0) UnmarshalBinary(b []byte) error { + var res BatchUpdateThresholdsDefaultBodyDetailsItems0 + if err := swag.ReadJSON(b, &res); err != nil { + return err + } + *o = res + return nil +} + +/* +BatchUpdateThresholdsOKBody batch update thresholds OK body +swagger:model BatchUpdateThresholdsOKBody +*/ +type BatchUpdateThresholdsOKBody struct { + // Thresholds that were set, in request order. Cleared ones are omitted. + Thresholds []*BatchUpdateThresholdsOKBodyThresholdsItems0 `json:"thresholds"` +} + +// Validate validates this batch update thresholds OK body +func (o *BatchUpdateThresholdsOKBody) Validate(formats strfmt.Registry) error { + var res []error + + if err := o.validateThresholds(formats); err != nil { + res = append(res, err) + } + + if len(res) > 0 { + return errors.CompositeValidationError(res...) + } + return nil +} + +func (o *BatchUpdateThresholdsOKBody) validateThresholds(formats strfmt.Registry) error { + if swag.IsZero(o.Thresholds) { // not required + return nil + } + + for i := 0; i < len(o.Thresholds); i++ { + if swag.IsZero(o.Thresholds[i]) { // not required + continue + } + + if o.Thresholds[i] != nil { + if err := o.Thresholds[i].Validate(formats); err != nil { + ve := new(errors.Validation) + if stderrors.As(err, &ve) { + return ve.ValidateName("batchUpdateThresholdsOk" + "." + "thresholds" + "." + strconv.Itoa(i)) + } + ce := new(errors.CompositeError) + if stderrors.As(err, &ce) { + return ce.ValidateName("batchUpdateThresholdsOk" + "." + "thresholds" + "." + strconv.Itoa(i)) + } + + return err + } + } + + } + + return nil +} + +// ContextValidate validate this batch update thresholds OK body based on the context it is used +func (o *BatchUpdateThresholdsOKBody) ContextValidate(ctx context.Context, formats strfmt.Registry) error { + var res []error + + if err := o.contextValidateThresholds(ctx, formats); err != nil { + res = append(res, err) + } + + if len(res) > 0 { + return errors.CompositeValidationError(res...) + } + return nil +} + +func (o *BatchUpdateThresholdsOKBody) contextValidateThresholds(ctx context.Context, formats strfmt.Registry) error { + for i := 0; i < len(o.Thresholds); i++ { + if o.Thresholds[i] != nil { + + if swag.IsZero(o.Thresholds[i]) { // not required + return nil + } + + if err := o.Thresholds[i].ContextValidate(ctx, formats); err != nil { + ve := new(errors.Validation) + if stderrors.As(err, &ve) { + return ve.ValidateName("batchUpdateThresholdsOk" + "." + "thresholds" + "." + strconv.Itoa(i)) + } + ce := new(errors.CompositeError) + if stderrors.As(err, &ce) { + return ce.ValidateName("batchUpdateThresholdsOk" + "." + "thresholds" + "." + strconv.Itoa(i)) + } + + return err + } + } + } + + return nil +} + +// MarshalBinary interface implementation +func (o *BatchUpdateThresholdsOKBody) MarshalBinary() ([]byte, error) { + if o == nil { + return nil, nil + } + return swag.WriteJSON(o) +} + +// UnmarshalBinary interface implementation +func (o *BatchUpdateThresholdsOKBody) UnmarshalBinary(b []byte) error { + var res BatchUpdateThresholdsOKBody + if err := swag.ReadJSON(b, &res); err != nil { + return err + } + *o = res + return nil +} + +/* +BatchUpdateThresholdsOKBodyThresholdsItems0 Threshold is one overridable parameter of one rule, as it applies to one target. +swagger:model BatchUpdateThresholdsOKBodyThresholdsItems0 +*/ +type BatchUpdateThresholdsOKBodyThresholdsItems0 struct { + // Identifier PMM assigned to the rule. Not unique within a response: rules duplicated + // in Grafana share it, so two entries can carry the same rule_id and param_name and + // differ only in which rule they came from. Do not key a map on it. + RuleID string `json:"rule_id,omitempty"` + + // Machine-readable name of the overridable parameter. + ParamName string `json:"param_name,omitempty"` + + // Short human-readable parameter summary, as it was when the rule was created. + Summary string `json:"summary,omitempty"` + + // ParamUnit represents template parameter unit. + // + // - PARAM_UNIT_UNSPECIFIED: Invalid, unknown or absent. + // - PARAM_UNIT_PERCENTAGE: % + // - PARAM_UNIT_SECONDS: s + // Enum: ["PARAM_UNIT_UNSPECIFIED","PARAM_UNIT_PERCENTAGE","PARAM_UNIT_SECONDS"] + Unit *string `json:"unit,omitempty"` + + // Value the rule falls back to when no override applies. + DefaultValue float64 `json:"default_value,omitempty"` + + // Value the rule currently evaluates this target against. + EffectiveValue float64 `json:"effective_value,omitempty"` + + // Whether effective_value comes from an override rather than the default. + IsOverridden bool `json:"is_overridden,omitempty"` + + // ThresholdScope says what a threshold override's target refers to. + // + // - THRESHOLD_SCOPE_NODE: Target is a Node ID. + // - THRESHOLD_SCOPE_SERVICE: Target is a Service ID. + // - THRESHOLD_SCOPE_CLUSTER: Target is a cluster label value. Unlike the others it names no inventory entity, + // so it cannot be validated for existence and is never removed by entity deletion. + // Enum: ["THRESHOLD_SCOPE_UNSPECIFIED","THRESHOLD_SCOPE_NODE","THRESHOLD_SCOPE_SERVICE","THRESHOLD_SCOPE_CLUSTER"] + Scope *string `json:"scope,omitempty"` + + // Target the effective override was set on. Empty when not overridden. + Target string `json:"target,omitempty"` +} + +// Validate validates this batch update thresholds OK body thresholds items0 +func (o *BatchUpdateThresholdsOKBodyThresholdsItems0) Validate(formats strfmt.Registry) error { + var res []error + + if err := o.validateUnit(formats); err != nil { + res = append(res, err) + } + + if err := o.validateScope(formats); err != nil { + res = append(res, err) + } + + if len(res) > 0 { + return errors.CompositeValidationError(res...) + } + return nil +} + +var batchUpdateThresholdsOkBodyThresholdsItems0TypeUnitPropEnum []any + +func init() { + var res []string + if err := json.Unmarshal([]byte(`["PARAM_UNIT_UNSPECIFIED","PARAM_UNIT_PERCENTAGE","PARAM_UNIT_SECONDS"]`), &res); err != nil { + panic(err) + } + for _, v := range res { + batchUpdateThresholdsOkBodyThresholdsItems0TypeUnitPropEnum = append(batchUpdateThresholdsOkBodyThresholdsItems0TypeUnitPropEnum, v) + } +} + +const ( + + // BatchUpdateThresholdsOKBodyThresholdsItems0UnitPARAMUNITUNSPECIFIED captures enum value "PARAM_UNIT_UNSPECIFIED" + BatchUpdateThresholdsOKBodyThresholdsItems0UnitPARAMUNITUNSPECIFIED string = "PARAM_UNIT_UNSPECIFIED" + + // BatchUpdateThresholdsOKBodyThresholdsItems0UnitPARAMUNITPERCENTAGE captures enum value "PARAM_UNIT_PERCENTAGE" + BatchUpdateThresholdsOKBodyThresholdsItems0UnitPARAMUNITPERCENTAGE string = "PARAM_UNIT_PERCENTAGE" + + // BatchUpdateThresholdsOKBodyThresholdsItems0UnitPARAMUNITSECONDS captures enum value "PARAM_UNIT_SECONDS" + BatchUpdateThresholdsOKBodyThresholdsItems0UnitPARAMUNITSECONDS string = "PARAM_UNIT_SECONDS" +) + +// prop value enum +func (o *BatchUpdateThresholdsOKBodyThresholdsItems0) validateUnitEnum(path, location string, value string) error { + if err := validate.EnumCase(path, location, value, batchUpdateThresholdsOkBodyThresholdsItems0TypeUnitPropEnum, true); err != nil { + return err + } + return nil +} + +func (o *BatchUpdateThresholdsOKBodyThresholdsItems0) validateUnit(formats strfmt.Registry) error { + if swag.IsZero(o.Unit) { // not required + return nil + } + + // value enum + if err := o.validateUnitEnum("unit", "body", *o.Unit); err != nil { + return err + } + + return nil +} + +var batchUpdateThresholdsOkBodyThresholdsItems0TypeScopePropEnum []any + +func init() { + var res []string + if err := json.Unmarshal([]byte(`["THRESHOLD_SCOPE_UNSPECIFIED","THRESHOLD_SCOPE_NODE","THRESHOLD_SCOPE_SERVICE","THRESHOLD_SCOPE_CLUSTER"]`), &res); err != nil { + panic(err) + } + for _, v := range res { + batchUpdateThresholdsOkBodyThresholdsItems0TypeScopePropEnum = append(batchUpdateThresholdsOkBodyThresholdsItems0TypeScopePropEnum, v) + } +} + +const ( + + // BatchUpdateThresholdsOKBodyThresholdsItems0ScopeTHRESHOLDSCOPEUNSPECIFIED captures enum value "THRESHOLD_SCOPE_UNSPECIFIED" + BatchUpdateThresholdsOKBodyThresholdsItems0ScopeTHRESHOLDSCOPEUNSPECIFIED string = "THRESHOLD_SCOPE_UNSPECIFIED" + + // BatchUpdateThresholdsOKBodyThresholdsItems0ScopeTHRESHOLDSCOPENODE captures enum value "THRESHOLD_SCOPE_NODE" + BatchUpdateThresholdsOKBodyThresholdsItems0ScopeTHRESHOLDSCOPENODE string = "THRESHOLD_SCOPE_NODE" + + // BatchUpdateThresholdsOKBodyThresholdsItems0ScopeTHRESHOLDSCOPESERVICE captures enum value "THRESHOLD_SCOPE_SERVICE" + BatchUpdateThresholdsOKBodyThresholdsItems0ScopeTHRESHOLDSCOPESERVICE string = "THRESHOLD_SCOPE_SERVICE" + + // BatchUpdateThresholdsOKBodyThresholdsItems0ScopeTHRESHOLDSCOPECLUSTER captures enum value "THRESHOLD_SCOPE_CLUSTER" + BatchUpdateThresholdsOKBodyThresholdsItems0ScopeTHRESHOLDSCOPECLUSTER string = "THRESHOLD_SCOPE_CLUSTER" +) + +// prop value enum +func (o *BatchUpdateThresholdsOKBodyThresholdsItems0) validateScopeEnum(path, location string, value string) error { + if err := validate.EnumCase(path, location, value, batchUpdateThresholdsOkBodyThresholdsItems0TypeScopePropEnum, true); err != nil { + return err + } + return nil +} + +func (o *BatchUpdateThresholdsOKBodyThresholdsItems0) validateScope(formats strfmt.Registry) error { + if swag.IsZero(o.Scope) { // not required + return nil + } + + // value enum + if err := o.validateScopeEnum("scope", "body", *o.Scope); err != nil { + return err + } + + return nil +} + +// ContextValidate validates this batch update thresholds OK body thresholds items0 based on context it is used +func (o *BatchUpdateThresholdsOKBodyThresholdsItems0) ContextValidate(ctx context.Context, formats strfmt.Registry) error { + return nil +} + +// MarshalBinary interface implementation +func (o *BatchUpdateThresholdsOKBodyThresholdsItems0) MarshalBinary() ([]byte, error) { + if o == nil { + return nil, nil + } + return swag.WriteJSON(o) +} + +// UnmarshalBinary interface implementation +func (o *BatchUpdateThresholdsOKBodyThresholdsItems0) UnmarshalBinary(b []byte) error { + var res BatchUpdateThresholdsOKBodyThresholdsItems0 + if err := swag.ReadJSON(b, &res); err != nil { + return err + } + *o = res + return nil +} + +/* +BatchUpdateThresholdsParamsBodyUpdatesItems0 ThresholdUpdate sets or clears one override. +swagger:model BatchUpdateThresholdsParamsBodyUpdatesItems0 +*/ +type BatchUpdateThresholdsParamsBodyUpdatesItems0 struct { + // ThresholdScope says what a threshold override's target refers to. + // + // - THRESHOLD_SCOPE_NODE: Target is a Node ID. + // - THRESHOLD_SCOPE_SERVICE: Target is a Service ID. + // - THRESHOLD_SCOPE_CLUSTER: Target is a cluster label value. Unlike the others it names no inventory entity, + // so it cannot be validated for existence and is never removed by entity deletion. + // Enum: ["THRESHOLD_SCOPE_UNSPECIFIED","THRESHOLD_SCOPE_NODE","THRESHOLD_SCOPE_SERVICE","THRESHOLD_SCOPE_CLUSTER"] + Scope *string `json:"scope,omitempty"` + + // target + Target string `json:"target,omitempty"` + + // rule id + RuleID string `json:"rule_id,omitempty"` + + // param name + ParamName string `json:"param_name,omitempty"` + + // Omit to clear the override rather than set it. + Value *float64 `json:"value,omitempty"` +} + +// Validate validates this batch update thresholds params body updates items0 +func (o *BatchUpdateThresholdsParamsBodyUpdatesItems0) Validate(formats strfmt.Registry) error { + var res []error + + if err := o.validateScope(formats); err != nil { + res = append(res, err) + } + + if len(res) > 0 { + return errors.CompositeValidationError(res...) + } + return nil +} + +var batchUpdateThresholdsParamsBodyUpdatesItems0TypeScopePropEnum []any + +func init() { + var res []string + if err := json.Unmarshal([]byte(`["THRESHOLD_SCOPE_UNSPECIFIED","THRESHOLD_SCOPE_NODE","THRESHOLD_SCOPE_SERVICE","THRESHOLD_SCOPE_CLUSTER"]`), &res); err != nil { + panic(err) + } + for _, v := range res { + batchUpdateThresholdsParamsBodyUpdatesItems0TypeScopePropEnum = append(batchUpdateThresholdsParamsBodyUpdatesItems0TypeScopePropEnum, v) + } +} + +const ( + + // BatchUpdateThresholdsParamsBodyUpdatesItems0ScopeTHRESHOLDSCOPEUNSPECIFIED captures enum value "THRESHOLD_SCOPE_UNSPECIFIED" + BatchUpdateThresholdsParamsBodyUpdatesItems0ScopeTHRESHOLDSCOPEUNSPECIFIED string = "THRESHOLD_SCOPE_UNSPECIFIED" + + // BatchUpdateThresholdsParamsBodyUpdatesItems0ScopeTHRESHOLDSCOPENODE captures enum value "THRESHOLD_SCOPE_NODE" + BatchUpdateThresholdsParamsBodyUpdatesItems0ScopeTHRESHOLDSCOPENODE string = "THRESHOLD_SCOPE_NODE" + + // BatchUpdateThresholdsParamsBodyUpdatesItems0ScopeTHRESHOLDSCOPESERVICE captures enum value "THRESHOLD_SCOPE_SERVICE" + BatchUpdateThresholdsParamsBodyUpdatesItems0ScopeTHRESHOLDSCOPESERVICE string = "THRESHOLD_SCOPE_SERVICE" + + // BatchUpdateThresholdsParamsBodyUpdatesItems0ScopeTHRESHOLDSCOPECLUSTER captures enum value "THRESHOLD_SCOPE_CLUSTER" + BatchUpdateThresholdsParamsBodyUpdatesItems0ScopeTHRESHOLDSCOPECLUSTER string = "THRESHOLD_SCOPE_CLUSTER" +) + +// prop value enum +func (o *BatchUpdateThresholdsParamsBodyUpdatesItems0) validateScopeEnum(path, location string, value string) error { + if err := validate.EnumCase(path, location, value, batchUpdateThresholdsParamsBodyUpdatesItems0TypeScopePropEnum, true); err != nil { + return err + } + return nil +} + +func (o *BatchUpdateThresholdsParamsBodyUpdatesItems0) validateScope(formats strfmt.Registry) error { + if swag.IsZero(o.Scope) { // not required + return nil + } + + // value enum + if err := o.validateScopeEnum("scope", "body", *o.Scope); err != nil { + return err + } + + return nil +} + +// ContextValidate validates this batch update thresholds params body updates items0 based on context it is used +func (o *BatchUpdateThresholdsParamsBodyUpdatesItems0) ContextValidate(ctx context.Context, formats strfmt.Registry) error { + return nil +} + +// MarshalBinary interface implementation +func (o *BatchUpdateThresholdsParamsBodyUpdatesItems0) MarshalBinary() ([]byte, error) { + if o == nil { + return nil, nil + } + return swag.WriteJSON(o) +} + +// UnmarshalBinary interface implementation +func (o *BatchUpdateThresholdsParamsBodyUpdatesItems0) UnmarshalBinary(b []byte) error { + var res BatchUpdateThresholdsParamsBodyUpdatesItems0 + if err := swag.ReadJSON(b, &res); err != nil { + return err + } + *o = res + return nil +} diff --git a/api/alerting/v1/json/client/alerting_service/clear_threshold_parameters.go b/api/alerting/v1/json/client/alerting_service/clear_threshold_parameters.go new file mode 100644 index 0000000000..7642a2a53b --- /dev/null +++ b/api/alerting/v1/json/client/alerting_service/clear_threshold_parameters.go @@ -0,0 +1,261 @@ +// Code generated by go-swagger; DO NOT EDIT. + +package alerting_service + +import ( + "context" + "net/http" + "time" + + "github.com/go-openapi/errors" + "github.com/go-openapi/runtime" + cr "github.com/go-openapi/runtime/client" + "github.com/go-openapi/strfmt" +) + +// NewClearThresholdParams creates a new ClearThresholdParams object, +// with the default timeout for this client. +// +// Default values are not hydrated, since defaults are normally applied by the API server side. +// +// To enforce default values in parameter, use SetDefaults or WithDefaults. +func NewClearThresholdParams() *ClearThresholdParams { + return &ClearThresholdParams{ + timeout: cr.DefaultTimeout, + } +} + +// NewClearThresholdParamsWithTimeout creates a new ClearThresholdParams object +// with the ability to set a timeout on a request. +func NewClearThresholdParamsWithTimeout(timeout time.Duration) *ClearThresholdParams { + return &ClearThresholdParams{ + timeout: timeout, + } +} + +// NewClearThresholdParamsWithContext creates a new ClearThresholdParams object +// with the ability to set a context for a request. +func NewClearThresholdParamsWithContext(ctx context.Context) *ClearThresholdParams { + return &ClearThresholdParams{ + Context: ctx, + } +} + +// NewClearThresholdParamsWithHTTPClient creates a new ClearThresholdParams object +// with the ability to set a custom HTTPClient for a request. +func NewClearThresholdParamsWithHTTPClient(client *http.Client) *ClearThresholdParams { + return &ClearThresholdParams{ + HTTPClient: client, + } +} + +/* +ClearThresholdParams contains all the parameters to send to the API endpoint + + for the clear threshold operation. + + Typically these are written to a http.Request. +*/ +type ClearThresholdParams struct { + // ParamName. + ParamName *string + + // RuleID. + RuleID *string + + /* Scope. + + - THRESHOLD_SCOPE_NODE: Target is a Node ID. + - THRESHOLD_SCOPE_SERVICE: Target is a Service ID. + - THRESHOLD_SCOPE_CLUSTER: Target is a cluster label value. Unlike the others it names no inventory entity, + so it cannot be validated for existence and is never removed by entity deletion. + + Default: "THRESHOLD_SCOPE_UNSPECIFIED" + */ + Scope *string + + // Target. + Target *string + + timeout time.Duration + Context context.Context + HTTPClient *http.Client +} + +// WithDefaults hydrates default values in the clear threshold params (not the query body). +// +// All values with no default are reset to their zero value. +func (o *ClearThresholdParams) WithDefaults() *ClearThresholdParams { + o.SetDefaults() + return o +} + +// SetDefaults hydrates default values in the clear threshold params (not the query body). +// +// All values with no default are reset to their zero value. +func (o *ClearThresholdParams) SetDefaults() { + scopeDefault := string("THRESHOLD_SCOPE_UNSPECIFIED") + + val := ClearThresholdParams{ + Scope: &scopeDefault, + } + + val.timeout = o.timeout + val.Context = o.Context + val.HTTPClient = o.HTTPClient + *o = val +} + +// WithTimeout adds the timeout to the clear threshold params +func (o *ClearThresholdParams) WithTimeout(timeout time.Duration) *ClearThresholdParams { + o.SetTimeout(timeout) + return o +} + +// SetTimeout adds the timeout to the clear threshold params +func (o *ClearThresholdParams) SetTimeout(timeout time.Duration) { + o.timeout = timeout +} + +// WithContext adds the context to the clear threshold params +func (o *ClearThresholdParams) WithContext(ctx context.Context) *ClearThresholdParams { + o.SetContext(ctx) + return o +} + +// SetContext adds the context to the clear threshold params +func (o *ClearThresholdParams) SetContext(ctx context.Context) { + o.Context = ctx +} + +// WithHTTPClient adds the HTTPClient to the clear threshold params +func (o *ClearThresholdParams) WithHTTPClient(client *http.Client) *ClearThresholdParams { + o.SetHTTPClient(client) + return o +} + +// SetHTTPClient adds the HTTPClient to the clear threshold params +func (o *ClearThresholdParams) SetHTTPClient(client *http.Client) { + o.HTTPClient = client +} + +// WithParamName adds the paramName to the clear threshold params +func (o *ClearThresholdParams) WithParamName(paramName *string) *ClearThresholdParams { + o.SetParamName(paramName) + return o +} + +// SetParamName adds the paramName to the clear threshold params +func (o *ClearThresholdParams) SetParamName(paramName *string) { + o.ParamName = paramName +} + +// WithRuleID adds the ruleID to the clear threshold params +func (o *ClearThresholdParams) WithRuleID(ruleID *string) *ClearThresholdParams { + o.SetRuleID(ruleID) + return o +} + +// SetRuleID adds the ruleId to the clear threshold params +func (o *ClearThresholdParams) SetRuleID(ruleID *string) { + o.RuleID = ruleID +} + +// WithScope adds the scope to the clear threshold params +func (o *ClearThresholdParams) WithScope(scope *string) *ClearThresholdParams { + o.SetScope(scope) + return o +} + +// SetScope adds the scope to the clear threshold params +func (o *ClearThresholdParams) SetScope(scope *string) { + o.Scope = scope +} + +// WithTarget adds the target to the clear threshold params +func (o *ClearThresholdParams) WithTarget(target *string) *ClearThresholdParams { + o.SetTarget(target) + return o +} + +// SetTarget adds the target to the clear threshold params +func (o *ClearThresholdParams) SetTarget(target *string) { + o.Target = target +} + +// WriteToRequest writes these params to a swagger request +func (o *ClearThresholdParams) WriteToRequest(r runtime.ClientRequest, reg strfmt.Registry) error { + if err := r.SetTimeout(o.timeout); err != nil { + return err + } + var res []error + + if o.ParamName != nil { + + // query param param_name + var qrParamName string + + if o.ParamName != nil { + qrParamName = *o.ParamName + } + qParamName := qrParamName + if qParamName != "" { + if err := r.SetQueryParam("param_name", qParamName); err != nil { + return err + } + } + } + + if o.RuleID != nil { + + // query param rule_id + var qrRuleID string + + if o.RuleID != nil { + qrRuleID = *o.RuleID + } + qRuleID := qrRuleID + if qRuleID != "" { + if err := r.SetQueryParam("rule_id", qRuleID); err != nil { + return err + } + } + } + + if o.Scope != nil { + + // query param scope + var qrScope string + + if o.Scope != nil { + qrScope = *o.Scope + } + qScope := qrScope + if qScope != "" { + if err := r.SetQueryParam("scope", qScope); err != nil { + return err + } + } + } + + if o.Target != nil { + + // query param target + var qrTarget string + + if o.Target != nil { + qrTarget = *o.Target + } + qTarget := qrTarget + if qTarget != "" { + if err := r.SetQueryParam("target", qTarget); err != nil { + return err + } + } + } + + if len(res) > 0 { + return errors.CompositeValidationError(res...) + } + return nil +} diff --git a/api/alerting/v1/json/client/alerting_service/clear_threshold_responses.go b/api/alerting/v1/json/client/alerting_service/clear_threshold_responses.go new file mode 100644 index 0000000000..af56989ef5 --- /dev/null +++ b/api/alerting/v1/json/client/alerting_service/clear_threshold_responses.go @@ -0,0 +1,411 @@ +// Code generated by go-swagger; DO NOT EDIT. + +package alerting_service + +import ( + "context" + "encoding/json" + stderrors "errors" + "fmt" + "io" + "strconv" + + "github.com/go-openapi/errors" + "github.com/go-openapi/runtime" + "github.com/go-openapi/strfmt" + "github.com/go-openapi/swag" +) + +// ClearThresholdReader is a Reader for the ClearThreshold structure. +type ClearThresholdReader struct { + formats strfmt.Registry +} + +// ReadResponse reads a server response into the received o. +func (o *ClearThresholdReader) ReadResponse(response runtime.ClientResponse, consumer runtime.Consumer) (any, error) { + switch response.Code() { + case 200: + result := NewClearThresholdOK() + if err := result.readResponse(response, consumer, o.formats); err != nil { + return nil, err + } + return result, nil + default: + result := NewClearThresholdDefault(response.Code()) + if err := result.readResponse(response, consumer, o.formats); err != nil { + return nil, err + } + if response.Code()/100 == 2 { + return result, nil + } + return nil, result + } +} + +// NewClearThresholdOK creates a ClearThresholdOK with default headers values +func NewClearThresholdOK() *ClearThresholdOK { + return &ClearThresholdOK{} +} + +/* +ClearThresholdOK describes a response with status code 200, with default header values. + +A successful response. +*/ +type ClearThresholdOK struct { + Payload any +} + +// IsSuccess returns true when this clear threshold Ok response has a 2xx status code +func (o *ClearThresholdOK) IsSuccess() bool { + return true +} + +// IsRedirect returns true when this clear threshold Ok response has a 3xx status code +func (o *ClearThresholdOK) IsRedirect() bool { + return false +} + +// IsClientError returns true when this clear threshold Ok response has a 4xx status code +func (o *ClearThresholdOK) IsClientError() bool { + return false +} + +// IsServerError returns true when this clear threshold Ok response has a 5xx status code +func (o *ClearThresholdOK) IsServerError() bool { + return false +} + +// IsCode returns true when this clear threshold Ok response a status code equal to that given +func (o *ClearThresholdOK) IsCode(code int) bool { + return code == 200 +} + +// Code gets the status code for the clear threshold Ok response +func (o *ClearThresholdOK) Code() int { + return 200 +} + +func (o *ClearThresholdOK) Error() string { + payload, _ := json.Marshal(o.Payload) + return fmt.Sprintf("[DELETE /v1/alerting/thresholds][%d] clearThresholdOk %s", 200, payload) +} + +func (o *ClearThresholdOK) String() string { + payload, _ := json.Marshal(o.Payload) + return fmt.Sprintf("[DELETE /v1/alerting/thresholds][%d] clearThresholdOk %s", 200, payload) +} + +func (o *ClearThresholdOK) GetPayload() any { + return o.Payload +} + +func (o *ClearThresholdOK) readResponse(response runtime.ClientResponse, consumer runtime.Consumer, formats strfmt.Registry) error { + // response payload + if err := consumer.Consume(response.Body(), &o.Payload); err != nil && !stderrors.Is(err, io.EOF) { + return err + } + + return nil +} + +// NewClearThresholdDefault creates a ClearThresholdDefault with default headers values +func NewClearThresholdDefault(code int) *ClearThresholdDefault { + return &ClearThresholdDefault{ + _statusCode: code, + } +} + +/* +ClearThresholdDefault describes a response with status code -1, with default header values. + +An unexpected error response. +*/ +type ClearThresholdDefault struct { + _statusCode int + + Payload *ClearThresholdDefaultBody +} + +// IsSuccess returns true when this clear threshold default response has a 2xx status code +func (o *ClearThresholdDefault) IsSuccess() bool { + return o._statusCode/100 == 2 +} + +// IsRedirect returns true when this clear threshold default response has a 3xx status code +func (o *ClearThresholdDefault) IsRedirect() bool { + return o._statusCode/100 == 3 +} + +// IsClientError returns true when this clear threshold default response has a 4xx status code +func (o *ClearThresholdDefault) IsClientError() bool { + return o._statusCode/100 == 4 +} + +// IsServerError returns true when this clear threshold default response has a 5xx status code +func (o *ClearThresholdDefault) IsServerError() bool { + return o._statusCode/100 == 5 +} + +// IsCode returns true when this clear threshold default response a status code equal to that given +func (o *ClearThresholdDefault) IsCode(code int) bool { + return o._statusCode == code +} + +// Code gets the status code for the clear threshold default response +func (o *ClearThresholdDefault) Code() int { + return o._statusCode +} + +func (o *ClearThresholdDefault) Error() string { + payload, _ := json.Marshal(o.Payload) + return fmt.Sprintf("[DELETE /v1/alerting/thresholds][%d] ClearThreshold default %s", o._statusCode, payload) +} + +func (o *ClearThresholdDefault) String() string { + payload, _ := json.Marshal(o.Payload) + return fmt.Sprintf("[DELETE /v1/alerting/thresholds][%d] ClearThreshold default %s", o._statusCode, payload) +} + +func (o *ClearThresholdDefault) GetPayload() *ClearThresholdDefaultBody { + return o.Payload +} + +func (o *ClearThresholdDefault) readResponse(response runtime.ClientResponse, consumer runtime.Consumer, formats strfmt.Registry) error { + o.Payload = new(ClearThresholdDefaultBody) + + // response payload + if err := consumer.Consume(response.Body(), o.Payload); err != nil && !stderrors.Is(err, io.EOF) { + return err + } + + return nil +} + +/* +ClearThresholdDefaultBody clear threshold default body +swagger:model ClearThresholdDefaultBody +*/ +type ClearThresholdDefaultBody struct { + // code + Code int32 `json:"code,omitempty"` + + // message + Message string `json:"message,omitempty"` + + // details + Details []*ClearThresholdDefaultBodyDetailsItems0 `json:"details"` +} + +// Validate validates this clear threshold default body +func (o *ClearThresholdDefaultBody) Validate(formats strfmt.Registry) error { + var res []error + + if err := o.validateDetails(formats); err != nil { + res = append(res, err) + } + + if len(res) > 0 { + return errors.CompositeValidationError(res...) + } + return nil +} + +func (o *ClearThresholdDefaultBody) validateDetails(formats strfmt.Registry) error { + if swag.IsZero(o.Details) { // not required + return nil + } + + for i := 0; i < len(o.Details); i++ { + if swag.IsZero(o.Details[i]) { // not required + continue + } + + if o.Details[i] != nil { + if err := o.Details[i].Validate(formats); err != nil { + ve := new(errors.Validation) + if stderrors.As(err, &ve) { + return ve.ValidateName("ClearThreshold default" + "." + "details" + "." + strconv.Itoa(i)) + } + ce := new(errors.CompositeError) + if stderrors.As(err, &ce) { + return ce.ValidateName("ClearThreshold default" + "." + "details" + "." + strconv.Itoa(i)) + } + + return err + } + } + + } + + return nil +} + +// ContextValidate validate this clear threshold default body based on the context it is used +func (o *ClearThresholdDefaultBody) ContextValidate(ctx context.Context, formats strfmt.Registry) error { + var res []error + + if err := o.contextValidateDetails(ctx, formats); err != nil { + res = append(res, err) + } + + if len(res) > 0 { + return errors.CompositeValidationError(res...) + } + return nil +} + +func (o *ClearThresholdDefaultBody) contextValidateDetails(ctx context.Context, formats strfmt.Registry) error { + for i := 0; i < len(o.Details); i++ { + if o.Details[i] != nil { + + if swag.IsZero(o.Details[i]) { // not required + return nil + } + + if err := o.Details[i].ContextValidate(ctx, formats); err != nil { + ve := new(errors.Validation) + if stderrors.As(err, &ve) { + return ve.ValidateName("ClearThreshold default" + "." + "details" + "." + strconv.Itoa(i)) + } + ce := new(errors.CompositeError) + if stderrors.As(err, &ce) { + return ce.ValidateName("ClearThreshold default" + "." + "details" + "." + strconv.Itoa(i)) + } + + return err + } + } + } + + return nil +} + +// MarshalBinary interface implementation +func (o *ClearThresholdDefaultBody) MarshalBinary() ([]byte, error) { + if o == nil { + return nil, nil + } + return swag.WriteJSON(o) +} + +// UnmarshalBinary interface implementation +func (o *ClearThresholdDefaultBody) UnmarshalBinary(b []byte) error { + var res ClearThresholdDefaultBody + if err := swag.ReadJSON(b, &res); err != nil { + return err + } + *o = res + return nil +} + +/* +ClearThresholdDefaultBodyDetailsItems0 clear threshold default body details items0 +swagger:model ClearThresholdDefaultBodyDetailsItems0 +*/ +type ClearThresholdDefaultBodyDetailsItems0 struct { + // at type + AtType string `json:"@type,omitempty"` + + // clear threshold default body details items0 + ClearThresholdDefaultBodyDetailsItems0 map[string]any `json:"-"` +} + +// UnmarshalJSON unmarshals this object with additional properties from JSON +func (o *ClearThresholdDefaultBodyDetailsItems0) UnmarshalJSON(data []byte) error { + // stage 1, bind the properties + var stage1 struct { + // at type + AtType string `json:"@type,omitempty"` + } + if err := json.Unmarshal(data, &stage1); err != nil { + return err + } + var rcv ClearThresholdDefaultBodyDetailsItems0 + + rcv.AtType = stage1.AtType + *o = rcv + + // stage 2, remove properties and add to map + stage2 := make(map[string]json.RawMessage) + if err := json.Unmarshal(data, &stage2); err != nil { + return err + } + + delete(stage2, "@type") + // stage 3, add additional properties values + if len(stage2) > 0 { + result := make(map[string]any) + for k, v := range stage2 { + var toadd any + if err := json.Unmarshal(v, &toadd); err != nil { + return err + } + result[k] = toadd + } + o.ClearThresholdDefaultBodyDetailsItems0 = result + } + + return nil +} + +// MarshalJSON marshals this object with additional properties into a JSON object +func (o ClearThresholdDefaultBodyDetailsItems0) MarshalJSON() ([]byte, error) { + var stage1 struct { + // at type + AtType string `json:"@type,omitempty"` + } + + stage1.AtType = o.AtType + + // make JSON object for known properties + props, err := json.Marshal(stage1) + if err != nil { + return nil, err + } + + if len(o.ClearThresholdDefaultBodyDetailsItems0) == 0 { // no additional properties + return props, nil + } + + // make JSON object for the additional properties + additional, err := json.Marshal(o.ClearThresholdDefaultBodyDetailsItems0) + if err != nil { + return nil, err + } + + if len(props) < 3 { // "{}": only additional properties + return additional, nil + } + + // concatenate the 2 objects + return swag.ConcatJSON(props, additional), nil +} + +// Validate validates this clear threshold default body details items0 +func (o *ClearThresholdDefaultBodyDetailsItems0) Validate(formats strfmt.Registry) error { + return nil +} + +// ContextValidate validates this clear threshold default body details items0 based on context it is used +func (o *ClearThresholdDefaultBodyDetailsItems0) ContextValidate(ctx context.Context, formats strfmt.Registry) error { + return nil +} + +// MarshalBinary interface implementation +func (o *ClearThresholdDefaultBodyDetailsItems0) MarshalBinary() ([]byte, error) { + if o == nil { + return nil, nil + } + return swag.WriteJSON(o) +} + +// UnmarshalBinary interface implementation +func (o *ClearThresholdDefaultBodyDetailsItems0) UnmarshalBinary(b []byte) error { + var res ClearThresholdDefaultBodyDetailsItems0 + if err := swag.ReadJSON(b, &res); err != nil { + return err + } + *o = res + return nil +} diff --git a/api/alerting/v1/json/client/alerting_service/list_thresholds_parameters.go b/api/alerting/v1/json/client/alerting_service/list_thresholds_parameters.go new file mode 100644 index 0000000000..b786a24ad9 --- /dev/null +++ b/api/alerting/v1/json/client/alerting_service/list_thresholds_parameters.go @@ -0,0 +1,241 @@ +// Code generated by go-swagger; DO NOT EDIT. + +package alerting_service + +import ( + "context" + "net/http" + "time" + + "github.com/go-openapi/errors" + "github.com/go-openapi/runtime" + cr "github.com/go-openapi/runtime/client" + "github.com/go-openapi/strfmt" +) + +// NewListThresholdsParams creates a new ListThresholdsParams object, +// with the default timeout for this client. +// +// Default values are not hydrated, since defaults are normally applied by the API server side. +// +// To enforce default values in parameter, use SetDefaults or WithDefaults. +func NewListThresholdsParams() *ListThresholdsParams { + return &ListThresholdsParams{ + timeout: cr.DefaultTimeout, + } +} + +// NewListThresholdsParamsWithTimeout creates a new ListThresholdsParams object +// with the ability to set a timeout on a request. +func NewListThresholdsParamsWithTimeout(timeout time.Duration) *ListThresholdsParams { + return &ListThresholdsParams{ + timeout: timeout, + } +} + +// NewListThresholdsParamsWithContext creates a new ListThresholdsParams object +// with the ability to set a context for a request. +func NewListThresholdsParamsWithContext(ctx context.Context) *ListThresholdsParams { + return &ListThresholdsParams{ + Context: ctx, + } +} + +// NewListThresholdsParamsWithHTTPClient creates a new ListThresholdsParams object +// with the ability to set a custom HTTPClient for a request. +func NewListThresholdsParamsWithHTTPClient(client *http.Client) *ListThresholdsParams { + return &ListThresholdsParams{ + HTTPClient: client, + } +} + +/* +ListThresholdsParams contains all the parameters to send to the API endpoint + + for the list thresholds operation. + + Typically these are written to a http.Request. +*/ +type ListThresholdsParams struct { + /* RuleID. + + Return only thresholds of this rule. + */ + RuleID *string + + /* Scope. + + Scope of the target to report thresholds for. Must be set together with target. + + - THRESHOLD_SCOPE_NODE: Target is a Node ID. + - THRESHOLD_SCOPE_SERVICE: Target is a Service ID. + - THRESHOLD_SCOPE_CLUSTER: Target is a cluster label value. Unlike the others it names no inventory entity, + so it cannot be validated for existence and is never removed by entity deletion. + + Default: "THRESHOLD_SCOPE_UNSPECIFIED" + */ + Scope *string + + /* Target. + + Target to report thresholds for. When set, every overridable parameter is returned + for that target, overridden or not. When empty, only existing overrides are + returned, since there is otherwise no bounded set to enumerate. + */ + Target *string + + timeout time.Duration + Context context.Context + HTTPClient *http.Client +} + +// WithDefaults hydrates default values in the list thresholds params (not the query body). +// +// All values with no default are reset to their zero value. +func (o *ListThresholdsParams) WithDefaults() *ListThresholdsParams { + o.SetDefaults() + return o +} + +// SetDefaults hydrates default values in the list thresholds params (not the query body). +// +// All values with no default are reset to their zero value. +func (o *ListThresholdsParams) SetDefaults() { + scopeDefault := string("THRESHOLD_SCOPE_UNSPECIFIED") + + val := ListThresholdsParams{ + Scope: &scopeDefault, + } + + val.timeout = o.timeout + val.Context = o.Context + val.HTTPClient = o.HTTPClient + *o = val +} + +// WithTimeout adds the timeout to the list thresholds params +func (o *ListThresholdsParams) WithTimeout(timeout time.Duration) *ListThresholdsParams { + o.SetTimeout(timeout) + return o +} + +// SetTimeout adds the timeout to the list thresholds params +func (o *ListThresholdsParams) SetTimeout(timeout time.Duration) { + o.timeout = timeout +} + +// WithContext adds the context to the list thresholds params +func (o *ListThresholdsParams) WithContext(ctx context.Context) *ListThresholdsParams { + o.SetContext(ctx) + return o +} + +// SetContext adds the context to the list thresholds params +func (o *ListThresholdsParams) SetContext(ctx context.Context) { + o.Context = ctx +} + +// WithHTTPClient adds the HTTPClient to the list thresholds params +func (o *ListThresholdsParams) WithHTTPClient(client *http.Client) *ListThresholdsParams { + o.SetHTTPClient(client) + return o +} + +// SetHTTPClient adds the HTTPClient to the list thresholds params +func (o *ListThresholdsParams) SetHTTPClient(client *http.Client) { + o.HTTPClient = client +} + +// WithRuleID adds the ruleID to the list thresholds params +func (o *ListThresholdsParams) WithRuleID(ruleID *string) *ListThresholdsParams { + o.SetRuleID(ruleID) + return o +} + +// SetRuleID adds the ruleId to the list thresholds params +func (o *ListThresholdsParams) SetRuleID(ruleID *string) { + o.RuleID = ruleID +} + +// WithScope adds the scope to the list thresholds params +func (o *ListThresholdsParams) WithScope(scope *string) *ListThresholdsParams { + o.SetScope(scope) + return o +} + +// SetScope adds the scope to the list thresholds params +func (o *ListThresholdsParams) SetScope(scope *string) { + o.Scope = scope +} + +// WithTarget adds the target to the list thresholds params +func (o *ListThresholdsParams) WithTarget(target *string) *ListThresholdsParams { + o.SetTarget(target) + return o +} + +// SetTarget adds the target to the list thresholds params +func (o *ListThresholdsParams) SetTarget(target *string) { + o.Target = target +} + +// WriteToRequest writes these params to a swagger request +func (o *ListThresholdsParams) WriteToRequest(r runtime.ClientRequest, reg strfmt.Registry) error { + if err := r.SetTimeout(o.timeout); err != nil { + return err + } + var res []error + + if o.RuleID != nil { + + // query param rule_id + var qrRuleID string + + if o.RuleID != nil { + qrRuleID = *o.RuleID + } + qRuleID := qrRuleID + if qRuleID != "" { + if err := r.SetQueryParam("rule_id", qRuleID); err != nil { + return err + } + } + } + + if o.Scope != nil { + + // query param scope + var qrScope string + + if o.Scope != nil { + qrScope = *o.Scope + } + qScope := qrScope + if qScope != "" { + if err := r.SetQueryParam("scope", qScope); err != nil { + return err + } + } + } + + if o.Target != nil { + + // query param target + var qrTarget string + + if o.Target != nil { + qrTarget = *o.Target + } + qTarget := qrTarget + if qTarget != "" { + if err := r.SetQueryParam("target", qTarget); err != nil { + return err + } + } + } + + if len(res) > 0 { + return errors.CompositeValidationError(res...) + } + return nil +} diff --git a/api/alerting/v1/json/client/alerting_service/list_thresholds_responses.go b/api/alerting/v1/json/client/alerting_service/list_thresholds_responses.go new file mode 100644 index 0000000000..19ad680136 --- /dev/null +++ b/api/alerting/v1/json/client/alerting_service/list_thresholds_responses.go @@ -0,0 +1,705 @@ +// Code generated by go-swagger; DO NOT EDIT. + +package alerting_service + +import ( + "context" + "encoding/json" + stderrors "errors" + "fmt" + "io" + "strconv" + + "github.com/go-openapi/errors" + "github.com/go-openapi/runtime" + "github.com/go-openapi/strfmt" + "github.com/go-openapi/swag" + "github.com/go-openapi/validate" +) + +// ListThresholdsReader is a Reader for the ListThresholds structure. +type ListThresholdsReader struct { + formats strfmt.Registry +} + +// ReadResponse reads a server response into the received o. +func (o *ListThresholdsReader) ReadResponse(response runtime.ClientResponse, consumer runtime.Consumer) (any, error) { + switch response.Code() { + case 200: + result := NewListThresholdsOK() + if err := result.readResponse(response, consumer, o.formats); err != nil { + return nil, err + } + return result, nil + default: + result := NewListThresholdsDefault(response.Code()) + if err := result.readResponse(response, consumer, o.formats); err != nil { + return nil, err + } + if response.Code()/100 == 2 { + return result, nil + } + return nil, result + } +} + +// NewListThresholdsOK creates a ListThresholdsOK with default headers values +func NewListThresholdsOK() *ListThresholdsOK { + return &ListThresholdsOK{} +} + +/* +ListThresholdsOK describes a response with status code 200, with default header values. + +A successful response. +*/ +type ListThresholdsOK struct { + Payload *ListThresholdsOKBody +} + +// IsSuccess returns true when this list thresholds Ok response has a 2xx status code +func (o *ListThresholdsOK) IsSuccess() bool { + return true +} + +// IsRedirect returns true when this list thresholds Ok response has a 3xx status code +func (o *ListThresholdsOK) IsRedirect() bool { + return false +} + +// IsClientError returns true when this list thresholds Ok response has a 4xx status code +func (o *ListThresholdsOK) IsClientError() bool { + return false +} + +// IsServerError returns true when this list thresholds Ok response has a 5xx status code +func (o *ListThresholdsOK) IsServerError() bool { + return false +} + +// IsCode returns true when this list thresholds Ok response a status code equal to that given +func (o *ListThresholdsOK) IsCode(code int) bool { + return code == 200 +} + +// Code gets the status code for the list thresholds Ok response +func (o *ListThresholdsOK) Code() int { + return 200 +} + +func (o *ListThresholdsOK) Error() string { + payload, _ := json.Marshal(o.Payload) + return fmt.Sprintf("[GET /v1/alerting/thresholds][%d] listThresholdsOk %s", 200, payload) +} + +func (o *ListThresholdsOK) String() string { + payload, _ := json.Marshal(o.Payload) + return fmt.Sprintf("[GET /v1/alerting/thresholds][%d] listThresholdsOk %s", 200, payload) +} + +func (o *ListThresholdsOK) GetPayload() *ListThresholdsOKBody { + return o.Payload +} + +func (o *ListThresholdsOK) readResponse(response runtime.ClientResponse, consumer runtime.Consumer, formats strfmt.Registry) error { + o.Payload = new(ListThresholdsOKBody) + + // response payload + if err := consumer.Consume(response.Body(), o.Payload); err != nil && !stderrors.Is(err, io.EOF) { + return err + } + + return nil +} + +// NewListThresholdsDefault creates a ListThresholdsDefault with default headers values +func NewListThresholdsDefault(code int) *ListThresholdsDefault { + return &ListThresholdsDefault{ + _statusCode: code, + } +} + +/* +ListThresholdsDefault describes a response with status code -1, with default header values. + +An unexpected error response. +*/ +type ListThresholdsDefault struct { + _statusCode int + + Payload *ListThresholdsDefaultBody +} + +// IsSuccess returns true when this list thresholds default response has a 2xx status code +func (o *ListThresholdsDefault) IsSuccess() bool { + return o._statusCode/100 == 2 +} + +// IsRedirect returns true when this list thresholds default response has a 3xx status code +func (o *ListThresholdsDefault) IsRedirect() bool { + return o._statusCode/100 == 3 +} + +// IsClientError returns true when this list thresholds default response has a 4xx status code +func (o *ListThresholdsDefault) IsClientError() bool { + return o._statusCode/100 == 4 +} + +// IsServerError returns true when this list thresholds default response has a 5xx status code +func (o *ListThresholdsDefault) IsServerError() bool { + return o._statusCode/100 == 5 +} + +// IsCode returns true when this list thresholds default response a status code equal to that given +func (o *ListThresholdsDefault) IsCode(code int) bool { + return o._statusCode == code +} + +// Code gets the status code for the list thresholds default response +func (o *ListThresholdsDefault) Code() int { + return o._statusCode +} + +func (o *ListThresholdsDefault) Error() string { + payload, _ := json.Marshal(o.Payload) + return fmt.Sprintf("[GET /v1/alerting/thresholds][%d] ListThresholds default %s", o._statusCode, payload) +} + +func (o *ListThresholdsDefault) String() string { + payload, _ := json.Marshal(o.Payload) + return fmt.Sprintf("[GET /v1/alerting/thresholds][%d] ListThresholds default %s", o._statusCode, payload) +} + +func (o *ListThresholdsDefault) GetPayload() *ListThresholdsDefaultBody { + return o.Payload +} + +func (o *ListThresholdsDefault) readResponse(response runtime.ClientResponse, consumer runtime.Consumer, formats strfmt.Registry) error { + o.Payload = new(ListThresholdsDefaultBody) + + // response payload + if err := consumer.Consume(response.Body(), o.Payload); err != nil && !stderrors.Is(err, io.EOF) { + return err + } + + return nil +} + +/* +ListThresholdsDefaultBody list thresholds default body +swagger:model ListThresholdsDefaultBody +*/ +type ListThresholdsDefaultBody struct { + // code + Code int32 `json:"code,omitempty"` + + // message + Message string `json:"message,omitempty"` + + // details + Details []*ListThresholdsDefaultBodyDetailsItems0 `json:"details"` +} + +// Validate validates this list thresholds default body +func (o *ListThresholdsDefaultBody) Validate(formats strfmt.Registry) error { + var res []error + + if err := o.validateDetails(formats); err != nil { + res = append(res, err) + } + + if len(res) > 0 { + return errors.CompositeValidationError(res...) + } + return nil +} + +func (o *ListThresholdsDefaultBody) validateDetails(formats strfmt.Registry) error { + if swag.IsZero(o.Details) { // not required + return nil + } + + for i := 0; i < len(o.Details); i++ { + if swag.IsZero(o.Details[i]) { // not required + continue + } + + if o.Details[i] != nil { + if err := o.Details[i].Validate(formats); err != nil { + ve := new(errors.Validation) + if stderrors.As(err, &ve) { + return ve.ValidateName("ListThresholds default" + "." + "details" + "." + strconv.Itoa(i)) + } + ce := new(errors.CompositeError) + if stderrors.As(err, &ce) { + return ce.ValidateName("ListThresholds default" + "." + "details" + "." + strconv.Itoa(i)) + } + + return err + } + } + + } + + return nil +} + +// ContextValidate validate this list thresholds default body based on the context it is used +func (o *ListThresholdsDefaultBody) ContextValidate(ctx context.Context, formats strfmt.Registry) error { + var res []error + + if err := o.contextValidateDetails(ctx, formats); err != nil { + res = append(res, err) + } + + if len(res) > 0 { + return errors.CompositeValidationError(res...) + } + return nil +} + +func (o *ListThresholdsDefaultBody) contextValidateDetails(ctx context.Context, formats strfmt.Registry) error { + for i := 0; i < len(o.Details); i++ { + if o.Details[i] != nil { + + if swag.IsZero(o.Details[i]) { // not required + return nil + } + + if err := o.Details[i].ContextValidate(ctx, formats); err != nil { + ve := new(errors.Validation) + if stderrors.As(err, &ve) { + return ve.ValidateName("ListThresholds default" + "." + "details" + "." + strconv.Itoa(i)) + } + ce := new(errors.CompositeError) + if stderrors.As(err, &ce) { + return ce.ValidateName("ListThresholds default" + "." + "details" + "." + strconv.Itoa(i)) + } + + return err + } + } + } + + return nil +} + +// MarshalBinary interface implementation +func (o *ListThresholdsDefaultBody) MarshalBinary() ([]byte, error) { + if o == nil { + return nil, nil + } + return swag.WriteJSON(o) +} + +// UnmarshalBinary interface implementation +func (o *ListThresholdsDefaultBody) UnmarshalBinary(b []byte) error { + var res ListThresholdsDefaultBody + if err := swag.ReadJSON(b, &res); err != nil { + return err + } + *o = res + return nil +} + +/* +ListThresholdsDefaultBodyDetailsItems0 list thresholds default body details items0 +swagger:model ListThresholdsDefaultBodyDetailsItems0 +*/ +type ListThresholdsDefaultBodyDetailsItems0 struct { + // at type + AtType string `json:"@type,omitempty"` + + // list thresholds default body details items0 + ListThresholdsDefaultBodyDetailsItems0 map[string]any `json:"-"` +} + +// UnmarshalJSON unmarshals this object with additional properties from JSON +func (o *ListThresholdsDefaultBodyDetailsItems0) UnmarshalJSON(data []byte) error { + // stage 1, bind the properties + var stage1 struct { + // at type + AtType string `json:"@type,omitempty"` + } + if err := json.Unmarshal(data, &stage1); err != nil { + return err + } + var rcv ListThresholdsDefaultBodyDetailsItems0 + + rcv.AtType = stage1.AtType + *o = rcv + + // stage 2, remove properties and add to map + stage2 := make(map[string]json.RawMessage) + if err := json.Unmarshal(data, &stage2); err != nil { + return err + } + + delete(stage2, "@type") + // stage 3, add additional properties values + if len(stage2) > 0 { + result := make(map[string]any) + for k, v := range stage2 { + var toadd any + if err := json.Unmarshal(v, &toadd); err != nil { + return err + } + result[k] = toadd + } + o.ListThresholdsDefaultBodyDetailsItems0 = result + } + + return nil +} + +// MarshalJSON marshals this object with additional properties into a JSON object +func (o ListThresholdsDefaultBodyDetailsItems0) MarshalJSON() ([]byte, error) { + var stage1 struct { + // at type + AtType string `json:"@type,omitempty"` + } + + stage1.AtType = o.AtType + + // make JSON object for known properties + props, err := json.Marshal(stage1) + if err != nil { + return nil, err + } + + if len(o.ListThresholdsDefaultBodyDetailsItems0) == 0 { // no additional properties + return props, nil + } + + // make JSON object for the additional properties + additional, err := json.Marshal(o.ListThresholdsDefaultBodyDetailsItems0) + if err != nil { + return nil, err + } + + if len(props) < 3 { // "{}": only additional properties + return additional, nil + } + + // concatenate the 2 objects + return swag.ConcatJSON(props, additional), nil +} + +// Validate validates this list thresholds default body details items0 +func (o *ListThresholdsDefaultBodyDetailsItems0) Validate(formats strfmt.Registry) error { + return nil +} + +// ContextValidate validates this list thresholds default body details items0 based on context it is used +func (o *ListThresholdsDefaultBodyDetailsItems0) ContextValidate(ctx context.Context, formats strfmt.Registry) error { + return nil +} + +// MarshalBinary interface implementation +func (o *ListThresholdsDefaultBodyDetailsItems0) MarshalBinary() ([]byte, error) { + if o == nil { + return nil, nil + } + return swag.WriteJSON(o) +} + +// UnmarshalBinary interface implementation +func (o *ListThresholdsDefaultBodyDetailsItems0) UnmarshalBinary(b []byte) error { + var res ListThresholdsDefaultBodyDetailsItems0 + if err := swag.ReadJSON(b, &res); err != nil { + return err + } + *o = res + return nil +} + +/* +ListThresholdsOKBody list thresholds OK body +swagger:model ListThresholdsOKBody +*/ +type ListThresholdsOKBody struct { + // thresholds + Thresholds []*ListThresholdsOKBodyThresholdsItems0 `json:"thresholds"` +} + +// Validate validates this list thresholds OK body +func (o *ListThresholdsOKBody) Validate(formats strfmt.Registry) error { + var res []error + + if err := o.validateThresholds(formats); err != nil { + res = append(res, err) + } + + if len(res) > 0 { + return errors.CompositeValidationError(res...) + } + return nil +} + +func (o *ListThresholdsOKBody) validateThresholds(formats strfmt.Registry) error { + if swag.IsZero(o.Thresholds) { // not required + return nil + } + + for i := 0; i < len(o.Thresholds); i++ { + if swag.IsZero(o.Thresholds[i]) { // not required + continue + } + + if o.Thresholds[i] != nil { + if err := o.Thresholds[i].Validate(formats); err != nil { + ve := new(errors.Validation) + if stderrors.As(err, &ve) { + return ve.ValidateName("listThresholdsOk" + "." + "thresholds" + "." + strconv.Itoa(i)) + } + ce := new(errors.CompositeError) + if stderrors.As(err, &ce) { + return ce.ValidateName("listThresholdsOk" + "." + "thresholds" + "." + strconv.Itoa(i)) + } + + return err + } + } + + } + + return nil +} + +// ContextValidate validate this list thresholds OK body based on the context it is used +func (o *ListThresholdsOKBody) ContextValidate(ctx context.Context, formats strfmt.Registry) error { + var res []error + + if err := o.contextValidateThresholds(ctx, formats); err != nil { + res = append(res, err) + } + + if len(res) > 0 { + return errors.CompositeValidationError(res...) + } + return nil +} + +func (o *ListThresholdsOKBody) contextValidateThresholds(ctx context.Context, formats strfmt.Registry) error { + for i := 0; i < len(o.Thresholds); i++ { + if o.Thresholds[i] != nil { + + if swag.IsZero(o.Thresholds[i]) { // not required + return nil + } + + if err := o.Thresholds[i].ContextValidate(ctx, formats); err != nil { + ve := new(errors.Validation) + if stderrors.As(err, &ve) { + return ve.ValidateName("listThresholdsOk" + "." + "thresholds" + "." + strconv.Itoa(i)) + } + ce := new(errors.CompositeError) + if stderrors.As(err, &ce) { + return ce.ValidateName("listThresholdsOk" + "." + "thresholds" + "." + strconv.Itoa(i)) + } + + return err + } + } + } + + return nil +} + +// MarshalBinary interface implementation +func (o *ListThresholdsOKBody) MarshalBinary() ([]byte, error) { + if o == nil { + return nil, nil + } + return swag.WriteJSON(o) +} + +// UnmarshalBinary interface implementation +func (o *ListThresholdsOKBody) UnmarshalBinary(b []byte) error { + var res ListThresholdsOKBody + if err := swag.ReadJSON(b, &res); err != nil { + return err + } + *o = res + return nil +} + +/* +ListThresholdsOKBodyThresholdsItems0 Threshold is one overridable parameter of one rule, as it applies to one target. +swagger:model ListThresholdsOKBodyThresholdsItems0 +*/ +type ListThresholdsOKBodyThresholdsItems0 struct { + // Identifier PMM assigned to the rule. Not unique within a response: rules duplicated + // in Grafana share it, so two entries can carry the same rule_id and param_name and + // differ only in which rule they came from. Do not key a map on it. + RuleID string `json:"rule_id,omitempty"` + + // Machine-readable name of the overridable parameter. + ParamName string `json:"param_name,omitempty"` + + // Short human-readable parameter summary, as it was when the rule was created. + Summary string `json:"summary,omitempty"` + + // ParamUnit represents template parameter unit. + // + // - PARAM_UNIT_UNSPECIFIED: Invalid, unknown or absent. + // - PARAM_UNIT_PERCENTAGE: % + // - PARAM_UNIT_SECONDS: s + // Enum: ["PARAM_UNIT_UNSPECIFIED","PARAM_UNIT_PERCENTAGE","PARAM_UNIT_SECONDS"] + Unit *string `json:"unit,omitempty"` + + // Value the rule falls back to when no override applies. + DefaultValue float64 `json:"default_value,omitempty"` + + // Value the rule currently evaluates this target against. + EffectiveValue float64 `json:"effective_value,omitempty"` + + // Whether effective_value comes from an override rather than the default. + IsOverridden bool `json:"is_overridden,omitempty"` + + // ThresholdScope says what a threshold override's target refers to. + // + // - THRESHOLD_SCOPE_NODE: Target is a Node ID. + // - THRESHOLD_SCOPE_SERVICE: Target is a Service ID. + // - THRESHOLD_SCOPE_CLUSTER: Target is a cluster label value. Unlike the others it names no inventory entity, + // so it cannot be validated for existence and is never removed by entity deletion. + // Enum: ["THRESHOLD_SCOPE_UNSPECIFIED","THRESHOLD_SCOPE_NODE","THRESHOLD_SCOPE_SERVICE","THRESHOLD_SCOPE_CLUSTER"] + Scope *string `json:"scope,omitempty"` + + // Target the effective override was set on. Empty when not overridden. + Target string `json:"target,omitempty"` +} + +// Validate validates this list thresholds OK body thresholds items0 +func (o *ListThresholdsOKBodyThresholdsItems0) Validate(formats strfmt.Registry) error { + var res []error + + if err := o.validateUnit(formats); err != nil { + res = append(res, err) + } + + if err := o.validateScope(formats); err != nil { + res = append(res, err) + } + + if len(res) > 0 { + return errors.CompositeValidationError(res...) + } + return nil +} + +var listThresholdsOkBodyThresholdsItems0TypeUnitPropEnum []any + +func init() { + var res []string + if err := json.Unmarshal([]byte(`["PARAM_UNIT_UNSPECIFIED","PARAM_UNIT_PERCENTAGE","PARAM_UNIT_SECONDS"]`), &res); err != nil { + panic(err) + } + for _, v := range res { + listThresholdsOkBodyThresholdsItems0TypeUnitPropEnum = append(listThresholdsOkBodyThresholdsItems0TypeUnitPropEnum, v) + } +} + +const ( + + // ListThresholdsOKBodyThresholdsItems0UnitPARAMUNITUNSPECIFIED captures enum value "PARAM_UNIT_UNSPECIFIED" + ListThresholdsOKBodyThresholdsItems0UnitPARAMUNITUNSPECIFIED string = "PARAM_UNIT_UNSPECIFIED" + + // ListThresholdsOKBodyThresholdsItems0UnitPARAMUNITPERCENTAGE captures enum value "PARAM_UNIT_PERCENTAGE" + ListThresholdsOKBodyThresholdsItems0UnitPARAMUNITPERCENTAGE string = "PARAM_UNIT_PERCENTAGE" + + // ListThresholdsOKBodyThresholdsItems0UnitPARAMUNITSECONDS captures enum value "PARAM_UNIT_SECONDS" + ListThresholdsOKBodyThresholdsItems0UnitPARAMUNITSECONDS string = "PARAM_UNIT_SECONDS" +) + +// prop value enum +func (o *ListThresholdsOKBodyThresholdsItems0) validateUnitEnum(path, location string, value string) error { + if err := validate.EnumCase(path, location, value, listThresholdsOkBodyThresholdsItems0TypeUnitPropEnum, true); err != nil { + return err + } + return nil +} + +func (o *ListThresholdsOKBodyThresholdsItems0) validateUnit(formats strfmt.Registry) error { + if swag.IsZero(o.Unit) { // not required + return nil + } + + // value enum + if err := o.validateUnitEnum("unit", "body", *o.Unit); err != nil { + return err + } + + return nil +} + +var listThresholdsOkBodyThresholdsItems0TypeScopePropEnum []any + +func init() { + var res []string + if err := json.Unmarshal([]byte(`["THRESHOLD_SCOPE_UNSPECIFIED","THRESHOLD_SCOPE_NODE","THRESHOLD_SCOPE_SERVICE","THRESHOLD_SCOPE_CLUSTER"]`), &res); err != nil { + panic(err) + } + for _, v := range res { + listThresholdsOkBodyThresholdsItems0TypeScopePropEnum = append(listThresholdsOkBodyThresholdsItems0TypeScopePropEnum, v) + } +} + +const ( + + // ListThresholdsOKBodyThresholdsItems0ScopeTHRESHOLDSCOPEUNSPECIFIED captures enum value "THRESHOLD_SCOPE_UNSPECIFIED" + ListThresholdsOKBodyThresholdsItems0ScopeTHRESHOLDSCOPEUNSPECIFIED string = "THRESHOLD_SCOPE_UNSPECIFIED" + + // ListThresholdsOKBodyThresholdsItems0ScopeTHRESHOLDSCOPENODE captures enum value "THRESHOLD_SCOPE_NODE" + ListThresholdsOKBodyThresholdsItems0ScopeTHRESHOLDSCOPENODE string = "THRESHOLD_SCOPE_NODE" + + // ListThresholdsOKBodyThresholdsItems0ScopeTHRESHOLDSCOPESERVICE captures enum value "THRESHOLD_SCOPE_SERVICE" + ListThresholdsOKBodyThresholdsItems0ScopeTHRESHOLDSCOPESERVICE string = "THRESHOLD_SCOPE_SERVICE" + + // ListThresholdsOKBodyThresholdsItems0ScopeTHRESHOLDSCOPECLUSTER captures enum value "THRESHOLD_SCOPE_CLUSTER" + ListThresholdsOKBodyThresholdsItems0ScopeTHRESHOLDSCOPECLUSTER string = "THRESHOLD_SCOPE_CLUSTER" +) + +// prop value enum +func (o *ListThresholdsOKBodyThresholdsItems0) validateScopeEnum(path, location string, value string) error { + if err := validate.EnumCase(path, location, value, listThresholdsOkBodyThresholdsItems0TypeScopePropEnum, true); err != nil { + return err + } + return nil +} + +func (o *ListThresholdsOKBodyThresholdsItems0) validateScope(formats strfmt.Registry) error { + if swag.IsZero(o.Scope) { // not required + return nil + } + + // value enum + if err := o.validateScopeEnum("scope", "body", *o.Scope); err != nil { + return err + } + + return nil +} + +// ContextValidate validates this list thresholds OK body thresholds items0 based on context it is used +func (o *ListThresholdsOKBodyThresholdsItems0) ContextValidate(ctx context.Context, formats strfmt.Registry) error { + return nil +} + +// MarshalBinary interface implementation +func (o *ListThresholdsOKBodyThresholdsItems0) MarshalBinary() ([]byte, error) { + if o == nil { + return nil, nil + } + return swag.WriteJSON(o) +} + +// UnmarshalBinary interface implementation +func (o *ListThresholdsOKBodyThresholdsItems0) UnmarshalBinary(b []byte) error { + var res ListThresholdsOKBodyThresholdsItems0 + if err := swag.ReadJSON(b, &res); err != nil { + return err + } + *o = res + return nil +} diff --git a/api/alerting/v1/json/client/alerting_service/set_threshold_parameters.go b/api/alerting/v1/json/client/alerting_service/set_threshold_parameters.go new file mode 100644 index 0000000000..79faf72993 --- /dev/null +++ b/api/alerting/v1/json/client/alerting_service/set_threshold_parameters.go @@ -0,0 +1,141 @@ +// Code generated by go-swagger; DO NOT EDIT. + +package alerting_service + +import ( + "context" + "net/http" + "time" + + "github.com/go-openapi/errors" + "github.com/go-openapi/runtime" + cr "github.com/go-openapi/runtime/client" + "github.com/go-openapi/strfmt" +) + +// NewSetThresholdParams creates a new SetThresholdParams object, +// with the default timeout for this client. +// +// Default values are not hydrated, since defaults are normally applied by the API server side. +// +// To enforce default values in parameter, use SetDefaults or WithDefaults. +func NewSetThresholdParams() *SetThresholdParams { + return &SetThresholdParams{ + timeout: cr.DefaultTimeout, + } +} + +// NewSetThresholdParamsWithTimeout creates a new SetThresholdParams object +// with the ability to set a timeout on a request. +func NewSetThresholdParamsWithTimeout(timeout time.Duration) *SetThresholdParams { + return &SetThresholdParams{ + timeout: timeout, + } +} + +// NewSetThresholdParamsWithContext creates a new SetThresholdParams object +// with the ability to set a context for a request. +func NewSetThresholdParamsWithContext(ctx context.Context) *SetThresholdParams { + return &SetThresholdParams{ + Context: ctx, + } +} + +// NewSetThresholdParamsWithHTTPClient creates a new SetThresholdParams object +// with the ability to set a custom HTTPClient for a request. +func NewSetThresholdParamsWithHTTPClient(client *http.Client) *SetThresholdParams { + return &SetThresholdParams{ + HTTPClient: client, + } +} + +/* +SetThresholdParams contains all the parameters to send to the API endpoint + + for the set threshold operation. + + Typically these are written to a http.Request. +*/ +type SetThresholdParams struct { + // Body. + Body SetThresholdBody + + timeout time.Duration + Context context.Context + HTTPClient *http.Client +} + +// WithDefaults hydrates default values in the set threshold params (not the query body). +// +// All values with no default are reset to their zero value. +func (o *SetThresholdParams) WithDefaults() *SetThresholdParams { + o.SetDefaults() + return o +} + +// SetDefaults hydrates default values in the set threshold params (not the query body). +// +// All values with no default are reset to their zero value. +func (o *SetThresholdParams) SetDefaults() { + // no default values defined for this parameter +} + +// WithTimeout adds the timeout to the set threshold params +func (o *SetThresholdParams) WithTimeout(timeout time.Duration) *SetThresholdParams { + o.SetTimeout(timeout) + return o +} + +// SetTimeout adds the timeout to the set threshold params +func (o *SetThresholdParams) SetTimeout(timeout time.Duration) { + o.timeout = timeout +} + +// WithContext adds the context to the set threshold params +func (o *SetThresholdParams) WithContext(ctx context.Context) *SetThresholdParams { + o.SetContext(ctx) + return o +} + +// SetContext adds the context to the set threshold params +func (o *SetThresholdParams) SetContext(ctx context.Context) { + o.Context = ctx +} + +// WithHTTPClient adds the HTTPClient to the set threshold params +func (o *SetThresholdParams) WithHTTPClient(client *http.Client) *SetThresholdParams { + o.SetHTTPClient(client) + return o +} + +// SetHTTPClient adds the HTTPClient to the set threshold params +func (o *SetThresholdParams) SetHTTPClient(client *http.Client) { + o.HTTPClient = client +} + +// WithBody adds the body to the set threshold params +func (o *SetThresholdParams) WithBody(body SetThresholdBody) *SetThresholdParams { + o.SetBody(body) + return o +} + +// SetBody adds the body to the set threshold params +func (o *SetThresholdParams) SetBody(body SetThresholdBody) { + o.Body = body +} + +// WriteToRequest writes these params to a swagger request +func (o *SetThresholdParams) WriteToRequest(r runtime.ClientRequest, reg strfmt.Registry) error { + if err := r.SetTimeout(o.timeout); err != nil { + return err + } + var res []error + if err := r.SetBodyParam(o.Body); err != nil { + return err + } + + if len(res) > 0 { + return errors.CompositeValidationError(res...) + } + return nil +} diff --git a/api/alerting/v1/json/client/alerting_service/set_threshold_responses.go b/api/alerting/v1/json/client/alerting_service/set_threshold_responses.go new file mode 100644 index 0000000000..3ad67187cc --- /dev/null +++ b/api/alerting/v1/json/client/alerting_service/set_threshold_responses.go @@ -0,0 +1,808 @@ +// Code generated by go-swagger; DO NOT EDIT. + +package alerting_service + +import ( + "context" + "encoding/json" + stderrors "errors" + "fmt" + "io" + "strconv" + + "github.com/go-openapi/errors" + "github.com/go-openapi/runtime" + "github.com/go-openapi/strfmt" + "github.com/go-openapi/swag" + "github.com/go-openapi/validate" +) + +// SetThresholdReader is a Reader for the SetThreshold structure. +type SetThresholdReader struct { + formats strfmt.Registry +} + +// ReadResponse reads a server response into the received o. +func (o *SetThresholdReader) ReadResponse(response runtime.ClientResponse, consumer runtime.Consumer) (any, error) { + switch response.Code() { + case 200: + result := NewSetThresholdOK() + if err := result.readResponse(response, consumer, o.formats); err != nil { + return nil, err + } + return result, nil + default: + result := NewSetThresholdDefault(response.Code()) + if err := result.readResponse(response, consumer, o.formats); err != nil { + return nil, err + } + if response.Code()/100 == 2 { + return result, nil + } + return nil, result + } +} + +// NewSetThresholdOK creates a SetThresholdOK with default headers values +func NewSetThresholdOK() *SetThresholdOK { + return &SetThresholdOK{} +} + +/* +SetThresholdOK describes a response with status code 200, with default header values. + +A successful response. +*/ +type SetThresholdOK struct { + Payload *SetThresholdOKBody +} + +// IsSuccess returns true when this set threshold Ok response has a 2xx status code +func (o *SetThresholdOK) IsSuccess() bool { + return true +} + +// IsRedirect returns true when this set threshold Ok response has a 3xx status code +func (o *SetThresholdOK) IsRedirect() bool { + return false +} + +// IsClientError returns true when this set threshold Ok response has a 4xx status code +func (o *SetThresholdOK) IsClientError() bool { + return false +} + +// IsServerError returns true when this set threshold Ok response has a 5xx status code +func (o *SetThresholdOK) IsServerError() bool { + return false +} + +// IsCode returns true when this set threshold Ok response a status code equal to that given +func (o *SetThresholdOK) IsCode(code int) bool { + return code == 200 +} + +// Code gets the status code for the set threshold Ok response +func (o *SetThresholdOK) Code() int { + return 200 +} + +func (o *SetThresholdOK) Error() string { + payload, _ := json.Marshal(o.Payload) + return fmt.Sprintf("[POST /v1/alerting/thresholds][%d] setThresholdOk %s", 200, payload) +} + +func (o *SetThresholdOK) String() string { + payload, _ := json.Marshal(o.Payload) + return fmt.Sprintf("[POST /v1/alerting/thresholds][%d] setThresholdOk %s", 200, payload) +} + +func (o *SetThresholdOK) GetPayload() *SetThresholdOKBody { + return o.Payload +} + +func (o *SetThresholdOK) readResponse(response runtime.ClientResponse, consumer runtime.Consumer, formats strfmt.Registry) error { + o.Payload = new(SetThresholdOKBody) + + // response payload + if err := consumer.Consume(response.Body(), o.Payload); err != nil && !stderrors.Is(err, io.EOF) { + return err + } + + return nil +} + +// NewSetThresholdDefault creates a SetThresholdDefault with default headers values +func NewSetThresholdDefault(code int) *SetThresholdDefault { + return &SetThresholdDefault{ + _statusCode: code, + } +} + +/* +SetThresholdDefault describes a response with status code -1, with default header values. + +An unexpected error response. +*/ +type SetThresholdDefault struct { + _statusCode int + + Payload *SetThresholdDefaultBody +} + +// IsSuccess returns true when this set threshold default response has a 2xx status code +func (o *SetThresholdDefault) IsSuccess() bool { + return o._statusCode/100 == 2 +} + +// IsRedirect returns true when this set threshold default response has a 3xx status code +func (o *SetThresholdDefault) IsRedirect() bool { + return o._statusCode/100 == 3 +} + +// IsClientError returns true when this set threshold default response has a 4xx status code +func (o *SetThresholdDefault) IsClientError() bool { + return o._statusCode/100 == 4 +} + +// IsServerError returns true when this set threshold default response has a 5xx status code +func (o *SetThresholdDefault) IsServerError() bool { + return o._statusCode/100 == 5 +} + +// IsCode returns true when this set threshold default response a status code equal to that given +func (o *SetThresholdDefault) IsCode(code int) bool { + return o._statusCode == code +} + +// Code gets the status code for the set threshold default response +func (o *SetThresholdDefault) Code() int { + return o._statusCode +} + +func (o *SetThresholdDefault) Error() string { + payload, _ := json.Marshal(o.Payload) + return fmt.Sprintf("[POST /v1/alerting/thresholds][%d] SetThreshold default %s", o._statusCode, payload) +} + +func (o *SetThresholdDefault) String() string { + payload, _ := json.Marshal(o.Payload) + return fmt.Sprintf("[POST /v1/alerting/thresholds][%d] SetThreshold default %s", o._statusCode, payload) +} + +func (o *SetThresholdDefault) GetPayload() *SetThresholdDefaultBody { + return o.Payload +} + +func (o *SetThresholdDefault) readResponse(response runtime.ClientResponse, consumer runtime.Consumer, formats strfmt.Registry) error { + o.Payload = new(SetThresholdDefaultBody) + + // response payload + if err := consumer.Consume(response.Body(), o.Payload); err != nil && !stderrors.Is(err, io.EOF) { + return err + } + + return nil +} + +/* +SetThresholdBody set threshold body +swagger:model SetThresholdBody +*/ +type SetThresholdBody struct { + // ThresholdScope says what a threshold override's target refers to. + // + // - THRESHOLD_SCOPE_NODE: Target is a Node ID. + // - THRESHOLD_SCOPE_SERVICE: Target is a Service ID. + // - THRESHOLD_SCOPE_CLUSTER: Target is a cluster label value. Unlike the others it names no inventory entity, + // so it cannot be validated for existence and is never removed by entity deletion. + // Enum: ["THRESHOLD_SCOPE_UNSPECIFIED","THRESHOLD_SCOPE_NODE","THRESHOLD_SCOPE_SERVICE","THRESHOLD_SCOPE_CLUSTER"] + Scope *string `json:"scope,omitempty"` + + // target + Target string `json:"target,omitempty"` + + // rule id + RuleID string `json:"rule_id,omitempty"` + + // param name + ParamName string `json:"param_name,omitempty"` + + // Must be finite and within the parameter's declared range. + Value float64 `json:"value,omitempty"` +} + +// Validate validates this set threshold body +func (o *SetThresholdBody) Validate(formats strfmt.Registry) error { + var res []error + + if err := o.validateScope(formats); err != nil { + res = append(res, err) + } + + if len(res) > 0 { + return errors.CompositeValidationError(res...) + } + return nil +} + +var setThresholdBodyTypeScopePropEnum []any + +func init() { + var res []string + if err := json.Unmarshal([]byte(`["THRESHOLD_SCOPE_UNSPECIFIED","THRESHOLD_SCOPE_NODE","THRESHOLD_SCOPE_SERVICE","THRESHOLD_SCOPE_CLUSTER"]`), &res); err != nil { + panic(err) + } + for _, v := range res { + setThresholdBodyTypeScopePropEnum = append(setThresholdBodyTypeScopePropEnum, v) + } +} + +const ( + + // SetThresholdBodyScopeTHRESHOLDSCOPEUNSPECIFIED captures enum value "THRESHOLD_SCOPE_UNSPECIFIED" + SetThresholdBodyScopeTHRESHOLDSCOPEUNSPECIFIED string = "THRESHOLD_SCOPE_UNSPECIFIED" + + // SetThresholdBodyScopeTHRESHOLDSCOPENODE captures enum value "THRESHOLD_SCOPE_NODE" + SetThresholdBodyScopeTHRESHOLDSCOPENODE string = "THRESHOLD_SCOPE_NODE" + + // SetThresholdBodyScopeTHRESHOLDSCOPESERVICE captures enum value "THRESHOLD_SCOPE_SERVICE" + SetThresholdBodyScopeTHRESHOLDSCOPESERVICE string = "THRESHOLD_SCOPE_SERVICE" + + // SetThresholdBodyScopeTHRESHOLDSCOPECLUSTER captures enum value "THRESHOLD_SCOPE_CLUSTER" + SetThresholdBodyScopeTHRESHOLDSCOPECLUSTER string = "THRESHOLD_SCOPE_CLUSTER" +) + +// prop value enum +func (o *SetThresholdBody) validateScopeEnum(path, location string, value string) error { + if err := validate.EnumCase(path, location, value, setThresholdBodyTypeScopePropEnum, true); err != nil { + return err + } + return nil +} + +func (o *SetThresholdBody) validateScope(formats strfmt.Registry) error { + if swag.IsZero(o.Scope) { // not required + return nil + } + + // value enum + if err := o.validateScopeEnum("body"+"."+"scope", "body", *o.Scope); err != nil { + return err + } + + return nil +} + +// ContextValidate validates this set threshold body based on context it is used +func (o *SetThresholdBody) ContextValidate(ctx context.Context, formats strfmt.Registry) error { + return nil +} + +// MarshalBinary interface implementation +func (o *SetThresholdBody) MarshalBinary() ([]byte, error) { + if o == nil { + return nil, nil + } + return swag.WriteJSON(o) +} + +// UnmarshalBinary interface implementation +func (o *SetThresholdBody) UnmarshalBinary(b []byte) error { + var res SetThresholdBody + if err := swag.ReadJSON(b, &res); err != nil { + return err + } + *o = res + return nil +} + +/* +SetThresholdDefaultBody set threshold default body +swagger:model SetThresholdDefaultBody +*/ +type SetThresholdDefaultBody struct { + // code + Code int32 `json:"code,omitempty"` + + // message + Message string `json:"message,omitempty"` + + // details + Details []*SetThresholdDefaultBodyDetailsItems0 `json:"details"` +} + +// Validate validates this set threshold default body +func (o *SetThresholdDefaultBody) Validate(formats strfmt.Registry) error { + var res []error + + if err := o.validateDetails(formats); err != nil { + res = append(res, err) + } + + if len(res) > 0 { + return errors.CompositeValidationError(res...) + } + return nil +} + +func (o *SetThresholdDefaultBody) validateDetails(formats strfmt.Registry) error { + if swag.IsZero(o.Details) { // not required + return nil + } + + for i := 0; i < len(o.Details); i++ { + if swag.IsZero(o.Details[i]) { // not required + continue + } + + if o.Details[i] != nil { + if err := o.Details[i].Validate(formats); err != nil { + ve := new(errors.Validation) + if stderrors.As(err, &ve) { + return ve.ValidateName("SetThreshold default" + "." + "details" + "." + strconv.Itoa(i)) + } + ce := new(errors.CompositeError) + if stderrors.As(err, &ce) { + return ce.ValidateName("SetThreshold default" + "." + "details" + "." + strconv.Itoa(i)) + } + + return err + } + } + + } + + return nil +} + +// ContextValidate validate this set threshold default body based on the context it is used +func (o *SetThresholdDefaultBody) ContextValidate(ctx context.Context, formats strfmt.Registry) error { + var res []error + + if err := o.contextValidateDetails(ctx, formats); err != nil { + res = append(res, err) + } + + if len(res) > 0 { + return errors.CompositeValidationError(res...) + } + return nil +} + +func (o *SetThresholdDefaultBody) contextValidateDetails(ctx context.Context, formats strfmt.Registry) error { + for i := 0; i < len(o.Details); i++ { + if o.Details[i] != nil { + + if swag.IsZero(o.Details[i]) { // not required + return nil + } + + if err := o.Details[i].ContextValidate(ctx, formats); err != nil { + ve := new(errors.Validation) + if stderrors.As(err, &ve) { + return ve.ValidateName("SetThreshold default" + "." + "details" + "." + strconv.Itoa(i)) + } + ce := new(errors.CompositeError) + if stderrors.As(err, &ce) { + return ce.ValidateName("SetThreshold default" + "." + "details" + "." + strconv.Itoa(i)) + } + + return err + } + } + } + + return nil +} + +// MarshalBinary interface implementation +func (o *SetThresholdDefaultBody) MarshalBinary() ([]byte, error) { + if o == nil { + return nil, nil + } + return swag.WriteJSON(o) +} + +// UnmarshalBinary interface implementation +func (o *SetThresholdDefaultBody) UnmarshalBinary(b []byte) error { + var res SetThresholdDefaultBody + if err := swag.ReadJSON(b, &res); err != nil { + return err + } + *o = res + return nil +} + +/* +SetThresholdDefaultBodyDetailsItems0 set threshold default body details items0 +swagger:model SetThresholdDefaultBodyDetailsItems0 +*/ +type SetThresholdDefaultBodyDetailsItems0 struct { + // at type + AtType string `json:"@type,omitempty"` + + // set threshold default body details items0 + SetThresholdDefaultBodyDetailsItems0 map[string]any `json:"-"` +} + +// UnmarshalJSON unmarshals this object with additional properties from JSON +func (o *SetThresholdDefaultBodyDetailsItems0) UnmarshalJSON(data []byte) error { + // stage 1, bind the properties + var stage1 struct { + // at type + AtType string `json:"@type,omitempty"` + } + if err := json.Unmarshal(data, &stage1); err != nil { + return err + } + var rcv SetThresholdDefaultBodyDetailsItems0 + + rcv.AtType = stage1.AtType + *o = rcv + + // stage 2, remove properties and add to map + stage2 := make(map[string]json.RawMessage) + if err := json.Unmarshal(data, &stage2); err != nil { + return err + } + + delete(stage2, "@type") + // stage 3, add additional properties values + if len(stage2) > 0 { + result := make(map[string]any) + for k, v := range stage2 { + var toadd any + if err := json.Unmarshal(v, &toadd); err != nil { + return err + } + result[k] = toadd + } + o.SetThresholdDefaultBodyDetailsItems0 = result + } + + return nil +} + +// MarshalJSON marshals this object with additional properties into a JSON object +func (o SetThresholdDefaultBodyDetailsItems0) MarshalJSON() ([]byte, error) { + var stage1 struct { + // at type + AtType string `json:"@type,omitempty"` + } + + stage1.AtType = o.AtType + + // make JSON object for known properties + props, err := json.Marshal(stage1) + if err != nil { + return nil, err + } + + if len(o.SetThresholdDefaultBodyDetailsItems0) == 0 { // no additional properties + return props, nil + } + + // make JSON object for the additional properties + additional, err := json.Marshal(o.SetThresholdDefaultBodyDetailsItems0) + if err != nil { + return nil, err + } + + if len(props) < 3 { // "{}": only additional properties + return additional, nil + } + + // concatenate the 2 objects + return swag.ConcatJSON(props, additional), nil +} + +// Validate validates this set threshold default body details items0 +func (o *SetThresholdDefaultBodyDetailsItems0) Validate(formats strfmt.Registry) error { + return nil +} + +// ContextValidate validates this set threshold default body details items0 based on context it is used +func (o *SetThresholdDefaultBodyDetailsItems0) ContextValidate(ctx context.Context, formats strfmt.Registry) error { + return nil +} + +// MarshalBinary interface implementation +func (o *SetThresholdDefaultBodyDetailsItems0) MarshalBinary() ([]byte, error) { + if o == nil { + return nil, nil + } + return swag.WriteJSON(o) +} + +// UnmarshalBinary interface implementation +func (o *SetThresholdDefaultBodyDetailsItems0) UnmarshalBinary(b []byte) error { + var res SetThresholdDefaultBodyDetailsItems0 + if err := swag.ReadJSON(b, &res); err != nil { + return err + } + *o = res + return nil +} + +/* +SetThresholdOKBody set threshold OK body +swagger:model SetThresholdOKBody +*/ +type SetThresholdOKBody struct { + // threshold + Threshold *SetThresholdOKBodyThreshold `json:"threshold,omitempty"` +} + +// Validate validates this set threshold OK body +func (o *SetThresholdOKBody) Validate(formats strfmt.Registry) error { + var res []error + + if err := o.validateThreshold(formats); err != nil { + res = append(res, err) + } + + if len(res) > 0 { + return errors.CompositeValidationError(res...) + } + return nil +} + +func (o *SetThresholdOKBody) validateThreshold(formats strfmt.Registry) error { + if swag.IsZero(o.Threshold) { // not required + return nil + } + + if o.Threshold != nil { + if err := o.Threshold.Validate(formats); err != nil { + ve := new(errors.Validation) + if stderrors.As(err, &ve) { + return ve.ValidateName("setThresholdOk" + "." + "threshold") + } + ce := new(errors.CompositeError) + if stderrors.As(err, &ce) { + return ce.ValidateName("setThresholdOk" + "." + "threshold") + } + + return err + } + } + + return nil +} + +// ContextValidate validate this set threshold OK body based on the context it is used +func (o *SetThresholdOKBody) ContextValidate(ctx context.Context, formats strfmt.Registry) error { + var res []error + + if err := o.contextValidateThreshold(ctx, formats); err != nil { + res = append(res, err) + } + + if len(res) > 0 { + return errors.CompositeValidationError(res...) + } + return nil +} + +func (o *SetThresholdOKBody) contextValidateThreshold(ctx context.Context, formats strfmt.Registry) error { + if o.Threshold != nil { + + if swag.IsZero(o.Threshold) { // not required + return nil + } + + if err := o.Threshold.ContextValidate(ctx, formats); err != nil { + ve := new(errors.Validation) + if stderrors.As(err, &ve) { + return ve.ValidateName("setThresholdOk" + "." + "threshold") + } + ce := new(errors.CompositeError) + if stderrors.As(err, &ce) { + return ce.ValidateName("setThresholdOk" + "." + "threshold") + } + + return err + } + } + + return nil +} + +// MarshalBinary interface implementation +func (o *SetThresholdOKBody) MarshalBinary() ([]byte, error) { + if o == nil { + return nil, nil + } + return swag.WriteJSON(o) +} + +// UnmarshalBinary interface implementation +func (o *SetThresholdOKBody) UnmarshalBinary(b []byte) error { + var res SetThresholdOKBody + if err := swag.ReadJSON(b, &res); err != nil { + return err + } + *o = res + return nil +} + +/* +SetThresholdOKBodyThreshold Threshold is one overridable parameter of one rule, as it applies to one target. +swagger:model SetThresholdOKBodyThreshold +*/ +type SetThresholdOKBodyThreshold struct { + // Identifier PMM assigned to the rule. Not unique within a response: rules duplicated + // in Grafana share it, so two entries can carry the same rule_id and param_name and + // differ only in which rule they came from. Do not key a map on it. + RuleID string `json:"rule_id,omitempty"` + + // Machine-readable name of the overridable parameter. + ParamName string `json:"param_name,omitempty"` + + // Short human-readable parameter summary, as it was when the rule was created. + Summary string `json:"summary,omitempty"` + + // ParamUnit represents template parameter unit. + // + // - PARAM_UNIT_UNSPECIFIED: Invalid, unknown or absent. + // - PARAM_UNIT_PERCENTAGE: % + // - PARAM_UNIT_SECONDS: s + // Enum: ["PARAM_UNIT_UNSPECIFIED","PARAM_UNIT_PERCENTAGE","PARAM_UNIT_SECONDS"] + Unit *string `json:"unit,omitempty"` + + // Value the rule falls back to when no override applies. + DefaultValue float64 `json:"default_value,omitempty"` + + // Value the rule currently evaluates this target against. + EffectiveValue float64 `json:"effective_value,omitempty"` + + // Whether effective_value comes from an override rather than the default. + IsOverridden bool `json:"is_overridden,omitempty"` + + // ThresholdScope says what a threshold override's target refers to. + // + // - THRESHOLD_SCOPE_NODE: Target is a Node ID. + // - THRESHOLD_SCOPE_SERVICE: Target is a Service ID. + // - THRESHOLD_SCOPE_CLUSTER: Target is a cluster label value. Unlike the others it names no inventory entity, + // so it cannot be validated for existence and is never removed by entity deletion. + // Enum: ["THRESHOLD_SCOPE_UNSPECIFIED","THRESHOLD_SCOPE_NODE","THRESHOLD_SCOPE_SERVICE","THRESHOLD_SCOPE_CLUSTER"] + Scope *string `json:"scope,omitempty"` + + // Target the effective override was set on. Empty when not overridden. + Target string `json:"target,omitempty"` +} + +// Validate validates this set threshold OK body threshold +func (o *SetThresholdOKBodyThreshold) Validate(formats strfmt.Registry) error { + var res []error + + if err := o.validateUnit(formats); err != nil { + res = append(res, err) + } + + if err := o.validateScope(formats); err != nil { + res = append(res, err) + } + + if len(res) > 0 { + return errors.CompositeValidationError(res...) + } + return nil +} + +var setThresholdOkBodyThresholdTypeUnitPropEnum []any + +func init() { + var res []string + if err := json.Unmarshal([]byte(`["PARAM_UNIT_UNSPECIFIED","PARAM_UNIT_PERCENTAGE","PARAM_UNIT_SECONDS"]`), &res); err != nil { + panic(err) + } + for _, v := range res { + setThresholdOkBodyThresholdTypeUnitPropEnum = append(setThresholdOkBodyThresholdTypeUnitPropEnum, v) + } +} + +const ( + + // SetThresholdOKBodyThresholdUnitPARAMUNITUNSPECIFIED captures enum value "PARAM_UNIT_UNSPECIFIED" + SetThresholdOKBodyThresholdUnitPARAMUNITUNSPECIFIED string = "PARAM_UNIT_UNSPECIFIED" + + // SetThresholdOKBodyThresholdUnitPARAMUNITPERCENTAGE captures enum value "PARAM_UNIT_PERCENTAGE" + SetThresholdOKBodyThresholdUnitPARAMUNITPERCENTAGE string = "PARAM_UNIT_PERCENTAGE" + + // SetThresholdOKBodyThresholdUnitPARAMUNITSECONDS captures enum value "PARAM_UNIT_SECONDS" + SetThresholdOKBodyThresholdUnitPARAMUNITSECONDS string = "PARAM_UNIT_SECONDS" +) + +// prop value enum +func (o *SetThresholdOKBodyThreshold) validateUnitEnum(path, location string, value string) error { + if err := validate.EnumCase(path, location, value, setThresholdOkBodyThresholdTypeUnitPropEnum, true); err != nil { + return err + } + return nil +} + +func (o *SetThresholdOKBodyThreshold) validateUnit(formats strfmt.Registry) error { + if swag.IsZero(o.Unit) { // not required + return nil + } + + // value enum + if err := o.validateUnitEnum("setThresholdOk"+"."+"threshold"+"."+"unit", "body", *o.Unit); err != nil { + return err + } + + return nil +} + +var setThresholdOkBodyThresholdTypeScopePropEnum []any + +func init() { + var res []string + if err := json.Unmarshal([]byte(`["THRESHOLD_SCOPE_UNSPECIFIED","THRESHOLD_SCOPE_NODE","THRESHOLD_SCOPE_SERVICE","THRESHOLD_SCOPE_CLUSTER"]`), &res); err != nil { + panic(err) + } + for _, v := range res { + setThresholdOkBodyThresholdTypeScopePropEnum = append(setThresholdOkBodyThresholdTypeScopePropEnum, v) + } +} + +const ( + + // SetThresholdOKBodyThresholdScopeTHRESHOLDSCOPEUNSPECIFIED captures enum value "THRESHOLD_SCOPE_UNSPECIFIED" + SetThresholdOKBodyThresholdScopeTHRESHOLDSCOPEUNSPECIFIED string = "THRESHOLD_SCOPE_UNSPECIFIED" + + // SetThresholdOKBodyThresholdScopeTHRESHOLDSCOPENODE captures enum value "THRESHOLD_SCOPE_NODE" + SetThresholdOKBodyThresholdScopeTHRESHOLDSCOPENODE string = "THRESHOLD_SCOPE_NODE" + + // SetThresholdOKBodyThresholdScopeTHRESHOLDSCOPESERVICE captures enum value "THRESHOLD_SCOPE_SERVICE" + SetThresholdOKBodyThresholdScopeTHRESHOLDSCOPESERVICE string = "THRESHOLD_SCOPE_SERVICE" + + // SetThresholdOKBodyThresholdScopeTHRESHOLDSCOPECLUSTER captures enum value "THRESHOLD_SCOPE_CLUSTER" + SetThresholdOKBodyThresholdScopeTHRESHOLDSCOPECLUSTER string = "THRESHOLD_SCOPE_CLUSTER" +) + +// prop value enum +func (o *SetThresholdOKBodyThreshold) validateScopeEnum(path, location string, value string) error { + if err := validate.EnumCase(path, location, value, setThresholdOkBodyThresholdTypeScopePropEnum, true); err != nil { + return err + } + return nil +} + +func (o *SetThresholdOKBodyThreshold) validateScope(formats strfmt.Registry) error { + if swag.IsZero(o.Scope) { // not required + return nil + } + + // value enum + if err := o.validateScopeEnum("setThresholdOk"+"."+"threshold"+"."+"scope", "body", *o.Scope); err != nil { + return err + } + + return nil +} + +// ContextValidate validates this set threshold OK body threshold based on context it is used +func (o *SetThresholdOKBodyThreshold) ContextValidate(ctx context.Context, formats strfmt.Registry) error { + return nil +} + +// MarshalBinary interface implementation +func (o *SetThresholdOKBodyThreshold) MarshalBinary() ([]byte, error) { + if o == nil { + return nil, nil + } + return swag.WriteJSON(o) +} + +// UnmarshalBinary interface implementation +func (o *SetThresholdOKBodyThreshold) UnmarshalBinary(b []byte) error { + var res SetThresholdOKBodyThreshold + if err := swag.ReadJSON(b, &res); err != nil { + return err + } + *o = res + return nil +} diff --git a/api/alerting/v1/json/v1.json b/api/alerting/v1/json/v1.json index 0aca3c38ae..e80fae8eb0 100644 --- a/api/alerting/v1/json/v1.json +++ b/api/alerting/v1/json/v1.json @@ -720,6 +720,561 @@ } } } + }, + "/v1/alerting/thresholds": { + "get": { + "tags": [ + "AlertingService" + ], + "summary": "ListThresholds returns per-target threshold overrides.", + "operationId": "ListThresholds", + "parameters": [ + { + "enum": [ + "THRESHOLD_SCOPE_UNSPECIFIED", + "THRESHOLD_SCOPE_NODE", + "THRESHOLD_SCOPE_SERVICE", + "THRESHOLD_SCOPE_CLUSTER" + ], + "type": "string", + "default": "THRESHOLD_SCOPE_UNSPECIFIED", + "description": "Scope of the target to report thresholds for. Must be set together with target.\n\n - THRESHOLD_SCOPE_NODE: Target is a Node ID.\n - THRESHOLD_SCOPE_SERVICE: Target is a Service ID.\n - THRESHOLD_SCOPE_CLUSTER: Target is a cluster label value. Unlike the others it names no inventory entity,\nso it cannot be validated for existence and is never removed by entity deletion.", + "name": "scope", + "in": "query" + }, + { + "type": "string", + "description": "Target to report thresholds for. When set, every overridable parameter is returned\nfor that target, overridden or not. When empty, only existing overrides are\nreturned, since there is otherwise no bounded set to enumerate.", + "name": "target", + "in": "query" + }, + { + "type": "string", + "description": "Return only thresholds of this rule.", + "name": "rule_id", + "in": "query" + } + ], + "responses": { + "200": { + "description": "A successful response.", + "schema": { + "type": "object", + "properties": { + "thresholds": { + "type": "array", + "items": { + "description": "Threshold is one overridable parameter of one rule, as it applies to one target.", + "type": "object", + "properties": { + "rule_id": { + "description": "Identifier PMM assigned to the rule. Not unique within a response: rules duplicated\nin Grafana share it, so two entries can carry the same rule_id and param_name and\ndiffer only in which rule they came from. Do not key a map on it.", + "type": "string", + "x-order": 0 + }, + "param_name": { + "description": "Machine-readable name of the overridable parameter.", + "type": "string", + "x-order": 1 + }, + "summary": { + "description": "Short human-readable parameter summary, as it was when the rule was created.", + "type": "string", + "x-order": 2 + }, + "unit": { + "description": "ParamUnit represents template parameter unit.\n\n - PARAM_UNIT_UNSPECIFIED: Invalid, unknown or absent.\n - PARAM_UNIT_PERCENTAGE: %\n - PARAM_UNIT_SECONDS: s", + "type": "string", + "default": "PARAM_UNIT_UNSPECIFIED", + "enum": [ + "PARAM_UNIT_UNSPECIFIED", + "PARAM_UNIT_PERCENTAGE", + "PARAM_UNIT_SECONDS" + ], + "x-order": 3 + }, + "default_value": { + "description": "Value the rule falls back to when no override applies.", + "type": "number", + "format": "double", + "x-order": 4 + }, + "effective_value": { + "description": "Value the rule currently evaluates this target against.", + "type": "number", + "format": "double", + "x-order": 5 + }, + "is_overridden": { + "description": "Whether effective_value comes from an override rather than the default.", + "type": "boolean", + "x-order": 6 + }, + "scope": { + "description": "ThresholdScope says what a threshold override's target refers to.\n\n - THRESHOLD_SCOPE_NODE: Target is a Node ID.\n - THRESHOLD_SCOPE_SERVICE: Target is a Service ID.\n - THRESHOLD_SCOPE_CLUSTER: Target is a cluster label value. Unlike the others it names no inventory entity,\nso it cannot be validated for existence and is never removed by entity deletion.", + "type": "string", + "default": "THRESHOLD_SCOPE_UNSPECIFIED", + "enum": [ + "THRESHOLD_SCOPE_UNSPECIFIED", + "THRESHOLD_SCOPE_NODE", + "THRESHOLD_SCOPE_SERVICE", + "THRESHOLD_SCOPE_CLUSTER" + ], + "x-order": 7 + }, + "target": { + "description": "Target the effective override was set on. Empty when not overridden.", + "type": "string", + "x-order": 8 + } + } + }, + "x-order": 0 + } + } + } + }, + "default": { + "description": "An unexpected error response.", + "schema": { + "type": "object", + "properties": { + "code": { + "type": "integer", + "format": "int32", + "x-order": 0 + }, + "message": { + "type": "string", + "x-order": 1 + }, + "details": { + "type": "array", + "items": { + "type": "object", + "properties": { + "@type": { + "type": "string", + "x-order": 0 + } + }, + "additionalProperties": {} + }, + "x-order": 2 + } + } + } + } + } + }, + "post": { + "tags": [ + "AlertingService" + ], + "summary": "SetThreshold overrides one rule parameter for one target.", + "operationId": "SetThreshold", + "parameters": [ + { + "name": "body", + "in": "body", + "required": true, + "schema": { + "type": "object", + "properties": { + "scope": { + "description": "ThresholdScope says what a threshold override's target refers to.\n\n - THRESHOLD_SCOPE_NODE: Target is a Node ID.\n - THRESHOLD_SCOPE_SERVICE: Target is a Service ID.\n - THRESHOLD_SCOPE_CLUSTER: Target is a cluster label value. Unlike the others it names no inventory entity,\nso it cannot be validated for existence and is never removed by entity deletion.", + "type": "string", + "default": "THRESHOLD_SCOPE_UNSPECIFIED", + "enum": [ + "THRESHOLD_SCOPE_UNSPECIFIED", + "THRESHOLD_SCOPE_NODE", + "THRESHOLD_SCOPE_SERVICE", + "THRESHOLD_SCOPE_CLUSTER" + ], + "x-order": 0 + }, + "target": { + "type": "string", + "x-order": 1 + }, + "rule_id": { + "type": "string", + "x-order": 2 + }, + "param_name": { + "type": "string", + "x-order": 3 + }, + "value": { + "description": "Must be finite and within the parameter's declared range.", + "type": "number", + "format": "double", + "x-order": 4 + } + } + } + } + ], + "responses": { + "200": { + "description": "A successful response.", + "schema": { + "type": "object", + "properties": { + "threshold": { + "description": "Threshold is one overridable parameter of one rule, as it applies to one target.", + "type": "object", + "properties": { + "rule_id": { + "description": "Identifier PMM assigned to the rule. Not unique within a response: rules duplicated\nin Grafana share it, so two entries can carry the same rule_id and param_name and\ndiffer only in which rule they came from. Do not key a map on it.", + "type": "string", + "x-order": 0 + }, + "param_name": { + "description": "Machine-readable name of the overridable parameter.", + "type": "string", + "x-order": 1 + }, + "summary": { + "description": "Short human-readable parameter summary, as it was when the rule was created.", + "type": "string", + "x-order": 2 + }, + "unit": { + "description": "ParamUnit represents template parameter unit.\n\n - PARAM_UNIT_UNSPECIFIED: Invalid, unknown or absent.\n - PARAM_UNIT_PERCENTAGE: %\n - PARAM_UNIT_SECONDS: s", + "type": "string", + "default": "PARAM_UNIT_UNSPECIFIED", + "enum": [ + "PARAM_UNIT_UNSPECIFIED", + "PARAM_UNIT_PERCENTAGE", + "PARAM_UNIT_SECONDS" + ], + "x-order": 3 + }, + "default_value": { + "description": "Value the rule falls back to when no override applies.", + "type": "number", + "format": "double", + "x-order": 4 + }, + "effective_value": { + "description": "Value the rule currently evaluates this target against.", + "type": "number", + "format": "double", + "x-order": 5 + }, + "is_overridden": { + "description": "Whether effective_value comes from an override rather than the default.", + "type": "boolean", + "x-order": 6 + }, + "scope": { + "description": "ThresholdScope says what a threshold override's target refers to.\n\n - THRESHOLD_SCOPE_NODE: Target is a Node ID.\n - THRESHOLD_SCOPE_SERVICE: Target is a Service ID.\n - THRESHOLD_SCOPE_CLUSTER: Target is a cluster label value. Unlike the others it names no inventory entity,\nso it cannot be validated for existence and is never removed by entity deletion.", + "type": "string", + "default": "THRESHOLD_SCOPE_UNSPECIFIED", + "enum": [ + "THRESHOLD_SCOPE_UNSPECIFIED", + "THRESHOLD_SCOPE_NODE", + "THRESHOLD_SCOPE_SERVICE", + "THRESHOLD_SCOPE_CLUSTER" + ], + "x-order": 7 + }, + "target": { + "description": "Target the effective override was set on. Empty when not overridden.", + "type": "string", + "x-order": 8 + } + }, + "x-order": 0 + } + } + } + }, + "default": { + "description": "An unexpected error response.", + "schema": { + "type": "object", + "properties": { + "code": { + "type": "integer", + "format": "int32", + "x-order": 0 + }, + "message": { + "type": "string", + "x-order": 1 + }, + "details": { + "type": "array", + "items": { + "type": "object", + "properties": { + "@type": { + "type": "string", + "x-order": 0 + } + }, + "additionalProperties": {} + }, + "x-order": 2 + } + } + } + } + } + }, + "delete": { + "tags": [ + "AlertingService" + ], + "summary": "ClearThreshold removes an override so the target falls back to the rule's default,\nor to a broader override still covering it.", + "operationId": "ClearThreshold", + "parameters": [ + { + "enum": [ + "THRESHOLD_SCOPE_UNSPECIFIED", + "THRESHOLD_SCOPE_NODE", + "THRESHOLD_SCOPE_SERVICE", + "THRESHOLD_SCOPE_CLUSTER" + ], + "type": "string", + "default": "THRESHOLD_SCOPE_UNSPECIFIED", + "description": " - THRESHOLD_SCOPE_NODE: Target is a Node ID.\n - THRESHOLD_SCOPE_SERVICE: Target is a Service ID.\n - THRESHOLD_SCOPE_CLUSTER: Target is a cluster label value. Unlike the others it names no inventory entity,\nso it cannot be validated for existence and is never removed by entity deletion.", + "name": "scope", + "in": "query" + }, + { + "type": "string", + "name": "target", + "in": "query" + }, + { + "type": "string", + "name": "rule_id", + "in": "query" + }, + { + "type": "string", + "name": "param_name", + "in": "query" + } + ], + "responses": { + "200": { + "description": "A successful response.", + "schema": { + "type": "object" + } + }, + "default": { + "description": "An unexpected error response.", + "schema": { + "type": "object", + "properties": { + "code": { + "type": "integer", + "format": "int32", + "x-order": 0 + }, + "message": { + "type": "string", + "x-order": 1 + }, + "details": { + "type": "array", + "items": { + "type": "object", + "properties": { + "@type": { + "type": "string", + "x-order": 0 + } + }, + "additionalProperties": {} + }, + "x-order": 2 + } + } + } + } + } + } + }, + "/v1/alerting/thresholds:batchUpdate": { + "post": { + "tags": [ + "AlertingService" + ], + "summary": "BatchUpdateThresholds applies several set and clear operations in one transaction.", + "operationId": "BatchUpdateThresholds", + "parameters": [ + { + "name": "body", + "in": "body", + "required": true, + "schema": { + "type": "object", + "properties": { + "updates": { + "description": "Applied in one transaction: either every update lands or none does. A client\nediting several rows at once cannot otherwise report which ones took effect.", + "type": "array", + "items": { + "description": "ThresholdUpdate sets or clears one override.", + "type": "object", + "properties": { + "scope": { + "description": "ThresholdScope says what a threshold override's target refers to.\n\n - THRESHOLD_SCOPE_NODE: Target is a Node ID.\n - THRESHOLD_SCOPE_SERVICE: Target is a Service ID.\n - THRESHOLD_SCOPE_CLUSTER: Target is a cluster label value. Unlike the others it names no inventory entity,\nso it cannot be validated for existence and is never removed by entity deletion.", + "type": "string", + "default": "THRESHOLD_SCOPE_UNSPECIFIED", + "enum": [ + "THRESHOLD_SCOPE_UNSPECIFIED", + "THRESHOLD_SCOPE_NODE", + "THRESHOLD_SCOPE_SERVICE", + "THRESHOLD_SCOPE_CLUSTER" + ], + "x-order": 0 + }, + "target": { + "type": "string", + "x-order": 1 + }, + "rule_id": { + "type": "string", + "x-order": 2 + }, + "param_name": { + "type": "string", + "x-order": 3 + }, + "value": { + "description": "Omit to clear the override rather than set it.", + "type": "number", + "format": "double", + "x-nullable": true, + "x-order": 4 + } + } + }, + "x-order": 0 + } + } + } + } + ], + "responses": { + "200": { + "description": "A successful response.", + "schema": { + "type": "object", + "properties": { + "thresholds": { + "description": "Thresholds that were set, in request order. Cleared ones are omitted.", + "type": "array", + "items": { + "description": "Threshold is one overridable parameter of one rule, as it applies to one target.", + "type": "object", + "properties": { + "rule_id": { + "description": "Identifier PMM assigned to the rule. Not unique within a response: rules duplicated\nin Grafana share it, so two entries can carry the same rule_id and param_name and\ndiffer only in which rule they came from. Do not key a map on it.", + "type": "string", + "x-order": 0 + }, + "param_name": { + "description": "Machine-readable name of the overridable parameter.", + "type": "string", + "x-order": 1 + }, + "summary": { + "description": "Short human-readable parameter summary, as it was when the rule was created.", + "type": "string", + "x-order": 2 + }, + "unit": { + "description": "ParamUnit represents template parameter unit.\n\n - PARAM_UNIT_UNSPECIFIED: Invalid, unknown or absent.\n - PARAM_UNIT_PERCENTAGE: %\n - PARAM_UNIT_SECONDS: s", + "type": "string", + "default": "PARAM_UNIT_UNSPECIFIED", + "enum": [ + "PARAM_UNIT_UNSPECIFIED", + "PARAM_UNIT_PERCENTAGE", + "PARAM_UNIT_SECONDS" + ], + "x-order": 3 + }, + "default_value": { + "description": "Value the rule falls back to when no override applies.", + "type": "number", + "format": "double", + "x-order": 4 + }, + "effective_value": { + "description": "Value the rule currently evaluates this target against.", + "type": "number", + "format": "double", + "x-order": 5 + }, + "is_overridden": { + "description": "Whether effective_value comes from an override rather than the default.", + "type": "boolean", + "x-order": 6 + }, + "scope": { + "description": "ThresholdScope says what a threshold override's target refers to.\n\n - THRESHOLD_SCOPE_NODE: Target is a Node ID.\n - THRESHOLD_SCOPE_SERVICE: Target is a Service ID.\n - THRESHOLD_SCOPE_CLUSTER: Target is a cluster label value. Unlike the others it names no inventory entity,\nso it cannot be validated for existence and is never removed by entity deletion.", + "type": "string", + "default": "THRESHOLD_SCOPE_UNSPECIFIED", + "enum": [ + "THRESHOLD_SCOPE_UNSPECIFIED", + "THRESHOLD_SCOPE_NODE", + "THRESHOLD_SCOPE_SERVICE", + "THRESHOLD_SCOPE_CLUSTER" + ], + "x-order": 7 + }, + "target": { + "description": "Target the effective override was set on. Empty when not overridden.", + "type": "string", + "x-order": 8 + } + } + }, + "x-order": 0 + } + } + } + }, + "default": { + "description": "An unexpected error response.", + "schema": { + "type": "object", + "properties": { + "code": { + "type": "integer", + "format": "int32", + "x-order": 0 + }, + "message": { + "type": "string", + "x-order": 1 + }, + "details": { + "type": "array", + "items": { + "type": "object", + "properties": { + "@type": { + "type": "string", + "x-order": 0 + } + }, + "additionalProperties": {} + }, + "x-order": 2 + } + } + } + } + } + } } }, "tags": [ diff --git a/api/swagger/swagger-dev.json b/api/swagger/swagger-dev.json index 286fbb5c12..273a84be1c 100644 --- a/api/swagger/swagger-dev.json +++ b/api/swagger/swagger-dev.json @@ -2709,6 +2709,561 @@ } } }, + "/v1/alerting/thresholds": { + "get": { + "tags": [ + "AlertingService" + ], + "summary": "ListThresholds returns per-target threshold overrides.", + "operationId": "ListThresholds", + "parameters": [ + { + "enum": [ + "THRESHOLD_SCOPE_UNSPECIFIED", + "THRESHOLD_SCOPE_NODE", + "THRESHOLD_SCOPE_SERVICE", + "THRESHOLD_SCOPE_CLUSTER" + ], + "type": "string", + "default": "THRESHOLD_SCOPE_UNSPECIFIED", + "description": "Scope of the target to report thresholds for. Must be set together with target.\n\n - THRESHOLD_SCOPE_NODE: Target is a Node ID.\n - THRESHOLD_SCOPE_SERVICE: Target is a Service ID.\n - THRESHOLD_SCOPE_CLUSTER: Target is a cluster label value. Unlike the others it names no inventory entity,\nso it cannot be validated for existence and is never removed by entity deletion.", + "name": "scope", + "in": "query" + }, + { + "type": "string", + "description": "Target to report thresholds for. When set, every overridable parameter is returned\nfor that target, overridden or not. When empty, only existing overrides are\nreturned, since there is otherwise no bounded set to enumerate.", + "name": "target", + "in": "query" + }, + { + "type": "string", + "description": "Return only thresholds of this rule.", + "name": "rule_id", + "in": "query" + } + ], + "responses": { + "200": { + "description": "A successful response.", + "schema": { + "type": "object", + "properties": { + "thresholds": { + "type": "array", + "items": { + "description": "Threshold is one overridable parameter of one rule, as it applies to one target.", + "type": "object", + "properties": { + "rule_id": { + "description": "Identifier PMM assigned to the rule. Not unique within a response: rules duplicated\nin Grafana share it, so two entries can carry the same rule_id and param_name and\ndiffer only in which rule they came from. Do not key a map on it.", + "type": "string", + "x-order": 0 + }, + "param_name": { + "description": "Machine-readable name of the overridable parameter.", + "type": "string", + "x-order": 1 + }, + "summary": { + "description": "Short human-readable parameter summary, as it was when the rule was created.", + "type": "string", + "x-order": 2 + }, + "unit": { + "description": "ParamUnit represents template parameter unit.\n\n - PARAM_UNIT_UNSPECIFIED: Invalid, unknown or absent.\n - PARAM_UNIT_PERCENTAGE: %\n - PARAM_UNIT_SECONDS: s", + "type": "string", + "default": "PARAM_UNIT_UNSPECIFIED", + "enum": [ + "PARAM_UNIT_UNSPECIFIED", + "PARAM_UNIT_PERCENTAGE", + "PARAM_UNIT_SECONDS" + ], + "x-order": 3 + }, + "default_value": { + "description": "Value the rule falls back to when no override applies.", + "type": "number", + "format": "double", + "x-order": 4 + }, + "effective_value": { + "description": "Value the rule currently evaluates this target against.", + "type": "number", + "format": "double", + "x-order": 5 + }, + "is_overridden": { + "description": "Whether effective_value comes from an override rather than the default.", + "type": "boolean", + "x-order": 6 + }, + "scope": { + "description": "ThresholdScope says what a threshold override's target refers to.\n\n - THRESHOLD_SCOPE_NODE: Target is a Node ID.\n - THRESHOLD_SCOPE_SERVICE: Target is a Service ID.\n - THRESHOLD_SCOPE_CLUSTER: Target is a cluster label value. Unlike the others it names no inventory entity,\nso it cannot be validated for existence and is never removed by entity deletion.", + "type": "string", + "default": "THRESHOLD_SCOPE_UNSPECIFIED", + "enum": [ + "THRESHOLD_SCOPE_UNSPECIFIED", + "THRESHOLD_SCOPE_NODE", + "THRESHOLD_SCOPE_SERVICE", + "THRESHOLD_SCOPE_CLUSTER" + ], + "x-order": 7 + }, + "target": { + "description": "Target the effective override was set on. Empty when not overridden.", + "type": "string", + "x-order": 8 + } + } + }, + "x-order": 0 + } + } + } + }, + "default": { + "description": "An unexpected error response.", + "schema": { + "type": "object", + "properties": { + "code": { + "type": "integer", + "format": "int32", + "x-order": 0 + }, + "message": { + "type": "string", + "x-order": 1 + }, + "details": { + "type": "array", + "items": { + "type": "object", + "properties": { + "@type": { + "type": "string", + "x-order": 0 + } + }, + "additionalProperties": {} + }, + "x-order": 2 + } + } + } + } + } + }, + "post": { + "tags": [ + "AlertingService" + ], + "summary": "SetThreshold overrides one rule parameter for one target.", + "operationId": "SetThreshold", + "parameters": [ + { + "name": "body", + "in": "body", + "required": true, + "schema": { + "type": "object", + "properties": { + "scope": { + "description": "ThresholdScope says what a threshold override's target refers to.\n\n - THRESHOLD_SCOPE_NODE: Target is a Node ID.\n - THRESHOLD_SCOPE_SERVICE: Target is a Service ID.\n - THRESHOLD_SCOPE_CLUSTER: Target is a cluster label value. Unlike the others it names no inventory entity,\nso it cannot be validated for existence and is never removed by entity deletion.", + "type": "string", + "default": "THRESHOLD_SCOPE_UNSPECIFIED", + "enum": [ + "THRESHOLD_SCOPE_UNSPECIFIED", + "THRESHOLD_SCOPE_NODE", + "THRESHOLD_SCOPE_SERVICE", + "THRESHOLD_SCOPE_CLUSTER" + ], + "x-order": 0 + }, + "target": { + "type": "string", + "x-order": 1 + }, + "rule_id": { + "type": "string", + "x-order": 2 + }, + "param_name": { + "type": "string", + "x-order": 3 + }, + "value": { + "description": "Must be finite and within the parameter's declared range.", + "type": "number", + "format": "double", + "x-order": 4 + } + } + } + } + ], + "responses": { + "200": { + "description": "A successful response.", + "schema": { + "type": "object", + "properties": { + "threshold": { + "description": "Threshold is one overridable parameter of one rule, as it applies to one target.", + "type": "object", + "properties": { + "rule_id": { + "description": "Identifier PMM assigned to the rule. Not unique within a response: rules duplicated\nin Grafana share it, so two entries can carry the same rule_id and param_name and\ndiffer only in which rule they came from. Do not key a map on it.", + "type": "string", + "x-order": 0 + }, + "param_name": { + "description": "Machine-readable name of the overridable parameter.", + "type": "string", + "x-order": 1 + }, + "summary": { + "description": "Short human-readable parameter summary, as it was when the rule was created.", + "type": "string", + "x-order": 2 + }, + "unit": { + "description": "ParamUnit represents template parameter unit.\n\n - PARAM_UNIT_UNSPECIFIED: Invalid, unknown or absent.\n - PARAM_UNIT_PERCENTAGE: %\n - PARAM_UNIT_SECONDS: s", + "type": "string", + "default": "PARAM_UNIT_UNSPECIFIED", + "enum": [ + "PARAM_UNIT_UNSPECIFIED", + "PARAM_UNIT_PERCENTAGE", + "PARAM_UNIT_SECONDS" + ], + "x-order": 3 + }, + "default_value": { + "description": "Value the rule falls back to when no override applies.", + "type": "number", + "format": "double", + "x-order": 4 + }, + "effective_value": { + "description": "Value the rule currently evaluates this target against.", + "type": "number", + "format": "double", + "x-order": 5 + }, + "is_overridden": { + "description": "Whether effective_value comes from an override rather than the default.", + "type": "boolean", + "x-order": 6 + }, + "scope": { + "description": "ThresholdScope says what a threshold override's target refers to.\n\n - THRESHOLD_SCOPE_NODE: Target is a Node ID.\n - THRESHOLD_SCOPE_SERVICE: Target is a Service ID.\n - THRESHOLD_SCOPE_CLUSTER: Target is a cluster label value. Unlike the others it names no inventory entity,\nso it cannot be validated for existence and is never removed by entity deletion.", + "type": "string", + "default": "THRESHOLD_SCOPE_UNSPECIFIED", + "enum": [ + "THRESHOLD_SCOPE_UNSPECIFIED", + "THRESHOLD_SCOPE_NODE", + "THRESHOLD_SCOPE_SERVICE", + "THRESHOLD_SCOPE_CLUSTER" + ], + "x-order": 7 + }, + "target": { + "description": "Target the effective override was set on. Empty when not overridden.", + "type": "string", + "x-order": 8 + } + }, + "x-order": 0 + } + } + } + }, + "default": { + "description": "An unexpected error response.", + "schema": { + "type": "object", + "properties": { + "code": { + "type": "integer", + "format": "int32", + "x-order": 0 + }, + "message": { + "type": "string", + "x-order": 1 + }, + "details": { + "type": "array", + "items": { + "type": "object", + "properties": { + "@type": { + "type": "string", + "x-order": 0 + } + }, + "additionalProperties": {} + }, + "x-order": 2 + } + } + } + } + } + }, + "delete": { + "tags": [ + "AlertingService" + ], + "summary": "ClearThreshold removes an override so the target falls back to the rule's default,\nor to a broader override still covering it.", + "operationId": "ClearThreshold", + "parameters": [ + { + "enum": [ + "THRESHOLD_SCOPE_UNSPECIFIED", + "THRESHOLD_SCOPE_NODE", + "THRESHOLD_SCOPE_SERVICE", + "THRESHOLD_SCOPE_CLUSTER" + ], + "type": "string", + "default": "THRESHOLD_SCOPE_UNSPECIFIED", + "description": " - THRESHOLD_SCOPE_NODE: Target is a Node ID.\n - THRESHOLD_SCOPE_SERVICE: Target is a Service ID.\n - THRESHOLD_SCOPE_CLUSTER: Target is a cluster label value. Unlike the others it names no inventory entity,\nso it cannot be validated for existence and is never removed by entity deletion.", + "name": "scope", + "in": "query" + }, + { + "type": "string", + "name": "target", + "in": "query" + }, + { + "type": "string", + "name": "rule_id", + "in": "query" + }, + { + "type": "string", + "name": "param_name", + "in": "query" + } + ], + "responses": { + "200": { + "description": "A successful response.", + "schema": { + "type": "object" + } + }, + "default": { + "description": "An unexpected error response.", + "schema": { + "type": "object", + "properties": { + "code": { + "type": "integer", + "format": "int32", + "x-order": 0 + }, + "message": { + "type": "string", + "x-order": 1 + }, + "details": { + "type": "array", + "items": { + "type": "object", + "properties": { + "@type": { + "type": "string", + "x-order": 0 + } + }, + "additionalProperties": {} + }, + "x-order": 2 + } + } + } + } + } + } + }, + "/v1/alerting/thresholds:batchUpdate": { + "post": { + "tags": [ + "AlertingService" + ], + "summary": "BatchUpdateThresholds applies several set and clear operations in one transaction.", + "operationId": "BatchUpdateThresholds", + "parameters": [ + { + "name": "body", + "in": "body", + "required": true, + "schema": { + "type": "object", + "properties": { + "updates": { + "description": "Applied in one transaction: either every update lands or none does. A client\nediting several rows at once cannot otherwise report which ones took effect.", + "type": "array", + "items": { + "description": "ThresholdUpdate sets or clears one override.", + "type": "object", + "properties": { + "scope": { + "description": "ThresholdScope says what a threshold override's target refers to.\n\n - THRESHOLD_SCOPE_NODE: Target is a Node ID.\n - THRESHOLD_SCOPE_SERVICE: Target is a Service ID.\n - THRESHOLD_SCOPE_CLUSTER: Target is a cluster label value. Unlike the others it names no inventory entity,\nso it cannot be validated for existence and is never removed by entity deletion.", + "type": "string", + "default": "THRESHOLD_SCOPE_UNSPECIFIED", + "enum": [ + "THRESHOLD_SCOPE_UNSPECIFIED", + "THRESHOLD_SCOPE_NODE", + "THRESHOLD_SCOPE_SERVICE", + "THRESHOLD_SCOPE_CLUSTER" + ], + "x-order": 0 + }, + "target": { + "type": "string", + "x-order": 1 + }, + "rule_id": { + "type": "string", + "x-order": 2 + }, + "param_name": { + "type": "string", + "x-order": 3 + }, + "value": { + "description": "Omit to clear the override rather than set it.", + "type": "number", + "format": "double", + "x-nullable": true, + "x-order": 4 + } + } + }, + "x-order": 0 + } + } + } + } + ], + "responses": { + "200": { + "description": "A successful response.", + "schema": { + "type": "object", + "properties": { + "thresholds": { + "description": "Thresholds that were set, in request order. Cleared ones are omitted.", + "type": "array", + "items": { + "description": "Threshold is one overridable parameter of one rule, as it applies to one target.", + "type": "object", + "properties": { + "rule_id": { + "description": "Identifier PMM assigned to the rule. Not unique within a response: rules duplicated\nin Grafana share it, so two entries can carry the same rule_id and param_name and\ndiffer only in which rule they came from. Do not key a map on it.", + "type": "string", + "x-order": 0 + }, + "param_name": { + "description": "Machine-readable name of the overridable parameter.", + "type": "string", + "x-order": 1 + }, + "summary": { + "description": "Short human-readable parameter summary, as it was when the rule was created.", + "type": "string", + "x-order": 2 + }, + "unit": { + "description": "ParamUnit represents template parameter unit.\n\n - PARAM_UNIT_UNSPECIFIED: Invalid, unknown or absent.\n - PARAM_UNIT_PERCENTAGE: %\n - PARAM_UNIT_SECONDS: s", + "type": "string", + "default": "PARAM_UNIT_UNSPECIFIED", + "enum": [ + "PARAM_UNIT_UNSPECIFIED", + "PARAM_UNIT_PERCENTAGE", + "PARAM_UNIT_SECONDS" + ], + "x-order": 3 + }, + "default_value": { + "description": "Value the rule falls back to when no override applies.", + "type": "number", + "format": "double", + "x-order": 4 + }, + "effective_value": { + "description": "Value the rule currently evaluates this target against.", + "type": "number", + "format": "double", + "x-order": 5 + }, + "is_overridden": { + "description": "Whether effective_value comes from an override rather than the default.", + "type": "boolean", + "x-order": 6 + }, + "scope": { + "description": "ThresholdScope says what a threshold override's target refers to.\n\n - THRESHOLD_SCOPE_NODE: Target is a Node ID.\n - THRESHOLD_SCOPE_SERVICE: Target is a Service ID.\n - THRESHOLD_SCOPE_CLUSTER: Target is a cluster label value. Unlike the others it names no inventory entity,\nso it cannot be validated for existence and is never removed by entity deletion.", + "type": "string", + "default": "THRESHOLD_SCOPE_UNSPECIFIED", + "enum": [ + "THRESHOLD_SCOPE_UNSPECIFIED", + "THRESHOLD_SCOPE_NODE", + "THRESHOLD_SCOPE_SERVICE", + "THRESHOLD_SCOPE_CLUSTER" + ], + "x-order": 7 + }, + "target": { + "description": "Target the effective override was set on. Empty when not overridden.", + "type": "string", + "x-order": 8 + } + } + }, + "x-order": 0 + } + } + } + }, + "default": { + "description": "An unexpected error response.", + "schema": { + "type": "object", + "properties": { + "code": { + "type": "integer", + "format": "int32", + "x-order": 0 + }, + "message": { + "type": "string", + "x-order": 1 + }, + "details": { + "type": "array", + "items": { + "type": "object", + "properties": { + "@type": { + "type": "string", + "x-order": 0 + } + }, + "additionalProperties": {} + }, + "x-order": 2 + } + } + } + } + } + } + }, "/v1/backups/artifacts": { "get": { "description": "Return a list of backup artifacts.", diff --git a/api/swagger/swagger.json b/api/swagger/swagger.json index 8287656f36..d9c754ec83 100644 --- a/api/swagger/swagger.json +++ b/api/swagger/swagger.json @@ -2192,6 +2192,561 @@ } } }, + "/v1/alerting/thresholds": { + "get": { + "tags": [ + "AlertingService" + ], + "summary": "ListThresholds returns per-target threshold overrides.", + "operationId": "ListThresholds", + "parameters": [ + { + "enum": [ + "THRESHOLD_SCOPE_UNSPECIFIED", + "THRESHOLD_SCOPE_NODE", + "THRESHOLD_SCOPE_SERVICE", + "THRESHOLD_SCOPE_CLUSTER" + ], + "type": "string", + "default": "THRESHOLD_SCOPE_UNSPECIFIED", + "description": "Scope of the target to report thresholds for. Must be set together with target.\n\n - THRESHOLD_SCOPE_NODE: Target is a Node ID.\n - THRESHOLD_SCOPE_SERVICE: Target is a Service ID.\n - THRESHOLD_SCOPE_CLUSTER: Target is a cluster label value. Unlike the others it names no inventory entity,\nso it cannot be validated for existence and is never removed by entity deletion.", + "name": "scope", + "in": "query" + }, + { + "type": "string", + "description": "Target to report thresholds for. When set, every overridable parameter is returned\nfor that target, overridden or not. When empty, only existing overrides are\nreturned, since there is otherwise no bounded set to enumerate.", + "name": "target", + "in": "query" + }, + { + "type": "string", + "description": "Return only thresholds of this rule.", + "name": "rule_id", + "in": "query" + } + ], + "responses": { + "200": { + "description": "A successful response.", + "schema": { + "type": "object", + "properties": { + "thresholds": { + "type": "array", + "items": { + "description": "Threshold is one overridable parameter of one rule, as it applies to one target.", + "type": "object", + "properties": { + "rule_id": { + "description": "Identifier PMM assigned to the rule. Not unique within a response: rules duplicated\nin Grafana share it, so two entries can carry the same rule_id and param_name and\ndiffer only in which rule they came from. Do not key a map on it.", + "type": "string", + "x-order": 0 + }, + "param_name": { + "description": "Machine-readable name of the overridable parameter.", + "type": "string", + "x-order": 1 + }, + "summary": { + "description": "Short human-readable parameter summary, as it was when the rule was created.", + "type": "string", + "x-order": 2 + }, + "unit": { + "description": "ParamUnit represents template parameter unit.\n\n - PARAM_UNIT_UNSPECIFIED: Invalid, unknown or absent.\n - PARAM_UNIT_PERCENTAGE: %\n - PARAM_UNIT_SECONDS: s", + "type": "string", + "default": "PARAM_UNIT_UNSPECIFIED", + "enum": [ + "PARAM_UNIT_UNSPECIFIED", + "PARAM_UNIT_PERCENTAGE", + "PARAM_UNIT_SECONDS" + ], + "x-order": 3 + }, + "default_value": { + "description": "Value the rule falls back to when no override applies.", + "type": "number", + "format": "double", + "x-order": 4 + }, + "effective_value": { + "description": "Value the rule currently evaluates this target against.", + "type": "number", + "format": "double", + "x-order": 5 + }, + "is_overridden": { + "description": "Whether effective_value comes from an override rather than the default.", + "type": "boolean", + "x-order": 6 + }, + "scope": { + "description": "ThresholdScope says what a threshold override's target refers to.\n\n - THRESHOLD_SCOPE_NODE: Target is a Node ID.\n - THRESHOLD_SCOPE_SERVICE: Target is a Service ID.\n - THRESHOLD_SCOPE_CLUSTER: Target is a cluster label value. Unlike the others it names no inventory entity,\nso it cannot be validated for existence and is never removed by entity deletion.", + "type": "string", + "default": "THRESHOLD_SCOPE_UNSPECIFIED", + "enum": [ + "THRESHOLD_SCOPE_UNSPECIFIED", + "THRESHOLD_SCOPE_NODE", + "THRESHOLD_SCOPE_SERVICE", + "THRESHOLD_SCOPE_CLUSTER" + ], + "x-order": 7 + }, + "target": { + "description": "Target the effective override was set on. Empty when not overridden.", + "type": "string", + "x-order": 8 + } + } + }, + "x-order": 0 + } + } + } + }, + "default": { + "description": "An unexpected error response.", + "schema": { + "type": "object", + "properties": { + "code": { + "type": "integer", + "format": "int32", + "x-order": 0 + }, + "message": { + "type": "string", + "x-order": 1 + }, + "details": { + "type": "array", + "items": { + "type": "object", + "properties": { + "@type": { + "type": "string", + "x-order": 0 + } + }, + "additionalProperties": {} + }, + "x-order": 2 + } + } + } + } + } + }, + "post": { + "tags": [ + "AlertingService" + ], + "summary": "SetThreshold overrides one rule parameter for one target.", + "operationId": "SetThreshold", + "parameters": [ + { + "name": "body", + "in": "body", + "required": true, + "schema": { + "type": "object", + "properties": { + "scope": { + "description": "ThresholdScope says what a threshold override's target refers to.\n\n - THRESHOLD_SCOPE_NODE: Target is a Node ID.\n - THRESHOLD_SCOPE_SERVICE: Target is a Service ID.\n - THRESHOLD_SCOPE_CLUSTER: Target is a cluster label value. Unlike the others it names no inventory entity,\nso it cannot be validated for existence and is never removed by entity deletion.", + "type": "string", + "default": "THRESHOLD_SCOPE_UNSPECIFIED", + "enum": [ + "THRESHOLD_SCOPE_UNSPECIFIED", + "THRESHOLD_SCOPE_NODE", + "THRESHOLD_SCOPE_SERVICE", + "THRESHOLD_SCOPE_CLUSTER" + ], + "x-order": 0 + }, + "target": { + "type": "string", + "x-order": 1 + }, + "rule_id": { + "type": "string", + "x-order": 2 + }, + "param_name": { + "type": "string", + "x-order": 3 + }, + "value": { + "description": "Must be finite and within the parameter's declared range.", + "type": "number", + "format": "double", + "x-order": 4 + } + } + } + } + ], + "responses": { + "200": { + "description": "A successful response.", + "schema": { + "type": "object", + "properties": { + "threshold": { + "description": "Threshold is one overridable parameter of one rule, as it applies to one target.", + "type": "object", + "properties": { + "rule_id": { + "description": "Identifier PMM assigned to the rule. Not unique within a response: rules duplicated\nin Grafana share it, so two entries can carry the same rule_id and param_name and\ndiffer only in which rule they came from. Do not key a map on it.", + "type": "string", + "x-order": 0 + }, + "param_name": { + "description": "Machine-readable name of the overridable parameter.", + "type": "string", + "x-order": 1 + }, + "summary": { + "description": "Short human-readable parameter summary, as it was when the rule was created.", + "type": "string", + "x-order": 2 + }, + "unit": { + "description": "ParamUnit represents template parameter unit.\n\n - PARAM_UNIT_UNSPECIFIED: Invalid, unknown or absent.\n - PARAM_UNIT_PERCENTAGE: %\n - PARAM_UNIT_SECONDS: s", + "type": "string", + "default": "PARAM_UNIT_UNSPECIFIED", + "enum": [ + "PARAM_UNIT_UNSPECIFIED", + "PARAM_UNIT_PERCENTAGE", + "PARAM_UNIT_SECONDS" + ], + "x-order": 3 + }, + "default_value": { + "description": "Value the rule falls back to when no override applies.", + "type": "number", + "format": "double", + "x-order": 4 + }, + "effective_value": { + "description": "Value the rule currently evaluates this target against.", + "type": "number", + "format": "double", + "x-order": 5 + }, + "is_overridden": { + "description": "Whether effective_value comes from an override rather than the default.", + "type": "boolean", + "x-order": 6 + }, + "scope": { + "description": "ThresholdScope says what a threshold override's target refers to.\n\n - THRESHOLD_SCOPE_NODE: Target is a Node ID.\n - THRESHOLD_SCOPE_SERVICE: Target is a Service ID.\n - THRESHOLD_SCOPE_CLUSTER: Target is a cluster label value. Unlike the others it names no inventory entity,\nso it cannot be validated for existence and is never removed by entity deletion.", + "type": "string", + "default": "THRESHOLD_SCOPE_UNSPECIFIED", + "enum": [ + "THRESHOLD_SCOPE_UNSPECIFIED", + "THRESHOLD_SCOPE_NODE", + "THRESHOLD_SCOPE_SERVICE", + "THRESHOLD_SCOPE_CLUSTER" + ], + "x-order": 7 + }, + "target": { + "description": "Target the effective override was set on. Empty when not overridden.", + "type": "string", + "x-order": 8 + } + }, + "x-order": 0 + } + } + } + }, + "default": { + "description": "An unexpected error response.", + "schema": { + "type": "object", + "properties": { + "code": { + "type": "integer", + "format": "int32", + "x-order": 0 + }, + "message": { + "type": "string", + "x-order": 1 + }, + "details": { + "type": "array", + "items": { + "type": "object", + "properties": { + "@type": { + "type": "string", + "x-order": 0 + } + }, + "additionalProperties": {} + }, + "x-order": 2 + } + } + } + } + } + }, + "delete": { + "tags": [ + "AlertingService" + ], + "summary": "ClearThreshold removes an override so the target falls back to the rule's default,\nor to a broader override still covering it.", + "operationId": "ClearThreshold", + "parameters": [ + { + "enum": [ + "THRESHOLD_SCOPE_UNSPECIFIED", + "THRESHOLD_SCOPE_NODE", + "THRESHOLD_SCOPE_SERVICE", + "THRESHOLD_SCOPE_CLUSTER" + ], + "type": "string", + "default": "THRESHOLD_SCOPE_UNSPECIFIED", + "description": " - THRESHOLD_SCOPE_NODE: Target is a Node ID.\n - THRESHOLD_SCOPE_SERVICE: Target is a Service ID.\n - THRESHOLD_SCOPE_CLUSTER: Target is a cluster label value. Unlike the others it names no inventory entity,\nso it cannot be validated for existence and is never removed by entity deletion.", + "name": "scope", + "in": "query" + }, + { + "type": "string", + "name": "target", + "in": "query" + }, + { + "type": "string", + "name": "rule_id", + "in": "query" + }, + { + "type": "string", + "name": "param_name", + "in": "query" + } + ], + "responses": { + "200": { + "description": "A successful response.", + "schema": { + "type": "object" + } + }, + "default": { + "description": "An unexpected error response.", + "schema": { + "type": "object", + "properties": { + "code": { + "type": "integer", + "format": "int32", + "x-order": 0 + }, + "message": { + "type": "string", + "x-order": 1 + }, + "details": { + "type": "array", + "items": { + "type": "object", + "properties": { + "@type": { + "type": "string", + "x-order": 0 + } + }, + "additionalProperties": {} + }, + "x-order": 2 + } + } + } + } + } + } + }, + "/v1/alerting/thresholds:batchUpdate": { + "post": { + "tags": [ + "AlertingService" + ], + "summary": "BatchUpdateThresholds applies several set and clear operations in one transaction.", + "operationId": "BatchUpdateThresholds", + "parameters": [ + { + "name": "body", + "in": "body", + "required": true, + "schema": { + "type": "object", + "properties": { + "updates": { + "description": "Applied in one transaction: either every update lands or none does. A client\nediting several rows at once cannot otherwise report which ones took effect.", + "type": "array", + "items": { + "description": "ThresholdUpdate sets or clears one override.", + "type": "object", + "properties": { + "scope": { + "description": "ThresholdScope says what a threshold override's target refers to.\n\n - THRESHOLD_SCOPE_NODE: Target is a Node ID.\n - THRESHOLD_SCOPE_SERVICE: Target is a Service ID.\n - THRESHOLD_SCOPE_CLUSTER: Target is a cluster label value. Unlike the others it names no inventory entity,\nso it cannot be validated for existence and is never removed by entity deletion.", + "type": "string", + "default": "THRESHOLD_SCOPE_UNSPECIFIED", + "enum": [ + "THRESHOLD_SCOPE_UNSPECIFIED", + "THRESHOLD_SCOPE_NODE", + "THRESHOLD_SCOPE_SERVICE", + "THRESHOLD_SCOPE_CLUSTER" + ], + "x-order": 0 + }, + "target": { + "type": "string", + "x-order": 1 + }, + "rule_id": { + "type": "string", + "x-order": 2 + }, + "param_name": { + "type": "string", + "x-order": 3 + }, + "value": { + "description": "Omit to clear the override rather than set it.", + "type": "number", + "format": "double", + "x-nullable": true, + "x-order": 4 + } + } + }, + "x-order": 0 + } + } + } + } + ], + "responses": { + "200": { + "description": "A successful response.", + "schema": { + "type": "object", + "properties": { + "thresholds": { + "description": "Thresholds that were set, in request order. Cleared ones are omitted.", + "type": "array", + "items": { + "description": "Threshold is one overridable parameter of one rule, as it applies to one target.", + "type": "object", + "properties": { + "rule_id": { + "description": "Identifier PMM assigned to the rule. Not unique within a response: rules duplicated\nin Grafana share it, so two entries can carry the same rule_id and param_name and\ndiffer only in which rule they came from. Do not key a map on it.", + "type": "string", + "x-order": 0 + }, + "param_name": { + "description": "Machine-readable name of the overridable parameter.", + "type": "string", + "x-order": 1 + }, + "summary": { + "description": "Short human-readable parameter summary, as it was when the rule was created.", + "type": "string", + "x-order": 2 + }, + "unit": { + "description": "ParamUnit represents template parameter unit.\n\n - PARAM_UNIT_UNSPECIFIED: Invalid, unknown or absent.\n - PARAM_UNIT_PERCENTAGE: %\n - PARAM_UNIT_SECONDS: s", + "type": "string", + "default": "PARAM_UNIT_UNSPECIFIED", + "enum": [ + "PARAM_UNIT_UNSPECIFIED", + "PARAM_UNIT_PERCENTAGE", + "PARAM_UNIT_SECONDS" + ], + "x-order": 3 + }, + "default_value": { + "description": "Value the rule falls back to when no override applies.", + "type": "number", + "format": "double", + "x-order": 4 + }, + "effective_value": { + "description": "Value the rule currently evaluates this target against.", + "type": "number", + "format": "double", + "x-order": 5 + }, + "is_overridden": { + "description": "Whether effective_value comes from an override rather than the default.", + "type": "boolean", + "x-order": 6 + }, + "scope": { + "description": "ThresholdScope says what a threshold override's target refers to.\n\n - THRESHOLD_SCOPE_NODE: Target is a Node ID.\n - THRESHOLD_SCOPE_SERVICE: Target is a Service ID.\n - THRESHOLD_SCOPE_CLUSTER: Target is a cluster label value. Unlike the others it names no inventory entity,\nso it cannot be validated for existence and is never removed by entity deletion.", + "type": "string", + "default": "THRESHOLD_SCOPE_UNSPECIFIED", + "enum": [ + "THRESHOLD_SCOPE_UNSPECIFIED", + "THRESHOLD_SCOPE_NODE", + "THRESHOLD_SCOPE_SERVICE", + "THRESHOLD_SCOPE_CLUSTER" + ], + "x-order": 7 + }, + "target": { + "description": "Target the effective override was set on. Empty when not overridden.", + "type": "string", + "x-order": 8 + } + } + }, + "x-order": 0 + } + } + } + }, + "default": { + "description": "An unexpected error response.", + "schema": { + "type": "object", + "properties": { + "code": { + "type": "integer", + "format": "int32", + "x-order": 0 + }, + "message": { + "type": "string", + "x-order": 1 + }, + "details": { + "type": "array", + "items": { + "type": "object", + "properties": { + "@type": { + "type": "string", + "x-order": 0 + } + }, + "additionalProperties": {} + }, + "x-order": 2 + } + } + } + } + } + } + }, "/v1/backups/artifacts": { "get": { "description": "Return a list of backup artifacts.", diff --git a/managed/models/threshold_resolver.go b/managed/models/threshold_resolver.go index 22024ef1b7..ade1884737 100644 --- a/managed/models/threshold_resolver.go +++ b/managed/models/threshold_resolver.go @@ -89,7 +89,38 @@ func (inv ThresholdInventory) targetNames(override *AlertRuleThresholdOverride) // resolves to defaultValue - which is what makes clearing an override a value change on // an existing series rather than the series disappearing. func ResolveThresholds(overrides []*AlertRuleThresholdOverride, defaultValue float64, inv ThresholdInventory) map[string]float64 { - resolved := make(map[string]float64, len(overrides)) + detailed := ResolveThresholdsDetailed(overrides, defaultValue, inv) + + resolved := make(map[string]float64, len(detailed)) + for name, threshold := range detailed { + resolved[name] = threshold.Value + } + + return resolved +} + +// ResolvedThreshold is the effective threshold for one target, with the override it came +// from. Source is nil when the value is the rule's default, which happens when every +// override covering the target has been cleared. +type ResolvedThreshold struct { + Value float64 + Source *AlertRuleThresholdOverride +} + +// IsOverridden reports whether the value comes from an override rather than the default. +func (r ResolvedThreshold) IsOverridden() bool { + return r.Source != nil +} + +// ResolveThresholdsDetailed applies precedence and reports which override won for each +// target. It is the one implementation of precedence; ResolveThresholds is a thin view +// over it, so the value the API reports and the value the collector emits cannot drift. +func ResolveThresholdsDetailed( + overrides []*AlertRuleThresholdOverride, + defaultValue float64, + inv ThresholdInventory, +) map[string]ResolvedThreshold { + resolved := make(map[string]ResolvedThreshold, len(overrides)) specificity := make(map[string]int, len(overrides)) var cleared []string @@ -112,7 +143,7 @@ func ResolveThresholds(overrides []*AlertRuleThresholdOverride, defaultValue flo continue } - resolved[name] = override.Value + resolved[name] = ResolvedThreshold{Value: override.Value, Source: override} specificity[name] = rank } } @@ -122,7 +153,7 @@ func ResolveThresholds(overrides []*AlertRuleThresholdOverride, defaultValue flo for _, name := range cleared { _, ok := resolved[name] if !ok { - resolved[name] = defaultValue + resolved[name] = ResolvedThreshold{Value: defaultValue} } } diff --git a/managed/services/alerting/threshold_overrides.go b/managed/services/alerting/threshold_overrides.go new file mode 100644 index 0000000000..238266b8db --- /dev/null +++ b/managed/services/alerting/threshold_overrides.go @@ -0,0 +1,511 @@ +// Copyright (C) 2023 Percona LLC +// +// This program is free software: you can redistribute it and/or modify +// it under the terms of the GNU Affero General Public License as published by +// the Free Software Foundation, either version 3 of the License, or +// (at your option) any later version. +// +// This program is distributed in the hope that it will be useful, +// but WITHOUT ANY WARRANTY; without even the implied warranty of +// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +// GNU Affero General Public License for more details. +// +// You should have received a copy of the GNU Affero General Public License +// along with this program. If not, see . + +package alerting + +import ( + "context" + "math" + "slices" + "sort" + + "google.golang.org/grpc/codes" + "google.golang.org/grpc/status" + "gopkg.in/reform.v1" + + alerting "github.com/percona/pmm/api/alerting/v1" + "github.com/percona/pmm/managed/models" + "github.com/percona/pmm/managed/services" +) + +// thresholdScopeFromAPI converts a scope from the wire, defaulting to node. +// +// Only node scope resolves in this increment. Service and cluster are already carried by +// the schema, the resolver and the proto, so enabling them later is a validation change +// rather than an API change - which is why they are rejected as unimplemented rather than +// as invalid. +func thresholdScopeFromAPI(scope alerting.ThresholdScope) (models.ThresholdScope, error) { + switch scope { + case alerting.ThresholdScope_THRESHOLD_SCOPE_UNSPECIFIED, alerting.ThresholdScope_THRESHOLD_SCOPE_NODE: + return models.ThresholdScopeNode, nil + case alerting.ThresholdScope_THRESHOLD_SCOPE_SERVICE: + return "", status.Error(codes.Unimplemented, "Service-scoped threshold overrides are not supported yet.") + case alerting.ThresholdScope_THRESHOLD_SCOPE_CLUSTER: + return "", status.Error(codes.Unimplemented, "Cluster-scoped threshold overrides are not supported yet.") + } + + // do not add `default:` to make exhaustive linter do its job + + return "", status.Errorf(codes.InvalidArgument, "Unknown threshold scope %q.", scope.String()) +} + +func thresholdScopeToAPI(scope models.ThresholdScope) alerting.ThresholdScope { + switch scope { + case models.ThresholdScopeNode: + return alerting.ThresholdScope_THRESHOLD_SCOPE_NODE + case models.ThresholdScopeService: + return alerting.ThresholdScope_THRESHOLD_SCOPE_SERVICE + case models.ThresholdScopeCluster: + return alerting.ThresholdScope_THRESHOLD_SCOPE_CLUSTER + } + + // do not add `default:` to make exhaustive linter do its job + + return alerting.ThresholdScope_THRESHOLD_SCOPE_UNSPECIFIED +} + +// checkThresholdTargetExists rejects an override aimed at something that is not there. +// A cluster is a label value rather than an inventory entity, so its existence cannot be +// checked - and should not be, since a cluster override may legitimately precede the +// services that will join it. +func checkThresholdTargetExists(q *reform.Querier, scope models.ThresholdScope, target string) error { + switch scope { + case models.ThresholdScopeNode: + _, err := models.FindNodeByID(q, target) + + return err + case models.ThresholdScopeService: + _, err := models.FindServiceByID(q, target) + + return err + case models.ThresholdScopeCluster: + return nil + } + + // do not add `default:` to make exhaustive linter do its job + + return nil +} + +// resolveThresholdRequest validates one set or clear against the rule registry, and +// returns the rule parameter it addresses. Checks run cheapest-first, so a malformed +// request never reaches the database. +func resolveThresholdRequest( + q *reform.Querier, + scope models.ThresholdScope, + target, ruleID, paramName string, + value *float64, +) (models.AlertRuleParam, error) { + var zero models.AlertRuleParam + + rule, err := models.FindAlertRuleByID(q, ruleID) + if err != nil { + return zero, err + } + + // The registry only holds parameters that were overridable when the rule was + // created, so a parameter missing here is either unknown or not overridable. + param, ok := rule.Params[paramName] + if !ok { + return zero, status.Errorf(codes.NotFound, + "Rule %q has no overridable parameter %q.", ruleID, paramName) + } + + if !slices.Contains(param.Scopes, string(scope)) { + return zero, status.Errorf(codes.InvalidArgument, + "Parameter %q cannot be overridden at %q scope.", paramName, scope) + } + + if value != nil { + err = checkThresholdValue(paramName, param, *value) + if err != nil { + return zero, err + } + } + + err = checkThresholdTargetExists(q, scope, target) + if err != nil { + return zero, err + } + + return param, nil +} + +// checkThresholdValue guards the value at the API as well as the database. The column's +// CHECK rejects non-finite values too, but reaching it would surface as an opaque +// internal error rather than a bad request. +func checkThresholdValue(paramName string, param models.AlertRuleParam, value float64) error { + if math.IsNaN(value) || math.IsInf(value, 0) { + return status.Errorf(codes.InvalidArgument, "Threshold for %q must be a finite number.", paramName) + } + + if param.Min != nil && value < *param.Min { + return status.Errorf(codes.InvalidArgument, + "Threshold for %q must be at least %v.", paramName, *param.Min) + } + + if param.Max != nil && value > *param.Max { + return status.Errorf(codes.InvalidArgument, + "Threshold for %q must be at most %v.", paramName, *param.Max) + } + + return nil +} + +// thresholdFromResolved builds the API view of one parameter as it applies to one target. +func thresholdFromResolved(ruleID, paramName string, param models.AlertRuleParam, resolved models.ResolvedThreshold) *alerting.Threshold { + threshold := &alerting.Threshold{ + RuleId: ruleID, + ParamName: paramName, + Summary: param.Summary, + Unit: convertParamUnit(models.ParamUnit(param.Unit)), + DefaultValue: param.Default, + EffectiveValue: resolved.Value, + IsOverridden: resolved.IsOverridden(), + } + + if resolved.Source != nil { + threshold.Scope = thresholdScopeToAPI(resolved.Source.Scope) + threshold.Target = resolved.Source.Target + } + + return threshold +} + +// ListThresholds returns per-target threshold overrides. +func (s *Service) ListThresholds(_ context.Context, req *alerting.ListThresholdsRequest) (*alerting.ListThresholdsResponse, error) { + settings, err := models.GetSettings(s.db) + if err != nil { + return nil, err + } + + if !settings.IsAlertingEnabled() { + return nil, services.ErrAlertingDisabled + } + + var thresholds []*alerting.Threshold + + errTx := s.db.InTransaction(func(tx *reform.TX) error { + var scope models.ThresholdScope + if req.Target != "" { + scope, err = thresholdScopeFromAPI(req.Scope) + if err != nil { + return err + } + } + + rules, err := s.thresholdRules(tx.Querier, req.RuleId) + if err != nil { + return err + } + + for _, rule := range rules { + overrides, err := models.FindThresholdOverridesByRule(tx.Querier, rule.RuleID) + if err != nil { + return err + } + + inv, err := loadThresholdInventory(tx.Querier, overrides) + if err != nil { + return err + } + + targetName, err := s.thresholdTargetName(tx.Querier, scope, req.Target, &inv) + if err != nil { + return err + } + + for paramName, param := range rule.Params { + resolved := models.ResolveThresholdsDetailed( + filterOverridesByParam(overrides, paramName), param.Default, inv, + ) + + if req.Target == "" { + // With no target there is no bounded set of targets to enumerate, so + // only what has actually been overridden is reported. + for _, entry := range resolved { + if entry.IsOverridden() { + thresholds = append(thresholds, thresholdFromResolved(rule.RuleID, paramName, param, entry)) + } + } + + continue + } + + entry, ok := resolved[targetName] + if !ok { + entry = models.ResolvedThreshold{Value: param.Default} + } + + thresholds = append(thresholds, thresholdFromResolved(rule.RuleID, paramName, param, entry)) + } + } + + return nil + }) + if errTx != nil { + return nil, errTx + } + + sortThresholds(thresholds) + + return &alerting.ListThresholdsResponse{Thresholds: thresholds}, nil +} + +// thresholdRules returns the registry rows to report on, honouring an optional filter. +func (s *Service) thresholdRules(q *reform.Querier, ruleID string) ([]*models.AlertRule, error) { + if ruleID != "" { + rule, err := models.FindAlertRuleByID(q, ruleID) + if err != nil { + return nil, err + } + + return []*models.AlertRule{rule}, nil + } + + return models.FindAlertRules(q) +} + +// thresholdTargetName resolves the requested target to its join-label value and makes +// sure the inventory carries it, so a target with no override of its own still resolves. +func (s *Service) thresholdTargetName( + q *reform.Querier, + scope models.ThresholdScope, + target string, + inv *models.ThresholdInventory, +) (string, error) { + if target == "" { + return "", nil + } + + switch scope { + case models.ThresholdScopeNode: + node, err := models.FindNodeByID(q, target) + if err != nil { + return "", err + } + + if inv.NodeNames == nil { + inv.NodeNames = make(map[string]string, 1) + } + inv.NodeNames[target] = node.NodeName + + return node.NodeName, nil + + case models.ThresholdScopeService: + service, err := models.FindServiceByID(q, target) + if err != nil { + return "", err + } + + if inv.ServiceNames == nil { + inv.ServiceNames = make(map[string]string, 1) + } + inv.ServiceNames[target] = service.ServiceName + + return service.ServiceName, nil + + case models.ThresholdScopeCluster: + return "", status.Error(codes.InvalidArgument, "Cluster is not a listable target.") + } + + // do not add `default:` to make exhaustive linter do its job + + return "", nil +} + +func filterOverridesByParam(overrides []*models.AlertRuleThresholdOverride, paramName string) []*models.AlertRuleThresholdOverride { + filtered := make([]*models.AlertRuleThresholdOverride, 0, len(overrides)) + for _, override := range overrides { + if override.ParamName == paramName { + filtered = append(filtered, override) + } + } + + return filtered +} + +// sortThresholds gives the response a stable order, since it is assembled from map +// iteration over a rule's parameters. +func sortThresholds(thresholds []*alerting.Threshold) { + sort.Slice(thresholds, func(i, j int) bool { + if thresholds[i].RuleId != thresholds[j].RuleId { + return thresholds[i].RuleId < thresholds[j].RuleId + } + + if thresholds[i].ParamName != thresholds[j].ParamName { + return thresholds[i].ParamName < thresholds[j].ParamName + } + + return thresholds[i].Target < thresholds[j].Target + }) +} + +// SetThreshold overrides one rule parameter for one target. +func (s *Service) SetThreshold(_ context.Context, req *alerting.SetThresholdRequest) (*alerting.SetThresholdResponse, error) { + settings, err := models.GetSettings(s.db) + if err != nil { + return nil, err + } + + if !settings.IsAlertingEnabled() { + return nil, services.ErrAlertingDisabled + } + + var threshold *alerting.Threshold + + errTx := s.db.InTransaction(func(tx *reform.TX) error { + scope, err := thresholdScopeFromAPI(req.Scope) + if err != nil { + return err + } + + param, err := resolveThresholdRequest(tx.Querier, scope, req.Target, req.RuleId, req.ParamName, &req.Value) + if err != nil { + return err + } + + _, err = models.UpsertThresholdOverride(tx.Querier, req.RuleId, req.ParamName, scope, req.Target, req.Value) + if err != nil { + return err + } + + threshold, err = s.readThreshold(tx.Querier, req.RuleId, req.ParamName, param, scope, req.Target) + + return err + }) + if errTx != nil { + return nil, errTx + } + + return &alerting.SetThresholdResponse{Threshold: threshold}, nil +} + +// ClearThreshold removes an override so the target falls back to the rule's default, or +// to a broader override still covering it. +func (s *Service) ClearThreshold(_ context.Context, req *alerting.ClearThresholdRequest) (*alerting.ClearThresholdResponse, error) { + settings, err := models.GetSettings(s.db) + if err != nil { + return nil, err + } + + if !settings.IsAlertingEnabled() { + return nil, services.ErrAlertingDisabled + } + + errTx := s.db.InTransaction(func(tx *reform.TX) error { + scope, err := thresholdScopeFromAPI(req.Scope) + if err != nil { + return err + } + + _, err = resolveThresholdRequest(tx.Querier, scope, req.Target, req.RuleId, req.ParamName, nil) + if err != nil { + return err + } + + return models.ClearThresholdOverride(tx.Querier, req.RuleId, req.ParamName, scope, req.Target) + }) + if errTx != nil { + return nil, errTx + } + + return &alerting.ClearThresholdResponse{}, nil +} + +// BatchUpdateThresholds applies several set and clear operations in one transaction, so +// a client editing many rows at once never lands a partial result it cannot report. +func (s *Service) BatchUpdateThresholds(_ context.Context, req *alerting.BatchUpdateThresholdsRequest) (*alerting.BatchUpdateThresholdsResponse, error) { + settings, err := models.GetSettings(s.db) + if err != nil { + return nil, err + } + + if !settings.IsAlertingEnabled() { + return nil, services.ErrAlertingDisabled + } + + thresholds := make([]*alerting.Threshold, 0, len(req.Updates)) + + errTx := s.db.InTransaction(func(tx *reform.TX) error { + thresholds = thresholds[:0] + + for _, update := range req.Updates { + scope, err := thresholdScopeFromAPI(update.Scope) + if err != nil { + return err + } + + param, err := resolveThresholdRequest(tx.Querier, scope, update.Target, update.RuleId, update.ParamName, update.Value) + if err != nil { + return err + } + + if update.Value == nil { + err = models.ClearThresholdOverride(tx.Querier, update.RuleId, update.ParamName, scope, update.Target) + if err != nil { + return err + } + + continue + } + + _, err = models.UpsertThresholdOverride(tx.Querier, update.RuleId, update.ParamName, scope, update.Target, *update.Value) + if err != nil { + return err + } + + threshold, err := s.readThreshold(tx.Querier, update.RuleId, update.ParamName, param, scope, update.Target) + if err != nil { + return err + } + + thresholds = append(thresholds, threshold) + } + + return nil + }) + if errTx != nil { + return nil, errTx + } + + return &alerting.BatchUpdateThresholdsResponse{Thresholds: thresholds}, nil +} + +// readThreshold reports a parameter as it stands for one target after a write, resolved +// through the same precedence the collector applies. +func (s *Service) readThreshold( + q *reform.Querier, + ruleID, paramName string, + param models.AlertRuleParam, + scope models.ThresholdScope, + target string, +) (*alerting.Threshold, error) { + overrides, err := models.FindThresholdOverridesByRule(q, ruleID) + if err != nil { + return nil, err + } + + overrides = filterOverridesByParam(overrides, paramName) + + inv, err := loadThresholdInventory(q, overrides) + if err != nil { + return nil, err + } + + targetName, err := s.thresholdTargetName(q, scope, target, &inv) + if err != nil { + return nil, err + } + + resolved := models.ResolveThresholdsDetailed(overrides, param.Default, inv) + + entry, ok := resolved[targetName] + if !ok { + entry = models.ResolvedThreshold{Value: param.Default} + } + + return thresholdFromResolved(ruleID, paramName, param, entry), nil +} diff --git a/managed/services/alerting/threshold_overrides_test.go b/managed/services/alerting/threshold_overrides_test.go new file mode 100644 index 0000000000..70879d4a86 --- /dev/null +++ b/managed/services/alerting/threshold_overrides_test.go @@ -0,0 +1,317 @@ +// Copyright (C) 2023 Percona LLC +// +// This program is free software: you can redistribute it and/or modify +// it under the terms of the GNU Affero General Public License as published by +// the Free Software Foundation, either version 3 of the License, or +// (at your option) any later version. +// +// This program is distributed in the hope that it will be useful, +// but WITHOUT ANY WARRANTY; without even the implied warranty of +// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +// GNU Affero General Public License for more details. +// +// You should have received a copy of the GNU Affero General Public License +// along with this program. If not, see . + +package alerting + +import ( + "math" + "testing" + + "github.com/AlekSi/pointer" + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + "google.golang.org/grpc/codes" + "google.golang.org/grpc/status" + "gopkg.in/reform.v1" + "gopkg.in/reform.v1/dialects/postgresql" + + alerting "github.com/percona/pmm/api/alerting/v1" + "github.com/percona/pmm/managed/models" + "github.com/percona/pmm/managed/utils/testdb" +) + +const thresholdTestRuleID = "threshold-api-rule" + +func setupThresholdAPI(t *testing.T) (*Service, *reform.DB, *models.Node) { + t.Helper() + + sqlDB := testdb.Open(t, models.SkipFixtures, nil) + t.Cleanup(func() { require.NoError(t, sqlDB.Close()) }) + db := reform.NewDB(sqlDB, postgresql.Dialect, reform.NewPrintfLogger(t.Logf)) + + svc, err := NewService(db, newMockGrafanaClient(t)) + require.NoError(t, err) + + // Alerting must be on, or every RPC short-circuits. + _, err = models.UpdateSettings(db, &models.ChangeSettingsParams{EnableAlerting: pointer.ToBool(true)}) + require.NoError(t, err) + + _, err = models.CreateAlertRule(db.Querier, &models.CreateAlertRuleParams{ + RuleID: thresholdTestRuleID, + Params: models.AlertRuleParams{ + "threshold": { + Default: 80, + JoinLabel: "node_name", + Scopes: []string{string(models.ThresholdScopeNode)}, + Unit: "%", + Summary: "A percentage from configured maximum", + Min: pointer.ToFloat64(0), + Max: pointer.ToFloat64(100), + }, + }, + }) + require.NoError(t, err) + + node, err := models.CreateNode(db.Querier, models.GenericNodeType, &models.CreateNodeParams{ + NodeName: "api-node-1", + Address: "api-node-1.example.com", + }) + require.NoError(t, err) + + return svc, db, node +} + +func TestSetThreshold(t *testing.T) { + ctx := t.Context() + + t.Run("sets an override and reports the effective value", func(t *testing.T) { + svc, _, node := setupThresholdAPI(t) + + res, err := svc.SetThreshold(ctx, &alerting.SetThresholdRequest{ + Scope: alerting.ThresholdScope_THRESHOLD_SCOPE_NODE, + Target: node.NodeID, + RuleId: thresholdTestRuleID, + ParamName: "threshold", + Value: 90, + }) + require.NoError(t, err) + + assert.InDelta(t, 90.0, res.Threshold.EffectiveValue, 0.0001) + assert.InDelta(t, 80.0, res.Threshold.DefaultValue, 0.0001) + assert.True(t, res.Threshold.IsOverridden) + assert.Equal(t, alerting.ThresholdScope_THRESHOLD_SCOPE_NODE, res.Threshold.Scope) + assert.Equal(t, node.NodeID, res.Threshold.Target) + assert.Equal(t, alerting.ParamUnit_PARAM_UNIT_PERCENTAGE, res.Threshold.Unit) + assert.Equal(t, "A percentage from configured maximum", res.Threshold.Summary) + }) + + t.Run("rejects a value outside the declared range", func(t *testing.T) { + svc, _, node := setupThresholdAPI(t) + + _, err := svc.SetThreshold(ctx, &alerting.SetThresholdRequest{ + Scope: alerting.ThresholdScope_THRESHOLD_SCOPE_NODE, Target: node.NodeID, + RuleId: thresholdTestRuleID, ParamName: "threshold", Value: 150, + }) + require.Error(t, err) + assert.Equal(t, codes.InvalidArgument, status.Code(err)) + }) + + t.Run("rejects non-finite values before they reach the database", func(t *testing.T) { + svc, _, node := setupThresholdAPI(t) + + for _, value := range []float64{math.NaN(), math.Inf(1), math.Inf(-1)} { + _, err := svc.SetThreshold(ctx, &alerting.SetThresholdRequest{ + Scope: alerting.ThresholdScope_THRESHOLD_SCOPE_NODE, Target: node.NodeID, + RuleId: thresholdTestRuleID, ParamName: "threshold", Value: value, + }) + require.Error(t, err) + assert.Equal(t, codes.InvalidArgument, status.Code(err), + "the database CHECK would surface as an opaque internal error instead") + } + }) + + t.Run("rejects an unknown parameter", func(t *testing.T) { + svc, _, node := setupThresholdAPI(t) + + _, err := svc.SetThreshold(ctx, &alerting.SetThresholdRequest{ + Scope: alerting.ThresholdScope_THRESHOLD_SCOPE_NODE, Target: node.NodeID, + RuleId: thresholdTestRuleID, ParamName: "not-overridable", Value: 90, + }) + require.Error(t, err) + assert.Equal(t, codes.NotFound, status.Code(err)) + }) + + t.Run("rejects an unknown rule", func(t *testing.T) { + svc, _, node := setupThresholdAPI(t) + + _, err := svc.SetThreshold(ctx, &alerting.SetThresholdRequest{ + Scope: alerting.ThresholdScope_THRESHOLD_SCOPE_NODE, Target: node.NodeID, + RuleId: "no-such-rule", ParamName: "threshold", Value: 90, + }) + require.Error(t, err) + assert.Equal(t, codes.NotFound, status.Code(err)) + }) + + t.Run("rejects a target that does not exist", func(t *testing.T) { + svc, _, _ := setupThresholdAPI(t) + + _, err := svc.SetThreshold(ctx, &alerting.SetThresholdRequest{ + Scope: alerting.ThresholdScope_THRESHOLD_SCOPE_NODE, Target: "no-such-node", + RuleId: thresholdTestRuleID, ParamName: "threshold", Value: 90, + }) + require.Error(t, err) + assert.Equal(t, codes.NotFound, status.Code(err)) + }) + + // Service and cluster scope are carried by the schema, resolver and proto already, so + // they report as not-yet-implemented rather than as a malformed request. + t.Run("reports unimplemented scopes distinctly from invalid ones", func(t *testing.T) { + svc, _, node := setupThresholdAPI(t) + + for _, scope := range []alerting.ThresholdScope{ + alerting.ThresholdScope_THRESHOLD_SCOPE_SERVICE, + alerting.ThresholdScope_THRESHOLD_SCOPE_CLUSTER, + } { + _, err := svc.SetThreshold(ctx, &alerting.SetThresholdRequest{ + Scope: scope, Target: node.NodeID, + RuleId: thresholdTestRuleID, ParamName: "threshold", Value: 90, + }) + require.Error(t, err) + assert.Equal(t, codes.Unimplemented, status.Code(err)) + } + }) +} + +func TestClearThreshold(t *testing.T) { + ctx := t.Context() + + t.Run("clearing returns the target to the default", func(t *testing.T) { + svc, db, node := setupThresholdAPI(t) + + _, err := svc.SetThreshold(ctx, &alerting.SetThresholdRequest{ + Scope: alerting.ThresholdScope_THRESHOLD_SCOPE_NODE, Target: node.NodeID, + RuleId: thresholdTestRuleID, ParamName: "threshold", Value: 90, + }) + require.NoError(t, err) + + _, err = svc.ClearThreshold(ctx, &alerting.ClearThresholdRequest{ + Scope: alerting.ThresholdScope_THRESHOLD_SCOPE_NODE, Target: node.NodeID, + RuleId: thresholdTestRuleID, ParamName: "threshold", + }) + require.NoError(t, err) + + // The row survives as a tombstone: that is what keeps the emitted series alive + // so the clear lands in one scrape rather than a lookbehind. + overrides, err := models.FindThresholdOverridesByRule(db.Querier, thresholdTestRuleID) + require.NoError(t, err) + require.Len(t, overrides, 1) + assert.True(t, overrides[0].IsCleared()) + + list, err := svc.ListThresholds(ctx, &alerting.ListThresholdsRequest{ + Scope: alerting.ThresholdScope_THRESHOLD_SCOPE_NODE, Target: node.NodeID, + }) + require.NoError(t, err) + require.Len(t, list.Thresholds, 1) + assert.InDelta(t, 80.0, list.Thresholds[0].EffectiveValue, 0.0001) + assert.False(t, list.Thresholds[0].IsOverridden, + "a tombstone must not read as an override, or every target ever tuned reads as tuned forever") + }) +} + +func TestListThresholds(t *testing.T) { + ctx := t.Context() + + t.Run("with a target, every overridable parameter is reported", func(t *testing.T) { + svc, _, node := setupThresholdAPI(t) + + res, err := svc.ListThresholds(ctx, &alerting.ListThresholdsRequest{ + Scope: alerting.ThresholdScope_THRESHOLD_SCOPE_NODE, Target: node.NodeID, + }) + require.NoError(t, err) + + require.Len(t, res.Thresholds, 1, "an untouched target still reports its default") + assert.InDelta(t, 80.0, res.Thresholds[0].EffectiveValue, 0.0001) + assert.False(t, res.Thresholds[0].IsOverridden) + }) + + t.Run("without a target, only actual overrides are reported", func(t *testing.T) { + svc, _, node := setupThresholdAPI(t) + + res, err := svc.ListThresholds(ctx, &alerting.ListThresholdsRequest{}) + require.NoError(t, err) + assert.Empty(t, res.Thresholds, "there is no bounded target set to enumerate") + + _, err = svc.SetThreshold(ctx, &alerting.SetThresholdRequest{ + Scope: alerting.ThresholdScope_THRESHOLD_SCOPE_NODE, Target: node.NodeID, + RuleId: thresholdTestRuleID, ParamName: "threshold", Value: 90, + }) + require.NoError(t, err) + + res, err = svc.ListThresholds(ctx, &alerting.ListThresholdsRequest{}) + require.NoError(t, err) + require.Len(t, res.Thresholds, 1) + assert.True(t, res.Thresholds[0].IsOverridden) + }) +} + +func TestBatchUpdateThresholds(t *testing.T) { + ctx := t.Context() + + t.Run("applies several updates", func(t *testing.T) { + svc, _, node := setupThresholdAPI(t) + + res, err := svc.BatchUpdateThresholds(ctx, &alerting.BatchUpdateThresholdsRequest{ + Updates: []*alerting.ThresholdUpdate{{ + Scope: alerting.ThresholdScope_THRESHOLD_SCOPE_NODE, Target: node.NodeID, + RuleId: thresholdTestRuleID, ParamName: "threshold", + Value: pointer.ToFloat64(95), + }}, + }) + require.NoError(t, err) + require.Len(t, res.Thresholds, 1) + assert.InDelta(t, 95.0, res.Thresholds[0].EffectiveValue, 0.0001) + }) + + t.Run("an update with no value clears instead of setting", func(t *testing.T) { + svc, db, node := setupThresholdAPI(t) + + _, err := svc.SetThreshold(ctx, &alerting.SetThresholdRequest{ + Scope: alerting.ThresholdScope_THRESHOLD_SCOPE_NODE, Target: node.NodeID, + RuleId: thresholdTestRuleID, ParamName: "threshold", Value: 90, + }) + require.NoError(t, err) + + res, err := svc.BatchUpdateThresholds(ctx, &alerting.BatchUpdateThresholdsRequest{ + Updates: []*alerting.ThresholdUpdate{{ + Scope: alerting.ThresholdScope_THRESHOLD_SCOPE_NODE, Target: node.NodeID, + RuleId: thresholdTestRuleID, ParamName: "threshold", + }}, + }) + require.NoError(t, err) + assert.Empty(t, res.Thresholds, "cleared entries are omitted from the response") + + overrides, err := models.FindThresholdOverridesByRule(db.Querier, thresholdTestRuleID) + require.NoError(t, err) + require.Len(t, overrides, 1) + assert.True(t, overrides[0].IsCleared()) + }) + + // The whole reason the batch endpoint exists: a client editing many rows at once + // must never land a partial result it cannot report. + t.Run("one bad update rolls the whole batch back", func(t *testing.T) { + svc, db, node := setupThresholdAPI(t) + + _, err := svc.BatchUpdateThresholds(ctx, &alerting.BatchUpdateThresholdsRequest{ + Updates: []*alerting.ThresholdUpdate{ + { + Scope: alerting.ThresholdScope_THRESHOLD_SCOPE_NODE, Target: node.NodeID, + RuleId: thresholdTestRuleID, ParamName: "threshold", + Value: pointer.ToFloat64(90), + }, + { + Scope: alerting.ThresholdScope_THRESHOLD_SCOPE_NODE, Target: node.NodeID, + RuleId: thresholdTestRuleID, ParamName: "threshold", + Value: pointer.ToFloat64(500), // out of range + }, + }, + }) + require.Error(t, err) + assert.Equal(t, codes.InvalidArgument, status.Code(err)) + + overrides, err := models.FindThresholdOverridesByRule(db.Querier, thresholdTestRuleID) + require.NoError(t, err) + assert.Empty(t, overrides, "the first update must not survive the second one failing") + }) +} diff --git a/managed/services/grafana/auth_server.go b/managed/services/grafana/auth_server.go index 7bb9927c30..b8659640e5 100644 --- a/managed/services/grafana/auth_server.go +++ b/managed/services/grafana/auth_server.go @@ -68,6 +68,7 @@ var rules = map[string]role{ "/v1/alerting": viewer, "/v1/alerting/rules": editor, + "/v1/alerting/thresholds": admin, "/v1/advisors": editor, "/v1/advisors/checks:": editor, "/v1/advisors/failedServices": editor, diff --git a/managed/services/grafana/auth_server_test.go b/managed/services/grafana/auth_server_test.go index a009ab5d18..a29f316476 100644 --- a/managed/services/grafana/auth_server_test.go +++ b/managed/services/grafana/auth_server_test.go @@ -71,11 +71,15 @@ func TestResolveRule(t *testing.T) { wantRole role }{ // Alerting: only listing templates is viewable; writes need editor. - {http.MethodGet, "/v1/alerting/templates", viewer}, // ListTemplates - {http.MethodPost, "/v1/alerting/templates", editor}, // CreateTemplate - {http.MethodPut, "/v1/alerting/templates/foo", editor}, // UpdateTemplate - {http.MethodDelete, "/v1/alerting/templates/foo", editor}, // DeleteTemplate - {http.MethodPost, "/v1/alerting/rules", editor}, // CreateRule + {http.MethodGet, "/v1/alerting/templates", viewer}, // ListTemplates + {http.MethodPost, "/v1/alerting/templates", editor}, // CreateTemplate + {http.MethodPut, "/v1/alerting/templates/foo", editor}, // UpdateTemplate + {http.MethodDelete, "/v1/alerting/templates/foo", editor}, // DeleteTemplate + {http.MethodPost, "/v1/alerting/rules", editor}, // CreateRule + {http.MethodGet, "/v1/alerting/thresholds", admin}, // ListThresholds + {http.MethodPost, "/v1/alerting/thresholds", admin}, // SetThreshold + {http.MethodDelete, "/v1/alerting/thresholds", admin}, // ClearThreshold + {http.MethodPost, "/v1/alerting/thresholds:batchUpdate", admin}, // BatchUpdateThresholds // No matching rule falls back to grafanaAdmin. {http.MethodGet, "/v1/unknown", grafanaAdmin}, } { From 154c35fc92a42af67fc2ab5d17f8b1f653043b15 Mon Sep 17 00:00:00 2001 From: Matej Kubinec Date: Fri, 28 Aug 2026 14:01:07 +0200 Subject: [PATCH 06/15] PMM-14912 Clean up orphaned threshold overrides Threshold overrides have no cascade to rely on. Their target column is polymorphic - a node id, a service id, or a cluster label value - so it carries no foreign key, and rows would otherwise outlive whatever they point at. Two paths close that: entity removal, and a sweep for rules deleted in Grafana. RemoveNode and RemoveService now delete a target's overrides in the same transaction as the entity itself, so an override can never survive its target even briefly. Removing a node reaches its services' overrides through the existing cascade into RemoveService, so there is no second cleanup path to keep in step. Cluster-scoped rows are deliberately never removed: a cluster with no services yet is dormant rather than stale, and should apply again when services join it. The reconciler reaps registry rows whose Grafana rule is gone, taking their overrides with them through the rule foreign key. It runs leader-only, since every replica shares one database and parallel sweeps would duplicate the same deletions. Rules are matched by the identity label PMM stamps on them rather than by Grafana UID, so a rule that was copied or renamed still counts as present. Two guards, both protecting configuration a user set by hand: - Rows younger than the grace period are spared. CreateRule writes the registry row before the rule reaches Grafana, so a sweep landing in that window would otherwise delete the row of a rule being created successfully. - A failed lookup aborts the sweep rather than being read as an empty Grafana, which would reap the entire registry. ListPMMRuleIDs is the only Grafana client surface this needs; the label constant moves to the services package so both sides reference one definition rather than repeating the string. Co-Authored-By: Claude Opus 5 (1M context) --- managed/cmd/pmm-managed/main.go | 7 + managed/models/alert_rule_helpers_test.go | 144 +++++++++++++++++ managed/models/node_helpers.go | 8 + managed/models/service_helpers.go | 6 + managed/services/alert_rule.go | 5 + managed/services/alerting/deps.go | 1 + .../alerting/mock_grafana_client_test.go | 30 ++++ managed/services/alerting/reconciler.go | 110 +++++++++++++ managed/services/alerting/reconciler_test.go | 145 ++++++++++++++++++ managed/services/alerting/service.go | 2 +- managed/services/grafana/client.go | 44 ++++++ 11 files changed, 501 insertions(+), 1 deletion(-) create mode 100644 managed/services/alerting/reconciler.go create mode 100644 managed/services/alerting/reconciler_test.go diff --git a/managed/cmd/pmm-managed/main.go b/managed/cmd/pmm-managed/main.go index 895937003f..dd9bb9f6fe 100644 --- a/managed/cmd/pmm-managed/main.go +++ b/managed/cmd/pmm-managed/main.go @@ -1157,6 +1157,13 @@ func main() { //nolint:gocognit,maintidx,cyclop return nil })) + // Leader-only: every replica shares one database, so several sweeps would duplicate + // the same deletions and race each other. + haService.AddLeaderService(ha.NewContextService("alert-rule-reconciler", func(ctx context.Context) error { + alertingService.RunReconciler(ctx) + return nil + })) + wg.Go(func() { updater.Run(ctx) }) diff --git a/managed/models/alert_rule_helpers_test.go b/managed/models/alert_rule_helpers_test.go index 63438e8089..f4460fd579 100644 --- a/managed/models/alert_rule_helpers_test.go +++ b/managed/models/alert_rule_helpers_test.go @@ -19,6 +19,7 @@ import ( "math" "testing" + "github.com/AlekSi/pointer" "github.com/google/uuid" "github.com/stretchr/testify/assert" "github.com/stretchr/testify/require" @@ -350,3 +351,146 @@ func TestThresholdOverrideRejectsNonFiniteValues(t *testing.T) { }) } } + +// TestThresholdOverridesFollowTargetRemoval covers the cleanup that has no cascade to +// rely on: the target column is polymorphic, so it carries no foreign key and rows must +// be removed by the removal API itself. +func TestThresholdOverridesFollowTargetRemoval(t *testing.T) { + sqlDB := testdb.Open(t, models.SkipFixtures, nil) + t.Cleanup(func() { + require.NoError(t, sqlDB.Close()) + }) + db := reform.NewDB(sqlDB, postgresql.Dialect, reform.NewPrintfLogger(t.Logf)) + + t.Run("removing a node removes its overrides", func(t *testing.T) { + tx, err := db.Begin() + require.NoError(t, err) + defer func() { + require.NoError(t, tx.Rollback()) + }() + q := tx.Querier + + rule := createTestAlertRule(t, q) + node, err := models.CreateNode(q, models.GenericNodeType, &models.CreateNodeParams{ + NodeName: "doomed-node", + Address: "doomed.example.com", + }) + require.NoError(t, err) + + _, err = models.UpsertThresholdOverride(q, rule.RuleID, "threshold", models.ThresholdScopeNode, node.NodeID, 90) + require.NoError(t, err) + + require.NoError(t, models.RemoveNode(q, node.NodeID, models.RemoveRestrict)) + + all, err := models.FindThresholdOverridesByRule(q, rule.RuleID) + require.NoError(t, err) + assert.Empty(t, all, "an override must not outlive the node it targets") + }) + + t.Run("removing a service removes its overrides", func(t *testing.T) { + tx, err := db.Begin() + require.NoError(t, err) + defer func() { + require.NoError(t, tx.Rollback()) + }() + q := tx.Querier + + rule := createTestAlertRule(t, q) + node, err := models.CreateNode(q, models.GenericNodeType, &models.CreateNodeParams{ + NodeName: "svc-host", + Address: "svc-host.example.com", + }) + require.NoError(t, err) + + service, err := models.AddNewService(q, models.MySQLServiceType, &models.AddDBMSServiceParams{ + ServiceName: "doomed-service", + NodeID: node.NodeID, + Address: pointer.ToString("127.0.0.1"), + Port: pointer.ToUint16(3306), + }) + require.NoError(t, err) + + _, err = models.UpsertThresholdOverride(q, rule.RuleID, "threshold", models.ThresholdScopeService, service.ServiceID, 70) + require.NoError(t, err) + + require.NoError(t, models.RemoveService(q, service.ServiceID, models.RemoveRestrict)) + + all, err := models.FindThresholdOverridesByRule(q, rule.RuleID) + require.NoError(t, err) + assert.Empty(t, all) + }) + + // Removing a node cascades into its services, which is where the service-scoped + // rows are reached from - there is no second cleanup path for them. + t.Run("removing a node cascades to its services' overrides", func(t *testing.T) { + tx, err := db.Begin() + require.NoError(t, err) + defer func() { + require.NoError(t, tx.Rollback()) + }() + q := tx.Querier + + rule := createTestAlertRule(t, q) + node, err := models.CreateNode(q, models.GenericNodeType, &models.CreateNodeParams{ + NodeName: "cascade-host", + Address: "cascade-host.example.com", + }) + require.NoError(t, err) + + service, err := models.AddNewService(q, models.MySQLServiceType, &models.AddDBMSServiceParams{ + ServiceName: "cascade-service", + NodeID: node.NodeID, + Address: pointer.ToString("127.0.0.1"), + Port: pointer.ToUint16(3306), + }) + require.NoError(t, err) + + _, err = models.UpsertThresholdOverride(q, rule.RuleID, "threshold", models.ThresholdScopeNode, node.NodeID, 90) + require.NoError(t, err) + _, err = models.UpsertThresholdOverride(q, rule.RuleID, "threshold", models.ThresholdScopeService, service.ServiceID, 70) + require.NoError(t, err) + + require.NoError(t, models.RemoveNode(q, node.NodeID, models.RemoveCascade)) + + all, err := models.FindThresholdOverridesByRule(q, rule.RuleID) + require.NoError(t, err) + assert.Empty(t, all, "the service's override must go with the node that hosted it") + }) + + // A cluster override with no matching services is dormant, not stale: services may + // join that cluster later, and the override should apply again when they do. + t.Run("a cluster override survives removal of its services", func(t *testing.T) { + tx, err := db.Begin() + require.NoError(t, err) + defer func() { + require.NoError(t, tx.Rollback()) + }() + q := tx.Querier + + rule := createTestAlertRule(t, q) + node, err := models.CreateNode(q, models.GenericNodeType, &models.CreateNodeParams{ + NodeName: "cluster-host", + Address: "cluster-host.example.com", + }) + require.NoError(t, err) + + service, err := models.AddNewService(q, models.MySQLServiceType, &models.AddDBMSServiceParams{ + ServiceName: "clustered-service", + NodeID: node.NodeID, + Cluster: "prod", + Address: pointer.ToString("127.0.0.1"), + Port: pointer.ToUint16(3306), + }) + require.NoError(t, err) + + _, err = models.UpsertThresholdOverride(q, rule.RuleID, "threshold", models.ThresholdScopeCluster, "prod", 60) + require.NoError(t, err) + + require.NoError(t, models.RemoveService(q, service.ServiceID, models.RemoveRestrict)) + + all, err := models.FindThresholdOverridesByRule(q, rule.RuleID) + require.NoError(t, err) + require.Len(t, all, 1) + assert.Equal(t, models.ThresholdScopeCluster, all[0].Scope) + }) +} diff --git a/managed/models/node_helpers.go b/managed/models/node_helpers.go index ccd63e85d7..e39545326b 100644 --- a/managed/models/node_helpers.go +++ b/managed/models/node_helpers.go @@ -349,6 +349,14 @@ func removeNode(q *reform.Querier, id string, mode RemoveMode, allowPMMServerNod } } + // Threshold overrides carry no foreign key - their target column is polymorphic, and + // a cluster target has no referent table to point at - so they are removed here, in + // the same transaction, rather than by a cascade. + err = DeleteThresholdOverridesForTarget(q, ThresholdScopeNode, id) + if err != nil { + return err + } + err = q.Delete(n) if err != nil { return fmt.Errorf("failed to delete Node: %w", err) diff --git a/managed/models/service_helpers.go b/managed/models/service_helpers.go index 659e1a25b9..f8890d84ea 100644 --- a/managed/models/service_helpers.go +++ b/managed/models/service_helpers.go @@ -425,6 +425,12 @@ func RemoveService(q *reform.Querier, id string, mode RemoveMode) error { //noli panic(fmt.Errorf("unhandled RemoveMode %v", mode)) } + // See RemoveNode: override rows are not reachable by cascade, so they go here. + err = DeleteThresholdOverridesForTarget(q, ThresholdScopeService, id) + if err != nil { + return err + } + err = q.Delete(s) if err != nil { return fmt.Errorf("failed to delete Service: %w", err) diff --git a/managed/services/alert_rule.go b/managed/services/alert_rule.go index e9c0823001..80addd7725 100644 --- a/managed/services/alert_rule.go +++ b/managed/services/alert_rule.go @@ -19,6 +19,11 @@ import "encoding/json" // This file contains grafana alerting API DTOs. +// PMMRuleIDLabel is the label carrying PMM's own identity for a rule whose thresholds can +// be overridden. It lives on the rule rather than being its Grafana UID, so the rule can +// be matched back to its overrides after being copied or renamed in Grafana. +const PMMRuleIDLabel = "pmm_rule_id" + // Rule represents grafana alerting rule. type Rule struct { GrafanaAlert GrafanaAlert `json:"grafana_alert"` diff --git a/managed/services/alerting/deps.go b/managed/services/alerting/deps.go index 57e9eee88e..2bae06f042 100644 --- a/managed/services/alerting/deps.go +++ b/managed/services/alerting/deps.go @@ -25,6 +25,7 @@ import ( type grafanaClient interface { CreateAlertRule(ctx context.Context, folderUID, groupName, interval string, rule *services.Rule) error + ListPMMRuleIDs(ctx context.Context) (map[string]struct{}, error) GetDatasourceUIDByName(ctx context.Context, name string) (string, error) GetFolderByUID(ctx context.Context, uid string) (*models.Folder, error) } diff --git a/managed/services/alerting/mock_grafana_client_test.go b/managed/services/alerting/mock_grafana_client_test.go index 72e1fde3e7..4c5d456354 100644 --- a/managed/services/alerting/mock_grafana_client_test.go +++ b/managed/services/alerting/mock_grafana_client_test.go @@ -92,6 +92,36 @@ func (_m *mockGrafanaClient) GetFolderByUID(ctx context.Context, uid string) (*m return r0, r1 } +// ListPMMRuleIDs provides a mock function with given fields: ctx +func (_m *mockGrafanaClient) ListPMMRuleIDs(ctx context.Context) (map[string]struct{}, error) { + ret := _m.Called(ctx) + + if len(ret) == 0 { + panic("no return value specified for ListPMMRuleIDs") + } + + var r0 map[string]struct{} + var r1 error + if rf, ok := ret.Get(0).(func(context.Context) (map[string]struct{}, error)); ok { + return rf(ctx) + } + if rf, ok := ret.Get(0).(func(context.Context) map[string]struct{}); ok { + r0 = rf(ctx) + } else { + if ret.Get(0) != nil { + r0 = ret.Get(0).(map[string]struct{}) + } + } + + if rf, ok := ret.Get(1).(func(context.Context) error); ok { + r1 = rf(ctx) + } else { + r1 = ret.Error(1) + } + + return r0, r1 +} + // newMockGrafanaClient creates a new instance of mockGrafanaClient. It also registers a testing interface on the mock and a cleanup function to assert the mocks expectations. // The first argument is typically a *testing.T value. func newMockGrafanaClient(t interface { diff --git a/managed/services/alerting/reconciler.go b/managed/services/alerting/reconciler.go new file mode 100644 index 0000000000..a42f1fefb4 --- /dev/null +++ b/managed/services/alerting/reconciler.go @@ -0,0 +1,110 @@ +// Copyright (C) 2023 Percona LLC +// +// This program is free software: you can redistribute it and/or modify +// it under the terms of the GNU Affero General Public License as published by +// the Free Software Foundation, either version 3 of the License, or +// (at your option) any later version. +// +// This program is distributed in the hope that it will be useful, +// but WITHOUT ANY WARRANTY; without even the implied warranty of +// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +// GNU Affero General Public License for more details. +// +// You should have received a copy of the GNU Affero General Public License +// along with this program. If not, see . + +package alerting + +import ( + "context" + "time" + + "gopkg.in/reform.v1" + + "github.com/percona/pmm/managed/models" +) + +const ( + // reconcileInterval is how often orphaned registry rows are reaped. Orphans are + // inert rather than harmful - the collector emits nothing for a rule that is gone - + // so this trades promptness for staying out of the way. + reconcileInterval = 15 * time.Minute + + // reconcileGracePeriod keeps a freshly created row safe from the sweep. CreateRule + // writes the registry row before the rule exists in Grafana, so without this a sweep + // landing in that window would delete the row of a rule being created successfully. + reconcileGracePeriod = 10 * time.Minute +) + +// RunReconciler reaps registry rows whose Grafana rule no longer exists, until the +// context is cancelled. +// +// It must run leader-only: every replica shares one database, so several sweeps would +// duplicate the same deletions and race each other. +func (s *Service) RunReconciler(ctx context.Context) { + ticker := time.NewTicker(reconcileInterval) + defer ticker.Stop() + + for { + select { + case <-ctx.Done(): + return + case <-ticker.C: + err := s.ReconcileAlertRules(ctx) + if err != nil { + s.l.WithError(err).Warn("Failed to reconcile alert rule registry") + } + } + } +} + +// ReconcileAlertRules deletes registry rows for rules that are no longer in Grafana, +// taking their threshold overrides with them through the foreign key. +// +// Rules are matched by the identity label PMM stamps on them rather than by Grafana UID, +// so a rule that was copied or renamed still counts as present. +func (s *Service) ReconcileAlertRules(ctx context.Context) error { + live, err := s.grafanaClient.ListPMMRuleIDs(ctx) + if err != nil { + return err + } + + cutoff := models.Now().Add(-reconcileGracePeriod) + + var reaped []string + + errTx := s.db.InTransaction(func(tx *reform.TX) error { + rules, err := models.FindAlertRules(tx.Querier) + if err != nil { + return err + } + + for _, rule := range rules { + _, exists := live[rule.RuleID] + if exists || rule.CreatedAt.After(cutoff) { + continue + } + + err = models.DeleteAlertRule(tx.Querier, rule.RuleID) + if err != nil { + return err + } + + reaped = append(reaped, rule.RuleID) + } + + return nil + }) + if errTx != nil { + return errTx + } + + if len(reaped) != 0 { + // Worth a log line: this deletes override configuration a user set by hand, so + // it should be explainable after the fact. + s.l.WithField("rule_ids", reaped). + Infof("Reaped %d alert rule registry rows whose rules no longer exist", len(reaped)) + } + + return nil +} diff --git a/managed/services/alerting/reconciler_test.go b/managed/services/alerting/reconciler_test.go new file mode 100644 index 0000000000..e667c62a56 --- /dev/null +++ b/managed/services/alerting/reconciler_test.go @@ -0,0 +1,145 @@ +// Copyright (C) 2023 Percona LLC +// +// This program is free software: you can redistribute it and/or modify +// it under the terms of the GNU Affero General Public License as published by +// the Free Software Foundation, either version 3 of the License, or +// (at your option) any later version. +// +// This program is distributed in the hope that it will be useful, +// but WITHOUT ANY WARRANTY; without even the implied warranty of +// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +// GNU Affero General Public License for more details. +// +// You should have received a copy of the GNU Affero General Public License +// along with this program. If not, see . + +package alerting + +import ( + "errors" + "testing" + "time" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/mock" + "github.com/stretchr/testify/require" + "gopkg.in/reform.v1" + "gopkg.in/reform.v1/dialects/postgresql" + + "github.com/percona/pmm/managed/models" + "github.com/percona/pmm/managed/utils/testdb" +) + +func setupReconciler(t *testing.T) (*Service, *mockGrafanaClient, *reform.DB) { + t.Helper() + + sqlDB := testdb.Open(t, models.SkipFixtures, nil) + t.Cleanup(func() { require.NoError(t, sqlDB.Close()) }) + db := reform.NewDB(sqlDB, postgresql.Dialect, reform.NewPrintfLogger(t.Logf)) + + m := newMockGrafanaClient(t) + svc, err := NewService(db, m) + require.NoError(t, err) + + return svc, m, db +} + +// createRegistryRow inserts a rule row and ages it past the grace period, so the +// reconciler is willing to consider it. +func createRegistryRow(t *testing.T, db *reform.DB, ruleID string, age time.Duration) { + t.Helper() + + _, err := models.CreateAlertRule(db.Querier, &models.CreateAlertRuleParams{ + RuleID: ruleID, + Params: models.AlertRuleParams{ + "threshold": {Default: 80, JoinLabel: "node_name", Scopes: []string{string(models.ThresholdScopeNode)}}, + }, + }) + require.NoError(t, err) + + _, err = db.Exec(`UPDATE alert_rules SET created_at = $1 WHERE rule_id = $2`, + models.Now().Add(-age), ruleID) + require.NoError(t, err) +} + +func TestReconcileAlertRules(t *testing.T) { + ctx := t.Context() + + t.Run("reaps a row whose rule is gone from Grafana", func(t *testing.T) { + svc, m, db := setupReconciler(t) + createRegistryRow(t, db, "gone-rule", time.Hour) + + m.On("ListPMMRuleIDs", mock.Anything).Return(map[string]struct{}{}, nil) + + require.NoError(t, svc.ReconcileAlertRules(ctx)) + + rules, err := models.FindAlertRules(db.Querier) + require.NoError(t, err) + assert.Empty(t, rules) + }) + + t.Run("keeps a row whose rule still exists", func(t *testing.T) { + svc, m, db := setupReconciler(t) + createRegistryRow(t, db, "live-rule", time.Hour) + + m.On("ListPMMRuleIDs", mock.Anything). + Return(map[string]struct{}{"live-rule": {}}, nil) + + require.NoError(t, svc.ReconcileAlertRules(ctx)) + + rules, err := models.FindAlertRules(db.Querier) + require.NoError(t, err) + require.Len(t, rules, 1) + assert.Equal(t, "live-rule", rules[0].RuleID) + }) + + // CreateRule writes the registry row before the rule exists in Grafana. Without the + // grace period a sweep landing in that window would delete the row of a rule that is + // being created perfectly successfully. + t.Run("spares a row still inside the creation grace period", func(t *testing.T) { + svc, m, db := setupReconciler(t) + createRegistryRow(t, db, "just-created", time.Minute) + + m.On("ListPMMRuleIDs", mock.Anything).Return(map[string]struct{}{}, nil) + + require.NoError(t, svc.ReconcileAlertRules(ctx)) + + rules, err := models.FindAlertRules(db.Querier) + require.NoError(t, err) + require.Len(t, rules, 1, "a row younger than the grace period must survive") + }) + + t.Run("reaping takes the rule's overrides with it", func(t *testing.T) { + svc, m, db := setupReconciler(t) + createRegistryRow(t, db, "gone-rule", time.Hour) + + _, err := models.UpsertThresholdOverride(db.Querier, "gone-rule", "threshold", + models.ThresholdScopeNode, "node-id-1", 90) + require.NoError(t, err) + + m.On("ListPMMRuleIDs", mock.Anything).Return(map[string]struct{}{}, nil) + + require.NoError(t, svc.ReconcileAlertRules(ctx)) + + overrides, err := models.FindAllThresholdOverrides(db.Querier) + require.NoError(t, err) + assert.Empty(t, overrides, "the foreign key cascade should have removed them") + }) + + // A failed lookup must not be read as "Grafana has no rules", which would reap the + // whole registry and destroy override configuration a user set by hand. + t.Run("a Grafana failure deletes nothing", func(t *testing.T) { + svc, m, db := setupReconciler(t) + createRegistryRow(t, db, "some-rule", time.Hour) + + m.On("ListPMMRuleIDs", mock.Anything). + Return(map[string]struct{}(nil), errors.New("grafana unreachable")) + + err := svc.ReconcileAlertRules(ctx) + require.Error(t, err) + + rules, err := models.FindAlertRules(db.Querier) + require.NoError(t, err) + require.Len(t, rules, 1, "an unreachable Grafana must never look like an empty one") + }) +} diff --git a/managed/services/alerting/service.go b/managed/services/alerting/service.go index 1a8581b93a..76a2b0e9fd 100644 --- a/managed/services/alerting/service.go +++ b/managed/services/alerting/service.go @@ -792,7 +792,7 @@ func (s *Service) CreateRule(ctx context.Context, req *alerting.CreateRuleReques // can still be matched back to its registry row after being copied or renamed in // Grafana. The stored Grafana UID is only a cache of where it currently lives. if ruleID != "" { - labels["pmm_rule_id"] = ruleID + labels[services.PMMRuleIDLabel] = ruleID } labelSourceRefID := queryRefForRuleLabels(alertTemplate) diff --git a/managed/services/grafana/client.go b/managed/services/grafana/client.go index 87e466a0e3..4e01258b8c 100644 --- a/managed/services/grafana/client.go +++ b/managed/services/grafana/client.go @@ -767,6 +767,50 @@ func (c *Client) CreateAlertRule(ctx context.Context, folderUID, groupName, inte return nil } +// ListPMMRuleIDs returns the identity label of every Grafana alert rule that carries one, +// which is how PMM-created rules identify themselves. +// +// The label is read rather than the rule's UID because a copied rule gets a new UID while +// keeping the label, so this reports which rules still exist in terms of the identity PMM +// keys its threshold overrides on. +func (c *Client) ListPMMRuleIDs(ctx context.Context) (map[string]struct{}, error) { + authHeaders, err := auth.GetHeadersFromContext(ctx) + if err != nil { + return nil, err + } + + // The ruler returns every folder's groups keyed by folder title. Only the rule + // labels matter here, so the rest of the payload is left unmodelled. + type rulerRule struct { + Labels map[string]string `json:"labels"` + } + + type rulerGroup struct { + Rules []rulerRule `json:"rules"` + } + + var folders map[string][]rulerGroup + + err = c.do(ctx, http.MethodGet, "/api/ruler/grafana/api/v1/rules", "", authHeaders, nil, &folders) + if err != nil { + return nil, err + } + + ids := make(map[string]struct{}) + for _, groups := range folders { + for _, group := range groups { + for _, rule := range group.Rules { + id := rule.Labels[services.PMMRuleIDLabel] + if id != "" { + ids[id] = struct{}{} + } + } + } + } + + return ids, nil +} + func validateDurations(intervalD, forD string) error { i, err := time.ParseDuration(intervalD) if err != nil { From a5ee6d061f5931d56823361d975a516b9d1d99f9 Mon Sep 17 00:00:00 2001 From: Matej Kubinec Date: Fri, 28 Aug 2026 15:04:57 +0200 Subject: [PATCH 07/15] PMM-14912 Make node CPU load threshold overridable Turn the feature on for the first shipped template and let clients discover which parameters it applies to. pmm_node_high_cpu_load qualifies because it is multi-expression and aggregates by node_name, so the injected threshold query has an unambiguous label to join on. A golden-list test pins exactly which built-in templates are overridable, because marking one is not a free change: a template whose observed query does not carry the join label would generate a rule that silently never matches. Of the other three multi-expression templates, postgresql_high_transaction_rollbacks has no parameter at all, and the remaining two aggregate by service_name, so they belong with the service-scope increment. ParamDefinition gains an overridable field. Without it a client listing templates cannot tell which parameters are tunable, which is exactly what a UI needs in order to know what to render as editable. The models layer already carried the flag; only the API surface was missing it, so no unit test noticed. Verified end to end on a live server: creating a rule from this template injects the threshold step and registers the rule, an override reaches the rule's query through the collector, and clearing it returns the target to the default in 12 seconds rather than the five minutes that signalling a clear by absence would cost. Co-Authored-By: Claude Opus 5 (1M context) --- api/alerting/v1/alerting.pb.go | 21 ++++++--- api/alerting/v1/alerting.pb.validate.go | 2 + api/alerting/v1/alerting.proto | 4 ++ .../batch_update_thresholds_responses.go | 6 +-- .../clear_threshold_parameters.go | 9 ++-- .../list_templates_responses.go | 5 +++ .../list_thresholds_parameters.go | 11 +++-- .../list_thresholds_responses.go | 3 +- .../set_threshold_responses.go | 6 +-- api/alerting/v1/json/v1.json | 19 +++++--- api/swagger/swagger-dev.json | 19 +++++--- api/swagger/swagger.json | 19 +++++--- .../alerting-templates/node_high_cpu_load.yml | 1 + managed/services/alerting/service.go | 9 ++-- .../alerting/service_threshold_test.go | 43 +++++++++++++++++++ 15 files changed, 126 insertions(+), 51 deletions(-) diff --git a/api/alerting/v1/alerting.pb.go b/api/alerting/v1/alerting.pb.go index c4b9a149dc..c7315e7bd8 100644 --- a/api/alerting/v1/alerting.pb.go +++ b/api/alerting/v1/alerting.pb.go @@ -147,8 +147,7 @@ const ( ThresholdScope_THRESHOLD_SCOPE_NODE ThresholdScope = 1 // Target is a Service ID. ThresholdScope_THRESHOLD_SCOPE_SERVICE ThresholdScope = 2 - // Target is a cluster label value. Unlike the others it names no inventory entity, - // so it cannot be validated for existence and is never removed by entity deletion. + // Target is a cluster label value. ThresholdScope_THRESHOLD_SCOPE_CLUSTER ThresholdScope = 3 ) @@ -368,7 +367,11 @@ type ParamDefinition struct { // *ParamDefinition_Bool // *ParamDefinition_Float // *ParamDefinition_String_ - Value isParamDefinition_Value `protobuf_oneof:"value"` + Value isParamDefinition_Value `protobuf_oneof:"value"` + // Whether this parameter's threshold can be overridden per target without editing the + // rule. Only set for templates that support it; the scopes it may be set at are + // reported per rule by ListThresholds. + Overridable bool `protobuf:"varint,8,opt,name=overridable,proto3" json:"overridable,omitempty"` unknownFields protoimpl.UnknownFields sizeCache protoimpl.SizeCache } @@ -465,6 +468,13 @@ func (x *ParamDefinition) GetString_() *StringParamDefinition { return nil } +func (x *ParamDefinition) GetOverridable() bool { + if x != nil { + return x.Overridable + } + return false +} + type isParamDefinition_Value interface { isParamDefinition_Value() } @@ -2149,7 +2159,7 @@ const file_alerting_v1_alerting_proto_rawDesc = "" + "\x15StringParamDefinition\x12\x1d\n" + "\adefault\x18\x01 \x01(\tH\x00R\adefault\x88\x01\x01B\n" + "\n" + - "\b_default\"\xe3\x02\n" + + "\b_default\"\x85\x03\n" + "\x0fParamDefinition\x12\x1b\n" + "\x04name\x18\x01 \x01(\tB\a\xfaB\x04r\x02\x10\x01R\x04name\x12!\n" + "\asummary\x18\x02 \x01(\tB\a\xfaB\x04r\x02\x10\x01R\asummary\x12*\n" + @@ -2157,7 +2167,8 @@ const file_alerting_v1_alerting_proto_rawDesc = "" + "\x04type\x18\x04 \x01(\x0e2\x16.alerting.v1.ParamTypeR\x04type\x126\n" + "\x04bool\x18\x05 \x01(\v2 .alerting.v1.BoolParamDefinitionH\x00R\x04bool\x129\n" + "\x05float\x18\x06 \x01(\v2!.alerting.v1.FloatParamDefinitionH\x00R\x05float\x12<\n" + - "\x06string\x18\a \x01(\v2\".alerting.v1.StringParamDefinitionH\x00R\x06stringB\a\n" + + "\x06string\x18\a \x01(\v2\".alerting.v1.StringParamDefinitionH\x00R\x06string\x12 \n" + + "\voverridable\x18\b \x01(\bR\voverridableB\a\n" + "\x05value\":\n" + "\rTemplateQuery\x12\x15\n" + "\x06ref_id\x18\x01 \x01(\tR\x05refId\x12\x12\n" + diff --git a/api/alerting/v1/alerting.pb.validate.go b/api/alerting/v1/alerting.pb.validate.go index 0ce9bb4233..9d9df14709 100644 --- a/api/alerting/v1/alerting.pb.validate.go +++ b/api/alerting/v1/alerting.pb.validate.go @@ -416,6 +416,8 @@ func (m *ParamDefinition) validate(all bool) error { // no validation rules for Type + // no validation rules for Overridable + switch v := m.Value.(type) { case *ParamDefinition_Bool: if v == nil { diff --git a/api/alerting/v1/alerting.proto b/api/alerting/v1/alerting.proto index e905bc56e6..e4f11bc64a 100644 --- a/api/alerting/v1/alerting.proto +++ b/api/alerting/v1/alerting.proto @@ -49,6 +49,10 @@ message ParamDefinition { // String value. StringParamDefinition string = 7; } + // Whether this parameter's threshold can be overridden per target without editing the + // rule. Only set for templates that support it; the scopes it may be set at are + // reported per rule by ListThresholds. + bool overridable = 8; } // TemplateSource defines template source. diff --git a/api/alerting/v1/json/client/alerting_service/batch_update_thresholds_responses.go b/api/alerting/v1/json/client/alerting_service/batch_update_thresholds_responses.go index 70435b6f4c..3305e647fd 100644 --- a/api/alerting/v1/json/client/alerting_service/batch_update_thresholds_responses.go +++ b/api/alerting/v1/json/client/alerting_service/batch_update_thresholds_responses.go @@ -673,8 +673,7 @@ type BatchUpdateThresholdsOKBodyThresholdsItems0 struct { // // - THRESHOLD_SCOPE_NODE: Target is a Node ID. // - THRESHOLD_SCOPE_SERVICE: Target is a Service ID. - // - THRESHOLD_SCOPE_CLUSTER: Target is a cluster label value. Unlike the others it names no inventory entity, - // so it cannot be validated for existence and is never removed by entity deletion. + // - THRESHOLD_SCOPE_CLUSTER: Target is a cluster label value. // Enum: ["THRESHOLD_SCOPE_UNSPECIFIED","THRESHOLD_SCOPE_NODE","THRESHOLD_SCOPE_SERVICE","THRESHOLD_SCOPE_CLUSTER"] Scope *string `json:"scope,omitempty"` @@ -825,8 +824,7 @@ type BatchUpdateThresholdsParamsBodyUpdatesItems0 struct { // // - THRESHOLD_SCOPE_NODE: Target is a Node ID. // - THRESHOLD_SCOPE_SERVICE: Target is a Service ID. - // - THRESHOLD_SCOPE_CLUSTER: Target is a cluster label value. Unlike the others it names no inventory entity, - // so it cannot be validated for existence and is never removed by entity deletion. + // - THRESHOLD_SCOPE_CLUSTER: Target is a cluster label value. // Enum: ["THRESHOLD_SCOPE_UNSPECIFIED","THRESHOLD_SCOPE_NODE","THRESHOLD_SCOPE_SERVICE","THRESHOLD_SCOPE_CLUSTER"] Scope *string `json:"scope,omitempty"` diff --git a/api/alerting/v1/json/client/alerting_service/clear_threshold_parameters.go b/api/alerting/v1/json/client/alerting_service/clear_threshold_parameters.go index 7642a2a53b..387bcbc9ad 100644 --- a/api/alerting/v1/json/client/alerting_service/clear_threshold_parameters.go +++ b/api/alerting/v1/json/client/alerting_service/clear_threshold_parameters.go @@ -65,12 +65,11 @@ type ClearThresholdParams struct { /* Scope. - - THRESHOLD_SCOPE_NODE: Target is a Node ID. - - THRESHOLD_SCOPE_SERVICE: Target is a Service ID. - - THRESHOLD_SCOPE_CLUSTER: Target is a cluster label value. Unlike the others it names no inventory entity, - so it cannot be validated for existence and is never removed by entity deletion. + - THRESHOLD_SCOPE_NODE: Target is a Node ID. + - THRESHOLD_SCOPE_SERVICE: Target is a Service ID. + - THRESHOLD_SCOPE_CLUSTER: Target is a cluster label value. - Default: "THRESHOLD_SCOPE_UNSPECIFIED" + Default: "THRESHOLD_SCOPE_UNSPECIFIED" */ Scope *string diff --git a/api/alerting/v1/json/client/alerting_service/list_templates_responses.go b/api/alerting/v1/json/client/alerting_service/list_templates_responses.go index b263d1e67e..b7d9682b6f 100644 --- a/api/alerting/v1/json/client/alerting_service/list_templates_responses.go +++ b/api/alerting/v1/json/client/alerting_service/list_templates_responses.go @@ -1020,6 +1020,11 @@ type ListTemplatesOKBodyTemplatesItems0ParamsItems0 struct { // Enum: ["PARAM_TYPE_UNSPECIFIED","PARAM_TYPE_BOOL","PARAM_TYPE_FLOAT","PARAM_TYPE_STRING"] Type *string `json:"type,omitempty"` + // Whether this parameter's threshold can be overridden per target without editing the + // rule. Only set for templates that support it; the scopes it may be set at are + // reported per rule by ListThresholds. + Overridable bool `json:"overridable,omitempty"` + // bool Bool *ListTemplatesOKBodyTemplatesItems0ParamsItems0Bool `json:"bool,omitempty"` diff --git a/api/alerting/v1/json/client/alerting_service/list_thresholds_parameters.go b/api/alerting/v1/json/client/alerting_service/list_thresholds_parameters.go index b786a24ad9..e9c03b979b 100644 --- a/api/alerting/v1/json/client/alerting_service/list_thresholds_parameters.go +++ b/api/alerting/v1/json/client/alerting_service/list_thresholds_parameters.go @@ -65,14 +65,13 @@ type ListThresholdsParams struct { /* Scope. - Scope of the target to report thresholds for. Must be set together with target. + Scope of the target to report thresholds for. Must be set together with target. - - THRESHOLD_SCOPE_NODE: Target is a Node ID. - - THRESHOLD_SCOPE_SERVICE: Target is a Service ID. - - THRESHOLD_SCOPE_CLUSTER: Target is a cluster label value. Unlike the others it names no inventory entity, - so it cannot be validated for existence and is never removed by entity deletion. + - THRESHOLD_SCOPE_NODE: Target is a Node ID. + - THRESHOLD_SCOPE_SERVICE: Target is a Service ID. + - THRESHOLD_SCOPE_CLUSTER: Target is a cluster label value. - Default: "THRESHOLD_SCOPE_UNSPECIFIED" + Default: "THRESHOLD_SCOPE_UNSPECIFIED" */ Scope *string diff --git a/api/alerting/v1/json/client/alerting_service/list_thresholds_responses.go b/api/alerting/v1/json/client/alerting_service/list_thresholds_responses.go index 19ad680136..60783b0512 100644 --- a/api/alerting/v1/json/client/alerting_service/list_thresholds_responses.go +++ b/api/alerting/v1/json/client/alerting_service/list_thresholds_responses.go @@ -561,8 +561,7 @@ type ListThresholdsOKBodyThresholdsItems0 struct { // // - THRESHOLD_SCOPE_NODE: Target is a Node ID. // - THRESHOLD_SCOPE_SERVICE: Target is a Service ID. - // - THRESHOLD_SCOPE_CLUSTER: Target is a cluster label value. Unlike the others it names no inventory entity, - // so it cannot be validated for existence and is never removed by entity deletion. + // - THRESHOLD_SCOPE_CLUSTER: Target is a cluster label value. // Enum: ["THRESHOLD_SCOPE_UNSPECIFIED","THRESHOLD_SCOPE_NODE","THRESHOLD_SCOPE_SERVICE","THRESHOLD_SCOPE_CLUSTER"] Scope *string `json:"scope,omitempty"` diff --git a/api/alerting/v1/json/client/alerting_service/set_threshold_responses.go b/api/alerting/v1/json/client/alerting_service/set_threshold_responses.go index 3ad67187cc..0512ee4b89 100644 --- a/api/alerting/v1/json/client/alerting_service/set_threshold_responses.go +++ b/api/alerting/v1/json/client/alerting_service/set_threshold_responses.go @@ -194,8 +194,7 @@ type SetThresholdBody struct { // // - THRESHOLD_SCOPE_NODE: Target is a Node ID. // - THRESHOLD_SCOPE_SERVICE: Target is a Service ID. - // - THRESHOLD_SCOPE_CLUSTER: Target is a cluster label value. Unlike the others it names no inventory entity, - // so it cannot be validated for existence and is never removed by entity deletion. + // - THRESHOLD_SCOPE_CLUSTER: Target is a cluster label value. // Enum: ["THRESHOLD_SCOPE_UNSPECIFIED","THRESHOLD_SCOPE_NODE","THRESHOLD_SCOPE_SERVICE","THRESHOLD_SCOPE_CLUSTER"] Scope *string `json:"scope,omitempty"` @@ -664,8 +663,7 @@ type SetThresholdOKBodyThreshold struct { // // - THRESHOLD_SCOPE_NODE: Target is a Node ID. // - THRESHOLD_SCOPE_SERVICE: Target is a Service ID. - // - THRESHOLD_SCOPE_CLUSTER: Target is a cluster label value. Unlike the others it names no inventory entity, - // so it cannot be validated for existence and is never removed by entity deletion. + // - THRESHOLD_SCOPE_CLUSTER: Target is a cluster label value. // Enum: ["THRESHOLD_SCOPE_UNSPECIFIED","THRESHOLD_SCOPE_NODE","THRESHOLD_SCOPE_SERVICE","THRESHOLD_SCOPE_CLUSTER"] Scope *string `json:"scope,omitempty"` diff --git a/api/alerting/v1/json/v1.json b/api/alerting/v1/json/v1.json index e80fae8eb0..93cdfa64d7 100644 --- a/api/alerting/v1/json/v1.json +++ b/api/alerting/v1/json/v1.json @@ -372,6 +372,11 @@ } }, "x-order": 6 + }, + "overridable": { + "description": "Whether this parameter's threshold can be overridden per target without editing the\nrule. Only set for templates that support it; the scopes it may be set at are\nreported per rule by ListThresholds.", + "type": "boolean", + "x-order": 7 } } }, @@ -738,7 +743,7 @@ ], "type": "string", "default": "THRESHOLD_SCOPE_UNSPECIFIED", - "description": "Scope of the target to report thresholds for. Must be set together with target.\n\n - THRESHOLD_SCOPE_NODE: Target is a Node ID.\n - THRESHOLD_SCOPE_SERVICE: Target is a Service ID.\n - THRESHOLD_SCOPE_CLUSTER: Target is a cluster label value. Unlike the others it names no inventory entity,\nso it cannot be validated for existence and is never removed by entity deletion.", + "description": "Scope of the target to report thresholds for. Must be set together with target.\n\n - THRESHOLD_SCOPE_NODE: Target is a Node ID.\n - THRESHOLD_SCOPE_SERVICE: Target is a Service ID.\n - THRESHOLD_SCOPE_CLUSTER: Target is a cluster label value.", "name": "scope", "in": "query" }, @@ -811,7 +816,7 @@ "x-order": 6 }, "scope": { - "description": "ThresholdScope says what a threshold override's target refers to.\n\n - THRESHOLD_SCOPE_NODE: Target is a Node ID.\n - THRESHOLD_SCOPE_SERVICE: Target is a Service ID.\n - THRESHOLD_SCOPE_CLUSTER: Target is a cluster label value. Unlike the others it names no inventory entity,\nso it cannot be validated for existence and is never removed by entity deletion.", + "description": "ThresholdScope says what a threshold override's target refers to.\n\n - THRESHOLD_SCOPE_NODE: Target is a Node ID.\n - THRESHOLD_SCOPE_SERVICE: Target is a Service ID.\n - THRESHOLD_SCOPE_CLUSTER: Target is a cluster label value.", "type": "string", "default": "THRESHOLD_SCOPE_UNSPECIFIED", "enum": [ @@ -882,7 +887,7 @@ "type": "object", "properties": { "scope": { - "description": "ThresholdScope says what a threshold override's target refers to.\n\n - THRESHOLD_SCOPE_NODE: Target is a Node ID.\n - THRESHOLD_SCOPE_SERVICE: Target is a Service ID.\n - THRESHOLD_SCOPE_CLUSTER: Target is a cluster label value. Unlike the others it names no inventory entity,\nso it cannot be validated for existence and is never removed by entity deletion.", + "description": "ThresholdScope says what a threshold override's target refers to.\n\n - THRESHOLD_SCOPE_NODE: Target is a Node ID.\n - THRESHOLD_SCOPE_SERVICE: Target is a Service ID.\n - THRESHOLD_SCOPE_CLUSTER: Target is a cluster label value.", "type": "string", "default": "THRESHOLD_SCOPE_UNSPECIFIED", "enum": [ @@ -969,7 +974,7 @@ "x-order": 6 }, "scope": { - "description": "ThresholdScope says what a threshold override's target refers to.\n\n - THRESHOLD_SCOPE_NODE: Target is a Node ID.\n - THRESHOLD_SCOPE_SERVICE: Target is a Service ID.\n - THRESHOLD_SCOPE_CLUSTER: Target is a cluster label value. Unlike the others it names no inventory entity,\nso it cannot be validated for existence and is never removed by entity deletion.", + "description": "ThresholdScope says what a threshold override's target refers to.\n\n - THRESHOLD_SCOPE_NODE: Target is a Node ID.\n - THRESHOLD_SCOPE_SERVICE: Target is a Service ID.\n - THRESHOLD_SCOPE_CLUSTER: Target is a cluster label value.", "type": "string", "default": "THRESHOLD_SCOPE_UNSPECIFIED", "enum": [ @@ -1040,7 +1045,7 @@ ], "type": "string", "default": "THRESHOLD_SCOPE_UNSPECIFIED", - "description": " - THRESHOLD_SCOPE_NODE: Target is a Node ID.\n - THRESHOLD_SCOPE_SERVICE: Target is a Service ID.\n - THRESHOLD_SCOPE_CLUSTER: Target is a cluster label value. Unlike the others it names no inventory entity,\nso it cannot be validated for existence and is never removed by entity deletion.", + "description": " - THRESHOLD_SCOPE_NODE: Target is a Node ID.\n - THRESHOLD_SCOPE_SERVICE: Target is a Service ID.\n - THRESHOLD_SCOPE_CLUSTER: Target is a cluster label value.", "name": "scope", "in": "query" }, @@ -1124,7 +1129,7 @@ "type": "object", "properties": { "scope": { - "description": "ThresholdScope says what a threshold override's target refers to.\n\n - THRESHOLD_SCOPE_NODE: Target is a Node ID.\n - THRESHOLD_SCOPE_SERVICE: Target is a Service ID.\n - THRESHOLD_SCOPE_CLUSTER: Target is a cluster label value. Unlike the others it names no inventory entity,\nso it cannot be validated for existence and is never removed by entity deletion.", + "description": "ThresholdScope says what a threshold override's target refers to.\n\n - THRESHOLD_SCOPE_NODE: Target is a Node ID.\n - THRESHOLD_SCOPE_SERVICE: Target is a Service ID.\n - THRESHOLD_SCOPE_CLUSTER: Target is a cluster label value.", "type": "string", "default": "THRESHOLD_SCOPE_UNSPECIFIED", "enum": [ @@ -1219,7 +1224,7 @@ "x-order": 6 }, "scope": { - "description": "ThresholdScope says what a threshold override's target refers to.\n\n - THRESHOLD_SCOPE_NODE: Target is a Node ID.\n - THRESHOLD_SCOPE_SERVICE: Target is a Service ID.\n - THRESHOLD_SCOPE_CLUSTER: Target is a cluster label value. Unlike the others it names no inventory entity,\nso it cannot be validated for existence and is never removed by entity deletion.", + "description": "ThresholdScope says what a threshold override's target refers to.\n\n - THRESHOLD_SCOPE_NODE: Target is a Node ID.\n - THRESHOLD_SCOPE_SERVICE: Target is a Service ID.\n - THRESHOLD_SCOPE_CLUSTER: Target is a cluster label value.", "type": "string", "default": "THRESHOLD_SCOPE_UNSPECIFIED", "enum": [ diff --git a/api/swagger/swagger-dev.json b/api/swagger/swagger-dev.json index 273a84be1c..9082819577 100644 --- a/api/swagger/swagger-dev.json +++ b/api/swagger/swagger-dev.json @@ -2360,6 +2360,11 @@ } }, "x-order": 6 + }, + "overridable": { + "description": "Whether this parameter's threshold can be overridden per target without editing the\nrule. Only set for templates that support it; the scopes it may be set at are\nreported per rule by ListThresholds.", + "type": "boolean", + "x-order": 7 } } }, @@ -2726,7 +2731,7 @@ ], "type": "string", "default": "THRESHOLD_SCOPE_UNSPECIFIED", - "description": "Scope of the target to report thresholds for. Must be set together with target.\n\n - THRESHOLD_SCOPE_NODE: Target is a Node ID.\n - THRESHOLD_SCOPE_SERVICE: Target is a Service ID.\n - THRESHOLD_SCOPE_CLUSTER: Target is a cluster label value. Unlike the others it names no inventory entity,\nso it cannot be validated for existence and is never removed by entity deletion.", + "description": "Scope of the target to report thresholds for. Must be set together with target.\n\n - THRESHOLD_SCOPE_NODE: Target is a Node ID.\n - THRESHOLD_SCOPE_SERVICE: Target is a Service ID.\n - THRESHOLD_SCOPE_CLUSTER: Target is a cluster label value.", "name": "scope", "in": "query" }, @@ -2799,7 +2804,7 @@ "x-order": 6 }, "scope": { - "description": "ThresholdScope says what a threshold override's target refers to.\n\n - THRESHOLD_SCOPE_NODE: Target is a Node ID.\n - THRESHOLD_SCOPE_SERVICE: Target is a Service ID.\n - THRESHOLD_SCOPE_CLUSTER: Target is a cluster label value. Unlike the others it names no inventory entity,\nso it cannot be validated for existence and is never removed by entity deletion.", + "description": "ThresholdScope says what a threshold override's target refers to.\n\n - THRESHOLD_SCOPE_NODE: Target is a Node ID.\n - THRESHOLD_SCOPE_SERVICE: Target is a Service ID.\n - THRESHOLD_SCOPE_CLUSTER: Target is a cluster label value.", "type": "string", "default": "THRESHOLD_SCOPE_UNSPECIFIED", "enum": [ @@ -2870,7 +2875,7 @@ "type": "object", "properties": { "scope": { - "description": "ThresholdScope says what a threshold override's target refers to.\n\n - THRESHOLD_SCOPE_NODE: Target is a Node ID.\n - THRESHOLD_SCOPE_SERVICE: Target is a Service ID.\n - THRESHOLD_SCOPE_CLUSTER: Target is a cluster label value. Unlike the others it names no inventory entity,\nso it cannot be validated for existence and is never removed by entity deletion.", + "description": "ThresholdScope says what a threshold override's target refers to.\n\n - THRESHOLD_SCOPE_NODE: Target is a Node ID.\n - THRESHOLD_SCOPE_SERVICE: Target is a Service ID.\n - THRESHOLD_SCOPE_CLUSTER: Target is a cluster label value.", "type": "string", "default": "THRESHOLD_SCOPE_UNSPECIFIED", "enum": [ @@ -2957,7 +2962,7 @@ "x-order": 6 }, "scope": { - "description": "ThresholdScope says what a threshold override's target refers to.\n\n - THRESHOLD_SCOPE_NODE: Target is a Node ID.\n - THRESHOLD_SCOPE_SERVICE: Target is a Service ID.\n - THRESHOLD_SCOPE_CLUSTER: Target is a cluster label value. Unlike the others it names no inventory entity,\nso it cannot be validated for existence and is never removed by entity deletion.", + "description": "ThresholdScope says what a threshold override's target refers to.\n\n - THRESHOLD_SCOPE_NODE: Target is a Node ID.\n - THRESHOLD_SCOPE_SERVICE: Target is a Service ID.\n - THRESHOLD_SCOPE_CLUSTER: Target is a cluster label value.", "type": "string", "default": "THRESHOLD_SCOPE_UNSPECIFIED", "enum": [ @@ -3028,7 +3033,7 @@ ], "type": "string", "default": "THRESHOLD_SCOPE_UNSPECIFIED", - "description": " - THRESHOLD_SCOPE_NODE: Target is a Node ID.\n - THRESHOLD_SCOPE_SERVICE: Target is a Service ID.\n - THRESHOLD_SCOPE_CLUSTER: Target is a cluster label value. Unlike the others it names no inventory entity,\nso it cannot be validated for existence and is never removed by entity deletion.", + "description": " - THRESHOLD_SCOPE_NODE: Target is a Node ID.\n - THRESHOLD_SCOPE_SERVICE: Target is a Service ID.\n - THRESHOLD_SCOPE_CLUSTER: Target is a cluster label value.", "name": "scope", "in": "query" }, @@ -3112,7 +3117,7 @@ "type": "object", "properties": { "scope": { - "description": "ThresholdScope says what a threshold override's target refers to.\n\n - THRESHOLD_SCOPE_NODE: Target is a Node ID.\n - THRESHOLD_SCOPE_SERVICE: Target is a Service ID.\n - THRESHOLD_SCOPE_CLUSTER: Target is a cluster label value. Unlike the others it names no inventory entity,\nso it cannot be validated for existence and is never removed by entity deletion.", + "description": "ThresholdScope says what a threshold override's target refers to.\n\n - THRESHOLD_SCOPE_NODE: Target is a Node ID.\n - THRESHOLD_SCOPE_SERVICE: Target is a Service ID.\n - THRESHOLD_SCOPE_CLUSTER: Target is a cluster label value.", "type": "string", "default": "THRESHOLD_SCOPE_UNSPECIFIED", "enum": [ @@ -3207,7 +3212,7 @@ "x-order": 6 }, "scope": { - "description": "ThresholdScope says what a threshold override's target refers to.\n\n - THRESHOLD_SCOPE_NODE: Target is a Node ID.\n - THRESHOLD_SCOPE_SERVICE: Target is a Service ID.\n - THRESHOLD_SCOPE_CLUSTER: Target is a cluster label value. Unlike the others it names no inventory entity,\nso it cannot be validated for existence and is never removed by entity deletion.", + "description": "ThresholdScope says what a threshold override's target refers to.\n\n - THRESHOLD_SCOPE_NODE: Target is a Node ID.\n - THRESHOLD_SCOPE_SERVICE: Target is a Service ID.\n - THRESHOLD_SCOPE_CLUSTER: Target is a cluster label value.", "type": "string", "default": "THRESHOLD_SCOPE_UNSPECIFIED", "enum": [ diff --git a/api/swagger/swagger.json b/api/swagger/swagger.json index d9c754ec83..00225c6fe9 100644 --- a/api/swagger/swagger.json +++ b/api/swagger/swagger.json @@ -1843,6 +1843,11 @@ } }, "x-order": 6 + }, + "overridable": { + "description": "Whether this parameter's threshold can be overridden per target without editing the\nrule. Only set for templates that support it; the scopes it may be set at are\nreported per rule by ListThresholds.", + "type": "boolean", + "x-order": 7 } } }, @@ -2209,7 +2214,7 @@ ], "type": "string", "default": "THRESHOLD_SCOPE_UNSPECIFIED", - "description": "Scope of the target to report thresholds for. Must be set together with target.\n\n - THRESHOLD_SCOPE_NODE: Target is a Node ID.\n - THRESHOLD_SCOPE_SERVICE: Target is a Service ID.\n - THRESHOLD_SCOPE_CLUSTER: Target is a cluster label value. Unlike the others it names no inventory entity,\nso it cannot be validated for existence and is never removed by entity deletion.", + "description": "Scope of the target to report thresholds for. Must be set together with target.\n\n - THRESHOLD_SCOPE_NODE: Target is a Node ID.\n - THRESHOLD_SCOPE_SERVICE: Target is a Service ID.\n - THRESHOLD_SCOPE_CLUSTER: Target is a cluster label value.", "name": "scope", "in": "query" }, @@ -2282,7 +2287,7 @@ "x-order": 6 }, "scope": { - "description": "ThresholdScope says what a threshold override's target refers to.\n\n - THRESHOLD_SCOPE_NODE: Target is a Node ID.\n - THRESHOLD_SCOPE_SERVICE: Target is a Service ID.\n - THRESHOLD_SCOPE_CLUSTER: Target is a cluster label value. Unlike the others it names no inventory entity,\nso it cannot be validated for existence and is never removed by entity deletion.", + "description": "ThresholdScope says what a threshold override's target refers to.\n\n - THRESHOLD_SCOPE_NODE: Target is a Node ID.\n - THRESHOLD_SCOPE_SERVICE: Target is a Service ID.\n - THRESHOLD_SCOPE_CLUSTER: Target is a cluster label value.", "type": "string", "default": "THRESHOLD_SCOPE_UNSPECIFIED", "enum": [ @@ -2353,7 +2358,7 @@ "type": "object", "properties": { "scope": { - "description": "ThresholdScope says what a threshold override's target refers to.\n\n - THRESHOLD_SCOPE_NODE: Target is a Node ID.\n - THRESHOLD_SCOPE_SERVICE: Target is a Service ID.\n - THRESHOLD_SCOPE_CLUSTER: Target is a cluster label value. Unlike the others it names no inventory entity,\nso it cannot be validated for existence and is never removed by entity deletion.", + "description": "ThresholdScope says what a threshold override's target refers to.\n\n - THRESHOLD_SCOPE_NODE: Target is a Node ID.\n - THRESHOLD_SCOPE_SERVICE: Target is a Service ID.\n - THRESHOLD_SCOPE_CLUSTER: Target is a cluster label value.", "type": "string", "default": "THRESHOLD_SCOPE_UNSPECIFIED", "enum": [ @@ -2440,7 +2445,7 @@ "x-order": 6 }, "scope": { - "description": "ThresholdScope says what a threshold override's target refers to.\n\n - THRESHOLD_SCOPE_NODE: Target is a Node ID.\n - THRESHOLD_SCOPE_SERVICE: Target is a Service ID.\n - THRESHOLD_SCOPE_CLUSTER: Target is a cluster label value. Unlike the others it names no inventory entity,\nso it cannot be validated for existence and is never removed by entity deletion.", + "description": "ThresholdScope says what a threshold override's target refers to.\n\n - THRESHOLD_SCOPE_NODE: Target is a Node ID.\n - THRESHOLD_SCOPE_SERVICE: Target is a Service ID.\n - THRESHOLD_SCOPE_CLUSTER: Target is a cluster label value.", "type": "string", "default": "THRESHOLD_SCOPE_UNSPECIFIED", "enum": [ @@ -2511,7 +2516,7 @@ ], "type": "string", "default": "THRESHOLD_SCOPE_UNSPECIFIED", - "description": " - THRESHOLD_SCOPE_NODE: Target is a Node ID.\n - THRESHOLD_SCOPE_SERVICE: Target is a Service ID.\n - THRESHOLD_SCOPE_CLUSTER: Target is a cluster label value. Unlike the others it names no inventory entity,\nso it cannot be validated for existence and is never removed by entity deletion.", + "description": " - THRESHOLD_SCOPE_NODE: Target is a Node ID.\n - THRESHOLD_SCOPE_SERVICE: Target is a Service ID.\n - THRESHOLD_SCOPE_CLUSTER: Target is a cluster label value.", "name": "scope", "in": "query" }, @@ -2595,7 +2600,7 @@ "type": "object", "properties": { "scope": { - "description": "ThresholdScope says what a threshold override's target refers to.\n\n - THRESHOLD_SCOPE_NODE: Target is a Node ID.\n - THRESHOLD_SCOPE_SERVICE: Target is a Service ID.\n - THRESHOLD_SCOPE_CLUSTER: Target is a cluster label value. Unlike the others it names no inventory entity,\nso it cannot be validated for existence and is never removed by entity deletion.", + "description": "ThresholdScope says what a threshold override's target refers to.\n\n - THRESHOLD_SCOPE_NODE: Target is a Node ID.\n - THRESHOLD_SCOPE_SERVICE: Target is a Service ID.\n - THRESHOLD_SCOPE_CLUSTER: Target is a cluster label value.", "type": "string", "default": "THRESHOLD_SCOPE_UNSPECIFIED", "enum": [ @@ -2690,7 +2695,7 @@ "x-order": 6 }, "scope": { - "description": "ThresholdScope says what a threshold override's target refers to.\n\n - THRESHOLD_SCOPE_NODE: Target is a Node ID.\n - THRESHOLD_SCOPE_SERVICE: Target is a Service ID.\n - THRESHOLD_SCOPE_CLUSTER: Target is a cluster label value. Unlike the others it names no inventory entity,\nso it cannot be validated for existence and is never removed by entity deletion.", + "description": "ThresholdScope says what a threshold override's target refers to.\n\n - THRESHOLD_SCOPE_NODE: Target is a Node ID.\n - THRESHOLD_SCOPE_SERVICE: Target is a Service ID.\n - THRESHOLD_SCOPE_CLUSTER: Target is a cluster label value.", "type": "string", "default": "THRESHOLD_SCOPE_UNSPECIFIED", "enum": [ diff --git a/managed/data/alerting-templates/node_high_cpu_load.yml b/managed/data/alerting-templates/node_high_cpu_load.yml index 598b3d3a74..fb1d0d5dc7 100644 --- a/managed/data/alerting-templates/node_high_cpu_load.yml +++ b/managed/data/alerting-templates/node_high_cpu_load.yml @@ -19,6 +19,7 @@ templates: type: float range: [0, 100] value: 80 + overridable: true for: 5m severity: warning annotations: diff --git a/managed/services/alerting/service.go b/managed/services/alerting/service.go index 76a2b0e9fd..b180ad1b02 100644 --- a/managed/services/alerting/service.go +++ b/managed/services/alerting/service.go @@ -640,10 +640,11 @@ func convertParamDefinitions(l *logrus.Entry, params models.AlertExprParamsDefin res := make([]*alerting.ParamDefinition, 0, len(params)) for _, p := range params { pd := &alerting.ParamDefinition{ - Name: p.Name, - Summary: p.Summary, - Unit: convertParamUnit(p.Unit), - Type: convertParamType(p.Type), + Name: p.Name, + Summary: p.Summary, + Unit: convertParamUnit(p.Unit), + Type: convertParamType(p.Type), + Overridable: p.Overridable, } switch p.Type { diff --git a/managed/services/alerting/service_threshold_test.go b/managed/services/alerting/service_threshold_test.go index 964946992d..8571b68531 100644 --- a/managed/services/alerting/service_threshold_test.go +++ b/managed/services/alerting/service_threshold_test.go @@ -17,6 +17,9 @@ package alerting import ( "errors" + "os" + "path/filepath" + "sort" "strings" "testing" @@ -296,3 +299,43 @@ func TestCollectOverridableParams(t *testing.T) { assert.Nil(t, params) }) } + +// TestBuiltInOverridableTemplates pins exactly which shipped templates expose an +// overridable threshold. Marking one is not a free change: the injected threshold query +// joins on a single label, so a template whose observed query does not carry that label +// would generate a rule that silently never matches. +func TestBuiltInOverridableTemplates(t *testing.T) { + t.Parallel() + + // Only node scope resolves so far, so a template qualifies when it is + // multi-expression and aggregates by node_name. + want := []string{"pmm_node_high_cpu_load"} + + files, err := filepath.Glob(filepath.Join("..", "..", "data", "alerting-templates", "*.yml")) + require.NoError(t, err) + require.NotEmpty(t, files) + + var got []string + + for _, file := range files { + b, err := os.ReadFile(file) //nolint:gosec + require.NoError(t, err) + + templates, err := alert.Parse(strings.NewReader(string(b)), &alert.ParseParams{ + DisallowUnknownFields: true, + DisallowInvalidTemplates: true, + }) + require.NoError(t, err, "built-in template %s must parse", filepath.Base(file)) + + for _, template := range templates { + if len(template.OverridableParams()) != 0 { + got = append(got, template.Name) + } + } + } + + sort.Strings(got) + assert.Equal(t, want, got, + "adding a template here needs a join label the threshold query can match on; "+ + "templates that aggregate by (cluster) and drop node_name must not be marked overridable") +} From 9c7482601238ed24879e26d14bd1c85be856058a Mon Sep 17 00:00:00 2001 From: Matej Kubinec Date: Fri, 28 Aug 2026 16:11:20 +0200 Subject: [PATCH 08/15] PMM-14912 Add alert thresholds UI Add the Inventory modal for viewing and editing per-node alert thresholds, wired to the generic thresholds API. Scope and target travel as fields rather than path segments, matching the backend: a cluster target is an arbitrary label value, so it cannot be carried safely in a URL path. This screen only ever addresses nodes, so it pins the scope and leaves the rest of the API's reach unused. The form submits as one batch call rather than a request per row. Firing N requests would leave the modal half-applied after a partial failure, with no way to report which rows took effect; the batch endpoint applies every change in one transaction. Emptying a field, or typing the default back in, clears the override rather than writing the default as a new one, so the target falls back to the rule default or to a broader override still covering it. Rule titles are joined from the Grafana rules API on the pmm_rule_id label. They are deliberately absent from the thresholds response: rule metadata churns on rename, so Grafana stays authoritative for it. Rows whose rule has since been deleted keep an empty title rather than disappearing. Two details that are easy to get wrong and would fail quietly: - Numeric and boolean fields are optional because proto3 JSON omits zero values. A threshold of 0, or a row that is not overridden, arrives with the field absent rather than 0/false, so the row mapping coerces it. - A row is identified by rule, parameter and index. Two rules duplicated in Grafana share a rule id, which the API explicitly permits, so the first two parts alone would collapse two legitimate rows into one. The modal's message listener re-subscribes on every render, which is documented in place: GrafanaProvider's cleanup empties the shared listener array rather than removing only its own, so an ordinary navigation discards this subscription. Making the messenger's register/unregister symmetric is tracked separately. Content strings live in AlertThresholds.messages per convention. The reset button gains an accessible name, which it previously lacked. Co-Authored-By: Claude Opus 5 (1M context) --- ui/apps/pmm-compat/src/compat.ts | 8 + ui/apps/pmm-compat/src/lib/events.ts | 4 + ui/apps/pmm/src/api/alerting.ts | 37 +++- .../AlertThresholds.constants.tsx | 85 ++++++++ .../AlertThresholds.messages.ts | 22 ++ .../alert-thresholds/AlertThresholds.tsx | 205 ++++++++++++++++++ .../alert-thresholds/AlertThresholds.types.ts | 16 ++ .../alert-thresholds/AlertThresholds.utils.ts | 48 ++++ .../src/components/alert-thresholds/index.ts | 1 + .../reset-value-cell/ResetValueCell.tsx | 28 +++ .../reset-value-cell/index.ts | 1 + .../pmm/src/components/main/MainWithNav.tsx | 2 + ui/apps/pmm/src/components/modal/Modal.tsx | 6 +- .../pmm/src/hooks/api/useNodeThresholds.ts | 54 +++++ ui/apps/pmm/src/types/alerting.types.ts | 47 ++++ ui/packages/shared/src/messenger.ts | 1 + ui/packages/shared/src/types.ts | 8 +- 17 files changed, 570 insertions(+), 3 deletions(-) create mode 100644 ui/apps/pmm/src/components/alert-thresholds/AlertThresholds.constants.tsx create mode 100644 ui/apps/pmm/src/components/alert-thresholds/AlertThresholds.messages.ts create mode 100644 ui/apps/pmm/src/components/alert-thresholds/AlertThresholds.tsx create mode 100644 ui/apps/pmm/src/components/alert-thresholds/AlertThresholds.types.ts create mode 100644 ui/apps/pmm/src/components/alert-thresholds/AlertThresholds.utils.ts create mode 100644 ui/apps/pmm/src/components/alert-thresholds/index.ts create mode 100644 ui/apps/pmm/src/components/alert-thresholds/reset-value-cell/ResetValueCell.tsx create mode 100644 ui/apps/pmm/src/components/alert-thresholds/reset-value-cell/index.ts create mode 100644 ui/apps/pmm/src/hooks/api/useNodeThresholds.ts diff --git a/ui/apps/pmm-compat/src/compat.ts b/ui/apps/pmm-compat/src/compat.ts index ef18ae9b45..d40048894a 100644 --- a/ui/apps/pmm-compat/src/compat.ts +++ b/ui/apps/pmm-compat/src/compat.ts @@ -41,6 +41,7 @@ import { SettingsUpdatedEvent, FrontendSettingsUpdatedEvent, TimeZoneUpdatedEvent, + OpenAlertThresholdsModalEvent, } from 'lib/events'; import { handleExternalLinks } from 'compat/links'; @@ -228,6 +229,13 @@ export const initialize = () => { }); }); + getAppEvents().subscribe(OpenAlertThresholdsModalEvent, (e) => + messenger.sendMessage({ + type: 'OPEN_ALERT_THRESHOLDS_MODAL', + payload: e.payload, + }) + ); + getAppEvents().subscribe(ServiceDeletedEvent, () => { messenger.sendMessage({ type: 'SERVICE_DELETED', diff --git a/ui/apps/pmm-compat/src/lib/events.ts b/ui/apps/pmm-compat/src/lib/events.ts index 44461b0e15..8ba01e2315 100644 --- a/ui/apps/pmm-compat/src/lib/events.ts +++ b/ui/apps/pmm-compat/src/lib/events.ts @@ -24,3 +24,7 @@ export class FrontendSettingsUpdatedEvent extends BusEventBase { export class TimeZoneUpdatedEvent extends BusEventBase { static type = 'timezone-updated-event'; } + +export class OpenAlertThresholdsModalEvent extends BusEventBase { + static type = 'open-alert-thresholds-modal-event'; +} diff --git a/ui/apps/pmm/src/api/alerting.ts b/ui/apps/pmm/src/api/alerting.ts index 2cdfa28b9b..3f961c1833 100644 --- a/ui/apps/pmm/src/api/alerting.ts +++ b/ui/apps/pmm/src/api/alerting.ts @@ -4,9 +4,13 @@ import { AlertmanagerSilence, GrafanaAlertQuery, GrafanaRulerRuleDTO, + BatchUpdateThresholdsResponse, + ListThresholdsResponse, PrometheusAlertRulesResponse, + ThresholdScope, + ThresholdUpdate, } from 'types/alerting.types'; -import { grafanaApi } from './api'; +import { api, grafanaApi } from './api'; export const getPrometheusAlertRules = async () => { const response = await grafanaApi.get( @@ -55,3 +59,34 @@ export const getRulerRule = async (uid: string) => { ); return res.data; }; + +export const getThresholds = async ( + scope: ThresholdScope, + target: string, + ruleId?: string +) => { + const res = await api.get('alerting/thresholds', { + params: { scope, target, ruleId }, + }); + return res.data; +}; + +export const setThreshold = async (update: Required) => { + const res = await api.post('alerting/thresholds', update); + return res.data; +}; + +export const clearThreshold = async ( + update: Omit +) => { + const res = await api.delete('alerting/thresholds', { params: update }); + return res.data; +}; + +export const batchUpdateThresholds = async (updates: ThresholdUpdate[]) => { + const res = await api.post( + 'alerting/thresholds:batchUpdate', + { updates } + ); + return res.data; +}; diff --git a/ui/apps/pmm/src/components/alert-thresholds/AlertThresholds.constants.tsx b/ui/apps/pmm/src/components/alert-thresholds/AlertThresholds.constants.tsx new file mode 100644 index 0000000000..ff0b7a6971 --- /dev/null +++ b/ui/apps/pmm/src/components/alert-thresholds/AlertThresholds.constants.tsx @@ -0,0 +1,85 @@ +import { TextInput } from '@percona/peak-ui'; +import type { MRT_ColumnDef } from '@percona/peak-ui'; +import type { AlertThresholdRow } from './AlertThresholds.types'; +import ResetValueCell from './reset-value-cell'; +import { Messages } from './AlertThresholds.messages'; +import { formatUnit } from './AlertThresholds.utils'; + +// Maps the backend ParamUnit enum to a display symbol. +export const UNIT_SYMBOLS: Record = { + PARAM_UNIT_PERCENTAGE: '%', + PARAM_UNIT_SECONDS: 's', +}; + +export const ALERT_THRESHOLDS_COLUMNS: MRT_ColumnDef[] = [ + { + accessorKey: 'ruleTitle', + header: Messages.table.columns.ruleTitle, + }, + { + accessorKey: 'summary', + header: Messages.table.columns.parameter, + Cell: ({ row: { original } }) => original.paramName, + }, + { + accessorKey: 'defaultValue', + header: Messages.table.columns.default, + enableColumnActions: false, + enableColumnFilter: false, + enableSorting: false, + muiTableHeadCellProps: { + sx: { + '.Mui-TableHeadCell-Content': { + height: 40, + }, + }, + }, + }, + { + accessorKey: 'effectiveValue', + header: Messages.table.columns.override, + enableColumnActions: false, + enableColumnFilter: false, + enableSorting: false, + muiTableHeadCellProps: { + sx: { + '.Mui-TableHeadCell-Content': { + height: 40, + }, + }, + }, + // Every row returned by the endpoint is overridable; the field is + // pre-filled with the effective value via react-hook-form defaults. + Cell: ({ row: { original } }) => ( + + ), + }, + { + id: 'unit', + size: 80, + grow: false, + header: Messages.table.columns.unit, + enableColumnActions: false, + muiTableHeadCellProps: { + sx: { + '.Mui-TableHeadCell-Content': { + height: 40, + }, + }, + }, + Cell: ({ row: { original } }) => formatUnit(original.unit), + }, + { + id: 'reset', + size: 80, + header: '', + enableColumnActions: false, + Cell: ({ row: { original } }) => , + }, +]; diff --git a/ui/apps/pmm/src/components/alert-thresholds/AlertThresholds.messages.ts b/ui/apps/pmm/src/components/alert-thresholds/AlertThresholds.messages.ts new file mode 100644 index 0000000000..a252eca535 --- /dev/null +++ b/ui/apps/pmm/src/components/alert-thresholds/AlertThresholds.messages.ts @@ -0,0 +1,22 @@ +export const Messages = { + title: (nodeName: string) => `Alert thresholds: ${nodeName}`, + loading: "Loading thresholds…", + empty: "No overridable thresholds for this node.", + actions: { + cancel: "Cancel and close", + submit: "Submit changes", + reset: "Reset to default", + }, + table: { + columns: { + ruleTitle: "Alert rule", + parameter: "Parameter", + default: "Default", + override: "Override", + unit: "Unit", + }, + }, + success: { + updated: "Alert thresholds updated", + }, +}; diff --git a/ui/apps/pmm/src/components/alert-thresholds/AlertThresholds.tsx b/ui/apps/pmm/src/components/alert-thresholds/AlertThresholds.tsx new file mode 100644 index 0000000000..886dfb4ab9 --- /dev/null +++ b/ui/apps/pmm/src/components/alert-thresholds/AlertThresholds.tsx @@ -0,0 +1,205 @@ +import Button from '@mui/material/Button'; +import Stack from '@mui/material/Stack'; +import Typography from '@mui/material/Typography'; +import { Table } from '@percona/peak-ui'; +import type { OpenAlertThresholdsModalMessage } from '@pmm/shared'; +import { Modal } from 'components/modal'; +import { + NODE_SCOPE, + useBatchUpdateNodeThresholds, + useNodeThresholds, +} from 'hooks/api/useNodeThresholds'; +import { usePrometheusAlertRules } from 'hooks/api/usePrometheusAlertRules'; +import messenger from 'lib/messenger'; +import { enqueueSnackbar } from 'notistack'; +import { useEffect, useMemo, useState } from 'react'; +import { FormProvider, useForm } from 'react-hook-form'; +import { ALERT_THRESHOLDS_COLUMNS } from './AlertThresholds.constants'; +import { Messages } from './AlertThresholds.messages'; +import type { + AlertThresholdRow, + AlertThresholdsFormValues, +} from './AlertThresholds.types'; +import type { + ListThresholdsResponse, + PrometheusAlertRulesResponse, + ThresholdUpdate, +} from 'types/alerting.types'; +import { getRows, getRuleTitles } from './AlertThresholds.utils'; + +const AlertThresholds = () => { + const [nodeId, setNodeId] = useState(); + const [nodeName, setNodeName] = useState(); + const [open, setIsOpen] = useState(false); + + const { data, isLoading } = useNodeThresholds(nodeId ?? '', { + enabled: open && !!nodeId, + }); + + const { data: rulesData } = usePrometheusAlertRules({ + enabled: open && !!nodeId, + }); + + // Rule titles live in Grafana, not in the thresholds response, so they are joined + // on the identity label PMM stamps on every rule it creates. + const ruleTitles = useMemo( + () => getRuleTitles(rulesData as PrometheusAlertRulesResponse), + [rulesData] + ); + + const rows = useMemo( + () => getRows(data as ListThresholdsResponse, ruleTitles), + [data, ruleTitles] + ); + + const initialValues = useMemo( + () => + rows.reduce((acc, row) => { + acc[row.id] = row.effectiveValue; + return acc; + }, {} as AlertThresholdsFormValues), + [rows] + ); + + const methods = useForm({ + defaultValues: initialValues, + }); + const { mutateAsync: applyThresholds } = useBatchUpdateNodeThresholds( + nodeId ?? '' + ); + + useEffect(() => { + methods.reset(initialValues); + }, [initialValues, methods]); + + // Deliberately has no dependency array, so it re-subscribes after every render. + // + // GrafanaProvider's cleanup calls messenger.unregister(), which empties the shared + // listener array rather than removing only its own listeners, and its effect depends + // on `navigate` - so an ordinary navigation discards this subscription along with + // everyone else's. Re-registering on each render is what puts it back; with `[]` the + // modal would work until the first navigation and then silently stop opening. + // + // The costs are a subscribe/unsubscribe cycle per render, and a narrow window between + // cleanup and re-subscribe in which an OPEN_ALERT_THRESHOLDS_MODAL message would be + // dropped. Both go away once unregister() is made the true inverse of register() - + // detaching the window listener and leaving the array to each component's own cleanup. + // Tracked as a follow-up; fixing it here would mean changing shared messenger + // behaviour that every other consumer relies on. + useEffect(() => { + const handler = messenger.addListener({ + type: 'OPEN_ALERT_THRESHOLDS_MODAL', + onMessage: (msg: OpenAlertThresholdsModalMessage) => { + setNodeId(msg.payload?.nodeId); + setNodeName(msg.payload?.nodeName); + setIsOpen(true); + }, + }); + + return () => messenger.removeListener(handler); + }); + + const handleClose = () => { + setNodeId(undefined); + setNodeName(undefined); + setIsOpen(false); + }; + + const handleSubmit = async (values: AlertThresholdsFormValues) => { + const updates: ThresholdUpdate[] = []; + + for (const row of rows) { + const raw = values[row.id]; + const parsed = + raw === undefined || (raw as unknown) === '' ? undefined : Number(raw); + const cleared = parsed === undefined || Number.isNaN(parsed); + + const base = { + scope: NODE_SCOPE, + target: nodeId ?? '', + ruleId: row.ruleId, + paramName: row.paramName, + }; + + // Emptying the field, or typing the default back in, returns the node to the + // rule default. That is only a change if an override exists today; omitting + // the value clears it rather than writing the default as a new override. + if (cleared || parsed === row.defaultValue) { + if (row.isOverridden) { + updates.push(base); + } + continue; + } + + if (parsed !== row.effectiveValue) { + updates.push({ ...base, value: parsed }); + } + } + + if (updates.length > 0) { + // One transactional call: either every row lands or none does. + await applyThresholds(updates); + enqueueSnackbar(Messages.success.updated, { variant: 'success' }); + } + + handleClose(); + }; + + if (!open || !nodeId) { + return null; + } + + return ( + + + + {isLoading ? ( + + {Messages.loading} + + ) : rows.length === 0 ? ( + + {Messages.empty} + + ) : ( + + )} + + + + + + + + ); +}; + +export default AlertThresholds; diff --git a/ui/apps/pmm/src/components/alert-thresholds/AlertThresholds.types.ts b/ui/apps/pmm/src/components/alert-thresholds/AlertThresholds.types.ts new file mode 100644 index 0000000000..e4736a3491 --- /dev/null +++ b/ui/apps/pmm/src/components/alert-thresholds/AlertThresholds.types.ts @@ -0,0 +1,16 @@ +import type { Threshold } from 'types/alerting.types'; + +// A table row is a Threshold plus a stable composite id used as the react-hook-form +// field name, and the rule's title. +// +// The title is not part of the thresholds response: rule metadata churns on rename, +// so Grafana stays authoritative for it and the UI joins it on the `pmm_rule_id` +// label. Rows whose rule has since been deleted keep an empty title rather than +// disappearing. +export interface AlertThresholdRow extends Threshold { + id: string; + ruleTitle: string; +} + +// Form values: composite row id -> override value (string while editing). +export type AlertThresholdsFormValues = Record; diff --git a/ui/apps/pmm/src/components/alert-thresholds/AlertThresholds.utils.ts b/ui/apps/pmm/src/components/alert-thresholds/AlertThresholds.utils.ts new file mode 100644 index 0000000000..4abb39aa0f --- /dev/null +++ b/ui/apps/pmm/src/components/alert-thresholds/AlertThresholds.utils.ts @@ -0,0 +1,48 @@ +import type { + ListThresholdsResponse, + PrometheusAlertRulesResponse, + Threshold, +} from 'types/alerting.types'; +import { UNIT_SYMBOLS } from './AlertThresholds.constants'; + +export const formatUnit = (unit?: string): string => + (unit && UNIT_SYMBOLS[unit]) || ''; + +// A rule can expose several overridable params, and two rules duplicated in Grafana +// share a rule id, so neither part alone identifies a row. The index disambiguates +// the duplicate case, which the API explicitly permits. +export const thresholdRowId = (t: Threshold, index: number): string => + `${t.ruleId}:${t.paramName}:${index}`; + +// Rule titles live in Grafana, not in the thresholds response, so they are joined +// on the identity label PMM stamps on every rule it creates. +export const getRuleTitles = ( + rulesData: PrometheusAlertRulesResponse +): Map => { + const titles = new Map(); + + for (const group of rulesData?.data?.groups ?? []) { + for (const rule of group.rules ?? []) { + const id = rule.labels?.pmm_rule_id; + if (id) { + titles.set(id, rule.name); + } + } + } + + return titles; +}; + +export const getRows = ( + data: ListThresholdsResponse | undefined, + ruleTitles: Map +) => + (data?.thresholds ?? []).map((t, index) => ({ + ...t, + // proto3 omits zero values, so absent means 0 rather than unknown. + defaultValue: t.defaultValue ?? 0, + effectiveValue: t.effectiveValue ?? 0, + isOverridden: t.isOverridden ?? false, + id: thresholdRowId(t, index), + ruleTitle: ruleTitles.get(t.ruleId) ?? '', + })); diff --git a/ui/apps/pmm/src/components/alert-thresholds/index.ts b/ui/apps/pmm/src/components/alert-thresholds/index.ts new file mode 100644 index 0000000000..c786945f8f --- /dev/null +++ b/ui/apps/pmm/src/components/alert-thresholds/index.ts @@ -0,0 +1 @@ +export { default } from './AlertThresholds'; diff --git a/ui/apps/pmm/src/components/alert-thresholds/reset-value-cell/ResetValueCell.tsx b/ui/apps/pmm/src/components/alert-thresholds/reset-value-cell/ResetValueCell.tsx new file mode 100644 index 0000000000..fe64741811 --- /dev/null +++ b/ui/apps/pmm/src/components/alert-thresholds/reset-value-cell/ResetValueCell.tsx @@ -0,0 +1,28 @@ +import type { FC } from 'react'; +import type { + AlertThresholdRow, + AlertThresholdsFormValues, +} from '../AlertThresholds.types'; +import IconButton from '@mui/material/IconButton'; +import RestartAltIcon from '@mui/icons-material/RestartAlt'; +import { useFormContext } from 'react-hook-form'; +import { Messages } from '../AlertThresholds.messages'; + +interface Props { + row: AlertThresholdRow; +} + +const ResetValueCell: FC = ({ row }) => { + const { setValue } = useFormContext(); + + return ( + setValue(row.id, row.defaultValue)} + > + + + ); +}; + +export default ResetValueCell; diff --git a/ui/apps/pmm/src/components/alert-thresholds/reset-value-cell/index.ts b/ui/apps/pmm/src/components/alert-thresholds/reset-value-cell/index.ts new file mode 100644 index 0000000000..11ae2360c2 --- /dev/null +++ b/ui/apps/pmm/src/components/alert-thresholds/reset-value-cell/index.ts @@ -0,0 +1 @@ +export { default } from './ResetValueCell'; diff --git a/ui/apps/pmm/src/components/main/MainWithNav.tsx b/ui/apps/pmm/src/components/main/MainWithNav.tsx index 734c153608..73bf1c6fe8 100644 --- a/ui/apps/pmm/src/components/main/MainWithNav.tsx +++ b/ui/apps/pmm/src/components/main/MainWithNav.tsx @@ -11,6 +11,7 @@ import { DelayedRender } from 'components/delayed-render'; import { SHOW_UPDATE_INFO_DELAY_MS } from 'lib/constants'; import { isRenderingServer } from '@pmm/shared'; import Header from './header/Header'; +import AlertThresholds from 'components/alert-thresholds'; const useMainNavVisible = () => { const { isLoggedIn } = useAuth(); @@ -52,6 +53,7 @@ export const MainWithNav = () => { + ); }; diff --git a/ui/apps/pmm/src/components/modal/Modal.tsx b/ui/apps/pmm/src/components/modal/Modal.tsx index 9be6fbc6ab..c4a50ea08a 100644 --- a/ui/apps/pmm/src/components/modal/Modal.tsx +++ b/ui/apps/pmm/src/components/modal/Modal.tsx @@ -38,7 +38,11 @@ export const Modal: FC = ({ pb: 0, }} > - + {title} diff --git a/ui/apps/pmm/src/hooks/api/useNodeThresholds.ts b/ui/apps/pmm/src/hooks/api/useNodeThresholds.ts new file mode 100644 index 0000000000..bf4421cfb1 --- /dev/null +++ b/ui/apps/pmm/src/hooks/api/useNodeThresholds.ts @@ -0,0 +1,54 @@ +import { useMutation, useQuery, useQueryClient } from "@tanstack/react-query"; +import type { + UseMutationOptions, + UseQueryOptions, +} from "@tanstack/react-query"; +import { batchUpdateThresholds, getThresholds } from "api/alerting"; +import type { + BatchUpdateThresholdsResponse, + ListThresholdsResponse, + ThresholdUpdate, +} from "types/alerting.types"; + +export const nodeThresholdsQueryKey = (nodeId: string) => [ + "alerting:nodeThresholds", + nodeId, +]; + +// Asking for a target returns every overridable parameter for it, overridden or not, +// which is what the modal lists. Asking without one would return only existing +// overrides. +export const useNodeThresholds = ( + nodeId: string, + options?: Partial>, +) => + useQuery({ + queryKey: nodeThresholdsQueryKey(nodeId), + queryFn: () => getThresholds("THRESHOLD_SCOPE_NODE", nodeId), + enabled: !!nodeId, + ...options, + }); + +// One transactional call for a whole form's worth of edits. Firing a request per row +// would leave the modal half-applied on a partial failure, with no way to report +// which rows took effect. +export const useBatchUpdateNodeThresholds = ( + nodeId: string, + options?: Partial< + UseMutationOptions + >, +) => { + const queryClient = useQueryClient(); + + return useMutation({ + mutationKey: ["alerting:batchUpdateNodeThresholds", nodeId], + mutationFn: (updates: ThresholdUpdate[]) => batchUpdateThresholds(updates), + ...options, + onSuccess: async (data, variables, onMutate, context) => { + await options?.onSuccess?.(data, variables, onMutate, context); + await queryClient.invalidateQueries({ + queryKey: nodeThresholdsQueryKey(nodeId), + }); + }, + }); +}; diff --git a/ui/apps/pmm/src/types/alerting.types.ts b/ui/apps/pmm/src/types/alerting.types.ts index b9d5afdff9..e42dfb826d 100644 --- a/ui/apps/pmm/src/types/alerting.types.ts +++ b/ui/apps/pmm/src/types/alerting.types.ts @@ -228,3 +228,50 @@ export interface GrafanaRulerRuleDTO { annotations?: GrafanaRulerAnnotations; labels?: GrafanaRulerLabels; } + +// Dynamic per-target alert thresholds (PMM backend /v1/alerting/thresholds API). +// Field names are camelCase because the shared `api` client applies +// axios-case-converter to the snake_case wire format. +export type ThresholdScope = + | 'THRESHOLD_SCOPE_UNSPECIFIED' + | 'THRESHOLD_SCOPE_NODE' + | 'THRESHOLD_SCOPE_SERVICE' + | 'THRESHOLD_SCOPE_CLUSTER'; + +// One overridable parameter of one rule, as it applies to one target. +// +// Numeric and boolean fields are optional because proto3 JSON omits zero values: +// a threshold of 0, or a row that is not overridden, arrives with the field absent +// rather than set to 0/false. +export interface Threshold { + ruleId: string; + paramName: string; + summary?: string; + // ParamUnit enum string, e.g. "PARAM_UNIT_PERCENTAGE". + unit?: string; + defaultValue?: number; + // Effective value for the target: the winning override, otherwise the default. + effectiveValue?: number; + isOverridden?: boolean; + // Scope and target the winning override was set at; absent when not overridden. + scope?: ThresholdScope; + target?: string; +} + +export interface ListThresholdsResponse { + thresholds?: Threshold[]; +} + +// One set-or-clear operation. Omitting `value` clears the override instead of +// setting it, returning the target to the rule default or to a broader override. +export interface ThresholdUpdate { + scope: ThresholdScope; + target: string; + ruleId: string; + paramName: string; + value?: number; +} + +export interface BatchUpdateThresholdsResponse { + thresholds?: Threshold[]; +} diff --git a/ui/packages/shared/src/messenger.ts b/ui/packages/shared/src/messenger.ts index 57c4b7afd8..a20c1f8433 100644 --- a/ui/packages/shared/src/messenger.ts +++ b/ui/packages/shared/src/messenger.ts @@ -44,6 +44,7 @@ export class CrossFrameMessenger { addListener(listener: MessageListener) { this.listeners.push(listener); + return listener; } removeListener(listener: MessageListener) { diff --git a/ui/packages/shared/src/types.ts b/ui/packages/shared/src/types.ts index ce3a9e9c9a..c57fc84bbd 100644 --- a/ui/packages/shared/src/types.ts +++ b/ui/packages/shared/src/types.ts @@ -14,7 +14,8 @@ export type MessageType = | 'FRONTEND_SETTINGS_CHANGED' | 'SERVICE_ADDED' | 'SERVICE_DELETED' - | 'TIMEZONE_CHANGED'; + | 'TIMEZONE_CHANGED' + | 'OPEN_ALERT_THRESHOLDS_MODAL'; export type LocationState = { fromGrafana?: boolean } | null; @@ -67,3 +68,8 @@ export type FrontendSettingsChangedMessage = Message<'FRONTEND_SETTINGS_CHANGED'>; export type ServiceAddedMessage = Message<'SERVICE_ADDED'>; + +export type OpenAlertThresholdsModalMessage = Message< + 'OPEN_ALERT_THRESHOLDS_MODAL', + { nodeId: string; nodeName: string } +>; From 601dd7543e41a7c83008f0a768302ebe71b726f1 Mon Sep 17 00:00:00 2001 From: Matej Kubinec Date: Tue, 1 Sep 2026 10:52:54 +0200 Subject: [PATCH 09/15] PMM-14912 Extract and test threshold form logic Move the submit diff out of the component and cover it, along with the row mapping, with unit tests. Deciding whether a row is a set, a clear or no change was inline in AlertThresholds, reachable only by rendering the modal. buildThresholdUpdates makes it a pure function alongside the existing row helpers, so the part most likely to be wrong is also the part that can be verified without a browser. The tests target the failure modes that would be silent rather than loud: - proto3 JSON omits zero values, so an absent default_value means 0 and not "unknown". Left uncoerced the table renders blanks where numbers belong. - Two rules duplicated in Grafana share a rule id, which the API explicitly permits, so rule and parameter together do not identify a row. Colliding ids would let one form field drive two rows. - Clearing omits `value` rather than sending the default. Sending it would pin the target to today's default and stop it following a later change to the rule. - An emptied input arrives as '' and must read as a clear, not as 0. - A title for a rule that no longer exists resolves to empty rather than breaking the row. Also normalises quote style in two files carried over from the previous commit, which make format rewrites and format-check would otherwise reject. Co-Authored-By: Claude Opus 5 (1M context) --- .../AlertThresholds.messages.ts | 22 +- .../alert-thresholds/AlertThresholds.tsx | 43 +--- .../AlertThresholds.utils.test.ts | 205 ++++++++++++++++++ .../alert-thresholds/AlertThresholds.utils.ts | 47 ++++ .../pmm/src/hooks/api/useNodeThresholds.ts | 18 +- 5 files changed, 283 insertions(+), 52 deletions(-) create mode 100644 ui/apps/pmm/src/components/alert-thresholds/AlertThresholds.utils.test.ts diff --git a/ui/apps/pmm/src/components/alert-thresholds/AlertThresholds.messages.ts b/ui/apps/pmm/src/components/alert-thresholds/AlertThresholds.messages.ts index a252eca535..75f44bc038 100644 --- a/ui/apps/pmm/src/components/alert-thresholds/AlertThresholds.messages.ts +++ b/ui/apps/pmm/src/components/alert-thresholds/AlertThresholds.messages.ts @@ -1,22 +1,22 @@ export const Messages = { title: (nodeName: string) => `Alert thresholds: ${nodeName}`, - loading: "Loading thresholds…", - empty: "No overridable thresholds for this node.", + loading: 'Loading thresholds…', + empty: 'No overridable thresholds for this node.', actions: { - cancel: "Cancel and close", - submit: "Submit changes", - reset: "Reset to default", + cancel: 'Cancel and close', + submit: 'Submit changes', + reset: 'Reset to default', }, table: { columns: { - ruleTitle: "Alert rule", - parameter: "Parameter", - default: "Default", - override: "Override", - unit: "Unit", + ruleTitle: 'Alert rule', + parameter: 'Parameter', + default: 'Default', + override: 'Override', + unit: 'Unit', }, }, success: { - updated: "Alert thresholds updated", + updated: 'Alert thresholds updated', }, }; diff --git a/ui/apps/pmm/src/components/alert-thresholds/AlertThresholds.tsx b/ui/apps/pmm/src/components/alert-thresholds/AlertThresholds.tsx index 886dfb4ab9..8535cd6e25 100644 --- a/ui/apps/pmm/src/components/alert-thresholds/AlertThresholds.tsx +++ b/ui/apps/pmm/src/components/alert-thresholds/AlertThresholds.tsx @@ -5,7 +5,6 @@ import { Table } from '@percona/peak-ui'; import type { OpenAlertThresholdsModalMessage } from '@pmm/shared'; import { Modal } from 'components/modal'; import { - NODE_SCOPE, useBatchUpdateNodeThresholds, useNodeThresholds, } from 'hooks/api/useNodeThresholds'; @@ -23,9 +22,12 @@ import type { import type { ListThresholdsResponse, PrometheusAlertRulesResponse, - ThresholdUpdate, } from 'types/alerting.types'; -import { getRows, getRuleTitles } from './AlertThresholds.utils'; +import { + buildThresholdUpdates, + getRows, + getRuleTitles, +} from './AlertThresholds.utils'; const AlertThresholds = () => { const [nodeId, setNodeId] = useState(); @@ -106,35 +108,12 @@ const AlertThresholds = () => { }; const handleSubmit = async (values: AlertThresholdsFormValues) => { - const updates: ThresholdUpdate[] = []; - - for (const row of rows) { - const raw = values[row.id]; - const parsed = - raw === undefined || (raw as unknown) === '' ? undefined : Number(raw); - const cleared = parsed === undefined || Number.isNaN(parsed); - - const base = { - scope: NODE_SCOPE, - target: nodeId ?? '', - ruleId: row.ruleId, - paramName: row.paramName, - }; - - // Emptying the field, or typing the default back in, returns the node to the - // rule default. That is only a change if an override exists today; omitting - // the value clears it rather than writing the default as a new override. - if (cleared || parsed === row.defaultValue) { - if (row.isOverridden) { - updates.push(base); - } - continue; - } - - if (parsed !== row.effectiveValue) { - updates.push({ ...base, value: parsed }); - } - } + const updates = buildThresholdUpdates( + rows, + values, + 'THRESHOLD_SCOPE_NODE', + nodeId ?? '' + ); if (updates.length > 0) { // One transactional call: either every row lands or none does. diff --git a/ui/apps/pmm/src/components/alert-thresholds/AlertThresholds.utils.test.ts b/ui/apps/pmm/src/components/alert-thresholds/AlertThresholds.utils.test.ts new file mode 100644 index 0000000000..9f11adf77e --- /dev/null +++ b/ui/apps/pmm/src/components/alert-thresholds/AlertThresholds.utils.test.ts @@ -0,0 +1,205 @@ +import type { + ListThresholdsResponse, + PrometheusAlertRulesResponse, +} from 'types/alerting.types'; +import type { AlertThresholdRow } from './AlertThresholds.types'; +import { + buildThresholdUpdates, + getRows, + getRuleTitles, +} from './AlertThresholds.utils'; + +const NODE = 'THRESHOLD_SCOPE_NODE' as const; + +const rulesResponse = ( + rules: { name: string; labels?: Record }[] +): PrometheusAlertRulesResponse => + ({ + data: { groups: [{ rules }] }, + }) as PrometheusAlertRulesResponse; + +const row = (over: Partial = {}): AlertThresholdRow => ({ + id: 'rule-1:threshold:0', + ruleId: 'rule-1', + ruleTitle: 'CPU load', + paramName: 'threshold', + defaultValue: 80, + effectiveValue: 80, + isOverridden: false, + ...over, +}); + +describe('getRuleTitles', () => { + it('indexes rule names by the identity label PMM stamps on them', () => { + const titles = getRuleTitles( + rulesResponse([ + { name: 'CPU load', labels: { pmm_rule_id: 'rule-1' } }, + { name: 'Connections', labels: { pmm_rule_id: 'rule-2' } }, + ]) + ); + + expect(titles.get('rule-1')).toBe('CPU load'); + expect(titles.get('rule-2')).toBe('Connections'); + }); + + it('ignores rules that PMM did not create', () => { + const titles = getRuleTitles( + rulesResponse([{ name: 'Someone else rule' }]) + ); + + expect(titles.size).toBe(0); + }); +}); + +describe('getRows', () => { + // proto3 JSON omits zero values, so a threshold of 0 and a row that is not + // overridden both arrive with the field absent rather than 0/false. Left uncoerced + // the table would render blanks instead of numbers. + it('reads omitted numeric fields as zero rather than undefined', () => { + const data = { + thresholds: [{ ruleId: 'rule-1', paramName: 'threshold' }], + } as ListThresholdsResponse; + + const [first] = getRows(data, new Map()); + + expect(first.defaultValue).toBe(0); + expect(first.effectiveValue).toBe(0); + expect(first.isOverridden).toBe(false); + }); + + it('joins the rule title, and tolerates a rule that no longer exists', () => { + const data = { + thresholds: [ + { ruleId: 'rule-1', paramName: 'threshold' }, + { ruleId: 'deleted-rule', paramName: 'threshold' }, + ], + } as ListThresholdsResponse; + + const rows = getRows(data, new Map([['rule-1', 'CPU load']])); + + expect(rows[0].ruleTitle).toBe('CPU load'); + expect(rows[1].ruleTitle).toBe(''); + }); + + // Two rules duplicated in Grafana share a rule id, which the API explicitly + // permits, so rule and parameter together do not identify a row. Colliding ids + // would make one form field drive two rows. + it('gives duplicated rules distinct row ids', () => { + const data = { + thresholds: [ + { ruleId: 'rule-1', paramName: 'threshold' }, + { ruleId: 'rule-1', paramName: 'threshold' }, + ], + } as ListThresholdsResponse; + + const rows = getRows(data, new Map()); + + expect(rows[0].id).not.toBe(rows[1].id); + }); + + it('returns nothing when the response carries no thresholds', () => { + expect(getRows(undefined, new Map())).toEqual([]); + }); +}); + +describe('buildThresholdUpdates', () => { + it('sets a changed value', () => { + const rows = [row()]; + + expect( + buildThresholdUpdates(rows, { [rows[0].id]: 95 }, NODE, 'node-1') + ).toEqual([ + { + scope: NODE, + target: 'node-1', + ruleId: 'rule-1', + paramName: 'threshold', + value: 95, + }, + ]); + }); + + it('sends nothing when the value is unchanged', () => { + const rows = [row({ isOverridden: true, effectiveValue: 95 })]; + + expect( + buildThresholdUpdates(rows, { [rows[0].id]: 95 }, NODE, 'node-1') + ).toEqual([]); + }); + + // Omitting `value` clears the override. Writing the default as an override instead + // would pin the target to today's default and stop it following a later change to + // the rule. + it('clears by omitting the value when the field is emptied', () => { + const rows = [row({ isOverridden: true, effectiveValue: 95 })]; + + const updates = buildThresholdUpdates( + rows, + { [rows[0].id]: undefined }, + NODE, + 'node-1' + ); + + expect(updates).toHaveLength(1); + expect(updates[0]).not.toHaveProperty('value'); + }); + + it('clears when the default is typed back in', () => { + const rows = [row({ isOverridden: true, effectiveValue: 95 })]; + + const updates = buildThresholdUpdates( + rows, + { [rows[0].id]: 80 }, + NODE, + 'node-1' + ); + + expect(updates).toHaveLength(1); + expect(updates[0]).not.toHaveProperty('value'); + }); + + it('sends nothing when a row that was never overridden is left at the default', () => { + const rows = [row()]; + + expect( + buildThresholdUpdates(rows, { [rows[0].id]: 80 }, NODE, 'node-1') + ).toEqual([]); + }); + + it('treats an emptied string field as a clear, not as zero', () => { + const rows = [row({ isOverridden: true, effectiveValue: 95 })]; + + const updates = buildThresholdUpdates( + rows, + { [rows[0].id]: '' as unknown as number }, + NODE, + 'node-1' + ); + + expect(updates).toHaveLength(1); + expect(updates[0]).not.toHaveProperty('value'); + }); + + it('batches a set and a clear from one submission', () => { + const rows = [ + row({ id: 'a', ruleId: 'rule-1' }), + row({ + id: 'b', + ruleId: 'rule-2', + isOverridden: true, + effectiveValue: 95, + }), + ]; + + const updates = buildThresholdUpdates( + rows, + { a: 60, b: undefined }, + NODE, + 'node-1' + ); + + expect(updates).toHaveLength(2); + expect(updates[0]).toHaveProperty('value', 60); + expect(updates[1]).not.toHaveProperty('value'); + }); +}); diff --git a/ui/apps/pmm/src/components/alert-thresholds/AlertThresholds.utils.ts b/ui/apps/pmm/src/components/alert-thresholds/AlertThresholds.utils.ts index 4abb39aa0f..aa436b8d9c 100644 --- a/ui/apps/pmm/src/components/alert-thresholds/AlertThresholds.utils.ts +++ b/ui/apps/pmm/src/components/alert-thresholds/AlertThresholds.utils.ts @@ -2,7 +2,12 @@ import type { ListThresholdsResponse, PrometheusAlertRulesResponse, Threshold, + ThresholdUpdate, } from 'types/alerting.types'; +import type { + AlertThresholdRow, + AlertThresholdsFormValues, +} from './AlertThresholds.types'; import { UNIT_SYMBOLS } from './AlertThresholds.constants'; export const formatUnit = (unit?: string): string => @@ -46,3 +51,45 @@ export const getRows = ( id: thresholdRowId(t, index), ruleTitle: ruleTitles.get(t.ruleId) ?? '', })); + +// Turns the submitted form into the smallest set of changes that expresses it. +// +// Emptying a field, or typing the default back in, returns the target to the rule +// default. That is only a change when an override exists today, and it is expressed by +// omitting `value` - writing the default as an override instead would pin the target to +// today's default and stop it following a later change to the rule. +export const buildThresholdUpdates = ( + rows: AlertThresholdRow[], + values: AlertThresholdsFormValues, + scope: ThresholdUpdate['scope'], + target: string +): ThresholdUpdate[] => { + const updates: ThresholdUpdate[] = []; + + for (const row of rows) { + const raw = values[row.id]; + const parsed = + raw === undefined || (raw as unknown) === '' ? undefined : Number(raw); + const cleared = parsed === undefined || Number.isNaN(parsed); + + const base = { + scope, + target, + ruleId: row.ruleId, + paramName: row.paramName, + }; + + if (cleared || parsed === row.defaultValue) { + if (row.isOverridden) { + updates.push(base); + } + continue; + } + + if (parsed !== row.effectiveValue) { + updates.push({ ...base, value: parsed }); + } + } + + return updates; +}; diff --git a/ui/apps/pmm/src/hooks/api/useNodeThresholds.ts b/ui/apps/pmm/src/hooks/api/useNodeThresholds.ts index bf4421cfb1..9560529e01 100644 --- a/ui/apps/pmm/src/hooks/api/useNodeThresholds.ts +++ b/ui/apps/pmm/src/hooks/api/useNodeThresholds.ts @@ -1,17 +1,17 @@ -import { useMutation, useQuery, useQueryClient } from "@tanstack/react-query"; +import { useMutation, useQuery, useQueryClient } from '@tanstack/react-query'; import type { UseMutationOptions, UseQueryOptions, -} from "@tanstack/react-query"; -import { batchUpdateThresholds, getThresholds } from "api/alerting"; +} from '@tanstack/react-query'; +import { batchUpdateThresholds, getThresholds } from 'api/alerting'; import type { BatchUpdateThresholdsResponse, ListThresholdsResponse, ThresholdUpdate, -} from "types/alerting.types"; +} from 'types/alerting.types'; export const nodeThresholdsQueryKey = (nodeId: string) => [ - "alerting:nodeThresholds", + 'alerting:nodeThresholds', nodeId, ]; @@ -20,11 +20,11 @@ export const nodeThresholdsQueryKey = (nodeId: string) => [ // overrides. export const useNodeThresholds = ( nodeId: string, - options?: Partial>, + options?: Partial> ) => useQuery({ queryKey: nodeThresholdsQueryKey(nodeId), - queryFn: () => getThresholds("THRESHOLD_SCOPE_NODE", nodeId), + queryFn: () => getThresholds('THRESHOLD_SCOPE_NODE', nodeId), enabled: !!nodeId, ...options, }); @@ -36,12 +36,12 @@ export const useBatchUpdateNodeThresholds = ( nodeId: string, options?: Partial< UseMutationOptions - >, + > ) => { const queryClient = useQueryClient(); return useMutation({ - mutationKey: ["alerting:batchUpdateNodeThresholds", nodeId], + mutationKey: ['alerting:batchUpdateNodeThresholds', nodeId], mutationFn: (updates: ThresholdUpdate[]) => batchUpdateThresholds(updates), ...options, onSuccess: async (data, variables, onMutate, context) => { From b2263a7fe3effdc47b34edaa3f92d2ea0637793d Mon Sep 17 00:00:00 2001 From: Matej Kubinec Date: Tue, 1 Sep 2026 11:10:03 +0200 Subject: [PATCH 10/15] PMM-14912 Add threshold override API tests Cover the threshold endpoints through the generated client, against a running server, so the HTTP surface is exercised rather than the service layer beneath it. The service-level tests already assert behaviour; these assert that it survives routing, JSON conversion and gRPC status mapping. Four tests, each pinning something that would fail quietly rather than loudly: - The lifecycle: an untouched target reports the rule default, setting an override reports it as effective, and clearing returns the target to the default while reporting it as no longer overridden. A tombstoned row reading as overridden would make every target ever tuned look tuned forever. - Validation, including that service and cluster scope answer Unimplemented rather than InvalidArgument. Both are already carried by the schema, the resolver and the proto, so the distinction is the difference between "not yet" and "malformed". - Batch: one invalid update rolls back the valid one alongside it. That transactional guarantee is the whole reason the endpoint exists, and it cannot be observed from a single-row test. - Removing a node takes its overrides with it. Those rows have no foreign key to cascade from - the target column is polymorphic - so this is the only thing proving the removal hook is wired. The fixture also asserts CreateRule returns a rule_id for an overridable template. Without one the feature is unreachable, and nothing else would notice if the field stopped being populated. Co-Authored-By: Claude Opus 5 (1M context) --- api-tests/alerting/thresholds_test.go | 358 ++++++++++++++++++++++++++ 1 file changed, 358 insertions(+) create mode 100644 api-tests/alerting/thresholds_test.go diff --git a/api-tests/alerting/thresholds_test.go b/api-tests/alerting/thresholds_test.go new file mode 100644 index 0000000000..3d887d21b0 --- /dev/null +++ b/api-tests/alerting/thresholds_test.go @@ -0,0 +1,358 @@ +// Copyright (C) 2023 Percona LLC +// +// This program is free software: you can redistribute it and/or modify +// it under the terms of the GNU Affero General Public License as published by +// the Free Software Foundation, either version 3 of the License, or +// (at your option) any later version. +// +// This program is distributed in the hope that it will be useful, +// but WITHOUT ANY WARRANTY; without even the implied warranty of +// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +// GNU Affero General Public License for more details. +// +// You should have received a copy of the GNU Affero General Public License +// along with this program. If not, see . + +package alerting + +import ( + "fmt" + "net/http" + "testing" + + "github.com/AlekSi/pointer" + "github.com/grafana/grafana-openapi-client-go/client/folders" + "github.com/grafana/grafana-openapi-client-go/models" + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + "google.golang.org/grpc/codes" + + pmmapitests "github.com/percona/pmm/api-tests" + alertingClient "github.com/percona/pmm/api/alerting/v1/json/client" + alerting "github.com/percona/pmm/api/alerting/v1/json/client/alerting_service" +) + +const ( + scopeNode = "THRESHOLD_SCOPE_NODE" + scopeService = "THRESHOLD_SCOPE_SERVICE" + scopeCluster = "THRESHOLD_SCOPE_CLUSTER" +) + +// thresholdFixture is a rule created from an overridable template, plus the node its +// thresholds are set against. +type thresholdFixture struct { + client alerting.ClientService + ruleID string + nodeID string +} + +// setupThresholdFixture registers an overridable template, creates a rule from it and a +// node to target, and cleans all three up afterwards. +func setupThresholdFixture(t *testing.T) *thresholdFixture { + t.Helper() + + client := alertingClient.Default.AlertingService + + floatType := "PARAM_TYPE_FLOAT" + severity := "SEVERITY_WARNING" + forceDelete := true + + templateName := pmmapitests.TestString(t, "test-threshold-template") + yml := fmt.Sprintf(`templates: + - name: %s + version: 1 + summary: Overridable threshold + queries: + - ref_id: A + expr: |- + (1 - avg by(node_name) (rate(node_cpu_seconds_total{mode="idle"}[5m]))) * 100 + expressions: + - ref_id: C + type: math + expression: "$A > [[ .threshold ]]" + condition: C + params: + - name: threshold + summary: A percentage from configured maximum + unit: "%%" + type: float + range: [0, 100] + value: 80 + overridable: true + for: 60s + severity: warning + annotations: + summary: overridable threshold +`, templateName) + + _, err := client.CreateTemplate(&alerting.CreateTemplateParams{ + Body: alerting.CreateTemplateBody{Yaml: yml}, + Context: pmmapitests.Context, + }) + require.NoError(t, err) + t.Cleanup(func() { deleteTemplate(t, client, templateName) }) + + gClient := pmmapitests.GetGrafanaClient(t) + createdFolder, err := gClient.Folders.CreateFolder(&models.CreateFolderCommand{ + Title: pmmapitests.TestString(t, "test-threshold-folder"), + }) + require.NoError(t, err) + folder := createdFolder.Payload + t.Cleanup(func() { + _, _ = gClient.Folders.DeleteFolder( + folders.NewDeleteFolderParams().WithFolderUID(folder.UID).WithForceDeleteRules(&forceDelete)) + }) + + created, err := client.CreateRule(&alerting.CreateRuleParams{ + Body: alerting.CreateRuleBody{ + TemplateName: templateName, + Name: pmmapitests.TestString(t, "test-threshold-rule"), + FolderUID: folder.UID, + Group: "test", + Interval: "10s", + For: "60s", + Severity: &severity, + Params: []*alerting.CreateRuleParamsBodyParamsItems0{ + {Name: "threshold", Type: &floatType, Float: 80}, + }, + }, + Context: pmmapitests.Context, + }) + require.NoError(t, err) + + // A rule built from an overridable template must come back with an identity to key + // its overrides on; without one the whole feature is unreachable. + require.NotEmpty(t, created.Payload.RuleID, + "CreateRule must return a rule_id for an overridable template") + + node := pmmapitests.AddGenericNode(t, pmmapitests.TestString(t, "test-threshold-node")) + t.Cleanup(func() { pmmapitests.RemoveNodes(t, node.NodeID) }) + + return &thresholdFixture{ + client: client, + ruleID: created.Payload.RuleID, + nodeID: node.NodeID, + } +} + +func (f *thresholdFixture) list(t *testing.T) []*alerting.ListThresholdsOKBodyThresholdsItems0 { + t.Helper() + + res, err := f.client.ListThresholds(&alerting.ListThresholdsParams{ + Scope: pointer.ToString(scopeNode), + Target: pointer.ToString(f.nodeID), + RuleID: pointer.ToString(f.ruleID), + Context: pmmapitests.Context, + }) + require.NoError(t, err) + + return res.Payload.Thresholds +} + +func (f *thresholdFixture) set(t *testing.T, value float64) (*alerting.SetThresholdOK, error) { + t.Helper() + + return f.client.SetThreshold(&alerting.SetThresholdParams{ + Body: alerting.SetThresholdBody{ + Scope: pointer.ToString(scopeNode), + Target: f.nodeID, + RuleID: f.ruleID, + ParamName: "threshold", + Value: value, + }, + Context: pmmapitests.Context, + }) +} + +func TestThresholdOverrideLifecycle(t *testing.T) { + t.Parallel() + + f := setupThresholdFixture(t) + + // An untouched target still reports the parameter, at the rule's default. + before := f.list(t) + require.Len(t, before, 1) + assert.InDelta(t, 80, before[0].DefaultValue, 0.0001) + assert.InDelta(t, 80, before[0].EffectiveValue, 0.0001) + assert.False(t, before[0].IsOverridden) + + set, err := f.set(t, 95) + require.NoError(t, err) + assert.InDelta(t, 95, set.Payload.Threshold.EffectiveValue, 0.0001) + assert.True(t, set.Payload.Threshold.IsOverridden) + require.NotNil(t, set.Payload.Threshold.Scope) + assert.Equal(t, scopeNode, *set.Payload.Threshold.Scope) + + after := f.list(t) + require.Len(t, after, 1) + assert.InDelta(t, 95, after[0].EffectiveValue, 0.0001) + assert.True(t, after[0].IsOverridden) + + // Clearing returns the target to the default. The override row survives as a + // tombstone so the emitted series keeps existing and merely changes value, but that + // is invisible from here - what the API must report is "not overridden". + _, err = f.client.ClearThreshold(&alerting.ClearThresholdParams{ + Scope: pointer.ToString(scopeNode), + Target: pointer.ToString(f.nodeID), + RuleID: pointer.ToString(f.ruleID), + ParamName: pointer.ToString("threshold"), + Context: pmmapitests.Context, + }) + require.NoError(t, err) + + cleared := f.list(t) + require.Len(t, cleared, 1) + assert.InDelta(t, 80, cleared[0].EffectiveValue, 0.0001) + assert.False(t, cleared[0].IsOverridden, + "a cleared override must not read as overridden, or every target ever tuned reads as tuned forever") +} + +func TestThresholdOverrideValidation(t *testing.T) { + t.Parallel() + + f := setupThresholdFixture(t) + + t.Run("value outside the declared range", func(t *testing.T) { + _, err := f.set(t, 150) + pmmapitests.AssertAPIErrorf(t, err, http.StatusBadRequest, codes.InvalidArgument, "") + }) + + t.Run("unknown parameter", func(t *testing.T) { + _, err := f.client.SetThreshold(&alerting.SetThresholdParams{ + Body: alerting.SetThresholdBody{ + Scope: pointer.ToString(scopeNode), Target: f.nodeID, + RuleID: f.ruleID, ParamName: "not-overridable", Value: 90, + }, + Context: pmmapitests.Context, + }) + pmmapitests.AssertAPIErrorf(t, err, http.StatusNotFound, codes.NotFound, "") + }) + + t.Run("unknown rule", func(t *testing.T) { + _, err := f.client.SetThreshold(&alerting.SetThresholdParams{ + Body: alerting.SetThresholdBody{ + Scope: pointer.ToString(scopeNode), Target: f.nodeID, + RuleID: "no-such-rule", ParamName: "threshold", Value: 90, + }, + Context: pmmapitests.Context, + }) + pmmapitests.AssertAPIErrorf(t, err, http.StatusNotFound, codes.NotFound, "") + }) + + t.Run("target that does not exist", func(t *testing.T) { + _, err := f.client.SetThreshold(&alerting.SetThresholdParams{ + Body: alerting.SetThresholdBody{ + Scope: pointer.ToString(scopeNode), Target: "no-such-node", + RuleID: f.ruleID, ParamName: "threshold", Value: 90, + }, + Context: pmmapitests.Context, + }) + pmmapitests.AssertAPIErrorf(t, err, http.StatusNotFound, codes.NotFound, "") + }) + + // Service and cluster are already carried by the schema, the resolver and the proto, + // so they report as not-yet-implemented rather than as a malformed request. + t.Run("scopes that are not implemented yet", func(t *testing.T) { + for _, scope := range []string{scopeService, scopeCluster} { + _, err := f.client.SetThreshold(&alerting.SetThresholdParams{ + Body: alerting.SetThresholdBody{ + Scope: pointer.ToString(scope), Target: f.nodeID, + RuleID: f.ruleID, ParamName: "threshold", Value: 90, + }, + Context: pmmapitests.Context, + }) + pmmapitests.AssertAPIErrorf(t, err, http.StatusNotImplemented, codes.Unimplemented, "") + } + }) +} + +func TestThresholdBatchUpdate(t *testing.T) { + t.Parallel() + + f := setupThresholdFixture(t) + + t.Run("sets through the batch endpoint", func(t *testing.T) { + res, err := f.client.BatchUpdateThresholds(&alerting.BatchUpdateThresholdsParams{ + Body: alerting.BatchUpdateThresholdsBody{ + Updates: []*alerting.BatchUpdateThresholdsParamsBodyUpdatesItems0{{ + Scope: pointer.ToString(scopeNode), Target: f.nodeID, + RuleID: f.ruleID, ParamName: "threshold", + Value: pointer.ToFloat64(70), + }}, + }, + Context: pmmapitests.Context, + }) + require.NoError(t, err) + require.Len(t, res.Payload.Thresholds, 1) + assert.InDelta(t, 70, res.Payload.Thresholds[0].EffectiveValue, 0.0001) + }) + + t.Run("an update with no value clears instead of setting", func(t *testing.T) { + res, err := f.client.BatchUpdateThresholds(&alerting.BatchUpdateThresholdsParams{ + Body: alerting.BatchUpdateThresholdsBody{ + Updates: []*alerting.BatchUpdateThresholdsParamsBodyUpdatesItems0{{ + Scope: pointer.ToString(scopeNode), Target: f.nodeID, + RuleID: f.ruleID, ParamName: "threshold", + }}, + }, + Context: pmmapitests.Context, + }) + require.NoError(t, err) + assert.Empty(t, res.Payload.Thresholds, "cleared entries are omitted from the response") + + current := f.list(t) + require.Len(t, current, 1) + assert.False(t, current[0].IsOverridden) + }) + + // The reason the batch endpoint exists: a client editing several rows at once must + // never land a partial result it cannot report. + t.Run("one invalid update rolls the whole batch back", func(t *testing.T) { + _, err := f.client.BatchUpdateThresholds(&alerting.BatchUpdateThresholdsParams{ + Body: alerting.BatchUpdateThresholdsBody{ + Updates: []*alerting.BatchUpdateThresholdsParamsBodyUpdatesItems0{ + { + Scope: pointer.ToString(scopeNode), Target: f.nodeID, + RuleID: f.ruleID, ParamName: "threshold", + Value: pointer.ToFloat64(60), + }, + { + Scope: pointer.ToString(scopeNode), Target: f.nodeID, + RuleID: f.ruleID, ParamName: "threshold", + Value: pointer.ToFloat64(500), // outside the declared range + }, + }, + }, + Context: pmmapitests.Context, + }) + pmmapitests.AssertAPIErrorf(t, err, http.StatusBadRequest, codes.InvalidArgument, "") + + current := f.list(t) + require.Len(t, current, 1) + assert.False(t, current[0].IsOverridden, + "the valid update must not survive the invalid one failing") + }) +} + +// TestThresholdOverrideRemovedWithNode covers the cleanup that has no cascade to rely on: +// the override's target column is polymorphic, so it carries no foreign key and the rows +// are removed by the node removal API itself. +func TestThresholdOverrideRemovedWithNode(t *testing.T) { + t.Parallel() + + f := setupThresholdFixture(t) + + _, err := f.set(t, 90) + require.NoError(t, err) + + pmmapitests.RemoveNodes(t, f.nodeID) + + // With the node gone the override is unreachable by target, so ask for every + // override of this rule instead. + res, err := f.client.ListThresholds(&alerting.ListThresholdsParams{ + RuleID: pointer.ToString(f.ruleID), + Context: pmmapitests.Context, + }) + require.NoError(t, err) + assert.Empty(t, res.Payload.Thresholds, "an override must not outlive the node it targets") +} From 5fc8e97cc32eab05742ce3e1d481718eadca1e61 Mon Sep 17 00:00:00 2001 From: Matej Kubinec Date: Wed, 2 Sep 2026 09:44:06 +0200 Subject: [PATCH 11/15] PMM-14912 Desugar single-expression templates Split a single-expression template apart at build time into the three steps a multi-expression template already produces - observed query, injected threshold, math comparison - so its threshold can be overridden per target. The template file is untouched; only the generated rule changes shape. This is what stood between 16 of the 43 shipped templates and an overridable threshold. Every one declares exactly one parameter, and it is always the threshold. The split is done on the PromQL AST rather than with a regexp. The parameter token is not valid PromQL, so it is replaced by a numeric sentinel padded to the token's exact byte length; AST positions then map 1:1 onto the original text, which lets the left-hand side be sliced out of the original string with the author's line breaks intact rather than reprinted from the AST. A regexp would have to special-case parentheses, the bool modifier and vector matching, and would fail silently by producing a plausible but wrong left-hand side. mysql_too_many_connections is the case that proves it: its `/ ignoring (job)` sits on a division inside the left-hand side, so any string search for vector matching rejects a template that is perfectly splittable. Two details the corpus forced: - The bool modifier is optional - 13 of the 16 use it, 3 do not - and is dropped either way, since Grafana math comparisons already yield 0/1. - The operator is carried across rather than assumed. Eight of the 16 compare with `<`, so assuming `>` would invert half of them. Grafana's `$value` is only a single scalar when a rule has one step, so a desugared rule would ship broken alert text. Annotations are repointed at the observed query: a bare reference also gains formatting, while one that pipes the value keeps its pipeline and is only renamed. The rename deliberately does not match `$values`, which shares its prefix and is already used by one template. Two things this surfaced rather than introduced: - The vector-matching guard the design sketch called for is unreachable. PromQL permits vector matching only between two instant vectors, and a threshold is a scalar, so such a template fails to parse whatever the threshold is - it could never have existed. Removed, with the reason recorded where it stood. - mongodb_replication_lag does not parse: `max([1m])` should be max_over_time. A pre-existing defect the splitter reports instead of hiding. The corpus test logs it rather than failing. No template is marked overridable yet, so nothing changes for anyone: desugaring only runs once a parameter opts in. Co-Authored-By: Claude Opus 5 (1M context) --- managed/pi/alert/overridable.go | 102 +++++++-- managed/pi/alert/overridable_test.go | 46 +++- managed/pi/alert/singleexpr_test.go | 203 ++++++++++++++++++ managed/services/alerting/rule_builder.go | 114 +++++++++- .../alerting/rule_builder_dynamic_test.go | 186 ++++++++++++++++ managed/services/alerting/service.go | 6 + 6 files changed, 638 insertions(+), 19 deletions(-) create mode 100644 managed/pi/alert/singleexpr_test.go diff --git a/managed/pi/alert/overridable.go b/managed/pi/alert/overridable.go index 9fd10e352c..85fb8910ae 100644 --- a/managed/pi/alert/overridable.go +++ b/managed/pi/alert/overridable.go @@ -16,8 +16,12 @@ package alert import ( + "errors" "fmt" "regexp" + "strings" + + "github.com/prometheus/prometheus/promql/parser" ) // ParamTokenRegexp returns a regexp matching a parameter's placeholder token, tolerating @@ -51,26 +55,100 @@ func (r *Template) OverridableParams() []Parameter { return params } +// SingleExprSplit is a single-expression template taken apart so the builder can emit the +// same three steps a multi-expression template produces: the observed query, an injected +// threshold, and a math comparison between them. +type SingleExprSplit struct { + // LHS is everything left of the comparison, sliced from the original text so the + // author's formatting survives. + LHS string + // Operator is the comparison the template used, e.g. "<" or ">=". + Operator string +} + +// SplitSingleExpr takes a single-expression template apart at its final comparison. +// +// The parameter token is not valid PromQL, so it is replaced by a numeric sentinel padded +// to the token's exact byte length. AST positions then map 1:1 onto the original text, +// which is what lets the left-hand side be sliced out of the original string rather than +// printed back from the AST - preserving line breaks and spacing exactly as written. +// +// Splitting on the AST rather than with a regexp is deliberate. A regexp has to special-case +// parentheses, the `bool` modifier, vector matching and nested comparisons, and it fails +// silently by producing a plausible but wrong left-hand side. +func SplitSingleExpr(expr, paramName string) (SingleExprSplit, error) { + var zero SingleExprSplit + + token := ParamTokenRegexp(paramName).FindString(expr) + if token == "" { + return zero, fmt.Errorf("parameter %q is not referenced in the expression", paramName) + } + + // "0" padded with spaces to the token's exact byte length: parseable, and leaves every + // following position unchanged. + sentinel := "0" + strings.Repeat(" ", len(token)-1) + probe := strings.Replace(expr, token, sentinel, 1) + + // Default options: templates are ordinary PromQL, so nothing experimental is enabled. + parsed, err := parser.NewParser(parser.Options{}).ParseExpr(probe) + if err != nil { + return zero, fmt.Errorf("failed to parse expression: %w", err) + } + + binary, ok := parsed.(*parser.BinaryExpr) + if !ok || !binary.Op.IsComparisonOperator() { + return zero, errors.New("an overridable parameter must be the right-hand side of the expression's top-level comparison") + } + + // Anything other than the bare sentinel means the token was used inside a larger + // expression, e.g. `foo > [[ .threshold ]] * 100`, which cannot become a threshold step. + number, ok := binary.RHS.(*parser.NumberLiteral) + if !ok || number.Val != 0 { + return zero, errors.New("an overridable parameter must be compared directly, not used inside a larger expression") + } + + // Vector matching on the comparison itself needs no guard: PromQL only allows it + // between two instant vectors, and the threshold is a scalar, so such an expression + // fails to parse above with "vector matching only allowed between instant vectors". + // Matching on an operator *inside* the left-hand side is untouched and stays valid. + + // The `bool` modifier is read and then dropped: Grafana math comparisons already yield + // 0/1, so carrying it across would be redundant. + position := binary.LHS.PositionRange() + + return SingleExprSplit{ + LHS: strings.TrimSpace(expr[position.Start:position.End]), + Operator: binary.Op.String(), + }, nil +} + // validateOverridableParams checks the constraints that depend on the template's shape, // rather than on the parameter alone. func (r *Template) validateOverridableParams() error { - for _, param := range r.Params { - if !param.Overridable { + overridable := r.OverridableParams() + + for _, param := range overridable { + if r.UsesMultipleExpressions() { + // The threshold is injected as a separate query step and referenced from the + // expression, so a parameter no expression mentions has nothing to override. + if !r.ParamReferencedInExpressions(param.Name) { + return fmt.Errorf("overridable parameter %q must be referenced by an expression step", param.Name) + } + continue } - // A single-expression template bakes its threshold into the RHS of a PromQL - // comparison, which has to be split apart before a threshold step can be - // injected. That is deliberately not supported yet, so reject it here instead - // of silently ignoring the flag and shipping a rule that never overrides. - if !r.UsesMultipleExpressions() { - return fmt.Errorf("parameter %q cannot be overridable: only multi-expression templates support overridable parameters", param.Name) + // A single-expression template is split apart at build time. Checking that here + // means an expression that cannot be split fails when the template is parsed, with + // the reason, rather than producing a rule whose threshold silently never applies. + if len(overridable) > 1 { + return fmt.Errorf( + "a single-expression template supports at most one overridable parameter, got %d", len(overridable)) } - // The threshold is injected as a separate query step and referenced from the - // expression, so a parameter that no expression mentions has nothing to override. - if !r.ParamReferencedInExpressions(param.Name) { - return fmt.Errorf("overridable parameter %q must be referenced by an expression step", param.Name) + _, err := SplitSingleExpr(r.Expr, param.Name) + if err != nil { + return fmt.Errorf("overridable parameter %q: %w", param.Name, err) } } diff --git a/managed/pi/alert/overridable_test.go b/managed/pi/alert/overridable_test.go index 34f48f18d7..1cb1c705dc 100644 --- a/managed/pi/alert/overridable_test.go +++ b/managed/pi/alert/overridable_test.go @@ -120,18 +120,54 @@ func TestValidateOverridableTemplate(t *testing.T) { require.NoError(t, template.Validate()) } -func TestValidateOverridableRejectsSingleExpression(t *testing.T) { - t.Parallel() - +// singleExprTemplate returns a valid single-expression template whose one param is +// overridable, so the desugaring constraints can be varied one at a time. +func singleExprTemplate() Template { template := overridableTemplate() template.Queries = nil template.Expressions = nil template.Condition = "" - template.Expr = "up > [[ .threshold ]]" + template.Expr = "up > bool [[ .threshold ]]" + + return template +} + +func TestValidateOverridableAcceptsSplittableSingleExpression(t *testing.T) { + t.Parallel() + + template := singleExprTemplate() + require.NoError(t, template.Validate()) +} + +// An expression that cannot be split must fail when the template is parsed. Accepting it +// would produce a rule whose threshold silently never applies. +func TestValidateOverridableRejectsUnsplittableSingleExpression(t *testing.T) { + t.Parallel() + + template := singleExprTemplate() + template.Expr = "up > bool [[ .threshold ]] * 100" + + err := template.Validate() + require.Error(t, err) + assert.Contains(t, err.Error(), "must be compared directly") +} + +func TestValidateOverridableRejectsTwoParamsOnSingleExpression(t *testing.T) { + t.Parallel() + + template := singleExprTemplate() + template.Expr = "up > bool [[ .threshold ]]" + template.Params = append(template.Params, Parameter{ + Name: "second", + Summary: "second", + Type: Float, + Value: 1, + Overridable: true, + }) err := template.Validate() require.Error(t, err) - assert.Contains(t, err.Error(), "only multi-expression templates") + assert.Contains(t, err.Error(), "at most one overridable parameter") } func TestValidateOverridableRejectsUnreferencedParam(t *testing.T) { diff --git a/managed/pi/alert/singleexpr_test.go b/managed/pi/alert/singleexpr_test.go new file mode 100644 index 0000000000..ab3b934dcf --- /dev/null +++ b/managed/pi/alert/singleexpr_test.go @@ -0,0 +1,203 @@ +// Copyright (C) 2023 Percona LLC +// +// This program is free software: you can redistribute it and/or modify +// it under the terms of the GNU Affero General Public License as published by +// the Free Software Foundation, either version 3 of the License, or +// (at your option) any later version. +// +// This program is distributed in the hope that it will be useful, +// but WITHOUT ANY WARRANTY; without even the implied warranty of +// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +// GNU Affero General Public License for more details. +// +// You should have received a copy of the GNU Affero General Public License +// along with this program. If not, see . + +package alert + +import ( + "os" + "path/filepath" + "strings" + "testing" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" +) + +func TestSplitSingleExpr(t *testing.T) { + t.Parallel() + + for _, tc := range []struct { + name string + expr string + param string + wantLHS string + wantOp string + wantErr string + }{ + { + name: "greater than with bool", + expr: "node_load1 > bool [[ .threshold ]]", + param: "threshold", + wantLHS: "node_load1", + wantOp: ">", + }, + { + // 8 of the 16 shipped candidates compare with `<`, so assuming `>` would + // invert half the corpus. + name: "less than preserves the operator", + expr: "node_memory_MemAvailable_bytes < bool [[ .threshold ]]", + param: "threshold", + wantLHS: "node_memory_MemAvailable_bytes", + wantOp: "<", + }, + { + name: "greater or equal", + expr: "proxysql_runtime_servers_status >= bool [[ .status ]]", + param: "status", + wantLHS: "proxysql_runtime_servers_status", + wantOp: ">=", + }, + { + name: "bool is optional", + expr: "(max by (cluster) (some_metric)) > [[ .threshold ]]", + param: "threshold", + wantLHS: "(max by (cluster) (some_metric))", + wantOp: ">", + }, + { + // The left-hand side is sliced from the original text, so line breaks and + // spacing survive rather than being reprinted from the AST. + name: "multi-line left-hand side keeps its formatting", + expr: "node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes\n* 100\n< bool [[ .threshold ]]", + param: "threshold", + wantLHS: "node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes\n* 100", + wantOp: "<", + }, + { + // Vector matching on an operator *inside* the left-hand side is fine; only + // matching on the comparison itself is not. A string search for "ignoring(" + // would wrongly reject this, which is why the split is done on the AST. + name: "vector matching inside the left-hand side is allowed", + expr: "max_over_time(a[5m]) / ignoring (job) b * 100 > bool [[ .threshold ]]", + param: "threshold", + wantLHS: "max_over_time(a[5m]) / ignoring (job) b * 100", + wantOp: ">", + }, + { + name: "tolerates whitespace-free tokens", + expr: "node_load1 > bool [[.threshold]]", + param: "threshold", + wantLHS: "node_load1", + wantOp: ">", + }, + { + name: "parameter not referenced", + expr: "node_load1 > bool 80", + param: "threshold", + wantErr: "is not referenced in the expression", + }, + { + name: "token used inside a larger expression", + expr: "node_load1 > bool [[ .threshold ]] * 100", + param: "threshold", + wantErr: "must be compared directly", + }, + { + name: "no top-level comparison", + expr: "node_load1 + [[ .threshold ]]", + param: "threshold", + wantErr: "must be the right-hand side of the expression's top-level comparison", + }, + { + // PromQL allows vector matching only between two instant vectors, and a + // threshold is a scalar, so this is rejected by the parser rather than needing + // a guard of our own - such a template could never be valid in the first place. + name: "vector matching on the comparison itself", + expr: "a > bool on (node_name) [[ .threshold ]]", + param: "threshold", + wantErr: "vector matching only allowed between instant vectors", + }, + { + name: "expression that is not valid PromQL", + expr: "max(some_metric[1m]) > [[ .threshold ]]", + param: "threshold", + wantErr: "failed to parse expression", + }, + } { + t.Run(tc.name, func(t *testing.T) { + t.Parallel() + + split, err := SplitSingleExpr(tc.expr, tc.param) + if tc.wantErr != "" { + require.Error(t, err) + assert.Contains(t, err.Error(), tc.wantErr) + + return + } + + require.NoError(t, err) + assert.Equal(t, tc.wantLHS, split.LHS) + assert.Equal(t, tc.wantOp, split.Operator) + }) + } +} + +// TestSplitShippedSingleExprTemplates runs the splitter over every shipped single-expression +// template that carries a parameter. These are the expressions desugaring has to handle, so +// the corpus itself is the test: a template reworded into a shape the splitter cannot take +// apart should fail here rather than when someone marks it overridable. +func TestSplitShippedSingleExprTemplates(t *testing.T) { + t.Parallel() + + files, err := filepath.Glob(filepath.Join("..", "..", "data", "alerting-templates", "*.yml")) + require.NoError(t, err) + require.NotEmpty(t, files) + + splittable := 0 + + for _, file := range files { + b, err := os.ReadFile(file) //nolint:gosec + require.NoError(t, err) + + templates, err := Parse(strings.NewReader(string(b)), &ParseParams{ + DisallowUnknownFields: true, + DisallowInvalidTemplates: true, + }) + require.NoErrorf(t, err, "%s must parse", filepath.Base(file)) + + for _, template := range templates { + if template.UsesMultipleExpressions() || len(template.Params) == 0 { + continue + } + + for _, param := range template.Params { + if !ParamTokenRegexp(param.Name).MatchString(template.Expr) { + continue + } + + split, err := SplitSingleExpr(template.Expr, param.Name) + if err != nil { + // mongodb_replication_lag uses `max([1m])`, which is not + // valid PromQL - a pre-existing bug the parser surfaces here. Report it + // rather than failing, so this test tracks the corpus instead of + // blocking on a defect it did not introduce. + t.Logf("NOT SPLITTABLE %s (%s): %v", template.Name, param.Name, err) + + continue + } + + splittable++ + assert.NotEmptyf(t, split.LHS, "%s: left-hand side must not be empty", template.Name) + assert.NotEmptyf(t, split.Operator, "%s: operator must not be empty", template.Name) + assert.NotContainsf(t, split.LHS, "[[", + "%s: the parameter token must not survive into the observed query", template.Name) + } + } + } + + t.Logf("splittable single-expression templates: %d", splittable) + assert.GreaterOrEqual(t, splittable, 15, + "the shipped corpus should be almost entirely splittable") +} diff --git a/managed/services/alerting/rule_builder.go b/managed/services/alerting/rule_builder.go index 519438c609..e8ad76ddf6 100644 --- a/managed/services/alerting/rule_builder.go +++ b/managed/services/alerting/rule_builder.go @@ -39,6 +39,11 @@ const ( // thresholdRefIDPrefix prefixes the ref ID of each injected threshold query. thresholdRefIDPrefix = "T_" + // Ref IDs used when a single-expression template is desugared. A multi-expression + // template names its own steps; a desugared one has none to inherit. + desugaredQueryRefID = "A" + desugaredConditionRefID = "C" + // The label the injected threshold query joins the observed query on. It follows // from the scope: an override targets a node by node_name, and a service - whether // named directly or reached through its cluster - by service_name. @@ -80,8 +85,10 @@ func buildGrafanaRuleData( return buildMultiExpressionRuleData(template, metricsDatasourceUID, ruleID, params, filters) } - // Overridable parameters are rejected on single-expression templates at parse time, - // so nothing reaches here needing a threshold step. + overridable := template.OverridableParams() + if ruleID != "" && len(overridable) != 0 { + return buildDesugaredRuleData(template, metricsDatasourceUID, ruleID, overridable[0], params, filters) + } expr, err := fillAndFilterExpr(template.Expr, params, filters) if err != nil { @@ -96,6 +103,74 @@ func buildGrafanaRuleData( return []services.Data{data}, "A", nil } +// buildDesugaredRuleData turns a single-expression template into the same three steps a +// multi-expression template produces, so its threshold can be overridden per target: +// the observed query, an injected threshold, and a math comparison between them. +func buildDesugaredRuleData( + template *alert.Template, + metricsDatasourceUID string, + ruleID string, + param alert.Parameter, + params map[string]string, + filters []*alertingv1.Filter, +) ([]services.Data, string, error) { + split, err := alert.SplitSingleExpr(template.Expr, param.Name) + if err != nil { + return nil, "", fmt.Errorf("failed to split expression for parameter %q: %w", param.Name, err) + } + + joinLabel, err := joinLabelForScopes(param.GetOverrideScopes()) + if err != nil { + return nil, "", fmt.Errorf("parameter %q: %w", param.Name, err) + } + + defaultValue, ok := params[param.Name] + if !ok { + return nil, "", fmt.Errorf("no value supplied for overridable parameter %q", param.Name) + } + + // The observed query carries the alert's filters; the threshold deliberately does not, + // since a filtered threshold would leave the targets the filter excludes with none. + observed, err := fillAndFilterExpr(split.LHS, params, filters) + if err != nil { + return nil, "", err + } + + fanOut, err := fillExprWithParams(split.LHS, params) + if err != nil { + return nil, "", err + } + + // A and C are fixed here, unlike the multi-expression path where the template chooses + // its own ref IDs, so only those two can be collided with. + taken := map[string]struct{}{ + desugaredQueryRefID: {}, + desugaredConditionRefID: {}, + } + thresholdRefID := allocateThresholdRefID(param.Name, taken) + + query, err := newPromQueryData(metricsDatasourceUID, desugaredQueryRefID, observed) + if err != nil { + return nil, "", err + } + + threshold, err := newPromQueryData(metricsDatasourceUID, thresholdRefID, + thresholdQueryExpr(ruleID, param.Name, joinLabel, fanOut, defaultValue)) + if err != nil { + return nil, "", err + } + + // The template's `bool` modifier, if any, is dropped: Grafana math comparisons already + // yield 0/1, so carrying it across would be redundant. + condition, err := newMathExpressionData(desugaredConditionRefID, + fmt.Sprintf("$%s %s $%s", desugaredQueryRefID, split.Operator, thresholdRefID)) + if err != nil { + return nil, "", err + } + + return []services.Data{query, threshold, condition}, desugaredConditionRefID, nil +} + func buildMultiExpressionRuleData( template *alert.Template, metricsDatasourceUID string, @@ -437,3 +512,38 @@ func swapOverridableTokens(expression string, injections []thresholdInjection) s return expression } + +// desugaredValueRegexp matches Grafana's `$value` variable, but not `$values`, whose name +// starts with it. A plain string replacement would turn `$values.A` into nonsense. +var desugaredValueRegexp = regexp.MustCompile(`\$value\b`) + +// desugaredBareValueRegexp matches a whole action that is nothing but `$value`, which is the +// case worth formatting rather than only renaming. +var desugaredBareValueRegexp = regexp.MustCompile(`\{\{\s*\$value\s*\}\}`) + +// isDesugaredRule reports whether this rule is built by splitting a single expression apart. +func isDesugaredRule(template *alert.Template, ruleID string) bool { + return ruleID != "" && !template.UsesMultipleExpressions() && len(template.OverridableParams()) != 0 +} + +// rewriteDesugaredAnnotations repoints Grafana's `$value` at the observed query. +// +// `$value` is only a single scalar when a rule has one step. A desugared rule has three, so +// the variable stops resolving and the alert text ships broken - which is why this runs for +// every desugared rule rather than only where it looks necessary. +// +// A bare `{{ $value }}` also gains formatting, since an unformatted float renders every +// digit it has. An action that pipes the value, such as `{{ $value | humanizeDuration }}`, +// keeps its pipeline and only has the variable renamed. +func rewriteDesugaredAnnotations(annotations map[string]string) { + for key, text := range annotations { + // Literal replacement throughout: `$values` would otherwise be read as a capture + // group reference and silently dropped. + rewritten := desugaredBareValueRegexp.ReplaceAllLiteralString(text, + `{{ printf "%.2f" $values.`+desugaredQueryRefID+`.Value }}`) + rewritten = desugaredValueRegexp.ReplaceAllLiteralString(rewritten, + `$values.`+desugaredQueryRefID+`.Value`) + + annotations[key] = rewritten + } +} diff --git a/managed/services/alerting/rule_builder_dynamic_test.go b/managed/services/alerting/rule_builder_dynamic_test.go index 07f29961b0..0ba55a13b6 100644 --- a/managed/services/alerting/rule_builder_dynamic_test.go +++ b/managed/services/alerting/rule_builder_dynamic_test.go @@ -347,3 +347,189 @@ func TestObservedQueryForParamErrors(t *testing.T) { assert.Contains(t, err.Error(), "not referenced by any expression") }) } + +// desugarTemplate returns a single-expression template whose one param is overridable - +// the shape desugaring exists to handle. +func desugarTemplate() *alert.Template { + return &alert.Template{ + Name: "test_single_expr", + Version: 1, + Summary: "summary", + Expr: "node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes\n* 100\n< bool [[ .threshold ]]", + Params: []alert.Parameter{{ + Name: "threshold", + Summary: "threshold", + Type: alert.Float, + Value: 20, + Overridable: true, + }}, + } +} + +func TestBuildDesugaredRuleData(t *testing.T) { + t.Parallel() + + data, condition, err := buildGrafanaRuleData( + desugarTemplate(), "metrics-uid", "rule-1", + map[string]string{"threshold": "20"}, nil, + ) + require.NoError(t, err) + assert.Equal(t, "C", condition) + require.Len(t, data, 3, "observed query, injected threshold, math condition") + + byRef := dataByRefID(t, data) + require.Contains(t, byRef, "T_threshold") + + // The observed query is the left-hand side, with the author's line breaks intact and + // the parameter token gone. + observed := exprOf(t, byRef["A"]) + assert.Equal(t, "node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes\n* 100", observed) + assert.NotContains(t, observed, "[[") + + // The operator is carried across and `bool` is dropped, since Grafana math already + // yields 0/1. + body := expressionOf(t, byRef["C"]) + assert.Equal(t, "$A < $T_threshold", body) + assert.NotContains(t, body, "bool") + assert.NotContains(t, body, "20", "the default must never be baked into the expression") + + // The threshold fans out over the same observed query, exactly as it does for a + // multi-expression template. + threshold := exprOf(t, byRef["T_threshold"]) + assert.Contains(t, threshold, `pmm_alert_threshold_override{rule_id="rule-1", param="threshold"}`) + assert.Contains(t, threshold, "* 0 + 20)") +} + +// A single-expression template with no PMM-minted rule ID must generate exactly what it did +// before desugaring existed: one query, condition A, default baked in. +func TestBuildDesugaredRuleDataWithoutRuleIDIsUnchanged(t *testing.T) { + t.Parallel() + + data, condition, err := buildGrafanaRuleData( + desugarTemplate(), "metrics-uid", "", + map[string]string{"threshold": "20"}, nil, + ) + require.NoError(t, err) + assert.Equal(t, "A", condition) + require.Len(t, data, 1) + assert.Contains(t, exprOf(t, data[0]), "< bool 20") +} + +func TestBuildDesugaredRuleDataWithoutOverridableParam(t *testing.T) { + t.Parallel() + + template := desugarTemplate() + template.Params[0].Overridable = false + + data, condition, err := buildGrafanaRuleData( + template, "metrics-uid", "rule-1", + map[string]string{"threshold": "20"}, nil, + ) + require.NoError(t, err) + assert.Equal(t, "A", condition) + require.Len(t, data, 1) +} + +// Filters narrow the observed query but not the threshold: a filtered threshold would leave +// the targets the filter excludes with no threshold at all. +func TestBuildDesugaredRuleDataDoesNotFilterTheThreshold(t *testing.T) { + t.Parallel() + + data, _, err := buildGrafanaRuleData( + desugarTemplate(), "metrics-uid", "rule-1", + map[string]string{"threshold": "20"}, + []*alertingv1.Filter{{ + Type: alertingv1.FilterType_FILTER_TYPE_MATCH, + Label: "node_name", + Regexp: "prod-.*", + }}, + ) + require.NoError(t, err) + + byRef := dataByRefID(t, data) + assert.Contains(t, exprOf(t, byRef["A"]), "label_match(") + assert.NotContains(t, exprOf(t, byRef["T_threshold"]), "label_match(") +} + +// The template chooses no ref IDs of its own, so only the fixed A and C can be collided with. +func TestBuildDesugaredRuleDataAvoidsFixedRefIDCollision(t *testing.T) { + t.Parallel() + + template := desugarTemplate() + template.Expr = "up < bool [[ .A ]]" + template.Params[0].Name = "A" + + data, _, err := buildGrafanaRuleData( + template, "metrics-uid", "rule-1", + map[string]string{"A": "20"}, nil, + ) + require.NoError(t, err) + + byRef := dataByRefID(t, data) + require.Contains(t, byRef, "T_A") + assert.Equal(t, "$A < $T_A", expressionOf(t, byRef["C"])) +} + +func TestRewriteDesugaredAnnotations(t *testing.T) { + t.Parallel() + + for _, tc := range []struct { + name string + in string + want string + }{ + { + // A bare value gains formatting: an unformatted float renders every digit. + name: "bare value is repointed and formatted", + in: "Memory is {{ $value }}% free.", + want: `Memory is {{ printf "%.2f" $values.A.Value }}% free.`, + }, + { + // mongodb_pbm_backup_stale pipes the value; the pipeline must survive. + name: "piped value keeps its pipeline", + in: "Backup is {{ $value | humanizeDuration }} old.", + want: "Backup is {{ $values.A.Value | humanizeDuration }} old.", + }, + { + // $values starts with $value, so a plain string replacement would corrupt it. + // mongodb_replication_lag already uses this form. + name: "an existing $values reference is left alone", + in: "Lag is {{ $values.A }}s.", + want: "Lag is {{ $values.A }}s.", + }, + { + name: "text with no value reference is untouched", + in: "{{ $labels.node_name }} is unhealthy.", + want: "{{ $labels.node_name }} is unhealthy.", + }, + { + name: "several references in one annotation", + in: "{{ $value }} and {{ $value | humanize }}", + want: `{{ printf "%.2f" $values.A.Value }} and {{ $values.A.Value | humanize }}`, + }, + } { + t.Run(tc.name, func(t *testing.T) { + t.Parallel() + + annotations := map[string]string{"description": tc.in} + rewriteDesugaredAnnotations(annotations) + assert.Equal(t, tc.want, annotations["description"]) + }) + } +} + +func TestIsDesugaredRule(t *testing.T) { + t.Parallel() + + assert.True(t, isDesugaredRule(desugarTemplate(), "rule-1")) + + assert.False(t, isDesugaredRule(desugarTemplate(), ""), + "a rule with no PMM identity is built as it always was") + + plain := desugarTemplate() + plain.Params[0].Overridable = false + assert.False(t, isDesugaredRule(plain, "rule-1")) + + assert.False(t, isDesugaredRule(overridableRuleTemplate(), "rule-1"), + "a multi-expression template is not desugared") +} diff --git a/managed/services/alerting/service.go b/managed/services/alerting/service.go index b180ad1b02..abdaa515db 100644 --- a/managed/services/alerting/service.go +++ b/managed/services/alerting/service.go @@ -766,6 +766,12 @@ func (s *Service) CreateRule(ctx context.Context, req *alerting.CreateRuleReques return nil, fmt.Errorf("failed to fill template annotations placeholders: %w", err) } + // A desugared rule has three steps, so Grafana's `$value` no longer resolves and the + // alert text would ship broken. + if isDesugaredRule(alertTemplate, ruleID) { + rewriteDesugaredAnnotations(annotations) + } + labels := make(map[string]string) // Copy labels form template err = transformMaps(req.CustomLabels, labels, paramsValues.AsStringMap()) From fd3a4dce96eaa7bee11714eb3cb7457560715db0 Mon Sep 17 00:00:00 2001 From: Matej Kubinec Date: Wed, 2 Sep 2026 16:06:53 +0200 Subject: [PATCH 12/15] PMM-14912 Address linter findings on thresholds Run golangci-lint under the toolchain go.mod pins. The binary in bin/ is a linux build and the host Go is newer than 1.26.6, which made the linter fail to typecheck the stdlib and silently suppress every other check. Extract thresholdsForRule from ListThresholds, which was over the cognitive complexity limit, and move the scope conversion ahead of the transaction so a bad scope is rejected without taking a connection. Replace hand-numbered query placeholders with whereAllEqual, which derives the numbering from column order. The numbers previously sat in a format string while the values they referred to were passed positionally to a separate call, so inserting a column meant renumbering by hand and a mistake matched on the wrong columns rather than erroring. Mark the validation subtests parallel. TestThresholdBatchUpdate stays sequential at both levels: its subtests drive one override row through set, clear and rollback in order, and the rollback case asserts against the state the clear left behind. Reword the doc comments godot flagged. Its autofix capitalises the first word, which renamed unexported identifiers in their own comments. --- api-tests/alerting/thresholds_test.go | 52 +++++---- managed/models/alert_rule_helpers.go | 22 +++- managed/models/alert_rule_helpers_test.go | 6 +- managed/models/threshold_resolver.go | 14 ++- managed/models/threshold_resolver_test.go | 2 +- managed/pi/alert/overridable.go | 3 +- managed/pi/alert/singleexpr_test.go | 2 +- managed/services/alerting/reconciler.go | 12 +- managed/services/alerting/rule_builder.go | 2 +- .../alerting/rule_builder_dynamic_test.go | 2 +- managed/services/alerting/service.go | 4 +- .../alerting/service_threshold_test.go | 8 +- .../services/alerting/threshold_metrics.go | 27 +++-- .../alerting/threshold_metrics_test.go | 16 ++- .../services/alerting/threshold_overrides.go | 106 +++++++++++------- .../alerting/threshold_overrides_test.go | 2 +- 16 files changed, 169 insertions(+), 111 deletions(-) diff --git a/api-tests/alerting/thresholds_test.go b/api-tests/alerting/thresholds_test.go index 3d887d21b0..31f5dc5c3c 100644 --- a/api-tests/alerting/thresholds_test.go +++ b/api-tests/alerting/thresholds_test.go @@ -100,7 +100,8 @@ func setupThresholdFixture(t *testing.T) *thresholdFixture { folder := createdFolder.Payload t.Cleanup(func() { _, _ = gClient.Folders.DeleteFolder( - folders.NewDeleteFolderParams().WithFolderUID(folder.UID).WithForceDeleteRules(&forceDelete)) + folders.NewDeleteFolderParams().WithFolderUID(folder.UID).WithForceDeleteRules(&forceDelete), + ) }) created, err := client.CreateRule(&alerting.CreateRuleParams{ @@ -139,9 +140,9 @@ func (f *thresholdFixture) list(t *testing.T) []*alerting.ListThresholdsOKBodyTh t.Helper() res, err := f.client.ListThresholds(&alerting.ListThresholdsParams{ - Scope: pointer.ToString(scopeNode), - Target: pointer.ToString(f.nodeID), - RuleID: pointer.ToString(f.ruleID), + Scope: new(scopeNode), + Target: new(f.nodeID), + RuleID: new(f.ruleID), Context: pmmapitests.Context, }) require.NoError(t, err) @@ -154,7 +155,7 @@ func (f *thresholdFixture) set(t *testing.T, value float64) (*alerting.SetThresh return f.client.SetThreshold(&alerting.SetThresholdParams{ Body: alerting.SetThresholdBody{ - Scope: pointer.ToString(scopeNode), + Scope: new(scopeNode), Target: f.nodeID, RuleID: f.ruleID, ParamName: "threshold", @@ -192,10 +193,10 @@ func TestThresholdOverrideLifecycle(t *testing.T) { // tombstone so the emitted series keeps existing and merely changes value, but that // is invisible from here - what the API must report is "not overridden". _, err = f.client.ClearThreshold(&alerting.ClearThresholdParams{ - Scope: pointer.ToString(scopeNode), - Target: pointer.ToString(f.nodeID), - RuleID: pointer.ToString(f.ruleID), - ParamName: pointer.ToString("threshold"), + Scope: new(scopeNode), + Target: new(f.nodeID), + RuleID: new(f.ruleID), + ParamName: new("threshold"), Context: pmmapitests.Context, }) require.NoError(t, err) @@ -213,14 +214,18 @@ func TestThresholdOverrideValidation(t *testing.T) { f := setupThresholdFixture(t) t.Run("value outside the declared range", func(t *testing.T) { + t.Parallel() + _, err := f.set(t, 150) pmmapitests.AssertAPIErrorf(t, err, http.StatusBadRequest, codes.InvalidArgument, "") }) t.Run("unknown parameter", func(t *testing.T) { + t.Parallel() + _, err := f.client.SetThreshold(&alerting.SetThresholdParams{ Body: alerting.SetThresholdBody{ - Scope: pointer.ToString(scopeNode), Target: f.nodeID, + Scope: new(scopeNode), Target: f.nodeID, RuleID: f.ruleID, ParamName: "not-overridable", Value: 90, }, Context: pmmapitests.Context, @@ -229,9 +234,11 @@ func TestThresholdOverrideValidation(t *testing.T) { }) t.Run("unknown rule", func(t *testing.T) { + t.Parallel() + _, err := f.client.SetThreshold(&alerting.SetThresholdParams{ Body: alerting.SetThresholdBody{ - Scope: pointer.ToString(scopeNode), Target: f.nodeID, + Scope: new(scopeNode), Target: f.nodeID, RuleID: "no-such-rule", ParamName: "threshold", Value: 90, }, Context: pmmapitests.Context, @@ -240,9 +247,11 @@ func TestThresholdOverrideValidation(t *testing.T) { }) t.Run("target that does not exist", func(t *testing.T) { + t.Parallel() + _, err := f.client.SetThreshold(&alerting.SetThresholdParams{ Body: alerting.SetThresholdBody{ - Scope: pointer.ToString(scopeNode), Target: "no-such-node", + Scope: new(scopeNode), Target: "no-such-node", RuleID: f.ruleID, ParamName: "threshold", Value: 90, }, Context: pmmapitests.Context, @@ -253,10 +262,12 @@ func TestThresholdOverrideValidation(t *testing.T) { // Service and cluster are already carried by the schema, the resolver and the proto, // so they report as not-yet-implemented rather than as a malformed request. t.Run("scopes that are not implemented yet", func(t *testing.T) { + t.Parallel() + for _, scope := range []string{scopeService, scopeCluster} { _, err := f.client.SetThreshold(&alerting.SetThresholdParams{ Body: alerting.SetThresholdBody{ - Scope: pointer.ToString(scope), Target: f.nodeID, + Scope: new(scope), Target: f.nodeID, RuleID: f.ruleID, ParamName: "threshold", Value: 90, }, Context: pmmapitests.Context, @@ -266,16 +277,17 @@ func TestThresholdOverrideValidation(t *testing.T) { }) } +// TestThresholdBatchUpdate is deliberately not parallel, at either level: its subtests +// drive one override row through set, clear and rollback in that order, and the rollback +// case asserts against the state the clear case left behind. func TestThresholdBatchUpdate(t *testing.T) { - t.Parallel() - f := setupThresholdFixture(t) t.Run("sets through the batch endpoint", func(t *testing.T) { res, err := f.client.BatchUpdateThresholds(&alerting.BatchUpdateThresholdsParams{ Body: alerting.BatchUpdateThresholdsBody{ Updates: []*alerting.BatchUpdateThresholdsParamsBodyUpdatesItems0{{ - Scope: pointer.ToString(scopeNode), Target: f.nodeID, + Scope: new(scopeNode), Target: f.nodeID, RuleID: f.ruleID, ParamName: "threshold", Value: pointer.ToFloat64(70), }}, @@ -291,7 +303,7 @@ func TestThresholdBatchUpdate(t *testing.T) { res, err := f.client.BatchUpdateThresholds(&alerting.BatchUpdateThresholdsParams{ Body: alerting.BatchUpdateThresholdsBody{ Updates: []*alerting.BatchUpdateThresholdsParamsBodyUpdatesItems0{{ - Scope: pointer.ToString(scopeNode), Target: f.nodeID, + Scope: new(scopeNode), Target: f.nodeID, RuleID: f.ruleID, ParamName: "threshold", }}, }, @@ -312,12 +324,12 @@ func TestThresholdBatchUpdate(t *testing.T) { Body: alerting.BatchUpdateThresholdsBody{ Updates: []*alerting.BatchUpdateThresholdsParamsBodyUpdatesItems0{ { - Scope: pointer.ToString(scopeNode), Target: f.nodeID, + Scope: new(scopeNode), Target: f.nodeID, RuleID: f.ruleID, ParamName: "threshold", Value: pointer.ToFloat64(60), }, { - Scope: pointer.ToString(scopeNode), Target: f.nodeID, + Scope: new(scopeNode), Target: f.nodeID, RuleID: f.ruleID, ParamName: "threshold", Value: pointer.ToFloat64(500), // outside the declared range }, @@ -350,7 +362,7 @@ func TestThresholdOverrideRemovedWithNode(t *testing.T) { // With the node gone the override is unreachable by target, so ask for every // override of this rule instead. res, err := f.client.ListThresholds(&alerting.ListThresholdsParams{ - RuleID: pointer.ToString(f.ruleID), + RuleID: new(f.ruleID), Context: pmmapitests.Context, }) require.NoError(t, err) diff --git a/managed/models/alert_rule_helpers.go b/managed/models/alert_rule_helpers.go index 90b57d9eeb..c62d4829c4 100644 --- a/managed/models/alert_rule_helpers.go +++ b/managed/models/alert_rule_helpers.go @@ -18,6 +18,7 @@ package models import ( "errors" "fmt" + "strings" "github.com/google/uuid" "google.golang.org/grpc/codes" @@ -93,7 +94,7 @@ func FindThresholdOverridesByRule(q *reform.Querier, ruleID string) ([]*AlertRul return nil, status.Error(codes.InvalidArgument, "Empty rule ID.") } - return selectThresholdOverrides(q, "WHERE rule_id = "+q.Placeholder(1), ruleID) + return selectThresholdOverrides(q, whereAllEqual(q, "rule_id"), ruleID) } // FindThresholdOverridesByTarget returns every override row for one target, tombstones included. @@ -107,11 +108,23 @@ func FindThresholdOverridesByTarget(q *reform.Querier, scope ThresholdScope, tar return nil, status.Error(codes.InvalidArgument, "Empty target.") } - tail := fmt.Sprintf("WHERE scope = %s AND target = %s", q.Placeholder(1), q.Placeholder(2)) + tail := whereAllEqual(q, "scope", "target") return selectThresholdOverrides(q, tail, string(scope), target) } +// whereAllEqual builds a WHERE clause matching every named column, numbering the +// placeholders in column order. Callers pass their arguments in that same order, so +// the numbering cannot drift out of step with them the way hand-written placeholders can. +func whereAllEqual(q *reform.Querier, columns ...string) string { + conditions := make([]string, len(columns)) + for i, column := range columns { + conditions[i] = column + " = " + q.Placeholder(i+1) + } + + return "WHERE " + strings.Join(conditions, " AND ") +} + func selectThresholdOverrides(q *reform.Querier, tail string, args ...any) ([]*AlertRuleThresholdOverride, error) { structs, err := q.SelectAllFrom(AlertRuleThresholdOverrideTable, tail, args...) if err != nil { @@ -127,8 +140,7 @@ func selectThresholdOverrides(q *reform.Querier, tail string, args ...any) ([]*A } func findThresholdOverride(q *reform.Querier, ruleID, paramName string, scope ThresholdScope, target string) (*AlertRuleThresholdOverride, error) { - tail := fmt.Sprintf("WHERE rule_id = %s AND param_name = %s AND scope = %s AND target = %s", - q.Placeholder(1), q.Placeholder(2), q.Placeholder(3), q.Placeholder(4)) + tail := whereAllEqual(q, "rule_id", "param_name", "scope", "target") override := &AlertRuleThresholdOverride{} err := q.SelectOneTo(override, tail, ruleID, paramName, string(scope), target) @@ -289,7 +301,7 @@ func DeleteThresholdOverridesForTarget(q *reform.Querier, scope ThresholdScope, return status.Error(codes.InvalidArgument, "Empty target.") } - tail := fmt.Sprintf("WHERE scope = %s AND target = %s", q.Placeholder(1), q.Placeholder(2)) + tail := whereAllEqual(q, "scope", "target") _, err = q.DeleteFrom(AlertRuleThresholdOverrideTable, tail, string(scope), target) if err != nil { return fmt.Errorf("failed to delete threshold overrides: %w", err) diff --git a/managed/models/alert_rule_helpers_test.go b/managed/models/alert_rule_helpers_test.go index f4460fd579..cf5a0d7930 100644 --- a/managed/models/alert_rule_helpers_test.go +++ b/managed/models/alert_rule_helpers_test.go @@ -405,7 +405,7 @@ func TestThresholdOverridesFollowTargetRemoval(t *testing.T) { service, err := models.AddNewService(q, models.MySQLServiceType, &models.AddDBMSServiceParams{ ServiceName: "doomed-service", NodeID: node.NodeID, - Address: pointer.ToString("127.0.0.1"), + Address: new("127.0.0.1"), Port: pointer.ToUint16(3306), }) require.NoError(t, err) @@ -440,7 +440,7 @@ func TestThresholdOverridesFollowTargetRemoval(t *testing.T) { service, err := models.AddNewService(q, models.MySQLServiceType, &models.AddDBMSServiceParams{ ServiceName: "cascade-service", NodeID: node.NodeID, - Address: pointer.ToString("127.0.0.1"), + Address: new("127.0.0.1"), Port: pointer.ToUint16(3306), }) require.NoError(t, err) @@ -478,7 +478,7 @@ func TestThresholdOverridesFollowTargetRemoval(t *testing.T) { ServiceName: "clustered-service", NodeID: node.NodeID, Cluster: "prod", - Address: pointer.ToString("127.0.0.1"), + Address: new("127.0.0.1"), Port: pointer.ToUint16(3306), }) require.NoError(t, err) diff --git a/managed/models/threshold_resolver.go b/managed/models/threshold_resolver.go index ade1884737..8148eb572a 100644 --- a/managed/models/threshold_resolver.go +++ b/managed/models/threshold_resolver.go @@ -28,10 +28,18 @@ package models // label set is what makes `or` prefer the left operand, but that reduction is exactly // what destroys the scope information needed to rank by. So precedence is resolved here, // in Go, and there is no backstop in the query if this function is wrong. +// +// The iota order below is the precedence chain itself, narrowest last. +const ( + thresholdSpecificityCluster = iota + 1 + thresholdSpecificityNode + thresholdSpecificityService +) + var thresholdScopeSpecificity = map[ThresholdScope]int{ - ThresholdScopeService: 3, - ThresholdScopeNode: 2, - ThresholdScopeCluster: 1, + ThresholdScopeService: thresholdSpecificityService, + ThresholdScopeNode: thresholdSpecificityNode, + ThresholdScopeCluster: thresholdSpecificityCluster, } // ThresholdInventory maps override targets onto the join-label values an alert rule diff --git a/managed/models/threshold_resolver_test.go b/managed/models/threshold_resolver_test.go index aef2038fb0..043fee0d8b 100644 --- a/managed/models/threshold_resolver_test.go +++ b/managed/models/threshold_resolver_test.go @@ -239,7 +239,7 @@ func BenchmarkResolveThresholds(b *testing.B) { ServicesByCluster: make(map[string][]string, 50), } - var overrides []*AlertRuleThresholdOverride + overrides := make([]*AlertRuleThresholdOverride, 0, 1000) for i := range 1000 { id := fmt.Sprintf("node-id-%d", i) inv.NodeNames[id] = fmt.Sprintf("node-%d", i) diff --git a/managed/pi/alert/overridable.go b/managed/pi/alert/overridable.go index 85fb8910ae..bcab1c93d7 100644 --- a/managed/pi/alert/overridable.go +++ b/managed/pi/alert/overridable.go @@ -143,7 +143,8 @@ func (r *Template) validateOverridableParams() error { // the reason, rather than producing a rule whose threshold silently never applies. if len(overridable) > 1 { return fmt.Errorf( - "a single-expression template supports at most one overridable parameter, got %d", len(overridable)) + "a single-expression template supports at most one overridable parameter, got %d", len(overridable), + ) } _, err := SplitSingleExpr(r.Expr, param.Name) diff --git a/managed/pi/alert/singleexpr_test.go b/managed/pi/alert/singleexpr_test.go index ab3b934dcf..4b749243df 100644 --- a/managed/pi/alert/singleexpr_test.go +++ b/managed/pi/alert/singleexpr_test.go @@ -158,7 +158,7 @@ func TestSplitShippedSingleExprTemplates(t *testing.T) { splittable := 0 for _, file := range files { - b, err := os.ReadFile(file) //nolint:gosec + b, err := os.ReadFile(file) require.NoError(t, err) templates, err := Parse(strings.NewReader(string(b)), &ParseParams{ diff --git a/managed/services/alerting/reconciler.go b/managed/services/alerting/reconciler.go index a42f1fefb4..7e7ffd20b1 100644 --- a/managed/services/alerting/reconciler.go +++ b/managed/services/alerting/reconciler.go @@ -25,14 +25,14 @@ import ( ) const ( - // reconcileInterval is how often orphaned registry rows are reaped. Orphans are - // inert rather than harmful - the collector emits nothing for a rule that is gone - - // so this trades promptness for staying out of the way. + // How often orphaned registry rows are reaped. Orphans are inert rather than + // harmful - the collector emits nothing for a rule that is gone - so this trades + // promptness for staying out of the way. reconcileInterval = 15 * time.Minute - // reconcileGracePeriod keeps a freshly created row safe from the sweep. CreateRule - // writes the registry row before the rule exists in Grafana, so without this a sweep - // landing in that window would delete the row of a rule being created successfully. + // Keeps a freshly created row safe from the sweep. CreateRule writes the registry + // row before the rule exists in Grafana, so without this a sweep landing in that + // window would delete the row of a rule being created successfully. reconcileGracePeriod = 10 * time.Minute ) diff --git a/managed/services/alerting/rule_builder.go b/managed/services/alerting/rule_builder.go index e8ad76ddf6..f58997817b 100644 --- a/managed/services/alerting/rule_builder.go +++ b/managed/services/alerting/rule_builder.go @@ -36,7 +36,7 @@ const ( queryIntervalMs = 1000 maxDataPoints = 43200 - // thresholdRefIDPrefix prefixes the ref ID of each injected threshold query. + // Prefixes the ref ID of each injected threshold query. thresholdRefIDPrefix = "T_" // Ref IDs used when a single-expression template is desugared. A multi-expression diff --git a/managed/services/alerting/rule_builder_dynamic_test.go b/managed/services/alerting/rule_builder_dynamic_test.go index 0ba55a13b6..0967873269 100644 --- a/managed/services/alerting/rule_builder_dynamic_test.go +++ b/managed/services/alerting/rule_builder_dynamic_test.go @@ -190,7 +190,7 @@ func TestThresholdQueryMatchesCollectorDescriptor(t *testing.T) { assert.Contains(t, expr, fqName[1]+"{", "the query must select the metric the collector registers") - for _, label := range strings.Split(labels[1], ",") { + for label := range strings.SplitSeq(labels[1], ",") { label = strings.TrimSpace(label) require.NotEmpty(t, label) assert.Contains(t, expr, label, "the query must reference every label the collector emits") diff --git a/managed/services/alerting/service.go b/managed/services/alerting/service.go index abdaa515db..b84e6c321a 100644 --- a/managed/services/alerting/service.go +++ b/managed/services/alerting/service.go @@ -875,7 +875,9 @@ func (s *Service) deleteRuleRegistration(ruleID string) { func collectOverridableParams(template *alert.Template, values AlertExprParamsValues) (models.AlertRuleParams, error) { overridable := template.OverridableParams() if len(overridable) == 0 { - return nil, nil + // A nil AlertRuleParams map is the valid "nothing to snapshot" result: the rule + // is registered without params rather than being an error. + return nil, nil //nolint:nilnil } byName := make(map[string]AlertExprParamValue, len(values)) diff --git a/managed/services/alerting/service_threshold_test.go b/managed/services/alerting/service_threshold_test.go index 8571b68531..9aded8d466 100644 --- a/managed/services/alerting/service_threshold_test.go +++ b/managed/services/alerting/service_threshold_test.go @@ -133,7 +133,7 @@ func TestCreateRuleRegistersOverridableRule(t *testing.T) { var captured *services.Rule m.On("CreateAlertRule", mock.Anything, "folder-uid", "test-group", mock.Anything, mock.Anything). - Run(func(args mock.Arguments) { captured = args.Get(4).(*services.Rule) }). //nolint:forcetypeassert + Run(func(args mock.Arguments) { captured = args.Get(4).(*services.Rule) }). Return(nil) res, err := svc.CreateRule(ctx, &alerting.CreateRuleRequest{ @@ -178,7 +178,7 @@ func TestCreateRuleRegistersOverridableRule(t *testing.T) { var captured *services.Rule m.On("CreateAlertRule", mock.Anything, "folder-uid", "test-group", mock.Anything, mock.Anything). - Run(func(args mock.Arguments) { captured = args.Get(4).(*services.Rule) }). //nolint:forcetypeassert + Run(func(args mock.Arguments) { captured = args.Get(4).(*services.Rule) }). Return(nil) _, err := svc.CreateRule(ctx, &alerting.CreateRuleRequest{ @@ -206,7 +206,7 @@ func TestCreateRuleRegistersOverridableRule(t *testing.T) { var captured *services.Rule m.On("CreateAlertRule", mock.Anything, "folder-uid", "test-group", mock.Anything, mock.Anything). - Run(func(args mock.Arguments) { captured = args.Get(4).(*services.Rule) }). //nolint:forcetypeassert + Run(func(args mock.Arguments) { captured = args.Get(4).(*services.Rule) }). Return(nil) before, err := models.FindAlertRules(db.Querier) @@ -318,7 +318,7 @@ func TestBuiltInOverridableTemplates(t *testing.T) { var got []string for _, file := range files { - b, err := os.ReadFile(file) //nolint:gosec + b, err := os.ReadFile(file) require.NoError(t, err) templates, err := alert.Parse(strings.NewReader(string(b)), &alert.ParseParams{ diff --git a/managed/services/alerting/threshold_metrics.go b/managed/services/alerting/threshold_metrics.go index 925ead357d..29a41dfba7 100644 --- a/managed/services/alerting/threshold_metrics.go +++ b/managed/services/alerting/threshold_metrics.go @@ -27,28 +27,27 @@ import ( ) const ( - // thresholdCollectTimeout bounds one scrape. The threshold collector shares - // /debug/metrics, and its 0.9 * MR budget, with the inventory and HA collectors, so - // overrunning here would take those down too. + // Bounds one scrape. The threshold collector shares /debug/metrics, and its + // 0.9 * MR budget, with the inventory and HA collectors, so overrunning here + // would take those down too. thresholdCollectTimeout = 3 * time.Second - // thresholdCtxCheckInterval is how often the emission loop re-checks the deadline. - // The queries are bounded by the context, but the loop that follows them is not, - // so without this a large enough result set could run past the scrape budget with - // nothing stopping it. + // How often the emission loop re-checks the deadline. The queries are bounded by + // the context, but the loop that follows them is not, so without this a large + // enough result set could run past the scrape budget with nothing stopping it. thresholdCtxCheckInterval = 1000 - // thresholdMetricName is the gauge the injected threshold query reads. It is shared - // with rule_builder.go on purpose: the metric name and its label set are a contract - // between the collector and the generated PromQL, and a rule pointing at a metric - // nobody emits fails silently - it simply never fires. + // The gauge the injected threshold query reads. Shared with rule_builder.go on + // purpose: the metric name and its label set are a contract between the collector + // and the generated PromQL, and a rule pointing at a metric nobody emits fails + // silently - it simply never fires. thresholdMetricName = "pmm_alert_threshold_override" thresholdRuleIDLabel = "rule_id" thresholdParamLabel = "param" - // thresholdTargetLabel is generic rather than node_name/service_name because one - // fixed descriptor has to serve every scope; the rule query maps it onto whichever - // label it joins on with label_replace. + // Generic rather than node_name/service_name because one fixed descriptor has to + // serve every scope; the rule query maps it onto whichever label it joins on + // with label_replace. thresholdTargetLabel = "target" ) diff --git a/managed/services/alerting/threshold_metrics_test.go b/managed/services/alerting/threshold_metrics_test.go index a197ff762d..233db6056c 100644 --- a/managed/services/alerting/threshold_metrics_test.go +++ b/managed/services/alerting/threshold_metrics_test.go @@ -42,7 +42,9 @@ func thresholdExposition(t *testing.T, c *AlertThresholdMetricsCollector, sample start := strings.Index(desc, `help: "`) require.GreaterOrEqual(t, start, 0) help := desc[start+len(`help: "`):] - help = help[:strings.Index(help, `"`)] + end := strings.Index(help, `"`) + require.GreaterOrEqual(t, end, 0) + help = help[:end] body := "\n# HELP " + thresholdMetricName + " " + help + "\n# TYPE " + thresholdMetricName + " gauge\n" + @@ -119,9 +121,11 @@ func createThresholdRule(t *testing.T, db *reform.DB) { require.NoError(t, err) } -func createThresholdNode(t *testing.T, db *reform.DB, name string) *models.Node { +func createThresholdNode(t *testing.T, db *reform.DB) *models.Node { t.Helper() + const name = "node-1" + node, err := models.CreateNode(db.Querier, models.GenericNodeType, &models.CreateNodeParams{ NodeName: name, Address: name + ".example.com", @@ -142,7 +146,7 @@ func TestThresholdCollectorEmitsNothingWithoutOverrides(t *testing.T) { func TestThresholdCollectorEmitsOverride(t *testing.T) { c, db := setupThresholdCollector(t) createThresholdRule(t, db) - node := createThresholdNode(t, db, "node-1") + node := createThresholdNode(t, db) _, err := models.UpsertThresholdOverride(db.Querier, testRuleID, "threshold", models.ThresholdScopeNode, node.NodeID, 90) require.NoError(t, err) @@ -159,7 +163,7 @@ func TestThresholdCollectorEmitsOverride(t *testing.T) { func TestThresholdCollectorEmitsDefaultForTombstone(t *testing.T) { c, db := setupThresholdCollector(t) createThresholdRule(t, db) - node := createThresholdNode(t, db, "node-1") + node := createThresholdNode(t, db) _, err := models.UpsertThresholdOverride(db.Querier, testRuleID, "threshold", models.ThresholdScopeNode, node.NodeID, 90) require.NoError(t, err) @@ -187,7 +191,7 @@ func TestThresholdCollectorSkipsDeletedTarget(t *testing.T) { func TestThresholdCollectorSkipsUnknownParam(t *testing.T) { c, db := setupThresholdCollector(t) createThresholdRule(t, db) - node := createThresholdNode(t, db, "node-1") + node := createThresholdNode(t, db) _, err := models.UpsertThresholdOverride(db.Querier, testRuleID, "gone", models.ThresholdScopeNode, node.NodeID, 90) require.NoError(t, err) @@ -207,7 +211,7 @@ func TestThresholdCollectorEmitsOnePerTargetAcrossParams(t *testing.T) { }) require.NoError(t, err) - node := createThresholdNode(t, db, "node-1") + node := createThresholdNode(t, db) for _, param := range []string{"threshold", "second"} { _, err = models.UpsertThresholdOverride(db.Querier, testRuleID, param, models.ThresholdScopeNode, node.NodeID, 42) require.NoError(t, err) diff --git a/managed/services/alerting/threshold_overrides.go b/managed/services/alerting/threshold_overrides.go index 238266b8db..d4b058aa83 100644 --- a/managed/services/alerting/threshold_overrides.go +++ b/managed/services/alerting/threshold_overrides.go @@ -185,62 +185,31 @@ func (s *Service) ListThresholds(_ context.Context, req *alerting.ListThresholds return nil, services.ErrAlertingDisabled } + // Converted before the transaction opens: it only reads the request, and a bad scope + // should be rejected without taking a connection. + var scope models.ThresholdScope + if req.Target != "" { + scope, err = thresholdScopeFromAPI(req.Scope) + if err != nil { + return nil, err + } + } + var thresholds []*alerting.Threshold errTx := s.db.InTransaction(func(tx *reform.TX) error { - var scope models.ThresholdScope - if req.Target != "" { - scope, err = thresholdScopeFromAPI(req.Scope) - if err != nil { - return err - } - } - rules, err := s.thresholdRules(tx.Querier, req.RuleId) if err != nil { return err } for _, rule := range rules { - overrides, err := models.FindThresholdOverridesByRule(tx.Querier, rule.RuleID) + ruleThresholds, err := s.thresholdsForRule(tx.Querier, rule, scope, req.Target) if err != nil { return err } - inv, err := loadThresholdInventory(tx.Querier, overrides) - if err != nil { - return err - } - - targetName, err := s.thresholdTargetName(tx.Querier, scope, req.Target, &inv) - if err != nil { - return err - } - - for paramName, param := range rule.Params { - resolved := models.ResolveThresholdsDetailed( - filterOverridesByParam(overrides, paramName), param.Default, inv, - ) - - if req.Target == "" { - // With no target there is no bounded set of targets to enumerate, so - // only what has actually been overridden is reported. - for _, entry := range resolved { - if entry.IsOverridden() { - thresholds = append(thresholds, thresholdFromResolved(rule.RuleID, paramName, param, entry)) - } - } - - continue - } - - entry, ok := resolved[targetName] - if !ok { - entry = models.ResolvedThreshold{Value: param.Default} - } - - thresholds = append(thresholds, thresholdFromResolved(rule.RuleID, paramName, param, entry)) - } + thresholds = append(thresholds, ruleThresholds...) } return nil @@ -254,6 +223,57 @@ func (s *Service) ListThresholds(_ context.Context, req *alerting.ListThresholds return &alerting.ListThresholdsResponse{Thresholds: thresholds}, nil } +// thresholdsForRule reports one registry row's thresholds. With no target it reports only +// what has actually been overridden; with a target it reports every parameter of the rule, +// falling back to that rule's own default where nothing overrides it. +func (s *Service) thresholdsForRule( + q *reform.Querier, rule *models.AlertRule, scope models.ThresholdScope, target string, +) ([]*alerting.Threshold, error) { + overrides, err := models.FindThresholdOverridesByRule(q, rule.RuleID) + if err != nil { + return nil, err + } + + inv, err := loadThresholdInventory(q, overrides) + if err != nil { + return nil, err + } + + targetName, err := s.thresholdTargetName(q, scope, target, &inv) + if err != nil { + return nil, err + } + + var thresholds []*alerting.Threshold + + for paramName, param := range rule.Params { + resolved := models.ResolveThresholdsDetailed( + filterOverridesByParam(overrides, paramName), param.Default, inv, + ) + + if target == "" { + // With no target there is no bounded set of targets to enumerate, so + // only what has actually been overridden is reported. + for _, entry := range resolved { + if entry.IsOverridden() { + thresholds = append(thresholds, thresholdFromResolved(rule.RuleID, paramName, param, entry)) + } + } + + continue + } + + entry, ok := resolved[targetName] + if !ok { + entry = models.ResolvedThreshold{Value: param.Default} + } + + thresholds = append(thresholds, thresholdFromResolved(rule.RuleID, paramName, param, entry)) + } + + return thresholds, nil +} + // thresholdRules returns the registry rows to report on, honouring an optional filter. func (s *Service) thresholdRules(q *reform.Querier, ruleID string) ([]*models.AlertRule, error) { if ruleID != "" { diff --git a/managed/services/alerting/threshold_overrides_test.go b/managed/services/alerting/threshold_overrides_test.go index 70879d4a86..9025ecc96b 100644 --- a/managed/services/alerting/threshold_overrides_test.go +++ b/managed/services/alerting/threshold_overrides_test.go @@ -45,7 +45,7 @@ func setupThresholdAPI(t *testing.T) (*Service, *reform.DB, *models.Node) { require.NoError(t, err) // Alerting must be on, or every RPC short-circuits. - _, err = models.UpdateSettings(db, &models.ChangeSettingsParams{EnableAlerting: pointer.ToBool(true)}) + _, err = models.UpdateSettings(db, &models.ChangeSettingsParams{EnableAlerting: new(true)}) require.NoError(t, err) _, err = models.CreateAlertRule(db.Querier, &models.CreateAlertRuleParams{ From ca1ece4a40ca896d116d2b9d087131e95e9e2335 Mon Sep 17 00:00:00 2001 From: Matej Kubinec Date: Wed, 2 Sep 2026 16:12:55 +0200 Subject: [PATCH 13/15] PMM-14912 Remove dynamic thresholds design documents The design and performance-testing notes were working documents for building the feature, not reference material for maintaining it. The reasoning they carried that still matters lives as comments on the code it explains; the rest stays in git history. --- dynamic-thresholds-main-decision.md | 1135 --------------------- dynamic-thresholds-performance-testing.md | 356 ------- 2 files changed, 1491 deletions(-) delete mode 100644 dynamic-thresholds-main-decision.md delete mode 100644 dynamic-thresholds-performance-testing.md diff --git a/dynamic-thresholds-main-decision.md b/dynamic-thresholds-main-decision.md deleted file mode 100644 index 2514f9fa34..0000000000 --- a/dynamic-thresholds-main-decision.md +++ /dev/null @@ -1,1135 +0,0 @@ -# Dynamic Alert Thresholds — decision and implementation plan (against `main`) - -Per-node (and later per-service, per-cluster) threshold overrides for alert rules created from -templates, specified against **`main` at `a807f56d8`**. Every file and line reference below was checked -against `main`, not against any feature branch. - -**Scope of this document** - -| In scope | Out of scope | -|---|---| -| Migration, reform models, CRUD helpers | UI (thresholds modal, hooks, Grafana-side trigger) | -| Threshold metric collector | Dedicated metrics endpoint + own scrape job (costed as a follow-up) | -| Rule builder: threshold-query injection and single-expression desugaring | Grafana `UpdateRule`/`DeleteRule` RPCs | -| gRPC/REST API for reading and writing overrides | Service and cluster **behaviour** (schema and API are generalised now; only node scope is implemented) | -| Reconciler for orphaned rule registry rows | | - -Written greenfield: it assumes nothing exists beyond `main`. - -> **On evidence.** Where this document states a fact as *measured*, it was measured on a live PMM -> server (Grafana 12.4.5, VictoriaMetrics v1.147.0). Where it *extrapolates* from those measurements, it -> says so. Unverified assumptions are listed in §8, and decisions still needed are in §9. ---- - -## Summary - -**Verdict.** Per-target threshold overrides are delivered as **data, not as rule edits**: the Grafana -rule is written once at creation and never rewritten, and changing a threshold is a single Postgres row. -Postgres is the source of truth, VictoriaMetrics is a derived transport carrying **only the overrides**, -and the default is fanned out at query time over PMM's existing node-inventory metric. This keeps -tuning one target from disturbing alert state on every other target of the same rule, which is the -failure mode that rules out the obvious alternative of rendering thresholds into the query. Estimated -3–5 weeks for the node-only increment; the schema and API are generalised for service and cluster scope -from the start, because routes and proto fields are additive-only. - -### How it fits together - -``` - PMM UI / API pmm-managed VictoriaMetrics Grafana - ──────────── ─────────── ─────────────── ─────── - set threshold ──POST──▶ ┌─────────────────────┐ - │ alert_rule_ │ - │ threshold_overrides│◀── canonical - └──────────┬──────────┘ - │ read once per scrape - ▼ - ┌─────────────────────┐ scrape ┌──────────────────┐ - │ threshold collector │────────────▶│ pmm_alert_ │ - │ (overrides only) │ /debug/ │ threshold_ │ - └─────────────────────┘ metrics │ override │ - └────────┬─────────┘ - ┌─────────────────────┐ scrape ┌────────┴─────────┐ - │ inventory collector │────────────▶│ pmm_managed_ │ - │ (already exists) │ │ inventory_nodes │ - └─────────────────────┘ └────────┬─────────┘ - │ - T_ reads both - ▼ - ┌──────────────────┐ - │ alert rule │──▶ Alertmanager - │ A / T / C │ - └──────────────────┘ -``` - -The rule never changes after creation. The only write path for a threshold change is the leftmost arrow. - -### Where state lives, and what is authoritative - -``` - ┌──────────────────────────┐ ┌──────────────────────────┐ ┌──────────────────────────┐ - │ PostgreSQL │ │ VictoriaMetrics │ │ Grafana rule │ - ├──────────────────────────┤ ├──────────────────────────┤ ├──────────────────────────┤ - │ AUTHORITATIVE for │ │ DERIVED transport │ │ AUTHORITATIVE for │ - │ · override values │ │ · one series per │ │ · which rules exist │ - │ · per-param snapshot │ │ override (tens) │ │ · title, folder, group │ - │ (default, join label, │ │ · rebuilt every scrape │ │ · template_name label │ - │ scopes, unit, range) │ │ · never authoritative │ │ DERIVED │ - │ │ │ │ │ · default as a literal │ - │ Lose it → overrides gone │ │ Lose it → falls back to │ │ · rule_id label │ - │ │ │ defaults, keeps alerting │ │ Written once │ - └──────────────────────────┘ └──────────────────────────┘ └──────────────────────────┘ -``` - -Nothing needs both stores to agree to be correct: a threshold missing from VictoriaMetrics degrades to -the rule's default rather than to silence, and a Postgres row whose rule or target no longer exists is -inert because the emit path resolves it away. - -### For implementers — the shape to build - -- **Metric:** `pmm_alert_threshold_override{rule_id, param, target}`, one series per override, value = effective threshold after precedence is resolved **in Go** (§4.6). -- **Rule:** three steps — `A` observed, `T_` threshold, `C` math `$A > $T_`. `T_` is **always emitted**, even with no overrides (§5.1). -- **Threshold expression:** `max by () (label_replace(override…)) or (max by () () * 0 + )` — every clause justified in §4.3. The default fans out over the **observed expression**, not over an inventory metric, so no `last_over_time` and no window (§4.7). -- **Clearing:** tombstone the row, never delete it — 14–21 s instead of 309 s (§4.4). -- **Schema:** `alert_rules` (5 columns) + `alert_rule_threshold_overrides` keyed `(rule_id, param_name, scope, target)` — §4.4. -- **Cleanup:** override rows are deleted by the node/service removal API, not a sweep (§4.8); the reconciler handles only registry rows for rules deleted in Grafana (§4.9). -- **Ten implementation steps with file references and effort: §6.** - ---- - -## 1. The problem - -Alert rules created from templates bake the threshold into the query (`$A > 80`). Changing the threshold -for one node means editing or recreating the rule. We want a per-node override that is a **data change**, -not a rule change. - -## 2. Recommendation in one paragraph - -Emit a **VictoriaMetrics gauge carrying only the overrides**, materialise the **default at query time by -fanning out over the rule's own observed expression**, and have the rule compare its observed query -against that combination. The Grafana rule is written **once at creation and never rewritten**, so tuning -a threshold never disturbs alert state. Postgres is the single source of truth; VictoriaMetrics is a -derived transport. - -```promql -# the injected threshold step, T_ -max by (node_name) ( - label_replace(pmm_alert_threshold_override{rule_id="", param=""}, - "node_name", "$1", "target", "(.*)") -) -or -(max by (node_name) () * 0 + ) -``` - -Two properties of this shape are load-bearing, and both are measured (§8): - -- **The default is fanned out over the observed expression, not over an inventory metric.** The threshold - can then never outlive or predecease `$A`, because both read the same series. This removes the - 15-minute silent-stop cliff *structurally* rather than bounding it, and it deletes `last_over_time` - from the query — there is no window left to tune. Cost: the observed expression is evaluated twice, - measured at **~1.25×**, not 2×. -- **Clearing an override is a value change, never a disappearance.** The row is tombstoned rather than - deleted (§4.4), so the series keeps being emitted and simply carries the resolved default. Measured - **14–21 s**, against **309 s** when a clear is signalled by absence. - -## 3. What `main` already provides - -This is the part that makes the estimate small. On `main` today: - -| Foundation | Where | -|---|---| -| **Multi-expression templates** (`queries:` / `expressions:` / `condition:`) | `managed/pi/alert/query.go:25` (`TemplateQuery`), `:38` (`UsesMultipleExpressions`), `:55`–`:131` (validation) | -| Rule builder with a multi-expression path | `managed/services/alerting/rule_builder.go:58` (`buildGrafanaRuleData`), `:81` (`buildMultiExpressionRuleData`) | -| Prom-query and math-expression step builders | `rule_builder.go:141` (`newPromQueryData`), `:165` (`newMathExpressionData`) | -| Filter application | `rule_builder.go:120` (`fillAndFilterExpr`) | -| Rule creation flow | `managed/services/alerting/service.go:679` (`CreateRule`), annotations filled at `:744` | -| Grafana rule creation | `managed/services/grafana/client.go:712` (`CreateAlertRule`) | -| **A per-node inventory metric, already scraped** | `managed/services/inventory/inventory_metrics.go:77-80` → `pmm_managed_inventory_nodes{node_id, node_type, node_name, container_name}` | -| A per-service/agent inventory metric | `inventory_metrics.go:71-75` → `pmm_managed_inventory_agents{…, service_id, service_name, node_id, node_name, …}` | -| Collector registration + `/debug/metrics` exposition | `managed/cmd/pmm-managed/main.go:942`, `debugAddr` at `:123` | -| That endpoint already scraped by VM | `managed/services/victoriametrics/scrape_configs.go:80` (job `pmm-managed`, interval `MR` = 10 s per `managed/models/settings.go:209`, timeout `0.9 × MR` per `scrape_configs.go:41`) | -| Stable identifiers on observed series | `Node.UnifiedLabels()` `managed/models/node_model.go:118`, `Service.UnifiedLabels()` `managed/models/service_model.go:115` | -| Leader-election hook for background work | `managed/services/ha/haservice.go:569` (`AddLeaderService`), `:597` (`IsLeader` — true when HA is disabled) | -| Migrations up to **118** | `managed/models/database.go:1184` → this feature takes **119** | - -What is **absent** on `main` and must be built: the `overridable` param flag, both tables, the collector, -threshold-query injection, the API, the reconciler, and (per the scope decision) single-expression -desugaring. `main` has **no** `UpdateAlertRule`, `ListAlertRules` or `DeleteAlertRule` — and the -recommended design needs only `ListAlertRules`, for the reconciler. - -Template inventory on `main`: **42 templates — 38 single-expression, 4 multi-expression.** -`pmm_node_high_cpu_load` is already multi-expression, so it becomes overridable with one YAML line. - -## 4. Design - -### 4.1 Rule shape - -Every overridable rule compiles to the same steps, whatever the template looked like: - -| refId | datasource | body | -|---|---|---| -| `A`, `B`, … | Metrics (VM) | the template's observed queries, unchanged | -| `T_` | Metrics (VM) | the threshold expression from §2, one per overridable param | -| `C` | `__expr__` math | the template's expression, with `[[ .param ]]` swapped for `$T_` | - -Written once. A threshold change touches one Postgres row and nothing in Grafana. - -### 4.2 Metric shape - -``` -pmm_alert_threshold_override{rule_id, param, target} # gauge -``` - -Exactly three labels. Value = the effective threshold for that target, in the param's native unit, -**after precedence resolution in Go**. - -| Label | Why | -|---|---| -| `rule_id` | Scopes the selector so one rule cannot pick up another's series | -| `param` | Required for multi-param rules | -| `target` | The **value of the rule's join label** (`node_name` / `service_name` / cluster value), resolved in Go from the ID stored in Postgres | - -Excluded deliberately: **`scope`** (precedence cannot be expressed in PromQL — §4.6), the join label by -name (its *name* varies per rule and a fixed `prom.NewDesc` cannot vary label names, so generic `target` -plus one `label_replace` keeps a **checked** collector), the default value (a literal in the rule), -and rule metadata (`template_name`, `rule_title` — they churn on rename; see §4.4 for why they are not -stored at all). - -```go -desc: prom.NewDesc( - "pmm_alert_threshold_override", - "Effective alert threshold override for a rule parameter and target. Emitted only where an "+ - "override or a tombstone exists; targets without either fall back to the rule's default, "+ - "which the rule query materialises by fanning out over its own observed expression.", - []string{"rule_id", "param", "target"}, - nil, -), -``` - -Implement `Describe` directly (`ch <- c.desc`) rather than via `prom.DescribeByCollect`, which would run -a full `Collect` — and therefore a database query — merely to describe the collector. - -**Cardinality:** one series per `(rule, param, target-covered-by-an-override-or-tombstone)`. Because -precedence is resolved in Go, a coarse-scope override **expands** — a cluster override over 200 nodes -would emit 200 series. Bounded by "targets actually covered, plus targets ever tuned", which is why this -never approaches `rules × params × nodes`. That bound is the whole point: §4.12 measures what happens when -it is removed. - -### 4.3 Why each clause of the threshold query - -| Clause | Job | -|---|---| -| `label_replace(…, "", "$1", "target", "(.*)")` | Maps generic `target` onto whichever label this rule joins on. One fixed descriptor serves every scope. | -| `max by ()` | Three jobs: **strips `instance`/`job`** (the scrape target is `pmm-server`/`pmm-managed`, which can never match an observed series' `instance`); **reduces both sides of `or` to identical label sets**, without which `or` returns both instead of preferring the left; and **collapses HA duplicates**. Measured cost: none — `samplesScanned` is identical with and without it. | -| `or` | Set union preferring the left: "override if present, else default". Requires identical label sets, which `max by` guarantees. | -| `* 0 + ` | Preserves the label set, replaces the value → one default series per target **that has observed data**. | - -> **There is no `last_over_time` in this query, and no window to choose.** An earlier revision fanned the -> default out over `pmm_managed_inventory_nodes` and needed `last_over_time(…[15m])` to keep it resolving -> across a pmm-managed restart. That made the window a *reliability* parameter and produced the cliff -> in §4.7. Fanning out over the observed expression removes the need entirely: if `$A` resolves, so does -> `T`. Verified live — see §8. - -> **Do not resurrect `samplesScanned` as the cost argument.** A previous revision justified `[15m]` with -> per-series scan counts (bare **61**, `[15m]` **91**, `[1h]` **361**, `[7d]` **11,246**) taken on VM -> v1.147.0. On **v1.149.0**, which PMM 3.7.1 ships, that counter under-reports and cannot distinguish the -> functions: `count_over_time(…[1h])` provably reads all 360 samples per series yet reports the **same -> 151** as `last_over_time(…[1h])`. Only `bare = 61` still reproduces. Cost claims must come from -> wall-clock or `vm_rows_read_per_query`, not from `samplesScanned`. - -> **Window length turned out not to matter anyway.** Measured at 1000 nodes with 2 h of history: -> `[15m]` **3.8 ms**, `[2h]` **4.5 ms**, bare **3.8 ms** — against a 60 s evaluation interval. Any scan -> argument for or against a window is noise at PMM's scale; decide these questions on failure modes. - -> **Never wrap the override series in `last_over_time`.** With tombstones (§4.4) a cleared override is a -> *value change*, so a window is unnecessary; and if a row is ever hard-deleted, a window would keep the -> deleted value resolving for its whole length. - -### 4.4 Postgres schema (migration 119) - -Generalised for the confirmed roadmap. The decisive constraint: **`cluster` is a label value, not an -entity** — there is no `clusters` table, so it can never have a foreign key, and the same holds for -`environment`, `replication_set` and custom labels. Keeping FKs for node/service but not cluster would -force per-scope branching through every query and handler, so the target column is polymorphic. - -```sql -CREATE TABLE alert_rules ( - rule_id VARCHAR NOT NULL, -- PMM-minted; the identity - grafana_rule_uid VARCHAR CHECK (grafana_rule_uid <> ''), -- cached handle, NOT identity - params JSONB NOT NULL, -- see below - created_at TIMESTAMP NOT NULL, - updated_at TIMESTAMP NOT NULL, - PRIMARY KEY (rule_id), - UNIQUE (grafana_rule_uid) -); - -CREATE TABLE alert_rule_threshold_overrides ( - id VARCHAR NOT NULL, - rule_id VARCHAR NOT NULL REFERENCES alert_rules (rule_id) ON DELETE CASCADE, - param_name VARCHAR NOT NULL CHECK (param_name <> ''), - scope VARCHAR NOT NULL CHECK (scope <> ''), -- 'node' | 'service' | 'cluster' - target VARCHAR NOT NULL CHECK (target <> ''), -- node_id | service_id | cluster label value - value DOUBLE PRECISION NOT NULL, - cleared_at TIMESTAMP, -- non-NULL => tombstone, see below - created_at TIMESTAMP NOT NULL, - updated_at TIMESTAMP NOT NULL, - PRIMARY KEY (id), - UNIQUE (rule_id, param_name, scope, target) -); - -CREATE INDEX alert_rule_threshold_overrides_target_idx - ON alert_rule_threshold_overrides (scope, target); -``` - -`CHECK (x <> '')` follows repo convention (55 such constraints in `database.go`). **No -`CHECK (scope IN (…))`** — validate the enum in Go so adding a scope needs no migration. - -#### Clearing is a tombstone, not a delete - -`cleared_at` exists because **absence is a slow signal**. If clearing an override deletes the row, the -series stops being emitted, and a stopped series stays queryable for VM's whole lookbehind — measured -**309 s** end to end through a real rule. If clearing instead marks the row, the series continues and its -*value* changes, which is visible on the next scrape: measured **14–21 s**. Same rule, same environment, -~15× apart (§8). - -Three rules make this safe: - -1. **The tombstone stores no value.** `value` keeps whatever the override was, for audit; the collector - ignores it once `cleared_at` is set and resolves the default from `params[p].default` instead. Storing - the default here instead would silently freeze cleared targets at the *old* default the next time a - default changes — and changing a default is a supported operation (§4.4), so this is a real defect, not - a hypothetical. -2. **The resolver decides, not the writer.** The emitted target set widens from "targets with an override" - to "targets with an override **or** a tombstone"; the §4.10 resolver then runs unchanged. For a - tombstoned target it recomputes across the remaining scopes and falls back to the default **only if - nothing survives**. So clearing a node override that sits under a cluster override correctly yields the - cluster value, not the default — no new precedence logic. -3. **Tombstones are never garbage-collected.** Once the value is not stored, a retained tombstone - self-corrects on a default change. And sweeping would be behaviourally invisible anyway, since the - tombstone's emitted value is identical to the fan-out default it would fall through to — so a sweeper - buys nothing but a leader-election question. Growth is bounded by *targets ever tuned*, not by - inventory. - -`ListNodeThresholds` and the UI **must** treat `cleared_at IS NOT NULL` as "not overridden", or every -target ever tuned will read as tuned forever. That is the likeliest place for this to ship a bug. - -#### The `params` snapshot - -Rule metadata that Grafana already holds — title, folder, rule group, template name — is **not** stored; -it is read from the rule, which is authoritative (§4.4). `params` is the one thing that is irreducible. It -is a snapshot, keyed by param name: - -```json -{ - "threshold": { - "default": 80, - "join_label": "node_name", - "scopes": ["node"], - "unit": "%", - "summary": "A percentage from configured maximum", - "min": 0, - "max": 100 - } -} -``` - -Nothing else can supply it. The default *is* in the rule query, but as a PromQL literal, and recovering it -means parsing generated text — rejected in §7. Taking it from the *current* template instead lets the API -report a default the rule does not actually use, if the template was edited after the rule was created. -And it cannot live on the overrides table, because a param with **no** override has no row there. - -#### This is also what makes changing a default possible later - -Because the effective default is stored *and* rendered into the rule, the two can be reconciled: update -`params[p].default`, re-render the `T_` step, `PUT` the rule. It is deliberately **not cheap** — a -rule-definition change resets alert state for that rule (§7) — but that is acceptable for a rare, -deliberate administrative act, unlike per-target tuning. Without the snapshot there would be nothing to -change *from*, only a literal buried in a query. - -If dynamic defaults ever become routine rather than rare, the answer is not to optimise this path but to -move the default into data as well — see §10, which treats that as the trigger to revisit the -thresholds-datasource alternative. - -#### How `scope` and `target` are used - -`scope` and `target` are not just descriptive — together they drive three different code paths. - -**Write path (API).** `scope` says which kind of thing is being targeted, `target` identifies it. -`SetNodeThreshold(node_id, …)` is the node-scope alias: it becomes `scope='node'`, -`target=` (§4.5). Validation checks that `scope` is legal for that param per its `scopes` list, -and that `target` exists where existence is checkable. `UNIQUE (rule_id, param_name, scope, target)` allows -a node override *and* a cluster override to coexist for the same param — precedence decides which wins. - -**Emit path (collector).** `scope` selects what to do with `target`: - -| `scope` | `target` holds | Collector action | -|---|---|---| -| `node` | `node_id` | resolve → `node_name` via `nodes`; skip if it no longer resolves | -| `service` | `service_id` | resolve → `service_name` via `services`; skip if it no longer resolves | -| `cluster` | the `cluster` label value | no lookup — but **expand** onto the param's join label (below) | - -The skip-if-unresolvable behaviour is what makes a stale row inert (§4.8), and the emitted label is always -the *resolved* value, never the stored ID, because the rule joins on names. - -**Read path (API).** `(scope, target)` is the lookup key: "all thresholds for this node" is -`WHERE scope='node' AND target=$1`, which is what the `(scope, target)` index exists for. The same -precedence function then produces the effective value (§4.10). - -##### Which scopes are legal for a param — a rule, not a list - -A coarse scope has to be projected onto the param's join label, and that projection must be -**unambiguous**. The general rule: - -> A scope `S` is legal for a param whose join label is `L` **iff the mapping `L → S` is a function** in the -> inventory — i.e. every value of `L` maps to at most one target at scope `S`. - -Applying it: - -| join label | scope | `L → S` | Legal? | -|---|---|---|---| -| `node_name` | `node` | `node_name → node_id` | ✅ one-to-one | -| `service_name` | `service` | `service_name → service_id` | ✅ one-to-one | -| `service_name` | `cluster` | `service_name → cluster` | ✅ each service has exactly one cluster, so a cluster override expands onto its services without conflict | -| `node_name` | `cluster` | `node_name → cluster` | ❌ a node can host services from several clusters | -| `node_name` | `service` | `node_name → service_id` | ❌ a node hosts many services, so two service overrides could claim the same node | - -So a node-joined param accepts `{node}`; a service-joined param accepts `{service, cluster}`. This is a -**parse-time validation** on `override_scopes`, not a runtime tie-break — the illegal rows must never -exist. It also generalises to labels not yet supported (`environment`, `replication_set`) without new -case-by-case reasoning. - -> **Consequence for the collector.** Expanding a coarse scope requires knowing the param's `join_label`, -> so the collector reads `alert_rules.params` **only once multi-scope behaviour ships**. In the node-only -> first increment every override is `scope='node'` onto `node_name`, resolution is driven by `scope` -> alone, and the collector touches only the overrides table plus inventory. - -#### `grafana_rule_uid` — a cache, never the identity - -`pmm_rule_id` stays the identity: PMM mints it, stamps it as a rule label, and it is stable by -construction. The Grafana UID is stored **only as a handle** for cheap direct addressing — -`GET/PUT/DELETE /api/v1/provisioning/alert-rules/{uid}` needs no folder or group, which is exactly what a -future default-change or delete path wants. - -It must be treated as a cache. Note first what is *not* the reason: PMM has no delete-and-recreate -lifecycle — no `DeleteRule` RPC, no `DeleteAlertRule` on the client, and this design never rewrites a rule -after creation. Nor is a plain user delete-then-create a problem, because the replacement carries no -`pmm_rule_id` label, so the old rule is simply *gone* and its registry row is garbage rather than "the -same rule with a new UID". - -The staleness paths that do exist are **Grafana-side copy operations that preserve labels while minting a -new UID**: - -- duplicating a rule in the Grafana UI; -- alert-rule export/import, including provisioning-file restore; -- Grafana backup/restore. - -So the handle can go stale, which is why: - -- the column is **nullable** — the value may not be known yet, and code must work without it; -- reads that matter fall back to matching on the `pmm_rule_id` label; -- a `404` on direct addressing means "refresh the handle", not "the rule is gone" — **re-resolve by label and update the stored UID**, which is self-healing and free, because the reconcile/read pass already holds both the UID and the label for every rule. - -#### The same copy operations break `pmm_rule_id` uniqueness — and mostly that is fine - -Duplicating a PMM rule in Grafana produces **two rules carrying the same `pmm_rule_id`**. -`UNIQUE (grafana_rule_uid)` does not help: the collision is on the identity *label*, inside Grafana, where -PMM can impose no constraint. - -The important realisation is that **the coupling lives in the query text, not in the label.** The copy's -`T_` step literally contains `rule_id=""`, so a duplicated rule evaluates against the -original's overrides no matter what identity scheme is used. No scheme can prevent that, because copying -copies the query. Treat it as intended behaviour: **duplicated rules share thresholds.** - -What must change is every place that assumed a 1:1 mapping: - -| Concern | Resolution | -|---|---| -| Caching the UID | If a `pmm_rule_id` maps to more than one Grafana rule, store **`NULL`** — refuse to cache an ambiguous handle rather than picking one arbitrarily. | -| Reconciliation | It only asks *"is this `pmm_rule_id` present at all?"* — a boolean, so duplicates are harmless. Do not add "which one". | -| `ListNodeThresholds` | May legitimately return the same threshold under two rule titles. That is honest — there really are two rules — so present both rather than silently collapsing them. | -| Operator visibility | Count duplicated `pmm_rule_id`s during the reconcile pass and expose the count as a gauge (plus a log line), so this is observable instead of surprising. | - -None of this requires preventing duplication, which is not preventable without read-only rules — rejected -in §7 as group-wide and irreversible. - -**Capturing it costs nothing.** The Ruler API POST already returns the UIDs it created — -`{"message":"rule group updated successfully","created":["ffvj7sosfptdsd"]}` — and `CreateAlertRule` -(`managed/services/grafana/client.go:712`) currently returns only `error`, discarding that body. Have it -parse and return the created UID instead of adding a second round-trip. Note an *update* returns -`"updated"` rather than `"created"`, so handle both. - -**What `target` holds:** the stable **ID** for node/service (`node_id`, `service_id` — never reused), and -the **label value** for cluster (which has no ID). The query joins on the *name*, because that is what -templates aggregate by, so the collector resolves ID → name at emit time, bounded by the override count: - -```go -overrides, _ := models.FindThresholdOverrides(tx.Querier) // V rows, one indexed query -nodes, _ := models.FindNodesByIDs(tx.Querier, nodeIDsOf(overrides)) // WHERE node_id IN (…) -// scope='cluster' needs no lookup: target IS the label value. -``` - -> **In the node-only increment the collector never reads `alert_rules`.** It emits only overrides, takes -> each value from the override row, and picks the resolution table from that row's `scope` — so it needs -> neither defaults nor `params`. (Coarse-scope expansion later requires `params[p].join_label`; see "Which -> scopes are legal for a param" below.) Either way, nothing in the emit path knows whether the rule still -> exists — which is exactly why an orphaned registry row keeps producing series. See §4.9. - -That resolution doubles as cleanup: an override whose target no longer resolves emits nothing, so a stale -row is inert and invisible, and GC becomes tidiness rather than correctness. - -**Note on `node_name` reuse.** `UNIQUE (node_name)` (`database.go:117`) guarantees uniqueness at any -instant but **not across time** — rebuild a host, re-register with the same hostname, and you get the same -`node_name` with a fresh `node_id`. Storing `node_id` and rendering `node_name` is what keeps a stale -override from silently attaching to a different machine. - -### 4.5 Value and API validation - -`DOUBLE PRECISION` accepts non-finite values, and this would be **`main`'s first float column**, so there -is no existing precedent to inherit. Guard at both layers. - -Database: - -```sql -CHECK (value = value -- rejects NaN (NaN <> NaN) - AND value > '-Infinity'::float8 - AND value < 'Infinity'::float8) -``` - -API, in this order, each with a defined code: - -| Check | Code | -|---|---| -| `value` is finite (not NaN, not ±Inf) | `InvalidArgument` | -| `value` within the param's declared `range` | `InvalidArgument` | -| param exists on the rule | `NotFound` | -| param is `overridable` | `FailedPrecondition` | -| `scope` is a supported value, and legal for this param per `override_scopes` | `InvalidArgument` | -| `target` exists, where existence is checkable (node/service; not cluster) | `NotFound` | -| rule exists in the registry | `NotFound` | - -**Alias conflict.** The API carries both `node_id` and the general `scope`/`target` pair (§6 step 7). -`node_id` is an alias for `scope='node', target=`. If both are supplied and disagree, reject with -`InvalidArgument` — do not silently prefer one. Freeze this rule in the proto comments before the API -ships, since the fields are permanent once released. - -### 4.6 Precedence is resolved in Go, not PromQL - -Three measurements settle this: - -| Test | Result | -|---|---| -| `or` across **different** label sets | returns **both** series — no precedence | -| `or` across **identical** label sets | left wins — so `or` precedence needs same-label-set terms | -| node override 50 + cluster override 90 under `max by (node_name)` | **90** — `max` picks the larger value, not the more specific scope | - -Reducing to a common label set is required for `or` to mean "prefer the left", but that reduction is -exactly what destroys the scope information needed to rank by. So the collector emits only the effective -value per target, and **`ListNodeThresholds` and the collector must share one precedence function** — -PromQL provides no backstop if they diverge. Order: `service` → `node` → `cluster` (§4.10). - -### 4.7 Semantics that fall out for free - -- **Set an override** → row inserted → series appears within one scrape → `or` prefers it. -- **Clear an override** → row **tombstoned**, not deleted (§4.4) → the series keeps being emitted and its value changes to the resolved default. Measured latency **14–21 s**. Deleting the row instead signals the clear by absence, which measured **309 s** — the same ~5 minutes an earlier revision of this document accepted as unavoidable. It is not unavoidable; it is a consequence of encoding "cleared" as absence. -- **New node** → the fan-out picks it up; nothing to write. -- **Node or service deleted** → **there is no cascade for the target.** The only foreign key is `rule_id → alert_rules`, so deleting a *rule* cascades its overrides; deleting a *node or service* does not. The polymorphic `target` column (§4.4) cannot carry an FK, because `cluster` targets have no referent table. Two consequences, and both matter: - - **Immediately harmless:** ID → name resolution fails for the deleted entity, so the collector emits nothing for it. The row is inert from the next scrape onward — no phantom series, no wrong threshold. - - **But the rows must still be cleared.** Left behind, they accumulate silently, they show up in any admin/debug listing of overrides, and they make `ListNodeThresholds`-style queries return entries for entities that no longer exist. **Removing a node or a service must clear its overrides** — see the garbage-collection spec below. -- **pmm-managed down** → **defaults keep resolving indefinitely; only overrides degrade.** Because the - default is fanned out over the observed expression (§2), `T` cannot go empty while `$A` still resolves. - One step of degradation remains, and it is bounded and safe-direction: - -| Elapsed | Behaviour | -|---|---| -| 0 – ~5 min | Overrides and defaults both resolve. Normal. | -| ~5 min | Override series age out of VM's lookbehind → **overridden targets revert to their defaults**. Behaviour change #1, and the only one. | -| indefinitely | Defaults keep resolving from the observed expression. Alerting continues, at defaults, for as long as `$A` flows. | - - **This is measured, not argued.** Two rules were run side by side against the same observed series while - the fan-out metric was cut off. The inventory-fan-out rule lost its raw series at **259 s**, lost `T` - at **856 s** when the `[15m]` window expired, and at **905 s** reported `state=inactive, health=ok` — - a green, healthy, non-alerting rule with live data breaching its threshold. The observed-query fan-out - rule held `T = 80` and stayed `firing` for the full 26-minute run (§8). - - **The old shape had a second, worse trigger than a crash.** The threshold collector and the inventory - collector share one `/debug/metrics` endpoint and one `0.9 × MR = 9 s` timeout. A threshold collector - slow enough to blow that budget takes `pmm_managed_inventory_nodes` down **with** it — measured, see - §4.12 — which under the old fan-out silently disabled *every* rule within 15 minutes. Fanning out over - the observed expression severs that coupling: the rule no longer reads anything pmm-managed publishes. - - Detection still needs no new plumbing: **`up{job="pmm-managed"}` already exists on `main`** and covers - the remaining override-revert step. The dedicated endpoint's `up{job="pmm-thresholds"}` would later - attribute it more precisely. - -### 4.8 Clearing overrides when nodes and services are removed - -`target` carries no foreign key, so deletion of the referenced entity is handled **in the removal API -itself** — not by a background sweep. PMM already does dependant cleanup this way, which makes the hook -idiomatic rather than novel. - -> **Two different operations, deliberately.** A user *clearing* an override **tombstones** the row so the -> series keeps resolving and the clear lands in 14–21 s (§4.4). A *node or service being removed* -> **hard-deletes** the rows, because there is no longer any target to emit for and nothing will ever query -> it again — a tombstone there would be pure residue. Clearing is a value change; removal is a deletion. - -**The mechanism: delete inside the existing removal transaction.** - -```go -// managed/models/node_helpers.go, in RemoveNode -DeleteThresholdOverridesForTarget(q, ThresholdScopeNode, id) - -// managed/models/service_helpers.go, in RemoveService -DeleteThresholdOverridesForTarget(q, ThresholdScopeService, id) -``` - -Three properties of the existing code make this sufficient on its own: - -1. **The DB restricts rather than cascades.** `services.node_id` is a plain `FOREIGN KEY (node_id) REFERENCES nodes (node_id)` with no `ON DELETE CASCADE`, so PMM must remove dependants explicitly — which is exactly why these chokepoints exist: `RemoveNode` (`node_helpers.go:255`), `RemoveService` (`service_helpers.go:354`), `RemoveAgent` (`agent_helpers.go:1444`), each taking a `RemoveMode`. -2. **The helpers compose.** `RemoveNode` with `RemoveCascade` finds the services on that node and calls `RemoveService(…, RemoveCascade)` for each, which in turn routes through `RemoveAgent`. So deleting a node clears its own node-scoped override **and** the service-scoped overrides of every service on it, with no second code path and no gap. -3. **It is atomic.** The delete runs in the same reform transaction as the removal, so there is no window in which overrides outlive their entity — unlike a sweep, which is eventually consistent by construction. - -It also sidesteps the awkwardness a sweep would have had: no leader-election question (the delete happens -wherever the removal API is served) and no lag before the UI reflects reality. - -**Retained as a free backstop, not as the mechanism:** the collector resolves `target` IDs to join-label -names and **skips anything that no longer resolves**, so a row missed by any future removal path — or -inserted by direct SQL — can never produce a wrong threshold. It is simply inert. - -**Never delete `scope='cluster'` rows.** There is no "delete a cluster" operation to hook, and more -importantly a cluster override with no matching services is **dormant, not stale**: services may be added -to that cluster later, and the override should then apply. Expansion yielding nothing already makes it -inert, so keeping the row is correct rather than untidy. - -**Tests:** removing a node deletes its node-scoped override in the same transaction; removing a node with -cascade also deletes the service-scoped overrides of its services; a cluster override survives every -removal; an override whose target was deleted out-of-band emits nothing. - -### 4.9 Cross-store creation and reconciliation - -Rule creation spans Grafana and Postgres, so the ordering and failure behaviour are part of the design: - -1. **Mint `rule_id` first.** It is the idempotency key for every subsequent step; a retry is a plain upsert on the same id. -2. **Write Postgres, then Grafana.** If the Grafana call fails, the registry row is orphaned — which is already the reconciler's job, so no bespoke compensation path is needed. -3. **Grafana rule without a registry row** (the reverse failure, possible if a row is lost) is not silently harmful: the default is a literal in the rule, so it keeps evaluating correctly at defaults, and only the override API fails. Repair from the `pmm_rule_id` and `template_name` labels already stamped on the rule. - -**The reconciler is a garbage collector, not a consistency mechanism — and it never writes to Grafana.** -Because Grafana is authoritative for which rules exist (§4.4) and both the API join and the collector's -ID→name resolution drop unresolvable rows at read time, it has **no correctness duty for alerting -behaviour**. It has two jobs, and only one of them costs anything real: - -There are two kinds of orphan, and only one of them is the reconciler's problem: - -| Orphan | Cleaned by | Cost of not cleaning it | -|---|---|---| -| Override rows whose **node/service** is gone | **The removal API hooks (§4.8)** — atomic, immediate | n/a — handled at the source | -| **Registry rows** whose Grafana **rule** is gone | **The reconciler.** Irreducible: rule deletion happens *in Grafana*, where PMM has no hook and gets no notification | **Real.** The collector keeps emitting `pmm_alert_threshold_override` series for a rule that no longer exists. Nothing queries them, but they consume VM ingestion and cardinality and grow without bound as rules churn | - -So the reconciler has exactly **one** deletion job, plus a refresh: - -It should also **refresh `grafana_rule_uid`** while it is there, since the listing already pairs each UID -with its `pmm_rule_id`. - -An optional periodic sweep for stray override rows (direct SQL, or a removal path that forgets the hook) -is cheap — one `DELETE … WHERE scope='node' AND target NOT IN (SELECT node_id FROM nodes)` — but it is -belt-and-braces, not required, because read-time resolution already makes such rows inert. It must never -touch `scope='cluster'` (§4.8). - -(The "an identical group re-POST is a no-op" measurement quoted in §7 belongs to the *rejected* inline -design, where a reconciler would re-render rules. It is not a licence for this one to write.) - -> **A background job may not be needed at all.** The read path already fetches the authoritative rule -> list from Grafana, so it could delete registry rows it finds absent as a side effect — no ticker. -> Trade-off: cleanup then happens only when someone exercises the API, and it still needs the leader check -> (`IsLeader()`), because a follower must not write. Worth weighing rather than assuming the ticker. - -Deletion safety rules: - -- Require a **successful, complete** listing. On any error or partial response, skip the cycle — never delete on incomplete data. -- Require absence in **K consecutive cycles** (K ≥ 2) before deleting, to survive transient inconsistency and creation races. -- **Use a single global listing** (`GET /api/ruler/grafana/api/v1/rules` returns every folder) and match on the `pmm_rule_id` label alone. **Do not scope the check to a per-rule folder** — that would delete live data: a user can move a rule between folders in Grafana, after which a folder-scoped check reports it absent and the reconciler removes the overrides of a rule that still exists. This is also the second reason `folder_uid` is not stored. - -### 4.10 Multi-scope resolution — the shared resolver - -Only node scope is implemented in the first increment, so the full resolver is a gate for the -**service/cluster increment**, not for the first merge. What must exist from day one is the **shared -function boundary**, so the API and the collector cannot drift: - -```go -// ResolveEffective returns exactly one value per target, precedence applied. -// The collector and ListNodeThresholds MUST both call this. Nothing else may -// implement precedence. -func ResolveEffective(rule *models.AlertRule, param string, - overrides []*models.AlertRuleThresholdOverride, - inv Inventory) map[string]float64 // join-label value -> effective threshold -``` - -Invariants to assert and table-test: - -- **Exactly one emitted series per `(rule, param, target)`.** This is not a tie-break preference: if the resolver ever emits two series with identical labels, the Prometheus gatherer errors and the **entire** `/metrics` response fails. `node_name` and `service_name` are both `UNIQUE` (`database.go:117`, `:139`), so live targets cannot collide — the risk is a resolver bug, so assert it. -- **Precedence `service` → `node` → `cluster`**, most specific first. A param legal at both service and cluster scope with both present resolves to **service**. - - Only the first relation is derivable: **a service runs on exactly one node and belongs to at most one cluster**, so a service override is strictly narrower than either and must win. `node` over `cluster` is a *convention*, not a containment — the two cross-cut, since a cluster spans several nodes while a node hosts services from several clusters (the same fact that constrains cluster scope in the next bullet). One machine is the narrower intent, so node wins, but nothing derives it. - - In practice the two rarely meet: node scope resolves to `node_name` while service and cluster scope resolve to `service_name`, and a rule joins on one label. They collide only when a node and a service share a name string — which nothing prevents, since each is unique within its own table but not across them. The ranking decides that case, so it must be right even though it is currently unreachable through the API. -- **Cluster scope is only legal for params whose join label is service-level.** A node can host services from several clusters, so "the cluster override for this node" has no unique answer. This is a validation rule on `override_scopes`, not a runtime tie-break. -- **Unresolvable targets are skipped**, never defaulted to something else. - -### 4.11 High availability - -Every pmm-managed node registers collectors unconditionally and scrapes its own localhost, labelled -`instance = PMM_HA_NODE_ID`, so an N-node cluster emits N copies. Postgres is shared/external in HA, so -the copies are identical and `max by ()` collapses them. - -> **Invariant:** never drop `max by ()`. It looks redundant once the metric is override-only, -> but it is what makes the design HA-safe — and its absence only manifests with more than one node. - -The **reconciler must be leader-only**: -`haService.AddLeaderService(ha.NewContextService("threshold-reconciler", fn))` (`haservice.go:569`). -`IsLeader()` returns true when HA is disabled, so single-node deployments are unaffected. - -### 4.12 The collector that already exists on this branch — and what to change - -`main` has none of this, but **this branch already carries a working collector** — -`managed/services/alerting/threshold_metrics.go`, registered at `managed/cmd/pmm-managed/main.go:1034`. -It is not the design above. It implements **emit-everything**: for every rule, for every param, for every -node in inventory, it emits `pmm_alert_threshold{rule_id, param, node_name}` — its own doc comment says -"emitting a value for every node ensures unoverridden nodes still evaluate against the default". So the -recommendation in §2 is a **change away from what is built**, and the emit-everything row in §7 is a -measurement of live code, not a thought experiment. - -**Measured on that collector**, 1011 nodes seeded into `nodes`, scrape timeout `0.9 × MR = 9 s`: - -| nodes × rules × params | series per scrape | median | worst | % of 9 s budget | -|---|---|---|---|---| -| 1011 × 1 × 1 | 1,011 | 76 ms | 85 ms | 0.9 % | -| 1011 × 20 × 1 | 21,231 | 385 ms | 514 ms | 5.7 % | -| 1011 × 20 × 5 | 102,111 | 1,870 ms | 2,070 ms | 23 % | -| 1011 × 42 × 5 | 213,321 | 4,121 ms | 4,511 ms | 50 % | -| 1011 × 20 × 10 | 203,211 | 4,395 ms | **6,209 ms** | **69 %** | -| 1011 × 42 × 20 | 850,251 | **26,964 ms** | — | **blown** | - -At the scenario §7 quotes — 1000 nodes × 20 rules, one param each — emit-everything costs **385 ms** and -is genuinely fine. The exposure is **multi-param rules**: `main` ships 42 templates, so the 50–69 % rows -are reachable rather than hypothetical, and `worst` is what matters because one blown scrape drops the -whole endpoint. - -**What a blown scrape actually does**, observed at 42 × 20: - -``` -scrape_duration_seconds{job="pmm-managed"} 9.001 -up{job="pmm-managed"} 0 -count(pmm_managed_inventory_nodes) EMPTY <- collateral -``` - -The threshold collector takes the **inventory collector down with it** — one endpoint, one timeout. Under -the inventory-fan-out shape this was a compounding failure: a slow threshold collector silently disabled -every rule within 15 minutes (§4.7). The observed-query fan-out removes that path. - -**Three defects to fix regardless of which emission policy wins:** - -1. **No timeout.** `Collect` uses bare `context.Background()`. The inventory collector next door bounds - itself with `requestTimeout = 3 * time.Second` (`inventory_metrics.go:33`). That missing bound is *why* - the scrape runs 27 s instead of returning partial data, and why the blast radius reaches other - collectors. -2. **`Describe` uses `prom.DescribeByCollect`** (`threshold_metrics.go:67`) — exactly what §4.2 says not - to do, since it runs a full `Collect`, and therefore a database query, merely to describe the collector. -3. **Node scope only.** Overrides are keyed on `o.NodeID`; there is no polymorphic `target`, no `scope`, - and no shared resolver, so §4.4, §4.6 and §4.10 are unbuilt. - -Confirmed from the source, matching §7's estimate: `2 + R` queries per scrape — `FindAlertRules`, -`FindNodes`, then `FindThresholdOverridesByRule` once per rule, all inside one transaction. - -Switching it to override-only is a change to the same `Collect` loop — emit where an override or tombstone -exists, drop the per-node fan-out — which makes an apples-to-apples comparison cheap on the same code -path, same DB, same endpoint. - ---- - -## 5. Worked examples - -### 5.1 Multi-expression template — one line to make it overridable - -`pmm_node_high_cpu_load` is already multi-expression on `main`. The whole template-side change: - -```yaml - params: - - name: threshold - summary: A percentage from configured maximum - unit: "%" - type: float - range: [0, 100] - value: 80 - overridable: true # <-- the only addition - override_scopes: [node] # <-- optional; defaults to [node] / node_name -``` - -Generated rule, with one override (`node-02` → 90) and default 80: - -| refId | body | -|---|---| -| `A` | `(1 - avg by(node_name) (rate(node_cpu_seconds_total{mode="idle"}[5m]))) * 100` *(unchanged)* | -| `T_threshold` | `max by (node_name) (label_replace(pmm_alert_threshold_override{rule_id="7f3a…", param="threshold"}, "node_name", "$1", "target", "(.*)")) or (max by (node_name) ((1 - avg by(node_name) (rate(node_cpu_seconds_total{mode="idle"}[5m]))) * 100) * 0 + 80)` | -| `C` | `$A > $T_threshold` | - -Resolves to `node-02 = 90`, every other **reporting** node `= 80`. Note the second clause is `A`'s own -expression with its value discarded by `* 0` — that is what makes the threshold share `$A`'s fate instead -of depending on a metric pmm-managed publishes (§4.7). The duplication is generated, never hand-written, -and §5.2 already parses the template's query on the AST for the single-expression case. - -> **The `T_` step is always emitted, including when no override rows exist.** Omitting it for -> untuned rules looks like a free optimisation — the rule would stay byte-identical to today's output — -> and it is **wrong**: a rule created without the step needs its definition changed to add one when the -> first override arrives, which resets alert state for every instance on that rule, the precise failure -> this design exists to avoid. With no overrides the step is still present and simply resolves to the -> default for every target. -> -> The honest cost: **every overridable rule pays the fan-out scan (~N × 91 samples per evaluation) -> whether or not anyone has tuned it.** The "untuned rules cost nothing" property does not exist. - -### 5.2 Single-expression template — desugared - -`pmm_mysql_too_many_connections` on `main` is single-expression: - -```yaml - expr: |- - max_over_time(mysql_global_status_threads_connected[5m]) / ignoring (job) - mysql_global_variables_max_connections - * 100 - > bool [[ .threshold ]] -``` - -Marking its param `overridable: true` triggers desugaring at build time — the template file keeps its -`expr:` form; only the generated rule changes: - -| refId | body | -|---|---| -| `A` | `max_over_time(mysql_global_status_threads_connected[5m]) / ignoring (job) mysql_global_variables_max_connections * 100` | -| `T_threshold` | the threshold expression (join label `service_name`, fan-out over `pmm_managed_inventory_agents{service_name!=""}`) | -| `C` | `$A > $T_threshold` | - -Two mechanical details: - -1. **`> bool` disappears** — Grafana math comparisons already yield 0/1. -2. **`{{ $value }}` must become `{{ printf "%.2f" $values.A.Value }}`** — with multiple refIDs `$value` is no longer a single scalar. This template's `description` uses `{{ $value }}`, so shipping desugaring without the rewrite ships broken alert text. Rewrite it during desugaring (annotations are filled at `service.go:744`). - -Constraints inherited from parsing an `expr:`: the token must be the RHS of the **final** comparison, and -a single expression admits **one** overridable param. Reject at parse time otherwise. - -#### Do the split on the AST, not with a regular expression - -A permissive regex split is the wrong tool: it mishandles parentheses, comparison modifiers (`bool`), -vector-matching clauses (`on`/`ignoring`, `group_left`), and nested comparisons, and it fails *silently* -by producing a plausible-but-wrong `A`. `github.com/prometheus/prometheus v0.313.0` is **already a direct -dependency on `main`** (`go.mod:61`), so the parser is available with no new dependency. - -The one wrinkle is that `[[ .threshold ]]` is not valid PromQL, so it must be replaced before parsing. -Pad the replacement to the **same byte length** as the token and AST positions map 1:1 onto the original -text — which means `A` can be sliced out of the *original* string, preserving the author's formatting -exactly: - -```go -// splitSingleExpr splits a single-expression template into the observed query and -// the comparison operator, so the builder can emit A + T_ + math C. -// -// The [[ .param ]] token is replaced by a byte-length-padded numeric sentinel so -// that PromQL positions map 1:1 onto the original text. -func splitSingleExpr(expr, paramName string) (lhs string, op parser.ItemType, isBool bool, err error) { - token := paramTokenRegexp(paramName).FindString(expr) - if token == "" { - return "", 0, false, fmt.Errorf("param %q is not referenced in the expression", paramName) - } - - // "0" plus spaces to the token's exact byte length: parseable, and position-preserving. - sentinel := "0" + strings.Repeat(" ", len(token)-1) - probe := strings.Replace(expr, token, sentinel, 1) - - parsed, err := parser.ParseExpr(probe) - if err != nil { - return "", 0, false, fmt.Errorf("failed to parse expression: %w", err) - } - - // The threshold must be the RHS of the outermost comparison. - bin, ok := parsed.(*parser.BinaryExpr) - if !ok || !bin.Op.IsComparisonOperator() { - return "", 0, false, errors.New("an overridable param must be the right-hand side of the expression's top-level comparison") - } - if num, ok := bin.RHS.(*parser.NumberLiteral); !ok || num.Val != 0 { - return "", 0, false, errors.New("an overridable param must be compared directly, not used inside a larger expression") - } - - // Vector matching on the comparison itself cannot survive the split into a - // Grafana math step, so reject it rather than silently changing semantics. - if bin.VectorMatching != nil { - return "", 0, false, errors.New("vector matching on the threshold comparison is not supported for overridable params") - } - - // Positions are byte offsets into probe, which is the same length as expr. - r := bin.LHS.PositionRange() - return strings.TrimSpace(expr[r.Start:r.End]), bin.Op, bin.ReturnBool, nil -} -``` - -Everything the regex approach would have to special-case is now either handled by the parser or rejected -with a precise message. Note `bin.ReturnBool` captures the `bool` modifier — which is then **dropped**, -because Grafana math comparisons already yield 0/1 (§5.2, gotcha 1). - -**Verification must precede implementation.** The filter-vs-0/1 semantic difference is not proven for -real alert lifecycle behaviour — `for:` pending/firing transitions, NoData, recovery, annotation -rendering. Desugaring an existing template before that check risks changing *alerting behaviour* while -believing you only changed *threshold delivery*. See §9, gate 3. - -### 5.3 Two params at two different scopes — validated live - -The case that proves the design generalises. Both params in one rule, with different join labels: - -```yaml - queries: - - ref_id: A # service-scoped (carries node_name too) - expr: |- - sum by (service_name, node_name) (pg_stat_database_numbackends) - / on (service_name, node_name) group_left() - max by (service_name, node_name) (pg_settings_max_connections) * 100 - - ref_id: B # node-scoped - expr: |- - (1 - avg by(node_name) (rate(node_cpu_seconds_total{mode="idle"}[5m]))) * 100 - expressions: - - ref_id: C - type: math - expression: "$A > [[ .connections_threshold ]] && $B > [[ .cpu_threshold ]]" - condition: C - params: - - name: connections_threshold - value: 80 - overridable: true - override_scopes: [service, cluster] # -> join label service_name - - name: cpu_threshold - value: 80 - overridable: true - override_scopes: [node] # -> join label node_name -``` - -Deployed as four rules on a live server (actuals: connections ≈ 0.7 %, CPU ≈ 5.4 %): - -| Case | Overrides | Effective thresholds | Result | -|---|---|---|---| -| none | — | conn **0.50** / cpu **3.00** (both fan-out defaults) | **fires** | -| node only | `cpu_threshold`@node = 90 | conn 0.50 / cpu **90** | does not fire | -| service only | `connections_threshold`@service = 90 | conn **90** / cpu 3.00 | does not fire | -| node **+** service | cpu@node = 1, conn@service = 0.1 | conn **0.10** / cpu **1.00** | **fires** | - -The last row is the load-bearing one: its **defaults are 90/90**, so it could never fire on defaults. It -fired, annotated `conn=0.70% (thr 0.10) cpu=5.36% (thr 1.00)` — two overrides at two different scopes -applied simultaneously in one rule. - -This also proved **Grafana math subset-matches across differing label sets**: `$A{node,service}` against -`$T{service}`, `$B{node}` against `$T{node}`, then `&&` across the two results. That had been an -assumption in every prior write-up. - -### 5.4 Annotations - -`{{ printf "%.0f" $values.T_.Value }}` renders the **effective per-node** threshold — verified, -including on a multi-param rule. Templates should use it rather than interpolating `[[ .threshold ]]`, -which freezes the default at creation time and is then wrong for every overridden target. Word it -neutrally (`"CPU load is X%, threshold Y%"`), because annotations are also rendered for tracked -non-firing instances. - ---- - -## 6. Implementation plan - -Ordered so each step is independently reviewable. All paths are `main` paths. - -| # | Step | Files | Effort | -|---|---|---|---| -| 1 | `overridable` + `override_scopes` on `Parameter`; validation (float only, must be a `[[ .name ]]` token in an expression step, at most one for a desugared single-expression template) | `managed/pi/alert/parameter.go:25`, `managed/pi/alert/query.go` validation, `template.go:113` | S | -| 2 | Migration 119 (**including `cleared_at`**, §4.4) + reform models + helpers (`FindThresholdOverrides`, `FindThresholdOverridesByTarget`, `UpsertThresholdOverride`, `ClearThresholdOverride` — tombstones, does **not** delete, `DeleteThresholdOverridesForTarget`, `FindAlertRules`, `CreateAlertRule`, `DeleteAlertRule`) | `managed/models/database.go:1184`, new `alert_rule*_model.go` + `alert_rule_helpers.go`, `make gen` | M | -| 3 | Threshold collector — **rewrite, not create: it already exists on this branch and emits every node** (§4.12). Switch to override-or-tombstone emission; one indexed query + bounded ID→name resolution; fixed 3-label desc. Three defects to fix in the same pass, independent of emission policy: **bound `Collect` with a 3 s timeout** (it uses bare `context.Background()`, which is why a slow scrape reached 27 s and took the inventory collector with it), replace **`prom.DescribeByCollect`** with `ch <- c.desc`, and add `scope` handling | `managed/services/alerting/threshold_metrics.go` (exists), registered `main.go:1034` | S–M | -| 4 | Rule builder: allocate `T_` refIDs, inject the threshold step, swap `[[ .param ]]` → `$T_`; thread `ruleID` through | `rule_builder.go:58`/`:81`, reusing `newPromQueryData:141` and `newMathExpressionData:165` | M | -| 5 | `CreateRule`: mint `rule_id`, stamp a `pmm_rule_id` label, persist the registry row with the `params` snapshot, return the id. Also have `CreateAlertRule` parse and return the created Grafana UID (already in the POST response body) and store it as `grafana_rule_uid` | `service.go:679`, `grafana/client.go:712` | S | -| 6 | Single-expression desugaring: split `expr` into `A` + `T_` + math `C`; rewrite `{{ $value }}` → `{{ $values.A.Value }}` | new `managed/pi/alert/overridable.go`, `rule_builder.go`, annotations at `service.go:744` | S–M | -| 7 | API: `overridable` on the param-definition message; `rule_id` on `CreateRuleResponse`; list/set/clear threshold RPCs with `scope`/`target` fields present but node-only behaviour — **clear tombstones the row rather than deleting it**, and list must report `cleared_at IS NOT NULL` as *not overridden* (§4.4); shared precedence function. **Route shape is pending §10 Q1** — node-centric vs generic — and that answer is needed before the proto freezes | `api/alerting/v1/alerting.proto`, new `managed/services/alerting/threshold_overrides.go`, `make gen` | M | -| 8 | `ListAlertRules` on the Grafana client + leader-only reconciler (registry-row orphans only). Separately, hook `DeleteThresholdOverridesForTarget` into `RemoveNode` / `RemoveService` (§4.8) | `managed/services/grafana/client.go:712` area, `deps.go`, `haservice.go:569`, `models/node_helpers.go:255`, `models/service_helpers.go:354` | S | -| 9 | Mark built-in templates overridable — `node_high_cpu_load.yml` (one line) and, once step 6 lands, `mysql_too_many_connections.yml` | `managed/data/alerting-templates/` | XS | -| 10 | Add `cluster` to the inventory services descriptor, before anything depends on that label set | `managed/services/inventory/inventory_metrics.go:84` | XS | - -**3–5 weeks** for one engineer for the node-only increment. - -The earlier figure of 2–3 weeks covered the ten steps above with unit tests and silently excluded -everything else. Stated properly: - -| Included | Excluded | -|---|---| -| The ten steps, with unit tests | UI | -| `make gen` cycles for proto and reform | Service and cluster **behaviour** (schema/API only) | -| AST-based desugaring (§5.2) and its rejection cases | Scale re-measurement on a large inventory | -| Cross-store failure handling and the reconciler (§4.9) | HA cluster testing beyond the `max by` invariant | -| GC for stale targets (§4.8) | API review turnaround | - -Assumes the §9 decision gates are resolved **in parallel** with implementation, not serially. If gate 1 -(NoData/availability) or gate 5 (precedence sign-off) blocks, the critical path extends by that wait -rather than by engineering time. - -Deliberately **not** needed: `UpdateAlertRule`, a per-group write mutex, `uid`-preserving JSON patching, -Grafana provenance, PromQL parsing of generated text, a VM write path, or a keep-alive writer. - -**Tombstones *are* needed** — an earlier revision listed them here as avoided. They are what makes a clear -take 14–21 s instead of 309 s (§4.4), and they cost one nullable column plus a filter in the list path, not -a writer or a sweeper. - -### Follow-ups, costed - -| Follow-up | Effort | Why not now | -|---|---|---| -| Dedicated endpoint + own scrape job, giving `up{job="pmm-thresholds"}` | XS–S | Isolation and hygiene, not behaviour. `promscrape.yml` is generated in pmm-managed (`victoriametrics.go` `populateConfig`), so it stays in-tree. | -| Service and cluster **behaviour** | M | Schema and API already generalised; needs the precedence function plus the `cluster` inventory label from step 10. | -| UI | M | Separate deliverable. | - ---- - -## 7. Alternatives rejected - -Each with the measurement that settled it. - -| Alternative | Why not | -|---|---| -| **Inline the overrides as literals in the rule query** (no metric) | A rule-definition change **resets alert state for every instance on that rule** — `activeAt` moved `14:09Z → 14:18Z` for a node whose own threshold was untouched, with `uid` *and* `guid` preserved. Under `for: 5m`, tuning one node blinds every other node on that rule for five minutes. Batching, one-rule-per-group, shortening `for:` and `keep_firing_for` all fail to preserve the timer, because Grafana keys state on the rule definition. | -| **Put the canonical state inside the rule** (no tables) | Same state-reset cost, plus no contractual home: either parseable PromQL (symmetric escaping of user-controlled `node_name`, stable float formatting, permanent support for old shapes) or a JSON sidecar in the query model. The sidecar survives both an API round-trip and a UI edit on Grafana 12.4.5 — but rests on an undocumented implementation detail that an upgrade could remove silently. | -| **Read-only rules via Grafana provenance** | Group-wide and irreversible: claiming provenance makes the Ruler API reject the **whole group** (`400`), and release fails with `409 alerting.provenanceMismatch`. Only exit is delete-and-recreate, minting a new UID and breaking silences. | -| **Emit a threshold for every node** (default or override) | `rules × params × nodes`, ~99 % of it identical defaults. **Not rejected on query cost** — measured at 1000 nodes it is 4.5 ms, indistinguishable from the recommendation (§4.3). Rejected on **scrape cost and blast radius**, measured on the collector already on this branch (§4.12): fine at one param per rule (385 ms), 69 % of the 9 s budget at 20 rules × 10 params, and at 42 × 20 it blows the timeout and takes `pmm_managed_inventory_nodes` down with it. Also loses the in-rule literal fallback, and turns an orphaned `alert_rules` row from one inert row into \|inventory\| live series. | -| **Push overrides into VM on change** (import API / remote-write) | Buys only propagation latency — immediate vs ≤ 1 scrape — which is noise against a 60 s evaluation interval. Costs a keep-alive writer and retry/ordering logic. (The tombstone this row once counted against it is now adopted anyway — §4.4 — so it is no longer a differentiator; the writer and the ordering logic still are.) | -| **Cache overrides in a `GaugeVec` mutated on write** | Needs restart warm-up and produces N duplicated series in HA, for no gain: after override-only emission the read is one indexed query over a tens-of-rows table. | -| **A Prometheus-compatible "PMM Thresholds" datasource** | Architecturally cleanest — zero duplication, immediate clears, precedence purely in Go — but the largest new surface (a hand-written API shim whose contract with Grafana's Prometheus datasource is version-sensitive), it touches `build/ansible`, and it puts a synchronous Postgres round-trip inside every rule evaluation. Revisit if thresholds must change without an API call (schedules, computed baselines). | -| **Exception-rule splitting** / thresholds as scrape-target labels | Rule splitting rewrites the base rule anyway and changes alert identity when a target moves between rules, breaking dedup and silences. The label approach needs a vmagent reload per override change and pollutes every scrape target. | - -### Known costs of the recommendation - -- **Two stores.** A rule deleted in Grafana orphans an `alert_rules` row; paid for by the reconciler (step 8), which is safe to run on a timer because an identical group re-POST is a genuine no-op (`"no changes detected in the rule group"`, `activeAt` preserved). -- **The observed expression is evaluated twice per rule evaluation** — once as `$A`, once as the fan-out that manufactures the default's label set. Measured **~1.25×** on a real heavy query (`rate(node_cpu_seconds_total[5m])`, 9.4 → 11.8 ms), not 2×. Re-measure for the heaviest overridable template before GA. -- **The default's target set is "has observed data", not "is in inventory".** A node in inventory whose exporter is down gets no threshold — and no `$A` either, so no alert instance. §8 already accepted that as harmless, but it is a semantic change, not a no-op. -- **A tombstone row per target ever tuned**, never garbage-collected (§4.4). Bounded by human action rather than inventory, but monotonic. -- **The default is a literal in the rule**, so changing a default for everyone is a rule edit. -- **Threshold history lives only as long as VM retention** (~30 days). - ---- - -## 8. Verification - -Verified on two live servers. **Env A:** Grafana 12.4.5, VM v1.147.0, 3 nodes / 1 service. **Env B** -(2026-08-20/21, all figures below unless marked A): PMM 3.7.1, Grafana 12.4.5, **VM v1.149.0**, MR = 10 s, -`latencyOffset=5s`, `disableCache=true`, 11 real nodes / 40 k real series, plus 1000 synthetic nodes with -2 h of backfilled history for the scale figures. - -| Verified | Result | -|---|---| -| Threshold expression: no overrides → all targets at default; one override → that target only | ✅ | -| Override that **raises** the bar suppresses a target the default would have fired | ✅ | -| Target in inventory but reporting no data → no alert instance | ✅ harmless | -| Node + service overrides in one rule, two join labels | ✅ §5.3 | -| Grafana math subset-matches across differing label sets | ✅ | -| Override series expire independently per `(rule, param, target)` | ✅ A | -| `$values.T_.Value` in annotations | ✅ A, multi-param | -| **Clear by absence** (row deleted) | ✅ **309 s**, reproducing env A's 280–300 s | -| **Clear by value change** (row tombstoned, §4.4) | ✅ **14–21 s**, two independent runs — ~15× faster | -| **Inventory fan-out under loss of the fan-out metric** | ✅ raw series gone at 259 s, `T` empty at 856 s, rule `state=inactive health=ok` at **905 s** while `$A` still breached — the silent stop, reproduced | -| **Observed-query fan-out under the same loss** | ✅ `T` held at 80, rule stayed `firing` for the full 26 min | -| Window cost at 1000 nodes: bare / `[15m]` / `[2h]` | ✅ 3.8 / 3.8 / **4.5 ms** — window length is immaterial | -| Observed expression evaluated twice | ✅ 9.4 → **11.8 ms** (~1.25×, not 2×) | -| `samplesScanned` as a cost proxy on VM v1.149.0 | ❌ **invalid** — `count_over_time[1h]` reads 360 samples/series yet reports the same 151 as `last_over_time[1h]`; only `bare = 61` reproduces | -| Emit-everything collector scrape cost vs the 9 s timeout | ✅ §4.12 — 385 ms at 20×1, 69 % of budget at 20×10, blown at 42×20 | -| A blown pmm-managed scrape takes the inventory metric with it | ✅ `up=0`, `scrape_duration=9.001`, `count(pmm_managed_inventory_nodes)` EMPTY | - -Still to verify: - -1. **Filter vs 0/1 semantics after desugaring** — a PromQL comparison without `bool` filters series, Grafana math yields 0/1. The firing set should match; confirm against a real rule including `for:` and NoData. Live check, not a unit test. -2. **Behaviour at scale — query side now measured, collector side partly.** The 1000-node figures above are real, but the nodes were synthetic rows and synthetic series, not registered inventory with live agents. Still open: the override-only collector's scrape cost at scale (§4.12 measured emit-everything; the override-only variant has not been built), and the whole picture under HA with more than one pmm-managed. -3. **Three scopes on a single param**, resolved most-specific-first in Go. -4. **`-promscrape.config.dryRun` acceptance** of a new scrape job, if the dedicated-endpoint follow-up is taken. - -Procedure: [`dev/docs/process/running-and-verifying-locally.md`](dev/docs/process/running-and-verifying-locally.md). - ---- - -## 9. Decision gates - -Promoted out of "open questions" because each can invalidate GA behaviour. None is an engineering -unknown; each needs a decision or a test result. - -| # | Gate | Owner | Blocks | -|---|---|---|---| -| 1 | **`no_data_state` / availability — largely retired by the §2 fan-out change, confirm and close.** The failure was real and measured: a rule with a missing threshold reports `state=inactive, health=ok` at 905 s while `$A` is still breaching. Fanning the default out over the observed expression makes `T` unable to vanish while `$A` resolves, so the cliff is removed structurally rather than bounded by a window — verified over 26 min (§8). Flipping to `NoData` remains non-viable (it would fire for every legitimately absent target). What is left to decide: whether the remaining step — overridden targets reverting to defaults after ~5 min of pmm-managed downtime, detected by `up{job="pmm-managed"}` — is acceptable for GA. | Product + Eng | GA | -| 2 | **Stale-target GC** implemented and tested per §4.8 | Eng | merge | -| 3 | **Desugaring semantics verified live** — firing, pending under `for:`, recovery, NoData, annotation rendering — *before* step 6 is implemented | Eng | step 6 | -| 4 | **Cross-store creation semantics** implemented per §4.9 (ordering, idempotency key, K-cycle absence rule) | Eng | merge | -| 4b | **Registry-row orphan GC** — rows for rules deleted in Grafana, so emitted series do not grow without bound. Narrowed: override rows for deleted nodes/services are handled by the removal API hooks (§4.8), not here | Eng | GA | -| 5 | **Multi-scope precedence signed off** (`service` → `node` → `cluster`; cluster legal only for service-level join labels) *before* the proto freezes | Product | API freeze | -| 5b | **API style chosen** — node-centric vs generic routes (§10 Q1). Routes are additive-only, so deciding late means carrying both families forever | Product + Eng | API freeze | -| 6 | **Shared resolver** (§4.10) is the single implementation of precedence, table-tested across scope combinations | Eng | service/cluster increment | -| 7 | **Fan-out re-measured at representative scale and under HA** | Eng | GA | -| 8 | **Every overridable param emits its `T_` step at creation** — no conditional omission | Eng | merge | - ---- - -## 10. Open questions - -Everything that gates GA or the API freeze has moved to §9, **except the first item below, which is -open but time-critical**: it must be settled before the proto is frozen. - -### 1. Which API style — node-centric routes, or generic scope/target routes? - -**Undecided, and urgent.** HTTP routes are additive-only, exactly like proto fields: whichever style -ships first can never be removed. So if node-centric routes ship now and generic routes are added when -service and cluster scope land, PMM carries **both families indefinitely**. Choosing later is not -neutral — it is choosing duplication. - -Note this is a *different* question from the one already settled. The decision to "generalise the schema -and API now, with node-only behaviour" covers the **message fields** (`scope`/`target` alongside -`node_id`). It does not cover the **paths**, and the paths are the half that cannot be changed afterwards. - -**Style A — node-centric** (what the feature branch implements): - -``` -GET /v1/alerting/nodes/{node_id}/thresholds -POST /v1/alerting/nodes/{node_id}/thresholds -DELETE /v1/alerting/nodes/{node_id}/thresholds/{rule_id}/{param_name} -``` - -**Style B — generic:** - -``` -GET /v1/alerting/thresholds?scope=node&target=[&rule_id=] # all filters optional -POST /v1/alerting/thresholds body: {scope, target, rule_id, param_name, value} -DELETE /v1/alerting/thresholds?scope=&target=&rule_id=¶m_name= -POST /v1/alerting/thresholds:batchUpdate body: {updates: [...]} # optional, see below -``` - -| | Style A — node-centric | Style B — generic | -|---|---|---| -| Adding service/cluster scope later | needs a second route family, kept forever | no new routes | -| Discoverability / REST shape | ✅ clearer resource hierarchy | ⚠️ filters are less self-describing | -| Serves admin and debug reads ("all overrides for rule R", "everything") | ✅ needs extra endpoints | ✅ one route, optional filters | -| UI must send `scope` explicitly | no | yes (trivial) | -| Matches the branch, so no UI rework | ✅ | ⚠️ small client change | - -**Three facts that constrain the choice regardless of which style wins:** - -1. **`target` cannot be a path segment.** `node_id` and `service_id` are opaque IDs, but a `cluster` target is an **arbitrary label value** — `prod/us-east` breaks grpc-gateway path matching outright (path params do not match `/` without `{target=**}`), and spaces or unicode need encoding. PMM's own convention agrees: every existing path segment in `api/` is an opaque ID, never a free-form value. So `/thresholds/{scope}/{target}` is not an option in either style. -2. **A batch method is idiomatic here.** PMM already uses AIP-style custom verbs — `/v1/accesscontrol/roles:assign`, `/v1/actions:startServiceAction`, `/v1/advisors/checks:batchChange`, `/v1/inventory/services:getTypes`. This matters because the UI diffs a whole modal of rows on submit and currently fires **N separate set/delete calls**, so a partial failure leaves the modal half-applied with no way to report which rows landed. `checks:batchChange` is the in-repo precedent for making that one transactional call. Worth deciding alongside the style, since it is the same proto change. -3. **`rule_id` is not a unique key in the response.** Duplicated rules in Grafana share a `pmm_rule_id` (§4.4), so a list can legitimately return two entries with the same `rule_id` *and* `param_name`, differing only in rule title. Nothing breaks — the field is `repeated` — but a client keying a map on `(rule_id, param_name)` would silently collapse them. State it in the proto comments rather than leaving it to be discovered. - -**Recommendation if a tie-break is needed:** Style B, on the strength of the additive-only argument -alone — the cost of being wrong is permanent route duplication, versus a one-off loss of REST -readability. But this is an API-review call, and it belongs with gate 5's proto freeze in §9. - -### Genuinely open, not blocking - -2. **Audit history.** Nothing records "the threshold when this fired" beyond VM retention (~30 days). If - that is required, a small append-only table beats shaping the primary design around it. -3. **Do any planned overridable templates resist the `$X [[ .param ]]` shape?** The AST split - (§5.2) rejects a threshold used inside arithmetic, in a subquery, in a nested comparison, or with - vector matching on the comparison itself. Worth sketching one candidate before step 6 lands, so the - rejection set is validated against real intent rather than assumed. -4. **Whether an immediate delete hook should accompany the reconciler GC** (§4.8) for faster UI feedback. - It cannot replace the reconciler pass — removals can happen while pmm-managed is down, and in HA only - the leader may write — so this is a UX question, not a correctness one. -5. **Dynamic defaults.** The default is a literal in the rule, so changing it for everyone is a rule - edit. If defaults ever need to change without one — schedules, computed baselines — that is the - trigger to revisit the thresholds-datasource alternative in §7, where the whole computation is in Go. diff --git a/dynamic-thresholds-performance-testing.md b/dynamic-thresholds-performance-testing.md deleted file mode 100644 index e5c71a8d40..0000000000 --- a/dynamic-thresholds-performance-testing.md +++ /dev/null @@ -1,356 +0,0 @@ -# Dynamic Alert Thresholds — performance testing session (2026-08-21) - -Companion to [`dynamic-thresholds-main-decision.md`](dynamic-thresholds-main-decision.md). That document's -§8 "Still to verify" item 2 flagged two open scale questions: **the override-only collector's cost has -never been measured (only emit-everything was measured, in §4.12)**, and **service/cluster scope has no -implementation to measure at all**. This session builds just enough of both to get real numbers, then -measures them live on the same kind of dev server §8's numbers came from. It does not attempt to finish -the feature — see [What was deliberately not done](#what-was-deliberately-not-done). - -> A chart-based visual companion to this document — the same measurements, plus the head-to-head -> comparison in [Scenario D](#scenario-d--override-only-vs-emit-everything-head-to-head) and the fixed -> real-world parameter sweep in [Scenario E](#scenario-e--fixed-real-world-parameter-sweep-1000-nodes-250-overrides-120-rules) -> — was published as an Artifact ("Threshold collector benchmarks"). - -## Environment - -Same dev container (`pmm-server`, `perconalab/pmm-server:3-dev-latest`) used for the live measurements -already in §8 of the decision doc, on this branch (`PMM-14912-dynamic-thresholds`, `73b7032db`). - -| | | -|---|---| -| PostgreSQL | 14.24 (Percona Distribution), `max_connections=2000` | -| VictoriaMetrics | scraped every `MR = 10 s` | -| Grafana | 12.4.5 | -| Real inventory | 11 nodes, 10 services (MySQL/PostgreSQL/Valkey/MongoDB test instances) | -| ClickHouse / qan-api2 | **down for this whole session** — ClickHouse is crash-looping on an unrelated `AccessControl` config error, pre-existing in this container, nothing to do with this feature. QAN has no dependency on alert thresholds, so this doesn't affect anything below; noted for completeness only. | - -Baseline `/debug/metrics` scrape before any change: **25 ms**, 499 lines, 58 KB, 0 threshold series. - -### Baseline infra snapshot (before synthetic load) - -| Store | Size | Detail | -|---|---|---| -| PostgreSQL | 10 MB total | Largest tables: `agents` 312 KB, `nodes` 280 KB, `alert_rules` 64 KB, `alert_rule_threshold_overrides` 32 KB (empty), `settings` 48 KB. 20 active connections. | -| VictoriaMetrics | ~59 MB on disk (`indexdb` 18.5 MB + `storage/small` 46.6 MB) | 42,323 active series, 467,147 label-value pairs, 59 scrape targets, ingesting ~5,745 rows/s, 892 MB resident. | -| ClickHouse | 116 KB | Crash-looping (see above); no QAN data this session. | -| pmm-managed process | 628 MB RSS, 162 goroutines | Sampled via `process_resident_memory_bytes{job="pmm-managed"}` / `go_goroutines{job="pmm-managed"}` through VictoriaMetrics. | -| Host | load1 ≈ 3.8, memory ≈ 31.6% used | Sampled via `node_load1` / `node_memory_*` for the `pmm-server` node — the same series PMM's own Node Overview / health dashboards render. The in-app Browser tool couldn't reach `https://localhost` (sandboxed-network policy blocks local-network navigation), so these were pulled directly from VictoriaMetrics' query API instead of a dashboard screenshot — same underlying data. | - -## What changed on this branch for this session - -The branch's existing collector (`threshold_metrics.go`) implemented **emit-everything** — the design -§7 of the decision doc rejects, and §4.12 measured. To benchmark the **recommended** design (override-only -emission, tombstones, multi-scope resolver) there was nothing to run yet, so this session built a real, -working version of the two pieces actually in scope (see the earlier scoping discussion in this -conversation): - -| File | Change | -|---|---| -| [`managed/models/database.go`](managed/models/database.go) | Migration 119's `alert_rule_threshold_overrides` rewritten to the polymorphic `scope`/`target` schema with a `cleared_at` tombstone column and the NaN/±Inf `CHECK`, per §4.4/§4.5 of the decision doc. | -| [`managed/models/alert_rule_threshold_override_model.go`](managed/models/alert_rule_threshold_override_model.go) | `ThresholdScope` type + `node`/`service`/`cluster` constants; struct fields `Scope`/`Target`/`ClearedAt` replacing `NodeID`. Regenerated via `go generate` (reform), not hand-edited. | -| [`managed/models/alert_rule_helpers.go`](managed/models/alert_rule_helpers.go) | `FindThresholdOverridesByTarget`, tombstone-aware `UpsertThresholdOverride`/`ClearThresholdOverride`, hard-delete `DeleteThresholdOverridesForTarget` (refuses `cluster` scope, per §4.8). | -| [`managed/models/threshold_resolver.go`](managed/models/threshold_resolver.go) *(new)* | `ResolveThresholds` — the shared precedence resolver from §4.10: node > service > cluster, tombstones contribute no candidate for their own scope, unresolvable targets are skipped. Table-tested in [`threshold_resolver_test.go`](managed/models/threshold_resolver_test.go) (7 cases) plus a microbenchmark. | -| [`managed/models/service_helpers.go`](managed/models/service_helpers.go) | `FindServicesByClusters` — bounded by the clusters actually queried, for cluster-scope expansion. | -| [`managed/services/alerting/threshold_metrics.go`](managed/services/alerting/threshold_metrics.go) | Full rewrite: override/tombstone-only emission (`pmm_alert_threshold_override{rule_id,param,target}`), one query for overrides + bounded ID→name/cluster→services resolution, 3 s `Collect` timeout, `Describe` sends the descriptor directly instead of `prom.DescribeByCollect` — the three defects §4.12/§6 step 3 called out, fixed in the same pass. **Both emission modes now live side by side** behind a `ThresholdEmitMode` parameter: `ThresholdEmitOverridesOnly` (default, recommended) and `ThresholdEmitEveryTarget` (the §7-rejected alternative, generalised from its original node-only shape to also emit for every service, so cluster/service scope can be A/B'd too). Selected via `PMM_DEV_THRESHOLD_EMIT_MODE=all-targets` — a dev-only toggle (`PMM_DEV_` prefix, never a GA knob), read once at startup in `managed/cmd/pmm-managed/main.go`. | -| [`managed/services/alerting/threshold_overrides.go`](managed/services/alerting/threshold_overrides.go) | Adapted to the new schema; `DeleteNodeThreshold` now calls `ClearThresholdOverride` (tombstone) instead of a hard delete, and `ListNodeThresholds` skips tombstoned rows — both required by §4.4. | - -`go build ./managed/...` and `go vet ./managed/...` are clean; `gofmt -l` reports nothing. Existing -`rule_builder_dynamic_test.go` tests (unaffected — they don't touch this schema) still pass. Two pre-existing, -unrelated failures were left alone: a host-only `mkdir /srv: read-only file system` failure in -`service_test.go` (macOS host has no `/srv`) and a flaky timing assertion in -`software_version_helpers_test.go`; neither touches thresholds. - -## Methodology - -Same technique as the decision doc's §4.12 emit-everything table: seed real Postgres rows in the dev -container, hot-swap the rebuilt `pmm-managed` binary (`make env-root TARGET=run-managed-ci`), and time -repeated `GET /debug/metrics` scrapes. Where the shared endpoint's total time was dominated by other, -unrelated collectors (see below), `EXPLAIN ANALYZE` isolates this collector's own two queries, and a Go -microbenchmark isolates `ResolveThresholds` in memory. All synthetic rows were prefixed `bench-` and -deleted at the end of the session; the container was left in its original state (11 nodes / 10 services / -1 pre-existing rule / 0 overrides). - -## Results - -### A — override count, node scope (2,000 synthetic nodes held constant) - -| Overrides | median | max | emitted series | scrape total lines | -|---|---|---|---|---| -| 0 | 108 ms | 133 ms | 0 | 2,530 | -| 100 | 119 ms | 175 ms | 100 | 2,643 | -| 1,000 | 124 ms | 141 ms | 1,000 | 3,549 | -| 2,000 (all nodes overridden) | 143 ms | 192 ms | 2,000 | 4,549 | - -Collector's own added cost going from 0 → 2,000 override series: **~35 ms**, not the multiplicative blowup -the old design showed. The 108 ms floor here is the pre-existing, unrelated inventory collector iterating -2,011 nodes on every scrape (see below) — not this feature. - -### A4 — same total overrides, fragmented across many rules/params - -The old design's real exposure (§4.12) was **rules × params**, not raw override count: 1011×42×20 blew the -9 s timeout at 26.9 s. This is the one number from that table worth reproducing under the new design: - -| Scenario | median | max | emitted series | -|---|---|---|---| -| 1,000 overrides, 1 rule × 1 param | 124 ms | 141 ms | 1,000 | -| 840 overrides, **42 rules × 20 params** | 121 ms | 136 ms | 840 | - -Statistically indistinguishable. Fragmenting the same override volume across 840 distinct `(rule, param)` -groups costs nothing extra — the collector still issues exactly one overrides query and one bounded -node lookup, then partitions in memory. **This is the direct answer to §4.12's worst row**: the -multi-param exposure was a property of emit-everything's node-multiplication, and it is structurally -gone under override-only emission, not just empirically smaller. - -### B — total inventory size, override count held constant at 50 - -| Total nodes | median | max | emitted series (this collector) | -|---|---|---|---| -| 2,000 | 108 ms | 118 ms | 50 | -| 6,000 | 324 ms | 382 ms | 50 | -| 10,000 | 461 ms | 508 ms | 50 | - -Emitted series from **this** collector stayed at 50 throughout — the ~350 ms of growth is entirely the -pre-existing `pmm_managed_inventory_nodes` collector (unrelated to this feature) still emitting one line -per node. Isolated with `EXPLAIN ANALYZE` at the 10,000-node point: - -- `SELECT * FROM alert_rule_threshold_overrides` (all 50 rows): **1.7 ms** -- `SELECT * FROM nodes WHERE node_id IN (<50 ids>)`, index-bounded: **3.6 ms** - -Under 6 ms combined, flat regardless of total inventory — confirming the design's central claim -(§4.12/§9 gate 7) that this collector's cost is bounded by *targets ever tuned*, not fleet size. - -### C — cluster-scope expansion (new: not measured anywhere before this session) - -Synthetic services seeded per cluster, one cluster-scope override per cluster on top of the scenario-B -inventory (so total scrape time below still carries that unrelated ~460 ms inventory-collector floor): - -| Cluster overrides | Services in those clusters | emitted series | scrape median | scrape max | -|---|---|---|---|---| -| 1 (100 services) | 100 | 100 | 799 ms | 886 ms | -| 2 (+1,000 services) | 1,100 | 1,100 | 823 ms | 1,033 ms | -| 3 (+5,000 services) | 6,100 | 6,100 | 845 ms | 1,055 ms | -| **50 small clusters × 200 services** (§4.2's stated worst case) | 10,000 | 10,000 | 1,482 ms | 1,566 ms | - -Isolated: - -- `SELECT * FROM services WHERE cluster IN (<3 clusters>)` at 6,100 matched rows: **23 ms** -- Same query for the 50-cluster case, 10,000 matched rows out of 16,110 total synthetic services: **47 ms** -- `ResolveThresholds` in memory, 50 cluster overrides expanding to 10,000 candidates (Go microbenchmark, - `BenchmarkResolveThresholds`): **~1 ms** - -§4.2's cardinality warning — "a cluster override over 200 nodes would emit 200 series" — is real and the -collector handles it correctly at 50× that scale, cheaply. The cluster-expansion query -(`services.cluster IN (...)`) is a **sequential scan** — there is no index on `services.cluster`. At -16,110 synthetic rows it's still only 47 ms. - -> **Revised in Scenario E below.** This session first flagged that scan as an "index it before cluster -> scope ships" action item. A more careful re-test — a *selective* override set (10% of clusters, not -> the ~100% coverage the number above was measured against) at 10,010 services — still executes in -> **6.6 ms**. Postgres reads a table this size in one or two pages regardless of a `WHERE` clause; an -> index cannot beat that. **The recommendation is retracted**: don't add the index on the strength of -> this testing. Revisit only if a real deployment's `services` table is one to two orders of magnitude -> larger than anything measured here — the scan cost is linear in table size, so re-test at that scale -> rather than assuming today's numbers still hold. - -## Scenario D — override-only vs. emit-everything, head to head - -Everything above measured the recommended design in isolation. To compare it directly against the -rejected alternative on identical data, the collector now carries **both** emission modes side by side -(see [What changed](#what-changed-on-this-branch-for-this-session) above) — a live toggle, not a -re-implementation from memory of §4.12's numbers. `ThresholdEmitEveryTarget` is a deliberate -generalisation of that historical, node-only design: it now also emits for every **service**, so the -comparison covers cluster/service scope too, not just node scope. - -### D1 — node scope, same override sweep as scenario A - -2,000-node inventory; two registered rules with default params (the pre-existing real rule plus -`bench-rule-1`) — same DB state, same moment, mode flipped via `PMM_DEV_THRESHOLD_EMIT_MODE` and a -restart between passes: - -| Overrides | override-only median / max | override-only series | all-targets median / max | all-targets series | -|---|---|---|---|---| -| 0 | 118 / 203 ms | 0 | 149 / 199 ms | **4,042** | -| 100 | 111 / 112 ms | 100 | 163 / 174 ms | **4,042** | -| 1,000 | 119 / 138 ms | 1,000 | 199 / 221 ms | **4,042** | -| 2,000 | 143 / 150 ms | 2,000 | 209 / 243 ms | **4,042** | - -The all-targets column is already the whole point: **4,042 emitted series regardless of override -count — including at zero.** That number is `2 rules × 1 param × (2,011 nodes + 10 services)`; it has -nothing to do with how many overrides exist, because this mode was never about overrides, it's about -inventory. Override-only's series count tracks the override count exactly, and its scrape time is lower -at every point on the sweep. A second, smaller effect: all-targets' own time still drifts upward with -override count (149→209 ms) even though its emitted-series count doesn't move — `ResolveThresholds` has -more candidates to fold into its per-target map, a real but minor cost next to the ~150 ms fixed floor of -walking the full inventory twice per rule. - -### D2 — cluster scope, same DB state under both modes - -2,011 nodes + 6,100 synthetic services across 3 clusters, one cluster-scope override per cluster: - -| Mode | median | max | emitted series | -|---|---|---|---| -| override-only | 550 ms | 925 ms | 6,100 | -| all-targets | 681 ms | 692 ms | **16,242** | - -All-targets pays for the full `2 rules × 1 param × (2,011 nodes + 6,100 services)` universe — 2.7× the -series override-only emits for the exact same three cluster overrides, because it was always going to emit -for every service whether or not a cluster override existed. - -### The extreme point, not a controlled pair - -Pushing all-targets to `2,011 nodes + 16,100 services` (50 small clusters × 200 services, matching §4.12's -worst-case shape but now over nodes *and* services): **36,242 emitted series, 1,790 ms median / 1,885 ms -max** — still under the 9 s budget for one param on one rule pair, but this is exactly the shape that hit -26.9 s at 42 rules × 20 params in §4.12's original, node-only measurement. The nearest override-only -comparison is scenario C's 50-cluster point (§C above): **10,000 series, 1,482 ms**, measured over a -different total inventory (10,000 nodes instead of 2,011). The two runs aren't a controlled pair — don't -read the exact ms gap as precise — but the series counts are exact and the qualitative result matches D1 -and D2 at every controlled point: override-only's cost is anchored to what was actually overridden, -all-targets' is anchored to total inventory size, full stop. - -*(Operational side note, not a thresholds finding: seeding 16,100 synthetic services made the unrelated -Advisor **checks** service log an error per service — "no available pmm agents" — on its periodic pass, -which briefly slowed `pmm-managed`'s graceful shutdown between mode-toggle restarts. Nothing to do with -the threshold collector; mentioned only because it's what made a restart briefly look stuck.)* - -## Scenario E — fixed real-world parameter sweep (1,000 nodes, 250 overrides, 120 rules) - -Requested directly: a single fixed, realistic parameter set — **1,000 nodes, 250 overrides, 100 rules with -1 param + 20 rules with 2 params** (140 `(rule, param)` groups total) — run through node, service, and -cluster scope, each under both emission modes. Unlike scenarios A–D, this inventory is **left seeded on -the dev container** for further exploration (see the note at the end of this section), rather than cleaned -up after measuring. - -### Setup - -| | | -|---|---| -| Nodes | 1,000 synthetic + 11 real = 1,011 | -| Services | 1,000 synthetic (100 clusters × 10) + 10 real = 1,010, later expanded to 10,010 (1,000 clusters × 10) for the cluster-selectivity re-test below | -| Alert rules | 100 with 1 param (`threshold`) + 20 with 2 params (`param_a`, `param_b`) = **140 `(rule, param)` groups** | -| Overrides | 250 rows, spread across all 140 groups (~1.8 per group on average), scope varied per pass | - -The 250 overrides were distributed with a fixed, reproducible mapping (`idx % 140` → group, `idx` → target) -so the same 250-row shape is tested identically at each scope, and it self-verified: no two rows landed on -the same `(rule, param, scope, target)` key, so all 250 inserted cleanly under the real -`UNIQUE (rule_id, param_name, scope, target)` constraint every time. - -### Results - -| Scope | Mode | Emitted series | Scrape median | Scrape max | Isolated DB cost | -|---|---|---|---|---|---| -| Node | overrides-only | **250** | 108 ms | 114 ms | overrides scan 1.6 ms + node lookup 2.1 ms | -| Node | all-targets | **284,961** | **6,907 ms** | 6,907 ms | dominated by the emission loop, not the query | -| Service | overrides-only | **250** | 109 ms | 111 ms | service lookup 2.3 ms | -| Service | all-targets | **284,961** | **7,394 ms** | 7,394 ms | same order as node scope — see below | -| Cluster | overrides-only, 100% of 100 clusters selected (1,010 services) | **2,500** | 156 ms | 164 ms | seq scan, matches ~100% of table: 5.9 ms | -| Cluster | overrides-only, 10% of 1,000 clusters selected (10,010 services) | **2,500** | 620 ms | 818 ms | seq scan, 10% selective: 6.6 ms | -| Cluster | all-targets | *not run — see below* | — | — | — | - -Two clean, exact formulas fall out of these six rows: - -- **override-only emits exactly `resolved distinct targets`** — 250 override rows resolve to 250 distinct - node/service names at node/service scope, and to **2,500** at cluster scope, because each of the 250 - cluster-scope rows expands onto its cluster's 10 services (250 × 10 = 2,500, exactly what was measured — - not an estimate). -- **all-targets emits exactly `groups × (nodes + services)`, independent of override count and scope** — - **141** groups (the 140 bench groups plus one pre-existing real alert rule left over from earlier in the - session, confirmed by `SELECT sum(...jsonb_object_keys(default_params)...) FROM alert_rules` = 141) × - `(1,011 + 1,010) = 2,021` = **284,961** — exact, not approximate; there is no rounding gap. Node scope and - service scope produced the **same** all-targets series count and the same order-of-magnitude scrape time - (6,907 ms vs 7,394 ms) — confirming empirically, not just architecturally, that all-targets' cost has - nothing to do with which scope is being overridden. - -**The headline number: 6.9–7.4 seconds of a 9-second scrape budget**, for just 140 `(rule, param)` groups — -the exact shape §4.12 warned about (there, 42 rules × 20 params on nodes alone hit 26.9 s and blew the -budget outright). This run didn't blow the budget, but it used **77–82% of it**, with only 120 rules in -play — a fraction of what a real PMM deployment (42+ shipped templates, more once overridable params -grow) would register once several are made overridable under this design. - -**A collector-implementation note surfaced by this run, not by earlier ones:** the 3 s `Collect` timeout -this session added (§4.12/§6 step 3) bounds the *DB query* phase, but `collectEveryTarget`'s emission loop -— the part that actually costs 6–7 s here — runs entirely in Go after the queries return, with no -`ctx.Done()` check. **The timeout does not actually cap all-targets' worst case.** This is a real gap in -the current implementation of the (already-rejected) alternative, not a design-doc gap — worth noting -precisely because it means the measured 6.9–7.4 s undersells the actual risk: without a check inside the -loop, a slightly larger rule count would run past 9 s with nothing stopping it. - -**Cluster-scope all-targets was deliberately not run at the 10,010-service inventory.** Extrapolating the -formula above: `141 × (1,011 + 10,010) = 1,553,961` series — over 5× the 284,961-series run that already -took ~7 s and ~425% host CPU. Running it risked destabilizing the shared dev container (other test -databases run alongside it) for a number whose conclusion is already obvious from the formula. Treat -1.5M as an estimate, not a measurement, and note it as exactly that if it's cited elsewhere. - -### The index question, revisited — retracted - -Scenario C flagged the missing index on `services.cluster` as an action item, measured against an override -set covering essentially the whole services table (100%-selective). Re-tested here deliberately -*selectively* — 100 of 1,000 clusters overridden, 10,010 total services — the sequential scan still runs in -**6.6 ms**. Postgres reads a table this size in a handful of pages regardless of the `WHERE` clause; there -is no query for an index to speed up yet. **The recommendation is retracted.** Postgres' sequential-scan -cost is linear in table size, so this conclusion should be re-tested rather than assumed if a real -deployment's `services` table is one to two orders of magnitude larger than anything measured here — but -nothing in this session's data supports adding the index now. - -### Left in place for further exploration - -Unlike scenarios A–D, this session's seed data was **not** cleaned up. As of this section, the dev -container carries: 1,011 nodes, 10,010 services (1,000 clusters × 10 + real), 121 alert rules, and 250 -cluster-scope overrides (the last configuration measured) resolving to 2,500 effective thresholds. The -collector is back on `overrides-only` (the recommended default) — `all-targets` was never left active. - -## PMM health during the load - -Sampled while the 10,000-node / 16,000-service synthetic inventory was in place (the heaviest point of -this session, scenario C's worst row): host load1 stayed the same order of magnitude, `pmm-managed` RSS -and goroutine count did not visibly spike attributable to this collector specifically — the dominant -cost throughout was the pre-existing inventory collectors serializing tens of thousands of unrelated -lines into the same `/debug/metrics` response, not anything this feature adds. No scrape exceeded the -`0.9 × MR = 9 s` budget at any point tested here. - -## What was deliberately not done - -Scoped out at the start of this session, not discovered as blockers partway through: - -- **`rule_builder.go` / PromQL generation is untouched.** The `T_` / `label_replace` / fan-out-over- - observed-expression query shape from §2/§4.3 was not wired up; the existing direct-join rule builder - and its tests are unaffected. Query-side cost for node scope was already measured live in §8 of the - decision doc and remains valid. There is **no** end-to-end (Grafana rule → collector → firing) test of - service/cluster scope from this session — only the collector and resolver were exercised directly - against Postgres. -- **No proto/API changes.** `scope`/`target` are not exposed externally; the existing node-centric - endpoints now route through `ThresholdScopeNode` internally, unchanged from the outside. -- **Removal hooks not wired.** `DeleteThresholdOverridesForTarget` exists but isn't called from - `RemoveNode`/`RemoveService` yet (§4.8). -- **Reconciler untouched**, **HA not tested** — this session used one `pmm-managed` instance; §9 gate 7's - "under HA" half is still open. -- Not a shippable increment by itself — this is benchmarking-oriented code sized to answer the two - questions asked (override-only collector cost, service/cluster resolver cost), not to complete the - ten-step plan in §6 of the decision doc. - -## Bottom line - -Both open scale questions from §8/§9 now have real numbers instead of none: - -1. **The override-only collector costs what the design assumed it would** — single-digit milliseconds of - DB time, flat with inventory size, flat with how many rules/params the same override volume is spread - across. The multi-param blowup that killed emit-everything (§4.12, 42×20 → 26.9 s) does not reproduce - under this design at any scale tried here. -2. **Cluster-scope expansion works and is cheap up to 10,000 expanded series.** No index needed on - `services.cluster` — retracted in Scenario E after a more careful re-test at realistic selectivity. -3. **Head to head, on identical data, override-only wins at every point tested** (scenario D) — and the - gap is structural, not incidental: emit-everything's cost is a function of *inventory size*, so it pays - the same ~4,042-series price whether zero or two thousand overrides exist, while override-only's cost - is a function of *what was actually tuned*. Both modes now live in the same binary - (`PMM_DEV_THRESHOLD_EMIT_MODE=all-targets`), so this isn't a claim resting on the branch's older, - now-replaced code — it's the same code path, same DB, same endpoint, mode flipped by one env var. -4. **At a fixed, realistic parameter set (1,000 nodes, 250 overrides, 140 rule×param groups — Scenario E), - all-targets consumed 77–82% of the 9 s scrape budget** regardless of scope, while override-only used - 1–7%. This is with only 120 rules; more overridable templates only makes the gap worse, and the - `Collect` timeout added this session does not actually bound this cost — a real implementation gap in - the (rejected) alternative, not a design gap. - -HA and the full query-side integration with service/cluster join labels remain open per -[§9](dynamic-thresholds-main-decision.md#9-decision-gates) — this session narrows, but does not close, -those gates. From cb361f19018c573d6a08445f63b7a2b8c7afbbc2 Mon Sep 17 00:00:00 2001 From: Matej Kubinec Date: Thu, 3 Sep 2026 09:01:26 +0200 Subject: [PATCH 14/15] PMM-14912 Cover threshold validation and ordering Add tests for the branches that carry logic rather than error plumbing: the minimum bound in checkThresholdValue, which was enforced but never asserted; the rule-id filter in thresholdRules; the comparator behind the table's row order; and both scope conversions, which only had their node arm exercised. Also cover targetNames for a target that has outlived its inventory entry, and IsOverridden, which reads as uncovered because it is called only from another package. --- managed/models/threshold_resolver_test.go | 40 ++++++ .../alerting/threshold_overrides_test.go | 136 ++++++++++++++++++ 2 files changed, 176 insertions(+) diff --git a/managed/models/threshold_resolver_test.go b/managed/models/threshold_resolver_test.go index 043fee0d8b..baf6c0e318 100644 --- a/managed/models/threshold_resolver_test.go +++ b/managed/models/threshold_resolver_test.go @@ -260,3 +260,43 @@ func BenchmarkResolveThresholds(b *testing.B) { ResolveThresholds(overrides, testDefault, inv) } } + +func TestResolvedThresholdIsOverridden(t *testing.T) { + t.Parallel() + + // The API reports this as `is_overridden`, and the UI uses it to decide whether the + // reset control is live, so a default that claims to be an override would offer a + // reset that does nothing. + assert.False(t, ResolvedThreshold{Value: 80}.IsOverridden()) + assert.True(t, ResolvedThreshold{Value: 90, Source: &AlertRuleThresholdOverride{}}.IsOverridden()) +} + +func TestTargetNamesSkipsTargetsOutsideInventory(t *testing.T) { + t.Parallel() + + // The override table is polymorphic and carries no foreign key, so a row can outlive + // its target. Such a row must resolve to nothing rather than to an empty join-label + // value, which would match every series the rule produces. + inv := ThresholdInventory{ + NodeNames: map[string]string{"node-id-1": "node-1"}, + ServiceNames: map[string]string{"service-id-1": "service-1"}, + ServicesByCluster: map[string][]string{"prod": {"service-1"}}, + } + + assert.Nil(t, inv.targetNames(&AlertRuleThresholdOverride{ + Scope: ThresholdScopeService, Target: "service-id-gone", + })) + assert.Nil(t, inv.targetNames(&AlertRuleThresholdOverride{ + Scope: ThresholdScopeNode, Target: "node-id-gone", + })) + assert.Nil(t, inv.targetNames(&AlertRuleThresholdOverride{ + Scope: ThresholdScope("nonsense"), Target: "prod", + })) + + assert.Equal(t, []string{"service-1"}, inv.targetNames(&AlertRuleThresholdOverride{ + Scope: ThresholdScopeService, Target: "service-id-1", + })) + assert.Equal(t, []string{"service-1"}, inv.targetNames(&AlertRuleThresholdOverride{ + Scope: ThresholdScopeCluster, Target: "prod", + })) +} diff --git a/managed/services/alerting/threshold_overrides_test.go b/managed/services/alerting/threshold_overrides_test.go index 9025ecc96b..2a3d19810f 100644 --- a/managed/services/alerting/threshold_overrides_test.go +++ b/managed/services/alerting/threshold_overrides_test.go @@ -315,3 +315,139 @@ func TestBatchUpdateThresholds(t *testing.T) { assert.Empty(t, overrides, "the first update must not survive the second one failing") }) } + +func TestThresholdScopeConversion(t *testing.T) { + t.Parallel() + + // Service and cluster already exist in the schema, the resolver and the proto, so + // they report as not-yet-implemented rather than as a malformed request. Enabling + // them later is then a validation change rather than an API change. + for _, scope := range []alerting.ThresholdScope{ + alerting.ThresholdScope_THRESHOLD_SCOPE_SERVICE, + alerting.ThresholdScope_THRESHOLD_SCOPE_CLUSTER, + } { + _, err := thresholdScopeFromAPI(scope) + require.Error(t, err, scope.String()) + assert.Equal(t, codes.Unimplemented, status.Code(err), scope.String()) + } + + // An unset scope means node, so a client that only ever deals with nodes need not + // send one. + got, err := thresholdScopeFromAPI(alerting.ThresholdScope_THRESHOLD_SCOPE_UNSPECIFIED) + require.NoError(t, err) + assert.Equal(t, models.ThresholdScopeNode, got) + + _, err = thresholdScopeFromAPI(alerting.ThresholdScope(-1)) + require.Error(t, err) + assert.Equal(t, codes.InvalidArgument, status.Code(err)) + + // The reverse mapping must cover every scope, not just the settable ones: an override + // stored at a scope the API cannot yet set still has to be reportable. + assert.Equal(t, alerting.ThresholdScope_THRESHOLD_SCOPE_NODE, thresholdScopeToAPI(models.ThresholdScopeNode)) + assert.Equal(t, alerting.ThresholdScope_THRESHOLD_SCOPE_SERVICE, thresholdScopeToAPI(models.ThresholdScopeService)) + assert.Equal(t, alerting.ThresholdScope_THRESHOLD_SCOPE_CLUSTER, thresholdScopeToAPI(models.ThresholdScopeCluster)) + assert.Equal(t, alerting.ThresholdScope_THRESHOLD_SCOPE_UNSPECIFIED, thresholdScopeToAPI(models.ThresholdScope("nonsense"))) +} + +func TestSortThresholds(t *testing.T) { + t.Parallel() + + // This order is what the table renders. ListThresholds gathers a rule's parameters + // from a map, so without a total order the rows would reshuffle between two reads of + // unchanged data. + thresholds := []*alerting.Threshold{ + {RuleId: "rule-b", ParamName: "threshold", Target: "node-1"}, + {RuleId: "rule-a", ParamName: "threshold", Target: "node-2"}, + {RuleId: "rule-a", ParamName: "threshold", Target: "node-1"}, + {RuleId: "rule-a", ParamName: "another", Target: "node-9"}, + } + + sortThresholds(thresholds) + + got := make([]string, 0, len(thresholds)) + for _, threshold := range thresholds { + got = append(got, threshold.RuleId+"/"+threshold.ParamName+"/"+threshold.Target) + } + + assert.Equal(t, []string{ + "rule-a/another/node-9", + "rule-a/threshold/node-1", + "rule-a/threshold/node-2", + "rule-b/threshold/node-1", + }, got) +} + +func TestSetThresholdRejectsValueBelowMinimum(t *testing.T) { + svc, _, node := setupThresholdAPI(t) + + // The maximum is covered by TestSetThreshold; the minimum is the other half of the + // same guard, and nothing else would catch it being wrong. + _, err := svc.SetThreshold(t.Context(), &alerting.SetThresholdRequest{ + Scope: alerting.ThresholdScope_THRESHOLD_SCOPE_NODE, Target: node.NodeID, + RuleId: thresholdTestRuleID, ParamName: "threshold", Value: -5, + }) + require.Error(t, err) + assert.Equal(t, codes.InvalidArgument, status.Code(err)) +} + +func TestSetThresholdRejectsScopeTheParameterDoesNotDeclare(t *testing.T) { + svc, db, node := setupThresholdAPI(t) + + // A parameter carries the scopes its template declared. This rule's parameter joins + // on service_name, so overriding it per node would produce a threshold series the + // rule can never match. + const ruleID = "service-scoped-rule" + + _, err := models.CreateAlertRule(db.Querier, &models.CreateAlertRuleParams{ + RuleID: ruleID, + Params: models.AlertRuleParams{ + "threshold": { + Default: 80, + JoinLabel: "service_name", + Scopes: []string{string(models.ThresholdScopeService)}, + }, + }, + }) + require.NoError(t, err) + + _, err = svc.SetThreshold(t.Context(), &alerting.SetThresholdRequest{ + Scope: alerting.ThresholdScope_THRESHOLD_SCOPE_NODE, Target: node.NodeID, + RuleId: ruleID, ParamName: "threshold", Value: 90, + }) + require.Error(t, err) + assert.Equal(t, codes.InvalidArgument, status.Code(err)) +} + +func TestListThresholdsFiltersByRule(t *testing.T) { + svc, db, node := setupThresholdAPI(t) + ctx := t.Context() + + const otherRuleID = "threshold-api-rule-2" + + _, err := models.CreateAlertRule(db.Querier, &models.CreateAlertRuleParams{ + RuleID: otherRuleID, + Params: models.AlertRuleParams{ + "threshold": { + Default: 50, + JoinLabel: "node_name", + Scopes: []string{string(models.ThresholdScopeNode)}, + }, + }, + }) + require.NoError(t, err) + + res, err := svc.ListThresholds(ctx, &alerting.ListThresholdsRequest{ + Scope: alerting.ThresholdScope_THRESHOLD_SCOPE_NODE, Target: node.NodeID, + }) + require.NoError(t, err) + require.Len(t, res.Thresholds, 2, "both registered rules apply to the target") + + res, err = svc.ListThresholds(ctx, &alerting.ListThresholdsRequest{ + Scope: alerting.ThresholdScope_THRESHOLD_SCOPE_NODE, Target: node.NodeID, + RuleId: otherRuleID, + }) + require.NoError(t, err) + require.Len(t, res.Thresholds, 1) + assert.Equal(t, otherRuleID, res.Thresholds[0].RuleId) + assert.InDelta(t, 50.0, res.Thresholds[0].DefaultValue, 0.0001) +} From ac4939093b0c233fbd46e3cebdb78725e7dfcb89 Mon Sep 17 00:00:00 2001 From: Matej Kubinec Date: Fri, 4 Sep 2026 12:49:29 +0200 Subject: [PATCH 15/15] PMM-14912 Document alert threshold API Add reference pages for the four threshold endpoints: the scope and precedence model, both list modes, set and clear, and the batch endpoint. Covers the parts a client gets wrong by default - that an absent numeric field means zero rather than unknown, that clearing an override is not the same as writing the default back, and that omitting value in a batch entry is what clears it. --- .../api/alerting/batch-update-thresholds.md | 66 ++++++++++++++ documentation/api/alerting/list-thresholds.md | 82 ++++++++++++++++++ documentation/api/alerting/overview.md | 56 ++++++++++++ .../api/alerting/set-clear-threshold.md | 85 +++++++++++++++++++ 4 files changed, 289 insertions(+) create mode 100644 documentation/api/alerting/batch-update-thresholds.md create mode 100644 documentation/api/alerting/list-thresholds.md create mode 100644 documentation/api/alerting/overview.md create mode 100644 documentation/api/alerting/set-clear-threshold.md diff --git a/documentation/api/alerting/batch-update-thresholds.md b/documentation/api/alerting/batch-update-thresholds.md new file mode 100644 index 0000000000..c2b1bc4ddc --- /dev/null +++ b/documentation/api/alerting/batch-update-thresholds.md @@ -0,0 +1,66 @@ +--- +title: Batch update alert thresholds +slug: batch-updating-alert-thresholds +category: + uri: alerting-api +position: 3 +--- + +## Batch update alert thresholds + +Applies several threshold changes in a **single transaction**: either every update lands or +none does. + +This is what a form editing several rows at once should use. Issuing the changes as separate +[Set](ref:setthreshold) and [Clear](ref:clearthreshold) calls risks a partial result that the +client cannot report coherently — some rows saved, one rejected, and no way to tell the user +which state the system is now in. + +```shell +curl --insecure -X POST \ + --header 'Authorization: Bearer XXXXX' \ + --header 'Content-Type: application/json' \ + --url https://127.0.0.1/v1/alerting/thresholds:batchUpdate \ + --data ' +{ + "updates": [ + { + "scope": "THRESHOLD_SCOPE_NODE", + "target": "dc1f7e40-1b1a-4c5d-9f2e-2b6a1e3f4c5d", + "rule_id": "1f8b2c34-5d6e-4a7b-8c9d-0e1f2a3b4c5d", + "param_name": "threshold", + "value": 95 + }, + { + "scope": "THRESHOLD_SCOPE_NODE", + "target": "dc1f7e40-1b1a-4c5d-9f2e-2b6a1e3f4c5d", + "rule_id": "2a9c3d45-6e7f-4b8c-9d0e-1f2a3b4c5d6e", + "param_name": "threshold" + } + ] +} +' +``` + +### Setting and clearing in one call + +Whether an entry sets or clears is decided by `value`: + +- **`value` present** — sets the override to that value. +- **`value` omitted** — clears the override. + +The second entry in the example above clears its threshold, because it has no `value`. + +> 🚧 Omit the field, do not send zero +> +> `value` is optional precisely so that omitting it can mean *clear*. Sending `"value": 0` sets the threshold to zero, which is a real and very different instruction. + +### Response + +The response lists the thresholds that are now overridden. **Cleared entries are omitted** — +after a successful clear there is no override to report, so a request of three sets and two +clears returns three thresholds. + +At least one update is required. Validation is the same as for +[Set Alert Threshold](ref:setthreshold), applied to every entry; one invalid entry rolls the +whole batch back and nothing is written. diff --git a/documentation/api/alerting/list-thresholds.md b/documentation/api/alerting/list-thresholds.md new file mode 100644 index 0000000000..bc899a2e01 --- /dev/null +++ b/documentation/api/alerting/list-thresholds.md @@ -0,0 +1,82 @@ +--- +title: List alert thresholds +slug: listing-alert-thresholds +category: + uri: alerting-api +position: 1 +--- + +## List alert thresholds + +Reports the threshold each overridable parameter is currently evaluated against, and whether +that value comes from an override or from the rule's default. + +The response depends on whether you name a target. + +### For one target + +Pass `scope` and `target` to get **every** overridable parameter that applies to it, whether +overridden or not. This is what a settings screen for a single Node needs — the untouched +parameters have to be shown alongside the changed ones. + +```shell +curl --insecure -X GET \ + --header 'Authorization: Bearer XXXXX' \ + --url 'https://127.0.0.1/v1/alerting/thresholds?scope=THRESHOLD_SCOPE_NODE&target=dc1f7e40-1b1a-4c5d-9f2e-2b6a1e3f4c5d' +``` + +```json +{ + "thresholds": [ + { + "rule_id": "1f8b2c34-5d6e-4a7b-8c9d-0e1f2a3b4c5d", + "param_name": "threshold", + "summary": "A percentage from configured maximum", + "unit": "PARAM_UNIT_PERCENTAGE", + "default_value": 80, + "effective_value": 95, + "is_overridden": true, + "scope": "THRESHOLD_SCOPE_NODE", + "target": "dc1f7e40-1b1a-4c5d-9f2e-2b6a1e3f4c5d" + } + ] +} +``` + +`scope` and `target` in the response describe where the **effective** value came from, which +is not necessarily the target you asked about — a node can inherit a cluster-scoped override. +Both fields are absent when `is_overridden` is false. + +### Across all targets + +Omit `target` to get only the overrides that actually exist: + +```shell +curl --insecure -X GET \ + --header 'Authorization: Bearer XXXXX' \ + --url 'https://127.0.0.1/v1/alerting/thresholds' +``` + +Defaults are not enumerated here. Without a target there is no bounded set of targets to +enumerate them for — every Node PMM has ever monitored would qualify. + +### Filtering by rule + +`rule_id` narrows either form to a single rule: + +```shell +curl --insecure -X GET \ + --header 'Authorization: Bearer XXXXX' \ + --url 'https://127.0.0.1/v1/alerting/thresholds?rule_id=1f8b2c34-5d6e-4a7b-8c9d-0e1f2a3b4c5d' +``` + +> 🚧 Do not key a map on rule_id +> +> Rules duplicated in Grafana share a `rule_id`, so two entries can carry the same `rule_id` and `param_name` and differ only in which rule they came from. + +### Reading zero values + +The API omits zero-valued fields, as JSON mapping for Protocol Buffers requires. A threshold +of `0` arrives with `default_value` or `effective_value` **absent**, not set to `0`, and an +entry that is not overridden has no `is_overridden` field at all. Treat an absent numeric +field as `0` rather than as unknown. diff --git a/documentation/api/alerting/overview.md b/documentation/api/alerting/overview.md new file mode 100644 index 0000000000..4d049923ed --- /dev/null +++ b/documentation/api/alerting/overview.md @@ -0,0 +1,56 @@ +--- +title: Overview +slug: pmm-alert-thresholds +category: + uri: alerting-api +position: 0 +--- + +## Alert thresholds + +An alert rule created from a template carries the threshold the template shipped with. Alert +threshold APIs let you change that threshold for one target — a single Node, for example — +without editing the template, duplicating the rule, or affecting anything else the rule +watches. + +A rule whose template marks a parameter as overridable is registered with an identifier when +you create it, returned as `rule_id` in the [Create Alert Rule](ref:createrule) response. That +identifier is what the threshold endpoints address. + +### The model + +An **override** is a value set for one parameter of one rule on one target. A target is +identified by a **scope** and an id: + +| Scope | Target is | +|---|---| +| `THRESHOLD_SCOPE_NODE` | a Node ID | +| `THRESHOLD_SCOPE_SERVICE` | a Service ID | +| `THRESHOLD_SCOPE_CLUSTER` | a cluster label value | + +When more than one override could apply to the same series, the narrowest one wins: +`service`, then `node`, then `cluster`. A service runs on exactly one node, so a service +override is strictly narrower than a node override covering the same service. + +> 🚧 Availability +> +> Only `THRESHOLD_SCOPE_NODE` is currently supported. Service and cluster scopes are accepted by the schema but return `501 Not Implemented`. Omitting the scope means node. + +Where no override applies, the rule evaluates against the template's default. Clearing an +override returns the target to that default — or to a broader override that still covers it. + +### Endpoints + +- [List Alert Thresholds](ref:listthresholds) reports the value each target is currently + evaluated against, and whether it comes from an override or the default. +- [Set Alert Threshold](ref:setthreshold) overrides one parameter for one target. +- [Clear Alert Threshold](ref:clearthreshold) removes one override. +- [Batch Update Alert Thresholds](ref:batchupdatethresholds) applies several sets and clears + in a single transaction. + +### Permissions + +These endpoints require **admin**. Unlike the rest of the alerting API, a viewer or editor +token cannot read or change thresholds. + +To get the authentication token, check [Authentication](ref:authentication). diff --git a/documentation/api/alerting/set-clear-threshold.md b/documentation/api/alerting/set-clear-threshold.md new file mode 100644 index 0000000000..2c931e1ad1 --- /dev/null +++ b/documentation/api/alerting/set-clear-threshold.md @@ -0,0 +1,85 @@ +--- +title: Set and clear an alert threshold +slug: setting-alert-thresholds +category: + uri: alerting-api +position: 2 +--- + +## Set an alert threshold + +Overrides one parameter of one rule for one target. The rule itself is not modified, and +every other target it watches keeps evaluating against the default. + +```shell +curl --insecure -X POST \ + --header 'Authorization: Bearer XXXXX' \ + --header 'Content-Type: application/json' \ + --url https://127.0.0.1/v1/alerting/thresholds \ + --data ' +{ + "scope": "THRESHOLD_SCOPE_NODE", + "target": "dc1f7e40-1b1a-4c5d-9f2e-2b6a1e3f4c5d", + "rule_id": "1f8b2c34-5d6e-4a7b-8c9d-0e1f2a3b4c5d", + "param_name": "threshold", + "value": 95 +} +' +``` + +The response returns the threshold as it now stands, in the same shape +[List Alert Thresholds](ref:listthresholds) uses. + +Setting a threshold on a target that already has one replaces it. There is no separate +create-versus-update call. + +### Validation + +`value` must be finite and within the range the parameter declared: + +| Condition | Status | +|---|---| +| Value outside the declared range, or not finite | `400 Bad Request` | +| Rule has no such overridable parameter | `404 Not Found` | +| Rule ID does not exist | `404 Not Found` | +| Target does not exist | `404 Not Found` | +| Parameter cannot be overridden at that scope | `400 Bad Request` | +| Scope is service or cluster | `501 Not Implemented` | + +A parameter is only overridable if its template said so. A rule created before a template +gained an overridable parameter does not acquire one — the range and default are captured +when the rule is created, so an edit to the template afterwards does not change what an +existing rule validates against. + +## Clear an alert threshold + +Removes an override, returning the target to the rule's default or to a broader override that +still covers it. + +```shell +curl --insecure -X DELETE \ + --header 'Authorization: Bearer XXXXX' \ + --header 'Content-Type: application/json' \ + --url https://127.0.0.1/v1/alerting/thresholds \ + --data ' +{ + "scope": "THRESHOLD_SCOPE_NODE", + "target": "dc1f7e40-1b1a-4c5d-9f2e-2b6a1e3f4c5d", + "rule_id": "1f8b2c34-5d6e-4a7b-8c9d-0e1f2a3b4c5d", + "param_name": "threshold" +} +' +``` + +Clearing is idempotent: clearing a threshold that is not overridden succeeds and changes +nothing. + +> 🚧 Clear rather than write the default back +> +> To return a target to the default, clear the override — do not set the threshold to the default value. Writing the default as an override pins that target to today's value, so it will not follow a later change to the rule. + +### Removing a target + +Deleting a Node removes its overrides along with it. Cluster-scoped overrides are not +removed this way, because a cluster is a label value rather than an inventory entity and has +no removal event to hook.