chore: Default cluster selector to multi-select in loki-mixin by tinitiuset · Pull Request #21768 · grafana/loki

tinitiuset · 2026-05-05T17:41:42Z

What this PR does / why we need it:

This PR sets loki-mixin dashboard cluster template variable to multi-select. This change will ease out searching for namespaces across different clusters.

Checklist

Reviewed the CONTRIBUTING.md guide (required)
Documentation added
Tests updated
Title matches the required conventional commits format, see here
Changes that require user attention or interaction to upgrade are documented in docs/sources/setup/upgrade/_index.md
If the change is deprecating or removing a configuration option, update the deprecated-config.yaml and deleted-config.yaml files respectively in the tools/deprecated-config-checker directory. Example PR

Note

Low Risk
Low risk dashboard-only changes; the main risk is unintended query broadening when selecting multiple clusters (or All), which could increase panel load and cardinality.

Overview
Grafana dashboard templating update: the cluster template variable is changed to multi-select and now supports an All option (includeAll: true, multi: true, allValue: ".+", defaulting to All).

Across the affected compiled dashboards, Prometheus/Loki expressions are updated from cluster="$cluster" to cluster=~"$cluster" (and related variable queries updated similarly) so panels work correctly with multi-value cluster selections.

^{Reviewed by Cursor Bugbot for commit 7d83269. Bugbot is set up for automated code reviews on this repo. Configure here.}

cla-assistant · 2026-05-05T17:42:01Z

All committers have signed the CLA.

cursor

Cursor Bugbot has reviewed your changes and found 2 potential issues.

Bugbot Autofix prepared fixes for both issues found in the latest run.

✅ Fixed: Chunks dashboard queries incompatible with multi-select cluster
- Updated all SSD chunks dashboard panel queries to use regex cluster matching for multi-select and All cluster values.
✅ Fixed: Recording rules tenant query incompatible with multi-select cluster
- Updated the SSD recording-rules tenant variable query to use regex cluster matching for multi-select cluster values.

Or push these changes by commenting:

@cursor push 98c57b7ceb

Preview (98c57b7ceb)

diff --git a/production/loki-mixin-compiled-ssd/dashboards/loki-chunks.json b/production/loki-mixin-compiled-ssd/dashboards/loki-chunks.json
--- a/production/loki-mixin-compiled-ssd/dashboards/loki-chunks.json
+++ b/production/loki-mixin-compiled-ssd/dashboards/loki-chunks.json
@@ -64,7 +64,7 @@
                   "span": 6,
                   "targets": [
                      {
-                        "expr": "sum(loki_ingester_memory_chunks{cluster=\"$cluster\", job=~\"$namespace/(loki.*|enterprise-logs)-write\"})",
+                        "expr": "sum(loki_ingester_memory_chunks{cluster=~\"$cluster\", job=~\"$namespace/(loki.*|enterprise-logs)-write\"})",
                         "format": "time_series",
                         "legendFormat": "series",
                         "legendLink": null
@@ -111,7 +111,7 @@
                   "span": 6,
                   "targets": [
                      {
-                        "expr": "sum(loki_ingester_memory_chunks{cluster=\"$cluster\", job=~\"$namespace/(loki.*|enterprise-logs)-write\"}) / sum(loki_ingester_memory_streams{cluster=\"$cluster\", job=~\"$namespace/(loki.*|enterprise-logs)-write\"})",
+                        "expr": "sum(loki_ingester_memory_chunks{cluster=~\"$cluster\", job=~\"$namespace/(loki.*|enterprise-logs)-write\"}) / sum(loki_ingester_memory_streams{cluster=~\"$cluster\", job=~\"$namespace/(loki.*|enterprise-logs)-write\"})",
                         "format": "time_series",
                         "legendFormat": "chunks",
                         "legendLink": null
@@ -171,19 +171,19 @@
                   "span": 6,
                   "targets": [
                      {
-                        "expr": "histogram_quantile(0.99, sum(rate(loki_ingester_chunk_utilization_bucket{cluster=\"$cluster\", job=~\"$namespace/(loki.*|enterprise-logs)-write\"}[$__rate_interval])) by (le)) * 1",
+                        "expr": "histogram_quantile(0.99, sum(rate(loki_ingester_chunk_utilization_bucket{cluster=~\"$cluster\", job=~\"$namespace/(loki.*|enterprise-logs)-write\"}[$__rate_interval])) by (le)) * 1",
                         "format": "time_series",
                         "legendFormat": "99th Percentile",
                         "refId": "A"
                      },
                      {
-                        "expr": "histogram_quantile(0.50, sum(rate(loki_ingester_chunk_utilization_bucket{cluster=\"$cluster\", job=~\"$namespace/(loki.*|enterprise-logs)-write\"}[$__rate_interval])) by (le)) * 1",
+                        "expr": "histogram_quantile(0.50, sum(rate(loki_ingester_chunk_utilization_bucket{cluster=~\"$cluster\", job=~\"$namespace/(loki.*|enterprise-logs)-write\"}[$__rate_interval])) by (le)) * 1",
                         "format": "time_series",
                         "legendFormat": "50th Percentile",
                         "refId": "B"
                      },
                      {
-                        "expr": "sum(rate(loki_ingester_chunk_utilization_sum{cluster=\"$cluster\", job=~\"$namespace/(loki.*|enterprise-logs)-write\"}[$__rate_interval])) * 1 / sum(rate(loki_ingester_chunk_utilization_count{cluster=\"$cluster\", job=~\"$namespace/(loki.*|enterprise-logs)-write\"}[$__rate_interval]))",
+                        "expr": "sum(rate(loki_ingester_chunk_utilization_sum{cluster=~\"$cluster\", job=~\"$namespace/(loki.*|enterprise-logs)-write\"}[$__rate_interval])) * 1 / sum(rate(loki_ingester_chunk_utilization_count{cluster=~\"$cluster\", job=~\"$namespace/(loki.*|enterprise-logs)-write\"}[$__rate_interval]))",
                         "format": "time_series",
                         "legendFormat": "Average",
                         "refId": "C"
@@ -249,19 +249,19 @@
                   "span": 6,
                   "targets": [
                      {
-                        "expr": "histogram_quantile(0.99, sum(rate(loki_ingester_chunk_age_seconds_bucket{cluster=\"$cluster\", job=~\"$namespace/(loki.*|enterprise-logs)-write\"}[$__rate_interval])) by (le)) * 1e3",
+                        "expr": "histogram_quantile(0.99, sum(rate(loki_ingester_chunk_age_seconds_bucket{cluster=~\"$cluster\", job=~\"$namespace/(loki.*|enterprise-logs)-write\"}[$__rate_interval])) by (le)) * 1e3",
                         "format": "time_series",
                         "legendFormat": "99th Percentile",
                         "refId": "A"
                      },
                      {
-                        "expr": "histogram_quantile(0.50, sum(rate(loki_ingester_chunk_age_seconds_bucket{cluster=\"$cluster\", job=~\"$namespace/(loki.*|enterprise-logs)-write\"}[$__rate_interval])) by (le)) * 1e3",
+                        "expr": "histogram_quantile(0.50, sum(rate(loki_ingester_chunk_age_seconds_bucket{cluster=~\"$cluster\", job=~\"$namespace/(loki.*|enterprise-logs)-write\"}[$__rate_interval])) by (le)) * 1e3",
                         "format": "time_series",
                         "legendFormat": "50th Percentile",
                         "refId": "B"
                      },
                      {
-                        "expr": "sum(rate(loki_ingester_chunk_age_seconds_sum{cluster=\"$cluster\", job=~\"$namespace/(loki.*|enterprise-logs)-write\"}[$__rate_interval])) * 1e3 / sum(rate(loki_ingester_chunk_age_seconds_count{cluster=\"$cluster\", job=~\"$namespace/(loki.*|enterprise-logs)-write\"}[$__rate_interval]))",
+                        "expr": "sum(rate(loki_ingester_chunk_age_seconds_sum{cluster=~\"$cluster\", job=~\"$namespace/(loki.*|enterprise-logs)-write\"}[$__rate_interval])) * 1e3 / sum(rate(loki_ingester_chunk_age_seconds_count{cluster=~\"$cluster\", job=~\"$namespace/(loki.*|enterprise-logs)-write\"}[$__rate_interval]))",
                         "format": "time_series",
                         "legendFormat": "Average",
                         "refId": "C"
@@ -339,19 +339,19 @@
                   "span": 6,
                   "targets": [
                      {
-                        "expr": "histogram_quantile(0.99, sum(rate(loki_ingester_chunk_entries_bucket{cluster=\"$cluster\", job=~\"$namespace/(loki.*|enterprise-logs)-write\"}[$__rate_interval])) by (le)) * 1",
+                        "expr": "histogram_quantile(0.99, sum(rate(loki_ingester_chunk_entries_bucket{cluster=~\"$cluster\", job=~\"$namespace/(loki.*|enterprise-logs)-write\"}[$__rate_interval])) by (le)) * 1",
                         "format": "time_series",
                         "legendFormat": "99th Percentile",
                         "refId": "A"
                      },
                      {
-                        "expr": "histogram_quantile(0.50, sum(rate(loki_ingester_chunk_entries_bucket{cluster=\"$cluster\", job=~\"$namespace/(loki.*|enterprise-logs)-write\"}[$__rate_interval])) by (le)) * 1",
+                        "expr": "histogram_quantile(0.50, sum(rate(loki_ingester_chunk_entries_bucket{cluster=~\"$cluster\", job=~\"$namespace/(loki.*|enterprise-logs)-write\"}[$__rate_interval])) by (le)) * 1",
                         "format": "time_series",
                         "legendFormat": "50th Percentile",
                         "refId": "B"
                      },
                      {
-                        "expr": "sum(rate(loki_ingester_chunk_entries_sum{cluster=\"$cluster\", job=~\"$namespace/(loki.*|enterprise-logs)-write\"}[$__rate_interval])) * 1 / sum(rate(loki_ingester_chunk_entries_count{cluster=\"$cluster\", job=~\"$namespace/(loki.*|enterprise-logs)-write\"}[$__rate_interval]))",
+                        "expr": "sum(rate(loki_ingester_chunk_entries_sum{cluster=~\"$cluster\", job=~\"$namespace/(loki.*|enterprise-logs)-write\"}[$__rate_interval])) * 1 / sum(rate(loki_ingester_chunk_entries_count{cluster=~\"$cluster\", job=~\"$namespace/(loki.*|enterprise-logs)-write\"}[$__rate_interval]))",
                         "format": "time_series",
                         "legendFormat": "Average",
                         "refId": "C"
@@ -416,7 +416,7 @@
                   "span": 6,
                   "targets": [
                      {
-                        "expr": "sum(rate(loki_chunk_store_index_entries_per_chunk_sum{cluster=\"$cluster\", job=~\"$namespace/(loki.*|enterprise-logs)-write\"}[$__rate_interval])) / sum(rate(loki_chunk_store_index_entries_per_chunk_count{cluster=\"$cluster\", job=~\"$namespace/(loki.*|enterprise-logs)-write\"}[$__rate_interval]))",
+                        "expr": "sum(rate(loki_chunk_store_index_entries_per_chunk_sum{cluster=~\"$cluster\", job=~\"$namespace/(loki.*|enterprise-logs)-write\"}[$__rate_interval])) / sum(rate(loki_chunk_store_index_entries_per_chunk_count{cluster=~\"$cluster\", job=~\"$namespace/(loki.*|enterprise-logs)-write\"}[$__rate_interval]))",
                         "format": "time_series",
                         "legendFormat": "Index Entries",
                         "legendLink": null
@@ -475,7 +475,7 @@
                   "span": 6,
                   "targets": [
                      {
-                        "expr": "loki_ingester_flush_queue_length{cluster=\"$cluster\", job=~\"$namespace/(loki.*|enterprise-logs)-write\"} or cortex_ingester_flush_queue_length{cluster=\"$cluster\", job=~\"$namespace/(loki.*|enterprise-logs)-write\"}",
+                        "expr": "loki_ingester_flush_queue_length{cluster=~\"$cluster\", job=~\"$namespace/(loki.*|enterprise-logs)-write\"} or cortex_ingester_flush_queue_length{cluster=~\"$cluster\", job=~\"$namespace/(loki.*|enterprise-logs)-write\"}",
                         "format": "time_series",
                         "legendFormat": "{{pod}}",
                         "legendLink": null
@@ -673,7 +673,7 @@
                   "stack": true,
                   "targets": [
                      {
-                        "expr": "sum by (status) (\n  label_replace(label_replace(rate(loki_ingester_chunk_age_seconds_count{cluster=\"$cluster\", job=~\"$namespace/(loki.*|enterprise-logs)-write\"}[$__rate_interval]),\n  \"status\", \"${1}xx\", \"status_code\", \"([0-9])..\"),\n  \"status\", \"${1}\", \"status_code\", \"([a-zA-Z]+)\"))\n",
+                        "expr": "sum by (status) (\n  label_replace(label_replace(rate(loki_ingester_chunk_age_seconds_count{cluster=~\"$cluster\", job=~\"$namespace/(loki.*|enterprise-logs)-write\"}[$__rate_interval]),\n  \"status\", \"${1}xx\", \"status_code\", \"([0-9])..\"),\n  \"status\", \"${1}\", \"status_code\", \"([a-zA-Z]+)\"))\n",
                         "format": "time_series",
                         "legendFormat": "{{status}}",
                         "refId": "A"
@@ -732,7 +732,7 @@
                   "span": 6,
                   "targets": [
                      {
-                        "expr": "sum(rate(loki_ingester_chunks_flushed_total{cluster=\"$cluster\", job=~\"$namespace/(loki.*|enterprise-logs)-write\"}[$__rate_interval]))",
+                        "expr": "sum(rate(loki_ingester_chunks_flushed_total{cluster=~\"$cluster\", job=~\"$namespace/(loki.*|enterprise-logs)-write\"}[$__rate_interval]))",
                         "format": "time_series",
                         "legendFormat": "{{pod}}",
                         "legendLink": null
@@ -780,7 +780,7 @@
                   "stack": true,
                   "targets": [
                      {
-                        "expr": "sum by (reason) (rate(loki_ingester_chunks_flushed_total{cluster=\"$cluster\", job=~\"$namespace/(loki.*|enterprise-logs)-write\"}[$__rate_interval])) / ignoring(reason) group_left sum(rate(loki_ingester_chunks_flushed_total{cluster=\"$cluster\", job=~\"$namespace/(loki.*|enterprise-logs)-write\"}[$__rate_interval]))",
+                        "expr": "sum by (reason) (rate(loki_ingester_chunks_flushed_total{cluster=~\"$cluster\", job=~\"$namespace/(loki.*|enterprise-logs)-write\"}[$__rate_interval])) / ignoring(reason) group_left sum(rate(loki_ingester_chunks_flushed_total{cluster=~\"$cluster\", job=~\"$namespace/(loki.*|enterprise-logs)-write\"}[$__rate_interval]))",
                         "format": "time_series",
                         "legendFormat": "{{reason}}",
                         "legendLink": null
@@ -843,7 +843,7 @@
                   "span": 12,
                   "targets": [
                      {
-                        "expr": "sum by (le) (rate(loki_ingester_chunk_utilization_bucket{cluster=\"$cluster\", job=~\"$namespace/(loki.*|enterprise-logs)-write\"}[$__rate_interval]))",
+                        "expr": "sum by (le) (rate(loki_ingester_chunk_utilization_bucket{cluster=~\"$cluster\", job=~\"$namespace/(loki.*|enterprise-logs)-write\"}[$__rate_interval]))",
                         "format": "heatmap",
                         "intervalFactor": 2,
                         "legendFormat": "{{le}}",
@@ -905,7 +905,7 @@
                   "span": 12,
                   "targets": [
                      {
-                        "expr": "sum(rate(loki_ingester_chunk_size_bytes_bucket{cluster=\"$cluster\", job=~\"$namespace/(loki.*|enterprise-logs)-write\"}[$__rate_interval])) by (le)",
+                        "expr": "sum(rate(loki_ingester_chunk_size_bytes_bucket{cluster=~\"$cluster\", job=~\"$namespace/(loki.*|enterprise-logs)-write\"}[$__rate_interval])) by (le)",
                         "format": "heatmap",
                         "intervalFactor": 2,
                         "legendFormat": "{{le}}",
@@ -981,19 +981,19 @@
                   "span": 12,
                   "targets": [
                      {
-                        "expr": "histogram_quantile(0.99, sum(rate(loki_ingester_chunk_size_bytes_bucket{cluster=\"$cluster\", job=~\"$namespace/(loki.*|enterprise-logs)-write\"}[$__rate_interval])) by (le))",
+                        "expr": "histogram_quantile(0.99, sum(rate(loki_ingester_chunk_size_bytes_bucket{cluster=~\"$cluster\", job=~\"$namespace/(loki.*|enterprise-logs)-write\"}[$__rate_interval])) by (le))",
                         "format": "time_series",
                         "legendFormat": "p99",
                         "legendLink": null
                      },
                      {
-                        "expr": "histogram_quantile(0.90, sum(rate(loki_ingester_chunk_size_bytes_bucket{cluster=\"$cluster\", job=~\"$namespace/(loki.*|enterprise-logs)-write\"}[$__rate_interval])) by (le))",
+                        "expr": "histogram_quantile(0.90, sum(rate(loki_ingester_chunk_size_bytes_bucket{cluster=~\"$cluster\", job=~\"$namespace/(loki.*|enterprise-logs)-write\"}[$__rate_interval])) by (le))",
                         "format": "time_series",
                         "legendFormat": "p90",
                         "legendLink": null
                      },
                      {
-                        "expr": "histogram_quantile(0.50, sum(rate(loki_ingester_chunk_size_bytes_bucket{cluster=\"$cluster\", job=~\"$namespace/(loki.*|enterprise-logs)-write\"}[$__rate_interval])) by (le))",
+                        "expr": "histogram_quantile(0.50, sum(rate(loki_ingester_chunk_size_bytes_bucket{cluster=~\"$cluster\", job=~\"$namespace/(loki.*|enterprise-logs)-write\"}[$__rate_interval])) by (le))",
                         "format": "time_series",
                         "legendFormat": "p50",
                         "legendLink": null
@@ -1052,19 +1052,19 @@
                   "span": 12,
                   "targets": [
                      {
-                        "expr": "histogram_quantile(0.5, sum(rate(loki_ingester_chunk_bounds_hours_bucket{cluster=\"$cluster\", job=~\"$namespace/(loki.*|enterprise-logs)-write\"}[$__rate_interval])) by (le))",
+                        "expr": "histogram_quantile(0.5, sum(rate(loki_ingester_chunk_bounds_hours_bucket{cluster=~\"$cluster\", job=~\"$namespace/(loki.*|enterprise-logs)-write\"}[$__rate_interval])) by (le))",
                         "format": "time_series",
                         "legendFormat": "p50",
                         "legendLink": null
                      },
                      {
-                        "expr": "histogram_quantile(0.99, sum(rate(loki_ingester_chunk_bounds_hours_bucket{cluster=\"$cluster\", job=~\"$namespace/(loki.*|enterprise-logs)-write\"}[$__rate_interval])) by (le))",
+                        "expr": "histogram_quantile(0.99, sum(rate(loki_ingester_chunk_bounds_hours_bucket{cluster=~\"$cluster\", job=~\"$namespace/(loki.*|enterprise-logs)-write\"}[$__rate_interval])) by (le))",
                         "format": "time_series",
                         "legendFormat": "p99",
                         "legendLink": null
                      },
                      {
-                        "expr": "sum(rate(loki_ingester_chunk_bounds_hours_sum{cluster=\"$cluster\", job=~\"$namespace/(loki.*|enterprise-logs)-write\"}[$__rate_interval])) / sum(rate(loki_ingester_chunk_bounds_hours_count{cluster=\"$cluster\", job=~\"$namespace/(loki.*|enterprise-logs)-write\"}[$__rate_interval]))",
+                        "expr": "sum(rate(loki_ingester_chunk_bounds_hours_sum{cluster=~\"$cluster\", job=~\"$namespace/(loki.*|enterprise-logs)-write\"}[$__rate_interval])) / sum(rate(loki_ingester_chunk_bounds_hours_count{cluster=~\"$cluster\", job=~\"$namespace/(loki.*|enterprise-logs)-write\"}[$__rate_interval]))",
                         "format": "time_series",
                         "legendFormat": "avg",
                         "legendLink": null

diff --git a/production/loki-mixin-compiled-ssd/dashboards/loki-mixin-recording-rules.json b/production/loki-mixin-compiled-ssd/dashboards/loki-mixin-recording-rules.json
--- a/production/loki-mixin-compiled-ssd/dashboards/loki-mixin-recording-rules.json
+++ b/production/loki-mixin-compiled-ssd/dashboards/loki-mixin-recording-rules.json
@@ -676,7 +676,7 @@
                "multi": false,
                "name": "tenant",
                "options": [ ],
-               "query": "query_result(sum by (id) (grafanacloud_logs_instance_info) and sum(label_replace(loki_tenant:active_streams{cluster=\"$cluster\",namespace=\"$namespace\"},\"id\",\"$1\",\"tenant\",\"(.*)\")) by(id))",
+               "query": "query_result(sum by (id) (grafanacloud_logs_instance_info) and sum(label_replace(loki_tenant:active_streams{cluster=~\"$cluster\",namespace=\"$namespace\"},\"id\",\"$1\",\"tenant\",\"(.*)\")) by(id))",
                "refresh": 0,
                "regex": "/\"([^\"]+)\"/",
                "sort": 1,

_{You can send follow-ups to the cloud agent here.}

^{Reviewed by Cursor Bugbot for commit 0fbdf70. Configure here.}

…initiuset/ft-loki-mixin

pull-request-size Bot added the size/XXL label May 5, 2026

tinitiuset changed the title ~~chore: loki-mixin: default cluster selector to multi-select~~ chore (loki-mixin): default cluster selector to multi-select May 5, 2026

tinitiuset force-pushed the tinitiuset/ft-loki-mixin branch from 4d8ef33 to acf7504 Compare May 5, 2026 17:42

tinitiuset changed the title ~~chore (loki-mixin): default cluster selector to multi-select~~ chore: Default cluster selector to multi-select in loki-mixin May 5, 2026

chore: Default cluster selector to multi-select in loki-mixin

cff2082

tinitiuset force-pushed the tinitiuset/ft-loki-mixin branch from acf7504 to cff2082 Compare May 5, 2026 17:44

tinitiuset marked this pull request as ready for review May 5, 2026 18:00

tinitiuset requested a review from a team as a code owner May 5, 2026 18:00

Merge branch 'main' into tinitiuset/ft-loki-mixin

0fbdf70

cursor Bot reviewed May 6, 2026

View reviewed changes

Comment thread production/loki-mixin-compiled-ssd/dashboards/loki-chunks.json

Comment thread production/loki-mixin-compiled-ssd/dashboards/loki-mixin-recording-rules.json

tinitiuset added 3 commits May 6, 2026 16:42

Address cursor feedback

8f50821

Merge remote-tracking branch 'origin/tinitiuset/ft-loki-mixin' into t…

fc80c89

…initiuset/ft-loki-mixin

Merge branch 'main' into tinitiuset/ft-loki-mixin

7d83269

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

chore: Default cluster selector to multi-select in loki-mixin#21768

chore: Default cluster selector to multi-select in loki-mixin#21768
tinitiuset wants to merge 5 commits intomainfrom
tinitiuset/ft-loki-mixin

tinitiuset commented May 5, 2026 •

edited by cursor Bot

Loading

Uh oh!

cla-assistant Bot commented May 5, 2026 •

edited

Loading

Uh oh!

cursor Bot left a comment •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

tinitiuset commented May 5, 2026 • edited by cursor Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cla-assistant Bot commented May 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cursor Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

tinitiuset commented May 5, 2026 •

edited by cursor Bot

Loading

cla-assistant Bot commented May 5, 2026 •

edited

Loading

cursor Bot left a comment •

edited

Loading