Skip to content

Fix two flaky FB test failures: /proc race and an N/A legend read as no data - #1176

Merged
yurkovychv merged 2 commits into
mainfrom
claude/kind-meitner-ectojz
Aug 13, 2026
Merged

Fix two flaky FB test failures: /proc race and an N/A legend read as no data#1176
yurkovychv merged 2 commits into
mainfrom
claude/kind-meitner-ectojz

Conversation

@claude

@claude claude Bot commented Aug 13, 2026

Copy link
Copy Markdown
Contributor

Failures fixed (investigator)

  • source: Percona-Lab/pmm-submodules#4520 — run 31651545058
  • tests:
    • codeceptjs-e2e/tests/qa-integration/pmm_pgsm_integration_test.js:253 / @pgsm-pmm-integration — Verify Postgresql Dashboard Instance Summary has Data with socket based service and Agent log
    • e2e_tests/tests/dashboards/mysql/mysqlDashboards.test.ts:77 / @pmm-ps-integration — PMM-T324 Verify MySQL - MySQL User Details dashboard

Two unrelated failures, both in pmm-qa's own test code. Neither is a PMM regression — the FB build under test (PMM-15310, gRPC keepalive) is not implicated in either.

1. @pgsm-pmm-integrationfind / racing /proc

The "docker exec pdpgsql_pmm_17_1 find / -name pmm-agent.log" command was expected
to run without any errors, but the error found: "find: '/proc/12730': No such file or directory"
    at Grafana.verifyCommand (tests/helper/grafana_helper.js:265:14)

find / descends into /proc. When a process exits mid-walk its /proc/<pid> directory
disappears, find writes a warning to stderr and exits 1, and verifyCommand asserts
code === 0 — so the scenario fails whenever a short-lived process happens to die during
the walk. Nothing to do with PMM; the log itself is always at /var/log/pmm-agent.log.

Fix: prune /proc from the walk, in both places that do this lookup (lines 253 and 644).

2. @pmm-ps-integration PMM-T324 — a legend entry read as "no data"

Error: Metrics without data are: Users Activity
    Expected length: 0
    Received array:  ["Users Activity"]
    at ../pages/dashboards/dashboards.page.ts:147   (verifyAllPanelsHaveData)

The panel had data. From the ARIA/DOM snapshot Playwright recorded in the failing run's
own trace artifact, the element the locator matched is a viz legend label:

<ul>
  <li><div data-testid="data-testid VizLegend series Passive"><button class="…-LegendLabel">Passive</button></div></li>
  <li><div data-testid="data-testid VizLegend series Active"><button  class="…-LegendLabel">Active</button></div></li>
  <li><div data-testid="data-testid VizLegend series N/A"><button     class="…-LegendLabel">N/A</button></div></li>
</ul>

Users Activity is a state timeline whose value mapping renders the null state as the
text N/A
(MySQL_User_Details.json, panel 1058). Grafana therefore adds an N/A
legend entry whenever the visible range contains nulls — routine on a freshly provisioned
server, where the series only starts partway into now-1h, or after any brief scrape gap.
noDataPanelName matches any element whose text is exactly "N/A" and reports the
enclosing panel's title, so that legend entry was read as "this panel has no data" while
the panel was plainly drawing its Active and Passive states.

Fix: exclude Grafana viz-legend labels from the no-data marker match. Genuine indicators
still match — a panel rendering No data, a stat whose value is N/A or -, and
data-testid Panel data error message.

This is deliberately not handled by adding Users Activity to the dashboard's
noDataMetrics: the data is there, so suppressing the panel would switch off a real
"MySQL User Details shows nothing" signal for good. The bug is in the detection.

Verification

One throwaway Linode VM running the FB build under test — server perconalab/pmm-server-fb:PR-4520-8bebd04
(digest sha256:d21bf9e1…), FB client tarball, each job's own WIZARD_ARGS
(--database pdpgsql, and --database ps,SETUP_TYPE=replication,MY_ROCKS=true --database ps,SETUP_TYPE=gr,QUERY_SOURCE=slowlog).

PGSM — the failing command, run against the real pdpgsql_pmm_17_1 container:

Command Non-zero exits
find / -name pmm-agent.log (current) 3 / 40, then 2 / 50 interleaved
find / -path /proc -prune -o -name pmm-agent.log -print (this PR) 0 / 60, then 0 / 50 interleaved

Same box, same container, same load; the A/B ran the two alternately so they saw identical
/proc churn. Failures reproduced the CI signature exactly (find: '/proc/<pid>': No such file or directory).
stdout is unchanged (/var/log/pmm-agent.log).

PMM-T324 — reproduced by putting a real 3.5-minute scrape gap inside the panel's window
(docker pause/unpause of the monitored node), which is what makes Grafana add the N/A
legend entry. Both locators evaluated against that live DOM:

Locator Flags
current main ["Users Activity"] — the CI failure
this PR []

with the panel's legend reading N/A, Passive, Active and the timeline fully drawn.
Without an induced gap the test passes on the same VM at main (as it does in most FB runs —
this failed in 1 of the last 9). The same locator change was also checked against the exact
legend markup from the failing run's trace, alongside a panel rendering No data, a stat
whose value is N/A, and a panel error message — only the legend match disappears.

npx eslint clean on both changed files.

What only a real CI run can confirm: that the rest of the @pgsm-pmm-integration and
@pmm-ps-integration suites stay green — the PGSM scenario and PMM-T324 were verified at
the command and locator level rather than by a full 22-minute suite pass.

Not in scope

  • #1174 also touches mysqlDashboards.test.ts:77, but for a
    different failure (panel titles trimmed by PMM-15308 Trim whitespace from dashboard titles pmm#5767). Different files, no overlap.
  • codeceptjs-e2e/tests/pages/dashboardPage.js carries the same marker list in
    verifyThereAreNoGraphsWithoutData, so it has the same latent blind spot — left alone here
    since it did not fail and could not be verified against a real failure.

Generated by Claude Code

claude added 2 commits August 13, 2026 00:34
docker exec ... find / -name pmm-agent.log walks /proc, so a process
that exits mid-walk makes find warn and exit 1. verifyCommand asserts a
zero exit, so the scenario fails at random. Prune /proc from the walk.

Signed-off-by: Claude <noreply@anthropic.com>
The MySQL User Details 'Users Activity' state timeline maps its null state
to the text N/A, so Grafana renders an N/A legend entry whenever the range
contains nulls -- normal on a freshly provisioned server, where the series
starts partway into the last hour. The no-data locator matched that legend
label and reported the panel as having no data even though it was drawing
Active and Passive states, failing PMM-T324 at random.

Signed-off-by: Claude <noreply@anthropic.com>
@yurkovychv
yurkovychv merged commit 3bfd21d into main Aug 13, 2026
32 checks passed
@yurkovychv
yurkovychv deleted the claude/kind-meitner-ectojz branch August 13, 2026 08:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants