Fail the psmdb-pbm image build when a download fails

[claude]

Failures fixed (investigator)

• source: Percona-Lab/pmm-submodules#4518 — run 31620553836
• tests:
  • CLI / Integration tests / CLI / Integration / PSMDB Replica 6.x — setup psmdb=6.0,SETUP_TYPE=pss, @psmdb
  • E2E / Backup Management MongoDB UI tests / e2e tests: @bm-mongo — setup psmdb,SETUP_TYPE=pss,COMPOSE_PROFILES=extra

What failed

Both jobs died in setup, before a single test ran:

+ docker compose -f docker-compose-rs.yaml exec -T rs101 mgodatagen -f /etc/datagen/replicaset.json ...
OCI runtime exec failed: exec failed: unable to start container process: exec: "mgodatagen": executable file not found in $PATH
ERROR: Setup script 'start-rs-only.sh' failed.

Root cause — the image build swallowed a failed download

The binary was missing because the image was built without it, and the build said
nothing. Both jobs' own build logs carry the answer, ~1000 lines earlier:

#19 6.966   % Total    % Received % Xferd  ...
#19 7.139 curl: (22) The requested URL returned error: 503
#19 DONE 8.0s

GitHub's release CDN was briefly 503ing in that window — the helm-tests job in the
same run hit it too, on .../percona-helm-charts/releases/download/pmm-1.9.0/pmm-1.9.0.tgz.
The URL is fine now (200), and the sibling PSMDB 7.x/8.x jobs, which build the same
image, downloaded it without trouble.

The transient 503 is not the bug. The bug is DONE on the next line. The install
RUN in qa-integration/pmm_psmdb-pbm_setup/Dockerfile ended:

    curl -Lf -o /tmp/mgodatagen.tar.gz https://github.com/.../mgodatagen_0.11.2_Linux_x86_64.tar.gz && \
    tar -xf /tmp/mgodatagen.tar.gz -C /usr/bin && \
    dnf clean all; \
    rm -rf /var/cache/dnf /var/cache/dnf /data/db && mkdir -p /data/db;

That ; before rm -rf ends the && chain, so the layer's exit status comes from
mkdir -p /data/db — which always succeeds. Every command in the chain is
effectively optional: curl reports the failure, the chain short-circuits, the layer is
marked DONE, and a broken image ships. It is not specific to mgodatagen — a failed
pmm-client tarball download is masked exactly the same way.

Fix

• End the chain with && so the build fails at the download instead of hours later, in
  someone else's test.
• --retry 5 --retry-delay 5 --retry-all-errors on both downloads, so a transient 5xx
  is ridden out rather than turned into a red job at all.
• Dropped the duplicated /var/cache/dnf on the line being rewritten.

Verification

Throwaway Linode VM, FB build for this PR
(perconalab/pmm-server-fb:PR-4518-984159a + its client tarball), PSMDB 6.0 —
the exact matrix entry that failed. --add-host github.com:127.0.0.1 stands in for the
503: same shape of failure (curl exits non-zero mid-chain), reproducible on demand.

Before (main, download unreachable) — CI's failure, reproduced:

#8 16.22 curl: (7) Failed to connect to github.com port 443: Connection refused
#8 DONE 16.4s
===== BUILD EXIT (main) = 0
repro/main:latest   1e954fc3565f   2.92GB
===== mgodatagen in image:
MGODATAGEN-MISSING

The build passes and hands over an image whose mgodatagen is gone — precisely what
the runners got.

After (this branch, same unreachable download) — retries, then a red build:

Warning: Problem (retrying all errors). Will retry in 5 seconds. 5 retries
... (5 attempts, 5s apart) ...
ERROR: failed to build: ... did not complete successfully: exit code: 7
IMAGE   ID   DISK USAGE   CONTENT SIZE      <- no image produced

After (this branch, network normal) — the real path, end to end:

+ docker compose -f docker-compose-rs.yaml exec -T rs101 mgodatagen -f /etc/datagen/replicaset.json --uri=...
MongoDB server version 6.0.29-23
| COLLECTION | COUNT | AVG OBJECT SIZE |
| test       | 10000 |            1064 |
run finished in 370ms

pmm-framework --database psmdb=6.0,SETUP_TYPE=pss completed, command -v mgodatagen
in rs101 gives /usr/bin/mgodatagen, and rs101/rs102/rs103 are registered in PMM's
inventory.

What this does and does not prove

The masking is proven directly: identical build inputs, one exits 0 with a broken image
and one fails. The retry path is proven to engage and to be supported by the image's
curl, but against connection-refused rather than a real 503 — nothing here can prove the
CDN would have answered on attempt 2 that afternoon.

Note the trade: builds that used to go quietly green with a missing binary will now go
red at the download. That is the point — it fails 20 minutes earlier and names its own
cause, instead of surfacing as executable file not found in $PATH in an unrelated test.

Also red in that run, not fixed here

Helm tests / helm-tests failed test 5 of 8 on the same GitHub-CDN 503, fetching
pmm-1.9.0.tgz via helm show values. Tests 6-8 install charts and passed, the fetch
succeeds now, and that job failed loudly with the real error rather than hiding it — a
transient, with no pmm-qa defect behind it. Left alone.

---

Generated by Claude Code