Fail the PSMDB image build when mgodatagen install fails

[claude]

Failures fixed (investigator)

• source: Percona-Lab/pmm-submodules#4518 — run 31620553836, check E2E / Backup Management MongoDB UI tests / e2e tests: @bm-mongo
• tests:
  • qa-integration/pmm_psmdb-pbm_setup/start-rs-only.sh / @bm-mongo — setup step, --database psmdb,SETUP_TYPE=pss,COMPOSE_PROFILES=extra

What failed

The job never reached a test. It died in setup:

+ docker compose -f docker-compose-rs.yaml exec -T rs101 mgodatagen -f /etc/datagen/replicaset.json ...
OCI runtime exec failed: exec failed: unable to start container process: exec: "mgodatagen": executable file not found in $PATH
ERROR: Setup script 'start-rs-only.sh' failed.

Root cause — the image build swallowed a failed download

4½ minutes earlier, in the same log, the replica_member/local build had already lost mgodatagen and carried on regardless:

#19 8.949 curl: (56) Connection died, tried 5 times before giving up
#19 DONE 11.4s

The download was the last link of an && chain that ended in dnf clean all; — a semicolon, so the RUN's exit status came from the trailing rm -rf … && mkdir -p /data/db, which always succeeds:

    curl -Lf -o /tmp/mgodatagen.tar.gz https://github.com/…/mgodatagen_0.11.2_Linux_x86_64.tar.gz && \
    tar -xf /tmp/mgodatagen.tar.gz -C /usr/bin && \
    dnf clean all; \                                  # <-- breaks the chain
    rm -rf /var/cache/dnf /var/cache/dnf /data/db && mkdir -p /data/db;

So a transient GitHub blip produced a green build of a broken image, and the failure surfaced minutes later, in a different script, as a misleading "executable file not found". (Unlike the first RUN in this file, this one has no set -e to save it.)

The trigger itself was external and transient — GitHub release-asset delivery was degraded in that window. The other failed check in the same run, Helm tests / helm-tests, is the same outage seen from a different angle and needs no pmm-qa change:

Error: INSTALLATION FAILED: failed to fetch
https://github.com/percona/percona-helm-charts/releases/download/pmm-1.9.0/pmm-1.9.0.tgz : 503 Service Unavailable

That job failed loudly and correctly. This one didn't — that is what this PR fixes.

The fix

Chain the whole RUN with && so the build stops at the download, retry transient GitHub errors instead of giving up on the first one, and assert the binary actually landed in case a future release tarball changes layout.

Verified on a throwaway Linode VM (fresh, Ubuntu 24.04, real Docker)

The outage was simulated with --add-host github.com:127.0.0.1 — a flag, no file edits — so the curl fails exactly where it failed in CI.

#	Condition	Before (main, 0a6fda9)	After (this branch)
A	GitHub unreachable during build	curl: (7) …Connection refused → #8 DONE 18.2s → BUILD_EXIT=0	BUILD_EXIT=1, did not complete successfully: exit code: 7
B	Image from A, run mgodatagen	ls: cannot access '/usr/bin/mgodatagen': No such file → exec: "mgodatagen": executable file not found in $PATH — byte-identical to CI	build never produces this image
C	Normal network	—	BUILD_EXIT=0, /usr/bin/mgodatagen present, mgodatagen 0.11.2

Then the full FB setup, same server image and client tarball as the failed run (perconalab/pmm-server-fb:PR-4518-984159a), same args (--database psmdb,SETUP_TYPE=pss,COMPOSE_PROFILES=extra):

+ docker compose -f docker-compose-rs.yaml exec -T rs101 mgodatagen -f /etc/datagen/replicaset.json ...
connecting to mongodb://…@127.0.0.1:27017/?replicaSet=rs
MongoDB server version 8.0.28-12
collection test: done  [============================] 100%
| COLLECTION | COUNT | AVG OBJECT SIZE |
| test       | 10000 |            1064 |

Setup completed with 0 errors; all six services registered (rs101–rs103, rs201–rs203), and command -v mgodatagen → /usr/bin/mgodatagen in the live container.

What this does not prove: the original 503/connection-death was external and transient, so it can't be re-triggered on demand — only simulated, as above. What the next real FB/nightly run confirms is that the normal path stays green; what this PR guarantees is that if the download fails again, the build fails there with a clear message instead of 4½ minutes later somewhere else.

---

Generated by Claude Code

[claude]

Heads-up before this merges: the --retry-all-errors flag added here needs curl 7.71.0, and oraclelinux:8 ships 7.61.1 — the unknown option aborts the image build outright.

That already happened. The equivalent change in #1171 merged on Aug 13 and took out all 7 OL8 jobs of the next PSMDB GSSAPI E2E Tests Matrix run (31764874766) with curl: option --retry-all-errors: is unknown; the OL9 jobs passed. #1183 replaces the flag with a portable retry loop that keeps this PR's intent (retrying mid-transfer failures, which plain --retry skips).

Since #1171 already landed the rest of this change, this PR is likely redundant now — but if any of it is still wanted, it needs rebasing onto #1183 rather than re-adding the flag.

---

Generated by Claude Code