fix(relay): async provisioning (VM + LKE) so a dropped connection is recoverable

.claude/agents/investigator.md

@@ -36,7 +36,7 @@ Once you have what you're investigating and a ref, use marker `## Failures fixed
 
 ## Workflow
 
-1. **Dedup (mandatory — stop if this is already tracked)** — start with: fetch open `percona/pmm-qa` PRs and read bodies (title alone isn't enough) with repo-scoped REST (`gh pr list --json` is GraphQL and 403s here — see `repos`): `gh api "repos/percona/pmm-qa/pulls?state=open&per_page=50" --jq '.[] | {number, title, body}'`; if any identifier from the failure list already appears under the `## Failures fixed (investigator)` marker → **stop immediately**, reply with that PR URL. For a question or suspected bug more likely to be a product problem: also check whether an existing Jira ticket already describes the same thing (still sitting in `New` or another non-delivered status) before investigating fresh — if one exists, link it instead of duplicating the work. Either way, if nothing turns up, continue.
+1. **Dedup (mandatory — stop if this is already tracked)** — start with: fetch open `percona/pmm-qa` PRs and read bodies (title alone isn't enough) with repo-scoped REST (`gh pr list --json` is GraphQL and 403s here — see `repos`): `gh api "repos/percona/pmm-qa/pulls?state=open&per_page=50" --jq '.[] | {number, title, body}'`; if any identifier from the failure list already appears under the `## Failures fixed (investigator)` marker → **stop immediately**, reply with that PR URL. For a question or suspected bug more likely to be a product problem: also check whether an existing Jira ticket already describes the same thing (still sitting in `New` or another non-delivered status) before investigating fresh — search via the **relay** (`jira` skill → `search` action, JQL, PMM-scoped), **not** the Atlassian MCP (its search needs interactive auth that isn't there in a Routine/headless run). If one exists, link it instead of duplicating the work. Either way, if nothing turns up, continue.
 2. **Reproduce** — Follow `linode-docker-provisioning` to bring up a throwaway Linode VM at the given ref, and run the exact command(s) or tests that failed, or walk through the described scenario step by step if this is a question from someone. Watch what actually happens — a question only skips reproduction when reading the code makes the answer completely unambiguous and needs zero investigation; memory alone never counts.
 3. **Classify from what you observed** — one decision tree:
    - **Didn't reproduce at all** → likely an infra flake (known failure) or not enough detail to confirm (a report). Say so and stop. If this was a bug relayed secondhand, ask for more specific repro steps (exact version, exact clicks/commands) rather than guessing further.

.claude/scripts/pmm-ui-login.js

@@ -5,8 +5,11 @@
 // Usage:
 //   PMM_URL='https://<linode-ip>'          node pmm-ui-login.js PMM-14576
 //   ADMIN_PASSWORD='...'                   (optional, defaults to 'pmm3admin!')
-//   PMM_CERT_PATH='runs/<run_id>/pmm_cert.pem'  (required -- pins PMM's own
-//     cert instead of trusting any cert; see pmm-linode-docker-provisioning skill)
+//   PMM_CERT_PATH='runs/<run_id>/pmm_cert.pem'  (pins PMM's own cert instead of
+//     trusting any cert; see linode-docker-provisioning skill) — REQUIRED unless
+//   PMM_UI_INSECURE=1  (HA/LKE only: PMM's cert is self-signed behind the egress
+//     MITM so pinning can't match; disables TLS verification, keeps the
+//     origin-redirect guard)
 //
 // Writes a reusable Playwright storage state to
 // .claude/scripts/.sessions/<SESSION_ID>.json — pass that file to a
@@ -55,24 +58,33 @@ async function main() {
 
   const authToken = Buffer.from(`admin:${adminPassword}`).toString("base64");
 
-  // This script only ever targets PMM, so there's no legitimate case for an
-  // insecure fallback -- require the cert pin rather than silently trusting
-  // any certificate on a connection that's about to carry the admin
-  // password. PMM's cert can't be known before the box exists, but once
-  // PMM is up, its cert is fetched over the already-pinned exec-server (see
-  // pmm-linode-docker-provisioning's readyz step) and passed here via PMM_CERT_PATH.
+  // Single-server (docker) path: PMM's cert is fetched over the already-pinned
+  // exec-server and pinned here (SPKI), so the admin password never rides an
+  // unverified connection. HA/LKE has no exec-server and PMM's cert is
+  // self-signed behind the egress MITM, so pinning can't match — PMM_UI_INSECURE=1
+  // opts out of the pin there. It's explicit and loud, and the origin-redirect
+  // refusal below stays as the compensating control against leaking the password
+  // to the wrong origin.
+  const insecure = process.env.PMM_UI_INSECURE === "1";
   const certPath = process.env.PMM_CERT_PATH;
-  if (!certPath) {
+  let launchArgs = [];
+  if (insecure) {
     console.error(
-      "PMM_CERT_PATH is required -- fetch PMM's cert after readyz (see pmm-linode-docker-provisioning skill) and pass it here.",
+      "WARNING: PMM_UI_INSECURE=1 — TLS verification disabled (no SPKI pin). Use only for HA/LKE (self-signed cert behind the egress MITM).",
     );
-    process.exit(1);
-  }
-  if (!fs.existsSync(certPath)) {
-    console.error(`PMM_CERT_PATH set but not found: ${certPath}`);
-    process.exit(1);
+  } else {
+    if (!certPath) {
+      console.error(
+        "PMM_CERT_PATH is required -- fetch PMM's cert after readyz (see linode-docker-provisioning skill), or set PMM_UI_INSECURE=1 for HA/LKE.",
+      );
+      process.exit(1);
+    }
+    if (!fs.existsSync(certPath)) {
+      console.error(`PMM_CERT_PATH set but not found: ${certPath}`);
+      process.exit(1);
+    }
+    launchArgs = [`--ignore-certificate-errors-spki-list=${spkiPinFromCertFile(certPath)}`];
   }
-  const launchArgs = [`--ignore-certificate-errors-spki-list=${spkiPinFromCertFile(certPath)}`];
 
   // Explicit executablePath: the pre-installed Chromium revision at
   // /opt/pw-browsers can drift from what a freshly `npm install`-ed
@@ -87,7 +99,7 @@ async function main() {
     proxy: proxyOpts.proxy,
   });
   const context = await browser.newContext({
-    ignoreHTTPSErrors: false,
+    ignoreHTTPSErrors: insecure,
     viewport: { width, height },
   });
   const page = await context.newPage();

.claude/scripts/pw-screenshot.js

@@ -5,6 +5,15 @@
 // Usage:
 //   node pw-screenshot.js <url> <output.png> [sessionId]
 //
+// Env knobs:
+//   PMM_UI_INSECURE=1   disable TLS verification (HA/LKE: self-signed cert
+//                       behind the egress MITM — pairs with pmm-ui-login.js's
+//                       same flag)
+//   PW_SCROLL=1         scroll the page top-to-bottom first to force Grafana's
+//                       virtualized panels to render before a fullPage shot
+//   PW_CLICK_TEXT='...' click an element by partial text before capturing
+//   PW_SETTLE_MS, PW_WAIT_SELECTOR, PMM_UI_WIDTH/HEIGHT   as before
+//
 // If <sessionId> is given and .claude/scripts/.sessions/<sessionId>.json
 // exists (written by pmm-ui-login.js), it is reused as the browser context's
 // storage state so PMM pages stay logged in.
@@ -33,21 +42,25 @@ async function main() {
   const width = Number(process.env.PMM_UI_WIDTH || 1920);
   const height = Number(process.env.PMM_UI_HEIGHT || 1080);
 
+  // PMM_UI_INSECURE=1 disables TLS verification (no pin) — for HA/LKE, where
+  // PMM's cert is self-signed and the egress gateway MITMs outbound TLS, so
+  // SPKI-pinning can't match. Strict pinning stays the default everywhere else.
+  const insecure = process.env.PMM_UI_INSECURE === "1";
   // PMM_CERT_PATH (see pmm-ui-login.js) pins PMM's own cert instead of
   // trusting any cert -- optional since this script also screenshots
   // non-PMM pages (e.g. a GitHub Actions run) with a real CA already, which
   // strict verification (the default below) already handles fine.
   const certPath = process.env.PMM_CERT_PATH;
   const launchArgs = [];
-  if (certPath) {
+  if (certPath && !insecure) {
     if (!fs.existsSync(certPath)) {
       console.error(`PMM_CERT_PATH set but not found: ${certPath}`);
       process.exit(1);
     }
     launchArgs.push(`--ignore-certificate-errors-spki-list=${spkiPinFromCertFile(certPath)}`);
   }
 
-  const contextOpts = { ignoreHTTPSErrors: false, viewport: { width, height } };
+  const contextOpts = { ignoreHTTPSErrors: insecure, viewport: { width, height } };
   if (sessionId) {
     if (!/^[A-Za-z0-9][A-Za-z0-9_-]*$/.test(sessionId)) {
       console.error(`invalid sessionId '${sessionId}' (letters, digits, '_', '-' only)`);
@@ -116,6 +129,40 @@ async function main() {
     await page.waitForTimeout(settleMs);
   }
 
+  // PW_CLICK_TEXT: click an element by (partial) text before capturing — e.g. to
+  // open a collapsed row or expand a section on the dashboard.
+  const clickText = process.env.PW_CLICK_TEXT;
+  if (clickText) {
+    try {
+      await page.getByText(clickText, { exact: false }).first().click({ timeout: 8000 });
+      await page.waitForTimeout(2500);
+    } catch (e) {
+      console.error("click failed:", clickText, e.message);
+    }
+  }
+
+  // PW_SCROLL=1: Grafana virtualizes panels, so a fullPage shot of a tall HA
+  // dashboard misses panels that never scrolled into view. Scroll through to
+  // force lazy render, then return to the top before capturing.
+  if (process.env.PW_SCROLL === "1") {
+    // Re-read scrollHeight each step: lazy panels expand the page as they render,
+    // so a height captured once up front would stop the loop short and miss lower
+    // panels. Continue until the bottom is reached and the height has settled.
+    // Cap the iterations so a page that keeps growing can't hang the shot.
+    let y = 0;
+    let prevH = -1;
+    for (let i = 0; i < 60; i++) {
+      const scrollH = await page.evaluate(() => document.body.scrollHeight);
+      if (y >= scrollH && scrollH === prevH) break;
+      prevH = scrollH;
+      await page.evaluate((yy) => window.scrollTo(0, yy), y);
+      await page.waitForTimeout(1000);
+      y += 700;
+    }
+    await page.evaluate(() => window.scrollTo(0, 0));
+    await page.waitForTimeout(1500);
+  }
+
   fs.mkdirSync(path.dirname(path.resolve(outputPath)), { recursive: true });
   await page.screenshot({ path: outputPath, fullPage: true });
 

.claude/skills/jira/SKILL.md

@@ -23,7 +23,10 @@ the relay roster-checks. The relay:
 
 Do **not** call the Atlassian MCP connector tools (kept documented below only
 for reference), and do **not** hit `perconadev.atlassian.net` directly — the
-token isn't in this environment. Connector approval is also still broken for
+token isn't in this environment. This includes **searching for existing
+tickets**: use the relay `search` action (JQL), not the Atlassian Rovo search —
+the connector needs interactive auth that isn't there in a Routine/headless run,
+so it fails closed. The relay `search` is the supported dedup path. Connector approval is also still broken for
 Routine grants ([claude-code#61015](https://github.com/anthropics/claude-code/issues/61015)),
 so the connector path stays disabled; the relay path has no approval gate.
 
@@ -112,6 +115,12 @@ J create "$(jq -n --arg s "..." '{issuetype:"Bug", summary:$s, fields:{customfie
 J read "$(jq -n --arg i PMM-15188 '{issue:$i}')"
 J read "$(jq -n --arg i PMM-15188 '{issue:$i,fieldsCsv:"summary,status"}')"
 
+# search — JQL to find existing tickets (e.g. dedup before create). The project
+# is FORCED to PMM, so write only the rest of the clause. Read-only. ORDER BY ok;
+# maxResults<=100 (default 20); fields optional. Use THIS, never the Atlassian MCP.
+J search "$(jq -n --arg q 'text ~ "cannot add MySQL 8.4" AND statusCategory != Done ORDER BY updated DESC' \
+      '{jql:$q, maxResults:20, fields:"summary,status,issuetype,updated"}')"
+
 # comment — visibility is FORCED to Developers by the relay; you cannot post public
 J comment "$(jq -n --arg i PMM-15188 --arg b "h2. QA results"$'\n'"..." '{issue:$i,body:$b}')"
 
@@ -127,8 +136,9 @@ J attach "$(jq -n --arg i PMM-15188 --arg f fb-checks.png \
       --arg c "$(base64 -w0 fb-checks.png)" '{issue:$i,filename:$f,content_b64:$c}')"
 ```
 
-Available actions: `create`, `read`, `comment`, `field`, `transitions`,
+Available actions: `create`, `read`, `search`, `comment`, `field`, `transitions`,
 `transition`, `attach` — the full set the old direct-REST path had **plus
-`create`** (project forced to `PMM`), minus delete (the relay refuses that by
-construction). The **mandatory Developers-only visibility rule** is enforced by
+`create`** (project forced to `PMM`) **and `search`** (JQL, also PMM-scoped, so
+dedup goes through the relay instead of the Atlassian MCP), minus delete (the
+relay refuses that by construction). The **mandatory Developers-only visibility rule** is enforced by
 the relay itself, so it holds even if a caller forgets it.

.claude/skills/linode-docker-provisioning/SKILL.md

@@ -49,31 +49,55 @@ possession gate; `X-Actor` is the identity.
 ```bash
 RELAY=https://139-162-176-43.ip.linodeusercontent.com   # fixed prod relay (reserved IP)
 RUN_ID=<run_id>                       # e.g. PMM-15196 (see "Pick a run_id")
-ROLE=<role>                           # test-runner or investigator (free text, tag only)
+ROLE=<role>                           # test-runner or investigator (safe id: [A-Za-z0-9._-], tag only)
 RUN_DIR="terraform/linode-runner/runs/$RUN_ID"
 mkdir -p "$RUN_DIR"
 ACTOR="$(gh api user --jq .login 2>/dev/null)"
 
 # ttl_hours + pmm_qa_ref are optional; add keep-alive handling below.
-curl -sS -m 600 --fail-with-body -X POST "$RELAY/linode/provision" \
+# 1) Kick off the build — returns immediately with {run_id, status:"provisioning"}.
+curl -sS -m 60 --fail-with-body -X POST "$RELAY/linode/provision" \
   -H "X-Relay-Secret: $RELAY_KEY" -H "X-Actor: $ACTOR" -H "Content-Type: application/json" \
-  -d "$(jq -n --arg r "$ROLE" --arg id "$RUN_ID" '{role:$r, run_id:$id}')" >"$RUN_DIR/provision.json"
-
-# Unpack what the session-side helpers (run.sh/sync.sh/extend.sh) need locally,
-# and tag the run so the SessionEnd hook can tear down exactly our own VMs.
-jq -r .ip            "$RUN_DIR/provision.json" >"$RUN_DIR/ip"
-jq -r .exec_token    "$RUN_DIR/provision.json" >"$RUN_DIR/exec_token"; chmod 600 "$RUN_DIR/exec_token"
-jq -r .exec_cert_pem "$RUN_DIR/provision.json" >"$RUN_DIR/exec_cert.pem"
-printf '%s' "$RELAY"                      >"$RUN_DIR/relay"        # marks this run relay-brokered (holds the relay URL for the SessionEnd hook)
+  -d "$(jq -n --arg r "$ROLE" --arg id "$RUN_ID" '{role:$r, run_id:$id}')" >"$RUN_DIR/provision-start.json"
+
+# Mark the run relay-brokered NOW so the SessionEnd hook can tear it down even if we lose the poll.
+printf '%s' "$RELAY"                      >"$RUN_DIR/relay"        # relay URL for the SessionEnd hook
 printf '%s' "${CLAUDE_CODE_SESSION_ID:-}" >"$RUN_DIR/session_id"   # scopes the SessionEnd hook
+
+# 2) Poll for the result — a dropped connection is recoverable (state is on the relay).
+deadline=$(( $(date +%s) + 900 ))
+while :; do
+  code=$(curl -sS -m 60 -o "$RUN_DIR/provision.json" -w '%{http_code}' -X POST "$RELAY/linode/provision-result" \
+    -H "X-Relay-Secret: $RELAY_KEY" -H "X-Actor: $ACTOR" -H "Content-Type: application/json" \
+    -d "$(jq -n --arg id "$RUN_ID" '{run_id:$id}')")
+  case "$code" in
+    200) echo "VM ready"; break;;
+    202) echo "provisioning…";;
+    502) echo "provisioning FAILED:"; jq -r '.detail // .' "$RUN_DIR/provision.json"; break;;
+    *)   echo "unexpected $code:"; cat "$RUN_DIR/provision.json";;
+  esac
+  [ "$(date +%s)" -lt "$deadline" ] || { echo "timed out"; break; }
+  sleep 15
+done
+
+# 3) Unpack what the session-side helpers (run.sh/sync.sh/extend.sh) need locally.
+if jq -e .exec_token "$RUN_DIR/provision.json" >/dev/null 2>&1; then
+  jq -r .ip            "$RUN_DIR/provision.json" >"$RUN_DIR/ip"
+  jq -r .exec_token    "$RUN_DIR/provision.json" >"$RUN_DIR/exec_token"; chmod 600 "$RUN_DIR/exec_token"
+  jq -r .exec_cert_pem "$RUN_DIR/provision.json" >"$RUN_DIR/exec_cert.pem"
+else
+  echo "no exec creds — tear the run down (Cleanup) before retrying"
+fi
 ```
 
-`role` is `test-runner` or `investigator` (free text, just for the tag). The relay:
+`role` is `test-runner` or `investigator` — a tag only, but it must be a safe
+identifier (`[A-Za-z0-9._-]`, no spaces or `..`); the relay rejects anything else
+with `400 bad_role`. The relay:
 - Creates a Linode VM (default `g6-standard-6`, Ubuntu 24.04) with a firewall open only on 443, tagged `pmm-qa-ephemeral`.
 - Waits for the exec-server to answer, then for cloud-init to finish installing Docker + Ansible and scheduling its own self-destruct timer (default 24h — see Cleanup below).
 - `git clone`s `percona/pmm-qa` onto the box at `/root/pmm-qa` — `main` by default, or pass `"pmm_qa_ref":"<branch>"` in the POST body (must already be pushed; see "Never code on the Linode VM" above).
 
-Works from the **default** proxied-HTTPS environment — no special network policy needed. Takes 2-4 minutes; the relay call blocks until the box is fully ready. After this, `run.sh`/`sync.sh`/`extend.sh` are addressed exactly as before by `<run_id>` — they use the local `exec_token` + `exec_cert.pem`, never the account token. **Teardown is the exception:** it goes through the relay's `/linode/destroy` (see Cleanup), not a local `down.sh`, since destroying the VM needs the account token that no longer lives in this environment.
+Works from the **default** proxied-HTTPS environment — no special network policy needed. Provisioning is **async**: the first call returns a `run_id`, then you poll `/linode/provision-result` until `200` (ready — creds in the body) or `502` (failed). The build runs on the relay and all state lives in its run dir, so a dropped connection is recoverable by re-polling the same `run_id` (usually 2-4 min). After this, `run.sh`/`sync.sh`/`extend.sh` are addressed exactly as before by `<run_id>` — they use the local `exec_token` + `exec_cert.pem`, never the account token. **Teardown is the exception:** it goes through the relay's `/linode/destroy` (see Cleanup), not a local `down.sh`, since destroying the VM needs the account token that no longer lives in this environment.
 
 **Keep-alive:** for an explicit "leave it running" request, add `"ttl_hours":<N>` to the POST body **and** `touch "$RUN_DIR/keep-alive"` — the marker tells the SessionEnd hook to leave this VM up (its on-box self-destruct timer still reaps it after `ttl_hours`).