ci(e2e): restore staging launchable validation #9351
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,6 +1,6 @@ | ||
| --- | ||
| name: nemoclaw-maintainer-e2e | ||
| description: Dispatches and verifies trusted GitHub Actions E2E for NemoClaw maintainers, including manual PR E2E for the latest PR commit. Use for requests such as run E2E for PR #123, run the E2E suite, run the Launchable E2E, run the full E2E suite, deploy pre-release full E2E, run pre-tag full E2E, or run release-candidate E2E. | ||
| --- | ||
|
|
||
| <!-- SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. --> | ||
|
|
@@ -34,14 +34,17 @@ The workflow does not rotate or revoke these API keys or messaging credentials. | |
| Live targets can create external resources. | ||
| After a failure, inspect the artifacts and remove resources that target cleanup did not remove. | ||
|
|
||
| `Exact staging Brev Launchable` reads these credentials from repository Actions secrets: | ||
|
|
||
| - `BREV_API_KEY` authenticates the trusted host-side Brev CLI for workspace operations in the organization identified by `BREV_ORG_ID`. Candidate code does not receive this API key. | ||
| - `NEMOCLAW_IMAGE_DISPATCH_TOKEN` is exposed as `GH_TOKEN` only to the trusted host script. It grants Actions read/write access to `brevdev/nemoclaw-image`, which the script uses to dispatch the image workflow, inspect its run, and download its handoff artifact. | ||
| - `NVIDIA_INFERENCE_API_KEY` is exported into the Brev guest for the full E2E process. Code in the baked candidate checkout can read and use it. | ||
|
|
||
| `brev login` writes `BREV_API_KEY` and `BREV_ORG_ID` to `$HOME/.brev/credentials.json` on the GitHub-hosted runner. Later trusted steps and processes in the same job can read that file. The workflow does not delete it explicitly; it remains on the ephemeral runner filesystem until runner teardown discards that filesystem. | ||
| These credentials remain valid until they expire or an administrator revokes them in their issuing services. If cleanup fails, remove the recorded Brev workspace. Rotate or revoke each credential to remove later access. | ||
|
Comment on lines
+43
to
+44
Contributor
There was a problem hiding this comment. 🔒 Security & Privacy | 🟡 Minor | ⚡ Quick win 🧩 Analysis chain🏁 Script executed: #!/usr/bin/env bash
set -euo pipefail
version='0.6.334'
archive="$(mktemp)"
binary="$(mktemp)"
trap 'rm -f "$archive" "$binary"' EXIT
curl -fsSL -o "$archive" \
"https://github.com/brevdev/brev-cli/releases/download/v${version}/brev-cli_${version}_linux_amd64.tar.gz"
tar -xOzf "$archive" brev >"$binary"
strings "$binary" | rg 'credentials\.json|BREV_API_KEY|BREV_ORG_ID'Repository: NVIDIA/NemoClaw Length of output: 195 🏁 Script executed: #!/usr/bin/env bash
set -euo pipefail
version='0.6.334'
archive="$(mktemp)"
binary="$(mktemp)"
trap 'rm -f "$archive" "$binary"' EXIT
curl -fsSL -o "$archive" \
"https://github.com/brevdev/brev-cli/releases/download/v${version}/brev-cli_${version}_linux_amd64.tar.gz"
tar -tzf "$archive"
tar -xOzf "$archive" brev >"$binary"
python3 - "$binary" <<'PY'
import re
import sys
data = open(sys.argv[1], "rb").read()
matches = sorted(set(
item.decode("ascii", "ignore")
for item in re.findall(rb"[\x20-\x7e]{4,}", data)
if any(token in item for token in (
b"credentials.json", b"BREV_API_KEY", b"BREV_ORG_ID",
b".brev", b"credentials", b"BREV_"
))
))
for item in matches:
print(item)
PYRepository: NVIDIA/NemoClaw Length of output: 24418 🏁 Script executed: #!/usr/bin/env bash
set -euo pipefail
tag='v0.6.334'
tree="$(mktemp)"
trap 'rm -f "$tree"' EXIT
curl -fsSL \
"https://api.github.com/repos/brevdev/brev-cli/git/trees/${tag}?recursive=1" \
-o "$tree"
python3 - "$tree" <<'PY'
import json
import sys
payload = json.load(open(sys.argv[1]))
for item in payload.get("tree", []):
path = item.get("path", "")
if any(term in path.lower() for term in ("login", "credential", "auth", "config")):
print(path)
PYRepository: NVIDIA/NemoClaw Length of output: 204 🌐 Web query:
💡 Result: In the Brev CLI (v0.6.324 and later, including v0.6.334), Citations:
🌐 Web query:
💡 Result: The terms api_key and org_id are associated with authentication and configuration within the Brev CLI (brev-cli), which is used to manage NVIDIA Brev cloud environments [1][2]. In the context of the Brev CLI, these identifiers are typically handled during the authentication process when a user or machine logs in via an API key instead of standard user-based OAuth flows [1][3]. Key details regarding their usage include: Authentication Method: The introduction of API Key-based authentication allows for non-interactive or service-account style access to Brev resources [1][3]. Credentials Storage: While authentication data is often stored locally (typically in a credentials.json file or similar store within the ~/.brev/ directory), the CLI uses these fields to differentiate between authenticated states—specifically distinguishing between standard user sessions (which may require user-linked organizations) and API key sessions (which may have restricted capabilities, such as the inability to list organizations) [4][1][5]. Functionality: When the CLI resolves authentication, it checks if an API key is present. If it is, the CLI adjusts its behavior; for instance, certain commands like listing organizations are restricted for API key users, while workspace-related commands are scoped to the active organization associated with that key [1][5]. If you are encountering these fields in a JSON configuration file, they are likely parameters used by the CLI's internal store to manage your session identity and the organization context in which your commands are being executed [4][5]. Citations:
Correct the credential-file description.
🧰 Tools🪛 LanguageTool[style] ~44-~44: Consider using a different verb to strengthen your wording. (REMOVE_REVOKE) 🪛 SkillSpector (2.5.1)[error] 43: [PE3] Credential Access: Code accesses credential files (SSH keys, AWS credentials, etc.). This could indicate credential theft attempts. Remediation: Remove references to credential paths. Use environment variables or secrets managers. For docs, use placeholder paths (e.g., /path/to/config). Never load .env or token files in production code paths. (Privilege Escalation (PE3)) 🤖 Prompt for AI AgentsSource: Linters/SAST tools |
||
| This Brev credential boundary applies only to trusted Launchable or full manual dispatches against `main`. It does not apply to `main` pushes or manual PR runs. | ||
|
|
||
| The `NEMOCLAW_STAGING_LAUNCHABLE_ID` repository Actions variable selects the standing Launchable. Keep its value equal to the Launchable ID in the default URL owned by [`nemoclaw-maintainer-validate-launchable`](../nemoclaw-maintainer-validate-launchable/SKILL.md). | ||
|
|
||
| For `managed-image-protected-runtime`, the workflow supplies the long-lived `NVIDIA_API_KEY` repository secret only to the trusted qualification step. Trusted host code uses it for NGC login and passes it as `NGC_API_KEY` and `NIM_NGC_API_KEY` to the temporary NIM container. Candidate managed sandboxes receive generated local route tokens instead of this key. The live fixture removes the temporary NIM container only if its exact ID, name, requested image, immutable image ID, cohort owner, and provider kind match the recorded authority. The test fails if evidence is missing or ambiguous, a name is reused, authority drifts, removal is indeterminate, or the exact ID or name remains. A cleanup refusal can leave the container and its API key in place until runner teardown. The final workflow step removes the job's isolated Docker credential directory and fails if that removal does not complete. The workflow does not revoke the NVIDIA API key. Revoke it, or rotate it and disable the old value, in the issuing NVIDIA service. Verify that the exposed key is no longer valid. | ||
|
|
||
|
|
@@ -67,7 +70,7 @@ Require a review reason containing 10 to 500 printable characters. | |
| Choose exactly one mode: | ||
|
|
||
| - For a PR revision run, leave `E2E_JOBS` empty. The run selects: | ||
| - every default-selected free-standing workflow E2E except `Exact staging Brev Launchable`; | ||
| - every shared credential-free test; and | ||
| - these controller-selected registry targets: `ubuntu-policy-custom-missing-presets-negative`, `ubuntu-repo-cloud-langchain-deepagents-code`, `ubuntu-repo-cloud-openclaw`, and `ubuntu-repo-docker-post-reboot-recovery`. | ||
| The run skips `jetson-nvmap-gpu` unless `allow_jetson_dispatch` is `true`. | ||
|
|
@@ -158,25 +161,20 @@ A changed head repository, head SHA, or base SHA invalidates the evidence and re | |
| | Request | Mode | `jobs` | `include_staging_brev_launchable` | | ||
| |---|---|---|---| | ||
| | “Run the E2E suite” | Ordinary | empty | `false` | | ||
| | “Run the Launchable E2E” | Launchable | `staging-brev-launchable` | `false` | | ||
| | “Run the full E2E suite” | Full | empty | `true` | | ||
| | “deploy pre-release full E2E” | Full | empty | `true` | | ||
| | “run pre-tag full E2E” | Full | empty | `true` | | ||
| | “run release-candidate E2E” | Full | empty | `true` | | ||
| | “run pre-tag E2E with an administrator job waiver” | Administrator-waived full | empty | `true` | | ||
|
|
||
| A generic E2E request must not authorize the Brev Launchable path. | ||
| Do not infer full mode from words such as “all” or “complete.” | ||
| Ask for clarification only when the request contains conflicting mode phrases. | ||
|
|
||
| Ordinary mode selects every default-selected workflow E2E except `Exact staging Brev Launchable`. | ||
| Launchable mode runs only `Exact staging Brev Launchable`. | ||
| Full mode adds `Exact staging Brev Launchable` to the default E2E selection in the same workflow run. | ||
| Administrator-waived full mode runs the full suite but omits the approved execution jobs from release qualification. | ||
| Every waived job still runs. | ||
| Use this mode only when a repository administrator explicitly authorizes the job IDs and supplies the reason. | ||
|
|
@@ -222,7 +220,7 @@ gh workflow run .github/workflows/e2e.yaml \ | |
| -f "correlation_id=${CORRELATION_ID}" | ||
| ``` | ||
|
|
||
| For Launchable mode: | ||
|
|
||
| ```bash | ||
| gh workflow run .github/workflows/e2e.yaml \ | ||
|
|
@@ -273,8 +271,8 @@ gh workflow run .github/workflows/e2e.yaml \ | |
| ``` | ||
|
|
||
| Do not set `jobs=staging-brev-launchable` for full mode. | ||
| Empty `jobs` and `targets` select every default-selected workflow E2E except `Exact staging Brev Launchable`. | ||
| The `include_staging_brev_launchable` input adds the Launchable E2E job to that same run. | ||
| The trusted `main` workflow verifies that the dispatching and rerunning actors have | ||
| repository `maintain` or `admin` permission before the Launchable path's source | ||
| checkout. That role check is the authorization. | ||
|
|
@@ -300,7 +298,7 @@ empty-selector manual run or enable explicit qualification selection. Set it | |
| only after a repository administrator confirms an online DGX Spark runner in | ||
| the authoritative runner inventory. | ||
| If GitHub pauses the qualification job for the `approve-dgx-spark-image-qualification` environment, an authorized environment reviewer must approve it before qualification starts. | ||
| `Exact staging Brev Launchable` does not require environment approval. | ||
|
|
||
| Find the run by its unique title: | ||
|
|
||
|
|
@@ -334,7 +332,7 @@ Wait for completion: | |
| gh run watch "$RUN_ID" --repo NVIDIA/NemoClaw | ||
| ``` | ||
|
|
||
| Launchable and full modes can wait in the non-cancelling Launchable concurrency queue. | ||
| Queued, waiting, or accepted dispatch state is not success. | ||
| Classify the completed workflow and `Release qualification` job with the checks below. | ||
|
|
||
|
|
@@ -356,21 +354,20 @@ Require `run-$RUN_ID.json` to report: | |
| - `head_sha` equal to `CANDIDATE_SHA`; | ||
| - `status` equal to `completed`. | ||
|
|
||
| For ordinary, Launchable, and unwaived full modes, require `conclusion` equal to `success`. | ||
| For administrator-waived full mode, permit `conclusion` equal to `success` or `failure`. | ||
| A `failure` conclusion is acceptable only when one completed, successful `Release qualification` job and a valid exact-run waiver artifact with at least one canonical waived job failure both exist. | ||
|
|
||
| For Launchable mode, also require `jobs-latest-$RUN_ID.json` to contain one completed, successful | ||
| `Exact staging Brev Launchable` job. Return the workflow and job URLs. | ||
|
|
||
| For a full run, with or without a job waiver, require `jobs-latest-$RUN_ID.json` to contain one completed, successful | ||
| `Release qualification` job. Return its job URL with the workflow URL. | ||
| In full mode, that job waits for every default-required E2E result, including `Exact staging Brev Launchable`. | ||
| The Launchable job directly verifies the candidate checkout, in-guest full E2E result, and workspace cleanup before it succeeds. | ||
| Its `launchable-e2e.json`, `full-e2e.log`, and `cleanup.json` artifacts remain available for diagnosis. | ||
| A skipped, cancelled, queued, or failed `Release qualification` job is not evidence. | ||
| A Launchable-only run is not full-mode or pre-tag release evidence. | ||
|
|
||
| For administrator-waived full mode, the job waits for every unwaived release-required result. | ||
| A waived execution job may fail without failing `Release qualification`. | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🩺 Stability & Availability | 🟠 Major | ⚡ Quick win
Remove the recorded Brev workspace before rerun.
When Launchable E2E or cleanup fails, inspect the artifacts and remove the recorded Brev workspace before rerunning. The current step requires artifact inspection and a rerun, but it does not require workspace removal. A failed cleanup can leave a live workspace and cause resource leakage on the next run.
As per path instructions: “On failure, inspect artifacts and remove the recorded Brev workspace.”
🤖 Prompt for AI Agents
Source: Path instructions