docs: GB10 deployment guide + coverage-enforcing release gate

.claude/skills/atlas-release/SKILL.md

@@ -0,0 +1,181 @@
+---
+name: atlas-release
+description: The Atlas build → verify → image → publish pipeline, plus the upstream-sync PR automation. Turns a merged commit into a serve-matrix-verified `avarok/atlas-gb10` image that users can pull. Use when cutting an image, closing the main→:latest staleness gap, wiring the release gate, or auto-syncing the fork and opening a MODEL.toml enablement PR. Codifies the REAL commands, containers, tags, and gates already in-tree — it does not invent a new release system.
+argument-hint: <build | verify | image | publish | gate | sync-pr> [target]
+allowed-tools: Bash, Read, Write, Grep, Glob, Agent, Edit
+---
+
+# /atlas-release — Atlas build → verify → ship pipeline
+
+One entry point for turning **merged code into a pullable, verified image**. The
+gap this closes: today `avarok/atlas-gb10:latest` is cut by hand on a GB10 build
+host and drifts weeks behind `main` because nothing rebuilds/re-verifies/re-tags on merge.
+This skill makes each stage explicit, gated, and reproducible so a new image can
+ship in minutes with proof it works.
+
+Four stages, each gates the next. **A stage's output is only valid if the prior
+stage passed its written bar.**
+
+| Stage | Question it answers | Produces | SSOT |
+|-------|--------------------|----------|------|
+| **1. Build** | Does the target(s) compile into one `spark` binary? | `spark` (PTX embedded) | `docker/gb10/Dockerfile`, `crates/atlas-kernels/build.rs` |
+| **2. Verify** | Does every model×quant boot, stay coherent, and hold the signals? | pass/fail verdict + results JSON | `tests/run_all_models.py`, `references/verify-matrix.md` |
+| **3. Image** | Package the *verified* binary, tag it to its exact source. | `avarok/atlas-gb10:<sha>` (+ `.tar` via `docker save`) | `docker/gb10/Dockerfile`, `references/pipeline.md` |
+| **4. Publish** | Make it pullable — **human-gated push** (never automated). | pushed `:sha`/`:dev`/`:latest` + recorded shipped SHA | maintainer runs `docker push`; `references/pipeline.md` |
+
+**Iron rule of this skill: no image is tagged shippable until the serve matrix
+passes.** `run_all_models.py` only writes per-model JSON — so **`tests/gate_results.py`**
+is the verdict: a pure scorer that turns those results into a real PASS/FAIL exit
+code. Crucially it enforces **full coverage** — `run_all_models.py` writes a
+`_manifest.json` of every model the run planned, and the gate fails any planned
+model that produced no result (a boot crash writes no JSON and would otherwise be
+an invisible false-green). `tests/test_gate_results.py` locks this behavior in.
+`references/verify-matrix.md` specifies the bars and the run order.
+
+**Two hard constraints, honored everywhere in this skill:**
+- **Never advance the shippable surface without a human gate.** No `docker push`
+  of any `avarok/atlas-gb10` tag and no moving of `:latest`/`:dev`/`:nightly` — the
+  pipeline builds, verifies, tags, and `docker save`s, and *prepares* the final
+  `docker push` for a maintainer to run (Stage 4 stops there). What ships to users
+  is a human decision on a verified artifact. **Additive git pushes are fine:**
+  `sync-pr` may push an enablement *branch* and open its PR — that is not a
+  shippable-surface change.
+- **No takeover PRs.** Sync-pr opens *enablement* PRs (MODEL.toml, new target) on
+  the fork, or pushes a trivial fix to an existing branch — never a parallel
+  reimplementation of someone's in-flight work. Contribute to the author's branch;
+  reference and comment rather than fork their work (see `CONTRIBUTING.md`).
+
+---
+
+## Commands
+
+### `/atlas-release build [<model-slug> | *]`  (default `*`)
+Compile the `spark` binary. `*` bakes **all** kernel targets into one multi-model
+binary (what `:latest` ships); a slug bakes one slim single-model binary.
+
+```bash
+# Multi-model sweep (the shippable binary). Host-native — the '38s incremental'.
+ATLAS_TARGET_HW=gb10 ATLAS_TARGET_MODEL='*' ATLAS_TARGET_QUANT='*' \
+  cargo build --release -p spark-server
+
+# Or inside the pinned builder image (what the Docker image does):
+docker build -f docker/gb10/Dockerfile -t atlas-gb10:build .
+```
+
+- `ATLAS_TARGET_MODEL` selects which `kernels/gb10/<model>/<quant>/` PTX + which
+  `MODEL.toml` sampling/behavior presets are **embedded** in the binary. It does
+  **not** bake weights — those always load at runtime from HF cache or
+  `--model-from-path`. Runtime picks the right target from the model's
+  `config.json` (`model_type` + `hidden_size`).
+- **Footgun (PCND):** a bare `cargo build` with the env vars *unset* defaults to
+  `qwen3-next-80b-a3b/nvfp4` **only** — not a sweep. The `*` is mandatory and
+  easy to forget. Always pass all three explicitly.
+- The "38s" figure is a **host cargo incremental** rebuild after Rust-only edits
+  (build.rs `rerun-if-changed` gating + its in-build nvcc dedup cache). A *Docker*
+  build is clean each time (~2–3 min; no BuildKit cache mount / sccache in-tree).
+- Full command reference, container split, and toolchain pins: `references/pipeline.md`.
+
+### `/atlas-release verify <image-or-url>`
+Run the **serve matrix** — the gate. A binary/image is "verified" only if it
+passes. Cheapest-sufficient path:
+
+1. **Single-model quick gate** (one model, one box) before the full matrix:
+   `scripts/dev/test_all_models.sh <hf-id> 8888` (boots the image, curls 6 fixed
+   coherence probes). Or the richer `tests/single_gpu_suite.py`.
+2. **Full serve matrix** across all model×quant rounds (head + optional worker):
+   `ATLAS_IMAGE=<tag> python3 tests/run_all_models.py`.
+3. **Coherence gate** (the `webserver_ok` idea, generalized): determinism (5×
+   temp=0 identical), tool reliability ≥8/10, no CJK/Cyrillic/`<think>`/
+   `<tool_call>` leak, streaming envelope valid — via `scripts/test_coherence.py`
+   and the parked `bench/fp8_dgx2_drift/harness` agentic probe.
+4. **Gate + record.** Compute a real pass/fail (Stage 2 in `references/verify-matrix.md`
+   turns the JSON into an exit code) and write the verdict next to the image tag.
+
+> Hand the numerical/behavioral faithfulness question to **`/coherence-parity
+> gate-release`** — that skill owns the KL / argmax / drift thresholds. This skill
+> owns the *operational* gate: does every model in the matrix boot and stay sane.
+
+### `/atlas-release image <git-sha>`
+Build the runtime image from the **verified** binary and tag it to its exact
+source. **Refuses if Stage 2 didn't pass for this SHA.**
+
+```bash
+SHA=$(git rev-parse --short=7 HEAD)
+docker build -f docker/gb10/Dockerfile \
+  --build-arg ATLAS_GIT_SHA="$SHA" \
+  -t avarok/atlas-gb10:"$SHA" .
+docker save avarok/atlas-gb10:"$SHA" | zstd -o atlas-gb10-"$SHA".tar.zst   # offline distribution
+```
+
+- Tag with the **git SHA first** — it is the only identifier that answers "is
+  `:latest` the merged code?" (Today three schemes drift: crate `0.1.0`, image
+  `alpha-2.x`/`3.0.0`, distro `vX.Y.Z`. The SHA tag is the anchor; moving tags
+  point at it.) See the version-provenance note in `references/pipeline.md`.
+- `--build-arg ATLAS_GIT_SHA` now stamps `org.opencontainers.image.revision` into
+  the image (the Dockerfile `ARG`+`LABEL` were added by this pass) — so
+  `docker inspect` answers "which commit is `:latest`?" directly. See
+  `references/pipeline.md` §Provenance.
+
+### `/atlas-release publish <tag>`  (human-gated)
+Move the moving tags onto a verified SHA image and **prepare** the push. This
+command does **not** run `docker push` — it prints the exact commands for the
+maintainer to run (or `docker save`s for offline transfer).
+
+```bash
+# Prepared for the maintainer to execute (all three moving tags advance together
+# onto the same verified SHA — see references/pipeline.md §Publish):
+docker tag avarok/atlas-gb10:$SHA avarok/atlas-gb10:nightly
+docker tag avarok/atlas-gb10:$SHA avarok/atlas-gb10:dev
+docker tag avarok/atlas-gb10:$SHA avarok/atlas-gb10:latest
+docker push avarok/atlas-gb10:$SHA && \
+  docker push avarok/atlas-gb10:nightly && \
+  docker push avarok/atlas-gb10:dev && \
+  docker push avarok/atlas-gb10:latest
+```
+
+After the maintainer pushes, record the shipped SHA (so "what's in `:latest`?"
+is answerable) — a one-line append to `docs/releases/` or the image label.
+
+### `/atlas-release gate`
+Standalone: run only the release gate (serve matrix + coherence probe) against a
+running image and return **PASS/FAIL** + the results table. This is the check
+that must be green before `publish`. `references/verify-matrix.md` is its spec.
+
+### `/atlas-release sync-pr`
+The upstream→fork automation. Watch `monumental/main` (upstream) for merges →
+fast-forward `origin` (Avarok fork) → build → verify → **if** the merge enables a
+new model (new `kernels/gb10/<model>/MODEL.toml` or loader), open the enablement
+PR. AI-attributed, CLA-clean, CI-green **before** submit. Full runbook incl. the
+remotes, the CLA allowlist cleanup, and the CI-green preflight: `references/pr-and-ci.md`.
+
+---
+
+## Operating principles
+- **SSOT:** serve config is owned by `atlas-recipes` recipe `defaults:` (not
+  hand-copied into every doc); the model registry is `kernels/gb10/<model>/MODEL.toml`;
+  the CI cap is `file-size-cap.yml` (≤500 LoC). Derive from these — never
+  restate a value that lives in one of them.
+- **PCND:** no stage runs on an implicit default. `ATLAS_TARGET_MODEL='*'` is
+  explicit; the verify bar is written before the run; the image tag is the SHA.
+  A bare `cargo build` (single-model default) is a bug, not a shortcut.
+- **SBIO:** the gate's decision logic (pass/fail from the results table) is pure;
+  the I/O is the results JSON + the container boot. Don't bake a host/port/tag
+  into the gate — pass them in (`ATLAS_IMAGE`, `--base-url`).
+- **CBD:** when the matrix fails, don't reship-and-pray — bisect. Which model,
+  which round, which signal (coherence vs tool vs determinism vs OOM). The
+  smoking gun is usually one model×quant, one signal. Hand numeric drift to
+  `/coherence-parity diagnose-drift`.
+- **Verify before publish, always.** A green CI (fmt/clippy/test) proves the code
+  compiles GPU-free; it does **not** prove the image serves. Only the serve
+  matrix does. CI-green ≠ shippable.
+
+## Integration
+- **`/coherence-parity`** owns numerical faithfulness (KL/argmax/drift + the
+  `gate-release` thresholds). `/atlas-release verify` calls it for the numeric
+  question and owns the operational "does it boot and stay sane" question.
+- **`/hypercompile`** produces the kernels this pipeline ships. A hypercompile
+  iteration isn't done until `/atlas-release verify` passes on the resulting image.
+- **`atlas-recipes`** is where a winning serve config lands as a recipe; this
+  pipeline ships the *engine* those recipes pin
+  (`container: avarok/atlas-gb10:latest`). Keeping the image fresh is what keeps
+  every `sparkrun run @atlas/*` honest.

.claude/skills/atlas-release/references/pipeline.md

@@ -0,0 +1,141 @@
+# references/pipeline.md — build → image → publish, exact contract
+
+Everything `/atlas-release build`, `image`, and `publish` do, grounded in the
+real files. Nothing here is aspirational unless flagged **[TODO]**.
+
+## 1. Build
+
+### The binary
+```bash
+# The shippable multi-model sweep (host-native; the "38s incremental" path):
+ATLAS_TARGET_HW=gb10 ATLAS_TARGET_MODEL='*' ATLAS_TARGET_QUANT='*' \
+  cargo build --release -p spark-server
+# -> target/release/spark   (PTX for every kernels/gb10/<model>/<quant>/ embedded)
+```
+
+- **Target selection** is three env vars read in `crates/atlas-kernels/build.rs`
+  (defaults if unset: `gb10` / `qwen3-next-80b-a3b` / `nvfp4`). `*` = "every
+  subdir with a `MODEL.toml`". `build.rs` compiles each `.cu` to PTX with
+  `nvcc --ptx -arch=sm_121f` (arch from `kernels/gb10/HARDWARE.toml`) and emits
+  `OUT_DIR/target_ptx.rs`; a content-addressed dedup cache turns ~1100 naive
+  nvcc calls into ~170 unique compiles + copies, run across cores.
+- **Weights are never compiled in.** `MODEL.toml` contributes sampling presets +
+  behavior + the `(model_type, hidden_size)` match table, not weights.
+- **PCND footgun:** unset env → single-model `qwen3-next-80b-a3b` binary, *not* a
+  sweep. `/atlas-release build` always passes all three explicitly.
+- **Incrementality:** `build.rs` emits `cargo:rerun-if-changed` on every consumed
+  `.cu`/`.toml` and `rerun-if-env-changed` on the three `ATLAS_TARGET_*`. Touch
+  only Rust → build.rs doesn't rerun, PTX is reused, ~38s relink. Touch a kernel
+  or flip a target env → PTX recompiles for the affected targets.
+- **Fast Rust-only loop (no nvcc, no kernels):** set `ATLAS_SKIP_BUILD=1
+  CUDARC_CUDA_VERSION=13000` (exactly what CI uses) — `atlas-kernels/build.rs` and
+  `spark-runtime/build.rs` emit a PTX stub, so `cargo check`/clippy run without
+  nvcc. This is a hygiene loop, **not** a runnable binary. **[TODO]** `scripts/check.sh`
+  currently exports `SKIP_ATLAS_BUILD` (no `ATLAS_` prefix) + `CUDARC_CUDA_VERSION=12000`;
+  that name is honored only by `spark-storage/build.rs`, so it does *not* stub the
+  kernel crates. Prefer the `ATLAS_SKIP_BUILD` form until check.sh is fixed.
+
+### The container split (`docker/gb10/Dockerfile`, multi-stage)
+- **builder:** `nvidia/cuda:13.0.0-devel-ubuntu24.04` (digest-pinned). Installs
+  rust + `cmake clang libclang-dev` (vendored xgrammar-rs), sets
+  `ATLAS_TARGET_HW=gb10 ATLAS_TARGET_MODEL=* ATLAS_TARGET_QUANT=*`, runs
+  `cargo build --release -p spark-server`.
+- **runtime:** `nvidia/cuda:13.0.0-runtime-ubuntu24.04` (digest-pinned). Installs
+  `libnccl2` (**fail-fast asserts NCCL ≥ 2.28** for symmetric-memory
+  `ncclMemAlloc`) + RDMA userspace (`libibverbs1 librdmacm1 ibverbs-providers`).
+  Then **`COPY --from=builder /build/target/release/spark /usr/local/bin/spark`**
+  + `jinja-templates/` + LICENSE/README. `ENTRYPOINT ["spark"]`, `EXPOSE 8888`.
+- **"Mount the result into a runtime container" is a multi-stage `COPY`, not a
+  volume mount.** The binary is baked at build time. No nvcc in the runtime image
+  — the host driver re-JITs the embedded PTX → SASS for the running SM (ADR-0009).
+- **Toolchain hazard [TODO]:** the sweep Dockerfile sets `RUSTUP_TOOLCHAIN=stable`
+  to override the `rust-toolchain.toml` pin (`1.93.1`); the single-model
+  Dockerfiles don't. Two image families can build under different rustc. Pin both
+  to the same toolchain for reproducibility.
+
+### Model images
+- `docker/gb10/Dockerfile` = the **all-models** image (`avarok/atlas-gb10:*`).
+- `docker/gb10/<model>/nvfp4/Dockerfile` = per-model slim images (10 exist). Each
+  hardcodes `ATLAS_TARGET_MODEL=<slug> ATLAS_TARGET_QUANT=nvfp4`. Note only a
+  `nvfp4/` subdir exists per model — the nvfp4 bundle carries the FP8 + BF16 code
+  paths, so FP8 models (`qwen3.6-27b`, `qwen3.6-35b-a3b`) live under `nvfp4/` too.
+- **COPY-set divergence [TODO]:** the sweep Dockerfile COPYs `vendor/` (required —
+  `Cargo.toml` has `[patch.crates-io] cudarc = { path = "vendor/cudarc" }`) and
+  `jinja-templates/`; the single-model `qwen3.5-35b-a3b` Dockerfile COPYs neither.
+  Verify single-model images still build before relying on them.
+
+## 2. Image — tag to the source
+
+```bash
+SHA=$(git rev-parse --short=7 HEAD)
+docker build -f docker/gb10/Dockerfile --build-arg ATLAS_GIT_SHA="$SHA" \
+  -t avarok/atlas-gb10:"$SHA" .
+```
+
+**Always tag with the git SHA first.** It is the only identifier that answers
+"is `:latest` the merged code?" The moving tags (`:dev`, `:nightly`, `:latest`)
+are *pointers* moved onto a verified `:sha` in Stage 4 — never built independently.
+
+### Provenance — now stamped into the image
+`scripts/release.sh` and `/atlas-release image` pass `--build-arg ATLAS_GIT_SHA=...`.
+The runtime stage of `docker/gb10/Dockerfile` now declares the matching `ARG` +
+`LABEL org.opencontainers.image.revision` (added by this pass), so the SHA is
+recorded in the image. Answer the staleness question directly:
+
+```bash
+docker inspect --format '{{ index .Config.Labels "org.opencontainers.image.revision" }}' avarok/atlas-gb10:latest
+```
+
+(The per-model single-model Dockerfiles don't yet carry the `ARG` — add the same
+two lines there if you ship a per-model image and want its provenance.)
+
+### Offline distribution (`docker save`)
+```bash
+docker save avarok/atlas-gb10:"$SHA" | zstd -T0 -o atlas-gb10-"$SHA".tar.zst
+# receiver:  zstd -dc atlas-gb10-<sha>.tar.zst | docker load
+```
+Use this when there's no registry access, or to hand a verified image to a user
+directly (RoCE-copy dgx1→dgx2, or attach to a GitHub Release alongside the distro
+tarballs).
+
+## 3. Publish — human-gated (never automated)
+
+The pipeline stops here and **prepares** the push. The maintainer runs it. All
+moving tags (`:dev`, `:nightly`, `:latest`) advance together onto the same
+verified `:sha` — they are never built independently.
+
+```bash
+for T in nightly dev latest; do docker tag avarok/atlas-gb10:"$SHA" avarok/atlas-gb10:"$T"; done
+for T in "$SHA" nightly dev latest; do docker push avarok/atlas-gb10:"$T"; done
+```
+
+Preconditions the skill enforces before printing this:
+1. Stage 2 verify **passed** for exactly this `$SHA` (not a different tree).
+2. The build came from the intended ref — **build from `main` tip**, not whatever
+   branch happens to be checked out on the build host (a real past footgun; that
+   checkout is often a feature branch). Verify: `git rev-parse --short=7 origin/main == $SHA`.
+3. Docker is logged in as `avarok` on the pushing host.
+
+After push, **record the shipped SHA** so `:latest`'s provenance is durable — append
+to `docs/releases/` or rely on the image label from the ARG fix above.
+
+## Closing the main→:latest gap (why this pipeline exists)
+
+Today, three things are all manual and none are chained:
+1. Nothing rebuilds the image on merge to `main`.
+2. Nothing verifies an image before it's tagged `:latest`.
+3. The push is a human running `docker build && docker push` by hand from an
+   arbitrary branch.
+
+`release.yml` automates **only** the relocatable **tarball** distros on a `v*`
+tag (via `scripts/package-distro.sh` on self-hosted `[gb10]`/`[strix]` runners) —
+it never touches the Docker image. So the image path has zero automation.
+
+**The minimum viable close** (respecting no-auto-push): a self-hosted-runner
+workflow on `push: [main]` that runs Stage 1 (build) + Stage 2 (verify serve
+matrix) + Stage 3 (build + tag `:sha` + `docker save` as an artifact), and
+**stops** — surfacing a green "ready to publish `<sha>`" the maintainer promotes
+with one `docker push`. That turns weeks of drift into a one-command publish of a
+pre-verified image, while keeping the decision to ship to users in human hands.
+See `references/pr-and-ci.md` for the self-hosted-runner shape (the release-distros
+job already proves self-hosted GB10 runners work).