sync: upstream catch-up (133 commits, base 2026-05-26) + re-home fork delta after supervisor split

.agents/skills/build-from-issue/SKILL.md

@@ -148,7 +148,8 @@ In the prompt, instruct the reviewer to:
    - **Medium**: Multiple files/components, some design decisions, but well-scoped
    - **High**: Cross-cutting changes, architectural decisions needed, significant unknowns
 8. Call out risks, unknowns, and decisions that need stakeholder input.
-9. Assess **LSM compatibility** — if the change touches process identity, `/proc` filesystem access, binary execution, or inter-process visibility, flag whether it will behave differently on hosts running SELinux (enforcing) or AppArmor. In particular, tests that fork+exec into system binaries will fail on SELinux-enforcing hosts due to cross-label `/proc/<pid>/exe` access restrictions.
+9. Assess **gateway config documentation impact** — if the change adds, removes, renames, or changes defaults for gateway TOML keys or driver-specific config options, the plan must include an update to `docs/reference/gateway-config.mdx`. If the change is surfaced through Helm or a compute-driver overview, also include `docs/reference/sandbox-compute-drivers.mdx` or the relevant deployment docs.
+10. Assess **LSM compatibility** — if the change touches process identity, `/proc` filesystem access, binary execution, or inter-process visibility, flag whether it will behave differently on hosts running SELinux (enforcing) or AppArmor. In particular, tests that fork+exec into system binaries will fail on SELinux-enforcing hosts due to cross-label `/proc/<pid>/exe` access restrictions.
 
 ### A2: Post the Plan Comment
 
@@ -436,6 +437,13 @@ Review the documentation requirements in `AGENTS.md` and update any affected
 docs as part of the implementation. Keep documentation changes scoped to the
 behavior or subsystem that changed.
 
+If the implementation changes gateway TOML parsing, `[openshell.gateway]`
+fields, `[openshell.drivers.<name>]` fields, driver config defaults, or Helm
+rendering of `gateway.toml`, update `docs/reference/gateway-config.mdx` in the
+same branch. If the change affects user-facing compute-driver setup, also
+update `docs/reference/sandbox-compute-drivers.mdx` or the relevant deployment
+page.
+
 ### Step 12: Commit and Push
 
 Commit all changes using conventional commit format. The `<type>` comes from the issue type in the plan:

.agents/skills/create-github-pr/SKILL.md

@@ -15,6 +15,15 @@ Create pull requests on GitHub using the `gh` CLI.
 
 ## Before Creating a PR
 
+### Check Config Documentation
+
+If the branch changes gateway TOML parsing, `[openshell.gateway]` fields,
+`[openshell.drivers.<name>]` fields, driver config defaults, or Helm rendering
+of `gateway.toml`, verify that `docs/reference/gateway-config.mdx` is updated
+in the same branch. If the change affects user-facing compute-driver setup,
+also update `docs/reference/sandbox-compute-drivers.mdx` or the relevant
+deployment docs.
+
 ### Run Pre-commit Checks
 
 Run the local pre-commit task before opening a PR:

.agents/skills/create-rfc/SKILL.md

@@ -0,0 +1,51 @@
+---
+name: create-rfc
+description: Create OpenShell RFC proposals in rfc/ from a design request. Use when the user asks to write, draft, start, create, or update an RFC, Request for Comments, architecture proposal, API proposal, process proposal, or cross-cutting design proposal that should follow the OpenShell RFC process and template.
+---
+
+# Create RFC
+
+## Workflow
+
+Create RFCs by following `rfc/README.md` and `rfc/0000-template/README.md`.
+Keep the template as the source of truth for section guidance.
+
+1. Read `rfc/README.md` to confirm when an RFC is appropriate, how to choose the
+   RFC number, and how the lifecycle works.
+2. Read `rfc/0000-template/README.md` before drafting. Follow its section
+   guidance, including scope, expected detail, and suggested section length.
+3. Choose the next available `NNNN` from the existing `rfc/NNNN-*` directories
+   unless the user provided a specific number.
+4. Create `rfc/NNNN-short-title/README.md` by copying the template and replacing
+   placeholders. Use a short hyphenated folder title.
+5. Fill in front matter with the RFC author, `state: draft`, and any related
+   links the user provided. If the author is unknown, use the requesting user's
+   GitHub handle when available or leave the template placeholder.
+6. Draft each section from the user's design context. Keep Summary concise,
+   Motivation readable by anyone, Non-goals explicit, Proposal focused on what
+   is being proposed, and Alternatives focused on credible competing approaches.
+7. Preserve uncertainty in Open questions instead of silently deciding unknowns.
+   If a missing decision blocks a coherent RFC, ask the user for that decision.
+8. Check the completed RFC against the template once more before finishing.
+
+## Writing Standards
+
+- Prefer concrete design statements over placeholder language.
+- Link to relevant issues, prior RFCs, and architecture docs when they provide
+  needed context.
+- Keep rejected or left-out designs in Alternatives, not Proposal.
+- Use Mermaid diagrams for architecture or data flow when a diagram would make
+  the proposal easier to review.
+- Do not update `architecture/` or published docs just because an RFC was
+  drafted. Those updates belong with implementation or with an accepted RFC when
+  the user asks for them.
+
+## Validation
+
+Before handing the RFC back to the user:
+
+- Verify the folder name and RFC number match the process in `rfc/README.md`.
+- Verify every template section is present or intentionally marked as not
+  applicable.
+- Run a Markdown formatting or lint check only if the repo already provides one
+  for Markdown-only changes.

.agents/skills/create-rfc/agents/openai.yaml

@@ -0,0 +1,4 @@
+interface:
+  display_name: "Create RFC"
+  short_description: "Create OpenShell RFC proposals"
+  default_prompt: "Use $create-rfc to draft an OpenShell RFC from this design."

.agents/skills/create-spike/SKILL.md

@@ -91,9 +91,11 @@ The prompt to the reviewer **must** instruct it to:
 
 9. **Check architecture docs** in the `architecture/` directory for relevant documentation about the affected subsystems.
 
-10. **Assess Linux Security Module (LSM) impact.** If the change involves process identity, `/proc` filesystem access, file labeling, binary execution, or inter-process visibility, call out whether it will behave differently on hosts running SELinux (enforcing) or AppArmor. For example: reading `/proc/<pid>/exe` across an SELinux domain boundary returns ENOENT, not EACCES. Tests that fork+exec into system binaries (different SELinux label) will fail on enforcing hosts. Flag any LSM-sensitive code paths and recommend mitigations.
+10. **Assess gateway config documentation impact.** If the change would add, remove, rename, or change defaults for gateway TOML keys or driver-specific config options, call out that `docs/reference/gateway-config.mdx` must be updated. If the change is surfaced through Helm or compute-driver setup docs, call out the relevant deployment or compute-driver docs too.
 
-11. **Determine the issue type:** `feat`, `fix`, `refactor`, `chore`, `perf`, or `docs`.
+11. **Assess Linux Security Module (LSM) impact.** If the change involves process identity, `/proc` filesystem access, file labeling, binary execution, or inter-process visibility, call out whether it will behave differently on hosts running SELinux (enforcing) or AppArmor. For example: reading `/proc/<pid>/exe` across an SELinux domain boundary returns ENOENT, not EACCES. Tests that fork+exec into system binaries (different SELinux label) will fail on enforcing hosts. Flag any LSM-sensitive code paths and recommend mitigations.
+
+12. **Determine the issue type:** `feat`, `fix`, `refactor`, `chore`, `perf`, or `docs`.
 
 ### What makes a good investigation prompt
 

.agents/skills/debug-openshell-cluster/SKILL.md

@@ -54,7 +54,7 @@ Use gateway metadata, deployment values, or the user's setup notes to identify t
 |---|---|
 | Docker | Gateway process logs, Docker daemon health, sandbox containers, image pulls. |
 | Podman | Podman socket, rootless networking, sandbox containers, image pulls. |
-| Kubernetes | Helm release, StatefulSet, service, secrets, sandbox pods, events. |
+| Kubernetes | Helm release, gateway workload, service, secrets, sandbox pods, events. |
 | VM | VM driver logs, rootfs availability, host virtualization support. |
 
 ### Step 3: Check Docker-Backed Gateways
@@ -131,12 +131,27 @@ Common findings:
 ```bash
 helm -n openshell status openshell
 helm -n openshell get values openshell
-kubectl -n openshell get statefulset,pod,svc,pvc
-kubectl -n openshell logs statefulset/openshell --tail=200
+kubectl -n openshell get deployment,statefulset,pod,svc,pvc
+kubectl -n openshell logs deployment/openshell -c openshell-gateway --tail=200
+kubectl -n openshell logs statefulset/openshell -c openshell-gateway --tail=200
+kubectl -n openshell rollout status deployment/openshell
 kubectl -n openshell rollout status statefulset/openshell
 ```
 
-Look for failed installs, unexpected values, missing namespace, wrong image tag, TLS settings that do not match the registered endpoint, and scheduling failures.
+Use the log and rollout commands for the workload kind that exists in the
+release. Look for failed installs, unexpected values, missing namespace, wrong
+image tag, TLS settings that do not match the registered endpoint, and
+scheduling failures.
+
+For HA or PostgreSQL-backed installs, also check the external database Secret
+referenced by `server.externalDbSecret` and the PostgreSQL workload if the test
+or operator deployed one in-cluster:
+
+```bash
+kubectl -n openshell get secret openshell-ha-pg -o yaml
+kubectl -n openshell get deployment,service,pod -l app.kubernetes.io/name=openshell-e2e-postgres
+kubectl -n openshell logs deployment/openshell-e2e-postgres --tail=200
+```
 
 Check required Helm deployment secrets:
 
@@ -148,17 +163,45 @@ kubectl -n openshell get secret \
   openshell-jwt-keys
 ```
 
+In cert-manager installs, `certManager.enabled=true` makes cert-manager own TLS
+generation. The Helm chart should still render the `openshell-certgen`
+pre-install/pre-upgrade hook in JWT-only mode to create `openshell-jwt-keys`,
+even if `pkiInitJob.enabled` remains true.
+If the gateway pod is pending with `MountVolume.SetUp failed for volume
+"sandbox-jwt"` and `openshell-jwt-keys` is absent, inspect the rendered
+`templates/certgen.yaml` output and the hook Job logs; cert-manager creates TLS
+Secrets but does not create the sandbox JWT signing Secret.
+
 If the gateway exits with `failed to read sandbox JWT signing key from
 /etc/openshell-jwt/signing.pem`, verify that `openshell-jwt-keys` contains
-`signing.pem`, `public.pem`, and `kid`, and that the StatefulSet mounts the
+`signing.pem`, `public.pem`, and `kid`, and that the gateway workload mounts the
 `sandbox-jwt` secret at `/etc/openshell-jwt`. The sandbox JWT mount is required
 even when local Helm values disable TLS.
 
+If `server.providerTokenGrants.spiffe.enabled=true`, the gateway should still
+render `[openshell.gateway.gateway_jwt]` and mount the `sandbox-jwt` Secret.
+SPIRE is used only by sandbox pods for dynamic provider token grants. Verify
+that SPIRE is installed, the CSI driver is available, and the Kubernetes driver
+config includes `provider_spiffe_workload_api_socket_path`:
+
+```bash
+helm -n openshell get values openshell | grep -E 'providerTokenGrants|workloadApiSocketPath'
+kubectl get pods -A | grep -E 'spire|spiffe'
+kubectl -n openshell get configmap openshell-config -o yaml | grep provider_spiffe_workload_api_socket_path
+```
+
+Sandbox pods using provider token grants should have an
+`openshell.io/sandbox-id` annotation, an `openshell.ai/managed-by=openshell`
+label, supervisor env vars `OPENSHELL_K8S_SA_TOKEN_FILE` and
+`OPENSHELL_PROVIDER_SPIFFE_WORKLOAD_API_SOCKET`, plus both the projected
+`openshell-sa-token` volume and the `spiffe-workload-api` CSI volume.
+
 Check the image references currently used by the gateway deployment:
 
 ```bash
+kubectl -n openshell get deployment openshell -o jsonpath="{.spec.template.spec.containers[*].image}{\"\n\"}{.spec.template.spec.containers[*].env[?(@.name==\"OPENSHELL_SUPERVISOR_IMAGE\")].value}{\"\n\"}"
 kubectl -n openshell get statefulset openshell -o jsonpath="{.spec.template.spec.containers[*].image}{\"\n\"}{.spec.template.spec.containers[*].env[?(@.name==\"OPENSHELL_SUPERVISOR_IMAGE\")].value}{\"\n\"}"
-helm -n openshell get values openshell | grep -E 'repository|tag|supervisorImage'
+helm -n openshell get values openshell | grep -E 'repository|tag|supervisorImage|workload'
 ```
 
 The gateway image built from `deploy/docker/Dockerfile.gateway` and the scratch supervisor image built from `deploy/docker/Dockerfile.supervisor` should use the same build tag in branch and E2E deploys. A stale supervisor image can make sandbox behavior lag behind gateway policy or proto changes.
@@ -201,7 +244,8 @@ If the gateway is healthy but sandbox creation fails:
 ```bash
 kubectl -n openshell get pods
 kubectl -n openshell get events --sort-by=.lastTimestamp | tail -n 50
-kubectl -n openshell logs statefulset/openshell --tail=200
+kubectl -n openshell logs deployment/openshell -c openshell-gateway --tail=200
+kubectl -n openshell logs statefulset/openshell -c openshell-gateway --tail=200
 ```
 
 Check the configured sandbox namespace:
@@ -249,7 +293,7 @@ openshell logs <sandbox-name>
 | Docker or Podman sandbox never registers | Wrong callback endpoint or supervisor startup failure | Gateway logs and sandbox container logs |
 | Docker GPU e2e fails before GPU sandbox comparison | NVIDIA CDI specs are missing or Docker has not discovered them | `docker info --format '{{json .DiscoveredDevices}}'`, `/etc/cdi`, `/var/run/cdi`, `nvidia-cdi-refresh.service` |
 | Kubernetes gateway pod pending | PVC unbound, taint, selector, or insufficient resources | `kubectl -n openshell describe pod <pod>` |
-| Kubernetes gateway pod crash loops | Missing secret, bad DB URL, bad TLS config | `kubectl -n openshell logs statefulset/openshell` |
+| Kubernetes gateway pod crash loops | Missing secret, bad DB URL, bad TLS config | `kubectl -n openshell logs deployment/openshell -c openshell-gateway` or `kubectl -n openshell logs statefulset/openshell -c openshell-gateway` |
 | CLI TLS error | Local mTLS bundle does not match server cert/CA | Check `~/.config/openshell/gateways/<name>/mtls/` |
 | Image pull failure | Gateway or sandbox image cannot be pulled | Runtime events and image pull credentials |
 | `K8s namespace not ready` with `envoy-gateway-openshell.yaml: the server could not find the requested resource` | Optional Gateway API manifest was applied without Envoy Gateway CRDs, or k3s Helm controller startup exceeded the namespace wait | Apply `deploy/kube/manifests/envoy-gateway-openshell.yaml` manually only after Envoy Gateway is installed and `grpcRoute` is enabled |

.agents/skills/helm-dev-environment/SKILL.md

@@ -1,6 +1,6 @@
 ---
 name: helm-dev-environment
-description: Start up, tear down, and configure the local Kubernetes development environment for OpenShell. Uses k3d (Docker-backed k3s) + Skaffold + Helm. Covers cluster lifecycle, optional add-ons (Keycloak OIDC, Envoy Gateway), and port mappings. Trigger keywords - local k8s, local cluster, k3d, skaffold, helm dev, start cluster, stop cluster, tear down cluster, delete cluster, create cluster, helm:k3s, helm:skaffold, local dev environment, dev cluster, k8s dev, envoy gateway local, keycloak local.
+description: Start up, tear down, and configure the local Kubernetes development environment for OpenShell. Uses k3d (Docker-backed k3s) + Skaffold + Helm. Covers cluster lifecycle, optional add-ons (Keycloak OIDC, Envoy Gateway), HA testing, and port mappings. Trigger keywords - local k8s, local cluster, k3d, skaffold, helm dev, start cluster, stop cluster, tear down cluster, delete cluster, create cluster, helm:k3s, helm:skaffold, local dev environment, dev cluster, k8s dev, envoy gateway local, keycloak local, high availability, HA.
 ---
 
 # Helm Dev Environment
@@ -26,9 +26,10 @@ mise run helm:k3s:create
 ```
 
 Creates a k3d cluster and merges its kubeconfig into the worktree-local `kubeconfig` file.
-Also applies base manifests (`deploy/kube/manifests/agent-sandbox.yaml`) and preloads the
-default community sandbox image into k3d so the first sandbox create does not wait on a
-large registry pull. Traefik is disabled at cluster creation time.
+Also applies the upstream agent-sandbox CRDs/controller (pinned via `AGENT_SANDBOX_VERSION`
+in `tasks/scripts/helm-k3s-local.sh`, fetched from `github.com/kubernetes-sigs/agent-sandbox`
+releases) and preloads the default community sandbox image into k3d so the first sandbox
+create does not wait on a large registry pull. Traefik is disabled at cluster creation time.
 
 **Multi-worktree support:** the cluster name is derived from the last component of the
 current git branch (e.g. branch `kube-support/local-dev/tmutch` → cluster
@@ -65,6 +66,11 @@ generates mTLS secrets on first install. Envoy Gateway opt-in; see the Optional
 
 The gateway Service uses ClusterIP. Access is via Envoy Gateway (port `8080`) or `kubectl port-forward`.
 
+**HA test deploy** (two gateway replicas + external PostgreSQL Secret): uncomment
+`#- ci/values-high-availability.yaml` in `deploy/helm/openshell/skaffold.yaml`,
+create the Secret named `openshell-ha-pg` with a `uri` key, then run
+`mise run helm:skaffold:run` or `mise run helm:skaffold:dev`.
+
 ### TLS behaviour
 
 `ci/values-skaffold.yaml` sets `server.disableTls: true`, so Skaffold-based deploys run
@@ -172,6 +178,23 @@ To remove Keycloak:
 mise run keycloak:k8s:teardown
 ```
 
+### SPIRE / SPIFFE Provider Token Grants
+
+Skaffold can install SPIRE with the SPIFFE hardened Helm charts. To activate
+SPIFFE JWT-SVIDs for dynamic provider token grants:
+
+1. Uncomment the `spire-crds` and `spire` releases in `deploy/helm/openshell/skaffold.yaml`
+2. Uncomment `#- ci/values-spire.yaml` in the OpenShell release values files
+3. Redeploy: `mise run helm:skaffold:run`
+
+`ci/values-spire-stack.yaml` configures the local SPIRE trust domain as
+`openshell.local` and adds a `ClusterSPIFFEID` that maps sandbox pod
+annotations to `spiffe://openshell.local/openshell/sandbox/<sandbox-id>`.
+OpenShell mounts the SPIFFE CSI Workload API socket at
+`/spiffe-workload-api/spire-agent.sock` into sandbox pods for provider token
+grants. Supervisor-to-gateway authentication remains on the Kubernetes
+ServiceAccount bootstrap and gateway-minted sandbox JWT path.
+
 ---
 
 ## Cluster Lifecycle (suspend/resume)
@@ -198,7 +221,10 @@ mise run helm:k3s:status
 | `deploy/helm/openshell/ci/values-skaffold.yaml` | Dev overrides (image pull policy, TLS disabled for local Skaffold) |
 | `deploy/helm/openshell/ci/values-cert-manager.yaml` | cert-manager PKI overlay (opt-in; disables pkiInitJob) |
 | `deploy/helm/openshell/ci/values-gateway.yaml` | Envoy Gateway GRPCRoute + Gateway overlay |
+| `deploy/helm/openshell/ci/values-high-availability.yaml` | HA test overlay (`replicaCount: 2` with external PostgreSQL Secret) |
 | `deploy/helm/openshell/ci/values-keycloak.yaml` | Keycloak OIDC overlay |
+| `deploy/helm/openshell/ci/values-spire.yaml` | SPIFFE/SPIRE provider token grant overlay |
+| `deploy/helm/openshell/ci/values-spire-stack.yaml` | SPIRE hardened chart values for local dev |
 | `deploy/helm/openshell/ci/values-tls-disabled.yaml` | Lint-only: TLS + auth disabled (reverse-proxy edge termination) |
 | `deploy/kube/manifests/envoy-gateway-openshell.yaml` | GatewayClass for Envoy Gateway (`mise run helm:gateway:apply`) |
 | `tasks/scripts/helm-k3s-local.sh` | k3d cluster create/delete/start/stop/status |

.agents/skills/openshell-cli/SKILL.md

@@ -495,8 +495,9 @@ openshell gateway remove local                         # Remove local registrati
 ```bash
 # Inspect a Kubernetes Helm release and gateway pod
 helm -n openshell status openshell
-kubectl -n openshell get pods,svc
-kubectl -n openshell logs statefulset/openshell --tail=100
+kubectl -n openshell get deployment,statefulset,pods,svc
+kubectl -n openshell logs deployment/openshell -c openshell-gateway --tail=100
+kubectl -n openshell logs statefulset/openshell -c openshell-gateway --tail=100
 ```
 
 For Docker, Podman, and VM-backed gateways, inspect the gateway process or container logs and the selected runtime directly.