diff --git a/oseps/0020-subpath-directory-initialization.md b/oseps/0020-subpath-directory-initialization.md new file mode 100644 index 000000000..b47191978 --- /dev/null +++ b/oseps/0020-subpath-directory-initialization.md @@ -0,0 +1,403 @@ +--- +title: Explicit SubPath Directory Initialization for PVC Volumes +authors: + - "@gegemeimingzi" +creation-date: 2026-08-19 +last-updated: 2026-08-19 +status: provisional +--- + +# OSEP-0020: Explicit SubPath Directory Initialization for PVC Volumes + + +- [Summary](#summary) +- [Motivation](#motivation) + - [Goals](#goals) + - [Non-Goals](#non-goals) +- [Requirements](#requirements) +- [Proposal](#proposal) + - [Notes/Constraints/Caveats](#notesconstraintscaveats) + - [Risks and Mitigations](#risks-and-mitigations) +- [Design Details](#design-details) + - [API Surface](#api-surface) + - [Validation Rules](#validation-rules) + - [Kubernetes Provider Behavior](#kubernetes-provider-behavior) + - [Error Semantics](#error-semantics) + - [Idempotency and Metadata](#idempotency-and-metadata) +- [Test Plan](#test-plan) +- [Drawbacks](#drawbacks) +- [Alternatives](#alternatives) +- [Infrastructure Needed](#infrastructure-needed) +- [Upgrade & Migration Strategy](#upgrade--migration-strategy) + + +## Summary + +OpenSandbox's `pvc` backend maps `Volume.subPath` to a Kubernetes +`volumeMounts.subPath`. When the referenced directory does not already exist +inside the PVC, Kubernetes does not create it for `subPath` mounts (unlike +`subPathExpr` with some CSI drivers, and unlike `hostPath` with +`DirectoryOrCreate`), so workloads fail to start or fail to write. + +This OSEP proposes an explicit, Kubernetes-only opt-in on the `pvc` backend — +`pvc.createSubPathIfMissing` — that guarantees the directory referenced by a +non-empty `Volume.subPath` exists before the main sandbox container starts, +implemented as an init container that runs before the sandbox container. The +opt-in defaults to `false`, preserves exact existing behavior when omitted, and +fails the create request (never silently downgrades) when the guarantee cannot +be established. + +## Motivation + +Callers today must pre-create a required subPath directory out of band, for +example by launching a temporary provisioner sandbox that runs +`mkdir -p /data/runs/42` and then exits. This is: + +1. **Fragile**: a second resource (the provisioner sandbox) must be created, + waited on, and cleaned up by the caller on every create path. +2. **Non-atomic**: the directory creation is not coupled to the sandbox create + request, so a retried or concurrent create can observe a half-prepared + state. +3. **Silently divergent across providers**: OSEP-0003 states that a missing + `subPath` is created (L495), but its Docker named-volume mapping requires + the resolved path to exist (L203) and its test plan expects rejection when + the resolved `subPath` does not exist (L514). The Kubernetes provider maps + `subPath` to `volumeMounts.subPath` without any existence guarantee, while + the Docker provider defers the outcome to container creation. No provider + implements the documented creation guarantee today. + +An explicit opt-in gives callers a single-request, fail-closed way to obtain +the documented guarantee, scoped to the Kubernetes provider where the +semantics are well-defined. + +### Goals + +- Provide a per-volume opt-in (`pvc.createSubPathIfMissing`) that, when + `true`, guarantees a non-empty `Volume.subPath` directory exists on a + Kubernetes PVC before the main sandbox container starts. +- The guarantee is all-or-nothing: the create request either succeeds with the + directory present, or fails with a stable, documented error. +- The opt-in defaults to `false`; omission preserves exact existing behavior. +- Initialization is idempotent: re-running against an existing directory is a + no-op that leaves metadata unchanged. +- Document the Kubernetes-only scope explicitly; Docker and other providers + reject the opt-in with a clear error rather than silently ignoring it. + +### Non-Goals + +- **Not** implementing directory initialization on the Docker provider or any + provider other than Kubernetes. +- **Not** changing `pvc.createIfNotExists` semantics: it remains backing-storage + creation only, never subPath directory creation. +- **Not** combining backing-storage creation and directory initialization in a + single request (see [Notes/Constraints/Caveats](#notesconstraintscaveats)). +- **Not** proposing repair for Pending, unbound, inaccessible, or otherwise + unusable PVCs. +- **Not** supporting Windows workloads, block-mode volumes, or read-only final + mounts. +- **Not** providing automatic cleanup of initialized directories when a sandbox + is deleted. + +## Requirements + +- The opt-in must be a new field on the `pvc` backend object in the `Volume` + schema (`specs/sandbox-lifecycle.yml`), additive and backward-compatible. +- `createSubPathIfMissing: true` with an empty `subPath` must be rejected by + validation. +- The opt-in applies the existing relative-path and no-`..` rules already + enforced by `ensure_valid_sub_path` in + `server/opensandbox_server/services/validators.py`. +- The Kubernetes provider must reject the opt-in when it cannot establish the + guarantee (e.g. unsupported runtime, unusable claim) **before** persistent + mutation or main-container start. +- Initialization must complete before the main sandbox container starts; if it + fails, create returns an error. +- `createIfNotExists: true` combined with `createSubPathIfMissing: true` in + the same request is rejected in the initial scope. +- The final mount must be read-write for the initial scope (a read-only final + mount combined with the opt-in is rejected). +- Newly created directories must have well-defined creator identity and initial + UID, GID, mode, umask, and ACL behavior, documented in the release notes and + resolved from the init container's `securityContext` and the PVC mount + options. +- Initialization retries and stable errors: transient init-container failures + are retried by the Kubernetes pod lifecycle (restartPolicy), and terminal + failures surface as a stable, documented sandbox error carrying the init + container's termination message. +- `volumes` remain incompatible with `extensions.poolRef` under the existing + Lifecycle schema; the opt-in does not change that constraint. + +## Proposal + +Add a boolean field `createSubPathIfMissing` (default `false`) to the `pvc` +backend object in the `Volume` schema. When `true` and the backend is +Kubernetes, the server provisions an init container in the sandbox pod that +creates the directory referenced by `Volume.subPath` (recursively, with +`mkdir -p`) before the main sandbox container starts. + +The init container runs with the same PVC mounted at a scratch mount point and +executes `mkdir -p ` inside that mount. Because the init container +shares the pod's PVC, the directory lands on the same storage that the final +mount targets, and the final mount's `subPath` resolves to the now-existing +directory. + +Illustrative request: + +```yaml +volumes: + - name: workspace + pvc: + claimName: workspace-data + createIfNotExists: false + createSubPathIfMissing: true + mountPath: /workspace + subPath: runs/42 + readOnly: false +``` + +### Notes/Constraints/Caveats + +- **Kubernetes-only**: The opt-in is defined and implemented only for the + Kubernetes provider. The Docker provider and any other backend must reject a + request with `createSubPathIfMissing: true` with a stable, documented error + (`UNSUPPORTED_SUBPATH_INIT`) rather than silently downgrading to ordinary + `subPath` mounting. +- **No silent downgrade**: An unsupported provider or server version must + reject the request before initialization or main-container start. +- **Backing-storage independence**: Backing-storage creation + (`createIfNotExists`) and directory initialization are explicitly + independent. The initial scope rejects combining them in one request to + avoid cross-resource sequencing and rollback across PVC creation, binding, + mountability, and directory initialization. A later design may allow both + only if it defines their ordering and failure semantics. +- **Multiple volumes**: When several `Volume` entries reference the same PVC + claim with different `subPath` values, each entry with the opt-in gets its + own initialization; the init container mounts the claim once and creates + each requested subPath. +- **Unknown-field handling / capability negotiation**: The runtime request + models (`server/opensandbox_server/api/schema.py`) do **not** set + `extra="forbid"` on their Pydantic `Config` (only `populate_by_name = True`), + so an unknown field on an older server is silently ignored — the OpenAPI + `additionalProperties: false` declaration is **not** runtime enforcement. + A client setting `createSubPathIfMissing: true` against a server that does + not support it would therefore silently lose the guarantee. The design must + add an explicit client-visible capability/version negotiation instead of + relying on schema strictness. Options to be resolved at implementation + time, in order of preference: + 1. **Server capability surface**: extend the server's version/diagnostics + response (e.g. the stable diagnostics API from #1553) with a feature + flag such as `capabilities.subpath_init = true`, which the SDK checks + before sending the opt-in and rejects client-side otherwise. + 2. **Runtime rejection with a stable error**: set `extra="forbid"` on the + `PVC` model (and affected request models) so servers that ship this + change reject the unknown field with a `4xx` instead of ignoring it. + This does not help clients talking to servers deployed before the + change, so (1) remains the primary mechanism for mixed versions. + 3. **SDK guard**: typed SDKs refuse to serialize `createSubPathIfMissing` + unless the connected server advertises support. + Server and SDK release notes must call out the minimum server version that + supports the field and the capability-flag contract. + +### Risks and Mitigations + +| Risk | Mitigation | +|------|------------| +| Path traversal via `subPath` (`..`, absolute paths) | Reuse the existing `ensure_valid_sub_path` validator; no new traversal surface. | +| Option-like or shell-special `subPath` names (e.g. `-m`, `` `rm -rf /` ``) interpreted by the init command | Invoke `mkdir` as an argv array with `--` option terminator, never through a shell; subPath is passed as a single argument, so shell metacharacters have no effect. Add option-like and shell-special names to the test plan. | +| Symlink escapes inside the PVC (a parent component of `subPath` is a symlink pointing outside the claim) | Documented as caller responsibility in the initial scope; the init container runs `mkdir -p` which follows symlinks. A stricter design (e.g. `O_NOFOLLOW` walking) is deferred; the guarantee is "directory exists at the resolved path", matching `volumeMounts.subPath` resolution semantics. | +| Concurrent creates racing to create the same directory | `mkdir -p` is idempotent and safe under concurrency; no locks required. | +| Init container image availability / trust boundary | Reuse the existing execd or a minimal busybox-style image already available in the sandbox image set; the image is server-configured and operator-controlled. | +| Init container needs write permission on the PVC | The init container runs as root or with the PVC's owning UID/GID; exact `securityContext` is a server-side configuration detail resolved at implementation time, consistent with how the sandbox container is configured. | +| Partial creation leaves persistent directories on failure | Documented: initialization is idempotent and `mkdir -p` leaves a valid directory on partial success; no automatic cleanup is provided (explicitly out of scope). | + +## Design Details + +### API Surface + +In `specs/sandbox-lifecycle.yml`, under the `PVC` object properties, add: + +```yaml +createSubPathIfMissing: + type: boolean + default: false + description: | + When true (Kubernetes provider only), ensures that the directory + referenced by the volume's `subPath` exists on the backing claim before + the main sandbox container starts. The directory is created recursively + by an init container if missing. Requires a non-empty `subPath` and + `createIfNotExists: false`. Rejected on providers other than Kubernetes. +``` + +Corresponding additions to the server API model in +`server/opensandbox_server/api/schema.py` (`PVC` class) and the Kubernetes +volume-helper path in `server/opensandbox_server/services/k8s/volume_helper.py` +are implementation details defined at implementation time, with tests per the +Test Plan below. + +### Validation Rules + +- `createSubPathIfMissing: true` with empty/absent `subPath` → `400` + `INVALID_SUB_PATH_INIT`: "createSubPathIfMissing requires a non-empty + subPath". +- `createSubPathIfMissing: true` with `createIfNotExists: true` → `400` + `INVALID_SUB_PATH_INIT`: "createSubPathIfMissing cannot be combined with + createIfNotExists in the initial scope". +- `createSubPathIfMissing: true` with `readOnly: true` final mount → `400` + `INVALID_SUB_PATH_INIT`: "createSubPathIfMissing requires a read-write final + mount". +- `createSubPathIfMissing: true` on a non-Kubernetes runtime (e.g. Docker) → + `400` `UNSUPPORTED_SUBPATH_INIT`: "createSubPathIfMissing is only supported + on the Kubernetes provider". + +### Kubernetes Provider Behavior + +When all validation passes, the server adds an init container to the sandbox +pod before the sandbox container: + +1. Mount the PVC at a scratch mount path (e.g. `/mnt/subpath-init`). +2. Run `mkdir -p -- ""` as an **argv array** (no shell), with `--` + as the option terminator so option-like names (e.g. `subPath: -m`) are + treated as paths, not `mkdir` options. The subPath must never be + interpolated into a shell command string. +3. Exit 0 on success; exit non-zero on failure, which fails the pod and + surfaces as a sandbox create/run failure with the init container's log + attached to the error. + +The final `volumeMounts.subPath` is unchanged; the init container only +guarantees the target directory exists. + +### Error Semantics + +- Validation errors are returned synchronously by the create request + (`4xx`), before any cluster mutation. +- Init-container failures surface as pod-level failures: the sandbox does not + reach Ready, and the server reports the init container's termination + message with a stable error code. +- **Init-container failure detection is a required implementation change**: + the existing readiness wait path + (`KubernetesSandboxService._wait_for_sandbox_ready`) recognizes only + Running/Allocated and unschedulable states; it neither detects a failed + init container nor fetches its logs, so without a change it would return + the generic `K8S_POD_READY_TIMEOUT` instead of the promised stable error. + The implementation must extend the wait path to inspect + `pod.status.initContainerStatuses` for `terminated.reason != Completed` + (or `CrashLoopBackOff`) and surface `SUBPATH_INIT_FAILED` with the init + container's log tail attached. +- Transient init-container failures (e.g. pod eviction, node pressure) are + retried by the Kubernetes pod lifecycle under the sandbox pod's + `restartPolicy`; the create request stays in a retrying state until success + or a terminal failure. +- Terminal failures (non-zero exit after retries, unusable claim, mount + failure) produce a stable, documented error code (e.g. + `SUBPATH_INIT_FAILED`) with the init container's logs attached, so callers + can distinguish "directory not initialized" from other sandbox failures. + +### Idempotency and Metadata + +- If the directory already exists, `mkdir -p` succeeds and leaves all metadata + (ownership, mode, umask-derived attributes, ACLs) unchanged. +- For newly created directories, ownership/UID/GID/mode follow the init + container's `securityContext` and the PVC's mount options, applied under the + init container's umask; exact defaults (including any default ACL inherited + from the PVC mount or filesystem) are resolved at implementation time and + documented in the release notes. +- Successful existence does not guarantee that the workload can write the + directory (e.g. permissions may still block writes); this is consistent with + ordinary `subPath` mounts. + +## Test Plan + +Unit tests (`server/tests/`): + +- Validation: empty `subPath` + opt-in → rejected; `createIfNotExists: true` + + opt-in → rejected; read-only final mount + opt-in → rejected; valid + request → accepted. +- Validator keeps rejecting absolute paths and `..` components even with the + opt-in set. +- **Option-like and shell-special `subPath` names**: `subPath: "-m"`, + `subPath: "--help"`, and `subPath: "a;rm -rf /"` produce a single argv + argument passed to `mkdir -p --` (never a shell string); the init container + spec uses an argv array and no shell. +- Kubernetes volume-helper: request with opt-in produces an init container + with `mkdir -p` and the correct scratch mount; request without opt-in + produces no init container (byte-for-byte parity with today's output). +- Docker provider: opt-in → `UNSUPPORTED_SUBPATH_INIT` rejection. +- **Capability negotiation**: with the chosen mechanism (capability flag or + strict model), a client against a server without the feature either gets a + `4xx` rejection or a client-side error before the request is sent; the + "silently discarded field" path is covered by a regression test. + +Integration tests: + +- Kind-based e2e (kubernetes `test/e2e/`): create a PVC-backed sandbox with + `createSubPathIfMissing: true` and a non-existent nested `subPath`; assert + the sandbox reaches Ready and the directory exists; assert a second create + against the same path is idempotent. +- Negative e2e: `createSubPathIfMissing: true` on a read-only mount → create + rejected before pod creation. +- **Init-container failure detection e2e**: make `mkdir` fail (e.g. mount the + PVC read-only via a test-only override, or point `subPath` into a + read-only mount), assert the sandbox transitions to + `SUBPATH_INIT_FAILED` with the init container's log tail attached — not the + generic `K8S_POD_READY_TIMEOUT`. +- Option-like path e2e: create a sandbox with `subPath: "-m"` and assert the + directory named `-m` is created (not parsed as a `mkdir` option). + +Manual verification: + +- `helm install` the server, run the e2e scenarios above against a Kind + cluster, and capture pod spec showing the init container. + +## Drawbacks + +- Adds a new API field and Kubernetes-only behavior, increasing the surface + area of the `pvc` backend. +- Init containers add a small startup latency to sandbox creation when the + opt-in is used (one extra container start). +- The guarantee is weaker than it appears: "directory exists" does not imply + "workload can write" (permissions), and symlink resolution inside the PVC is + caller responsibility. + +## Alternatives + +1. **Document that callers must pre-create directories** (status quo). Rejected: + leaves the OSEP-0003 contradiction unresolved and imposes fragile + out-of-band provisioning on every affected caller. +2. **Use `subPathExpr`** with a fixed expression. Rejected: `subPathExpr` does + not create directories either; it only expands variables. +3. **Init container per provider, general mechanism**: implement the same + opt-in for Docker via named-volume path initialization. Deferred: Docker + named volumes have different mount semantics and the OSEP-0003 contract for + them expects rejection on missing paths; a separate OSEP is needed. +4. **Server-side directory creation via a Kubernetes Job**: heavier, requires + Job lifecycle management and race handling; init container is simpler and + naturally tied to the pod lifecycle. + +## Infrastructure Needed + +- No new repositories, services, or third-party dependencies. +- The init container image is drawn from the existing sandbox image set + (server-configured), so no new image build pipeline is required. +- Kind-based e2e already exists in `kubernetes/test/e2e/`; no new cluster + infrastructure. + +## Upgrade & Migration Strategy + +- **Backward compatibility**: existing valid requests that omit + `createSubPathIfMissing` retain exact current semantics. The new field + defaults to `false`. +- **Server version gating / capability negotiation**: servers that implement + the new field accept the opt-in; older servers must be detected by a + client-visible capability mechanism (see + [Notes/Constraints/Caveats](#notesconstraintscaveats)), because the runtime + request models do not set `extra="forbid"` and an unknown field is silently + ignored rather than rejected. The guarantee must never be silently lost; + either the capability flag is absent (client refuses to send the opt-in) or + the server rejects the request with a `4xx`. +- **Docs and release notes**: update the volume documentation in `docs/`, + SDK reference, and release notes to state the Kubernetes-only scope, the + error codes, the capability-flag contract, and the minimum server version + required. +- **SDKs**: add the field to SDK request builders and regenerate clients from + the updated spec, so typed SDKs expose `createSubPathIfMissing` and refuse + to serialize it against servers that do not advertise support. diff --git a/oseps/README.md b/oseps/README.md index 6c6646ea2..74c8ab655 100644 --- a/oseps/README.md +++ b/oseps/README.md @@ -25,3 +25,4 @@ This is the complete list of OpenSandbox Enhancement Proposals: | [OSEP-0017](0017-resilient-sdk-transport.md) | Resilient SDK Transport | implementing | 2026-07-22 | | [OSEP-0018](0018-execd-as-sandbox-init.md) | execd as Sandbox Init | draft | 2026-07-27 | | [OSEP-0019](0019-node-agent-sandbox-collection.md) | Node Agent for Node-Level Sandbox Collection | implementing | 2026-08-11 | +| [OSEP-0020](0020-subpath-directory-initialization.md) | Explicit SubPath Directory Initialization for PVC Volumes | provisional | 2026-08-19 |