opensandbox-group · hittyt · Sep 3, 2026 · Aug 31, 2026 · Aug 31, 2026 · Aug 31, 2026
@@ -12,6 +12,7 @@ This page lists the OpenTelemetry metrics currently implemented in egress.
 |---|---|---|---|
 | `egress.dns.query.duration` | Histogram | `s` | Upstream DNS forward latency (recorded for allowed queries). |
 | `egress.dns.query.failed_total` | Counter | - | Queries the proxy could not resolve, by `reason`. |
+| `egress.dns.reply.failed_total` | Counter | - | Reply writes that failed after a decision, by `stage`. A nonzero count means a query was handled but its answer never reached the client. |
 | `egress.policy.denied_total` | Counter | - | Number of DNS queries denied by policy. |
 | `egress.nftables.rules.count` | Observable Gauge | `{element}` | Approximate policy size after last successful static apply (fleet profile: summed across every installed subject's policy, 0 while deny-first). |
 | `egress.nftables.updates.count` | Counter | - | Number of successful nftables updates (static apply + dynamic IP add). |
@@ -66,15 +67,21 @@ queried name nor the error text is ever attached:
 
 `egress.nftables.updates.failed_total` covers the other silent failure. Its `operation`
 attribute is one of `static_apply`, `dynamic_add`, `remove`, or — in the fleet profile
-(OSEP-0022) — `deny_first`, `dispatch_update`, `reset`; `dynamic_add` is the one to
+(OSEP-0022) — `deny_first`, `reset`; `dynamic_add` is the one to
 alert on, because a failed add means the kernel never learned about IPs the policy allows,
 so the chain drops traffic that should pass — which looks exactly like a policy denial from
 inside the sandbox while `egress.policy.denied_total` stays flat.
 
-The per-sandbox netns layer (fleet profile) counts its updates under the same operations;
-two expected cases are deliberately NOT counted as failures: a sandbox-layer removal whose
-netns is already destroyed (the rules died with it), and the startup recovery sweep of
-netns that never had a table installed.
+`egress.dns.reply.failed_total` covers the last silent failure class: a query that was
+**handled** (decided, maybe forwarded and answered upstream) whose reply write then failed.
+Until the write error was surfaced, such windows were indistinguishable from "query never
+handled" — the fleet-profile case where guest-originated DNS is answered in the proxy but
+the reply never reaches the sandbox (issue #1704). Its `stage` attribute is one of
+`malformed`, `unknown_source`, `deny`, `upstream_error`, `answer`, and every failure also
+emits a `[dns] reply write failed (stage=… remote=… question=…)` warning with the remote
+address and query name, so the counter pinpoints the condition and the log line the flow.
+Alert on any nonzero value: like `dynamic_add`, an `answer`-stage failure means traffic the
+policy allows is not reaching the client.
 
 A `static_apply` failure happens during startup, where the sidecar logs and exits. Metrics
 leave through a periodic reader and `os.Exit` skips the deferred shutdown, so that path

@@ -9,76 +9,102 @@ kernel rules, and credentials.
 The sidecar profile differs (single policy, `hook output`, iptables DNS
 REDIRECT on 15353); only the fleet model is drawn here.
 
-## 1. Control plane: policy and credential push
+## 1. Subject lifecycle: fastlet action protocol → deny-first → active
 
-The server pushes policy and vault revisions over fast-sandbox's own proxy
-route. The egress listener binds the Pod netns loopback only — fastlet-proxy
-is the only peer and injects `X-Fast-Sandbox-Uid` to route the push to a
-subject. There is no sandbox-reachable policy surface.
+The Fastlet is the sole lifecycle dispatcher (Sandbox Actions Handler
+protocol, `sandbox.fast.io/actions/v1`). The egress Handler implements
+`GET /_fastlet/v1/actions/status` (process incarnation probe) and
+`POST /_fastlet/v1/actions` (`SET_BINDING` / `LIFECYCLE_HOOK` /
+`REMOVE_BINDING`). The binding input **is the policy** (declarative, carried
+by the Sandbox CRD `actionBindings`); the attachment block carries the
+network identity (source IP, gateway, veth, private CIDR). There is no
+file-driven observation source.
+
+A subject is fail-closed from the moment `SET_BINDING` registers it until its
+`sandbox.data-plane-ready` Hook succeeds: registration installs deny-first
+rules immediately, the policy is held pending, and DNS keeps denying.
+
+```mermaid
+stateDiagram-v2
+    [*] --> absent
+    absent --> denying: SET_BINDING (input = policy, deny-first install)
+    denying --> denying: deny-first install failing (Fastlet retries)
+    denying --> active: LIFECYCLE_HOOK sandbox.data-plane-ready (policy applied)
+    active --> denying: SET_BINDING null input (binding removed, revert to deny-first)
+    active --> denying: rebind (new runtimeInstanceId/attachmentId) - policy discarded
+    active --> absent: REMOVE_BINDING (unload: chain+sets removed)
+    denying --> absent: REMOVE_BINDING
+```
+
+Recovery after egress restart: `ApplyReset` wipes the table, the Handler
+serves a new `instanceId`, the Fastlet detects it and replays the latest
+`SET_BINDING` followed by the already-reached Hooks (every live subject
+re-enters `denying` through the same registration path), and the server's
+reconciliation re-pushes credential revisions.
+
+## 2. Control plane: policy and credential push
+
+The egress listener binds the Pod netns loopback only. It serves the action
+endpoints (Fastlet) and the proxy-route policy/credential surfaces
+(fastlet-proxy is the only peer and injects `X-Fast-Sandbox-Uid` to route a
+push to a subject). There is no sandbox-reachable policy surface.
+
+Policy itself rides `SET_BINDING` (the CRD binding is the complete desired
+value; updates arrive as new bindings). The proxy route carries credential
+vault revisions (memory-only, OSEP-0012 — the binding input is NOT a secret
+transport) and runtime policy operations.
 
 ```mermaid
 sequenceDiagram
     autonumber
-    participant S as Server (OpenSandbox)
-    participant P as fastlet-proxy
+    participant F as Fastlet
     participant E as Egress listener<br/>(127.0.0.1:18080, loopback)
     participant R as Subject registry<br/>(memory)
     participant N as nftables<br/>(table opensandbox-fleet)
     participant D as DNS proxy<br/>(gateway:53, shared)
+    participant S as Server (OpenSandbox)
+    participant P as fastlet-proxy
 
-    Note over S: ResolveEndpoint(egress) -> route + route credential
-    S->>P: PUT /v1/sandboxfleets/{sid}/egress/policy
+    F->>E: SET_BINDING (sandbox, revision, attachment, policy input)
+    E->>R: RegisterAndEnforce (denying, deny-first)
+    E->>N: deny-first install (empty sets + drop chain + dispatch rule)
+    Note over E: policy stored PENDING - DNS still denies
+    F->>E: LIFECYCLE_HOOK sandbox.runtime-ready (confirm)
+    F->>E: LIFECYCLE_HOOK sandbox.data-plane-ready
+    E->>R: ApplyPolicy (denying -> active, effective = user + always rules)
+    E->>N: atomic swap (subject chain + static sets, single nft -f)
+    E->>D: per-query selector now returns this subject's policy
+    Note over S,P: create-then-configure (server side)
+    S->>P: PUT /v1/sandboxfleets/{sid}/egress/credential-vault
     P->>E: forward (credential verified, X-Fast-Sandbox-Uid added)
-    alt subject registered (slot observed)
-        E->>R: ApplyPolicy (denying -> active, effective = user + always rules)
-        E->>N: atomic swap (subject chain + static sets, single nft -f)
-        E->>D: per-query selector now returns this subject's policy
-    else slot not observed yet (create-then-configure race)
+    alt subject registered
+        E->>E: apply vault revision (memory-only per subject)
+    else binding not observed yet (race)
         E->>E: cache as pending (TTL, X-Fast-Sandbox-Generation check)
-        E-->>P: 202 Accepted (push will be applied on registration)
+        E-->>P: 202 Accepted (push applied on registration)
     end
 ```
 
-## 2. Subject lifecycle: deny-first → active
-
-A subject is fail-closed from the moment its slot is observed until its own
-policy lands. Registration installs deny-first rules (empty sets, drop chain,
-resolv.conf → gateway) before the policy push can arrive.
-
-```mermaid
-stateDiagram-v2
-    [*] --> absent
-    absent --> denying: slot Bound observed (poll of slot store)
-    denying --> denying: deny-first install failing (retry with backoff)
-    denying --> active: policy push lands (atomic nft swap)
-    active --> denying: rebind (new generation) - policy discarded, nft reset
-    active --> absent: slot file deleted (unload: element+chain+sets removed)
-    denying --> absent: slot file deleted
-```
-
-Recovery after egress restart: `ApplyReset` wipes the table, the watch
-re-delivers every bound slot (all subjects re-enter `denying`), and the
-server's reconciliation re-pushes policies.
-
 ## 3. Data plane: outbound traffic flow
 
-Two enforcement layers per sandbox: the authoritative Pod netns `forward`
-hook (below), plus a per-sandbox netns OUTPUT chain mirroring the same policy
-as defense in depth (`nsenter` from the host, table `opensandbox-fleet-ns`).
-The sandbox layer allows loopback, DNS to the slot gateway only (dport 53,
-gateway-scoped — the Pod layer enforces DNS policy via the proxy), and the
-mirrored deny/dyn/allow verdicts; it catches traffic the forward hook never
-sees (sandbox → host-local destinations take the INPUT path). DNS-learned
-leases are refreshed in lockstep between both layers by the per-subject
-connection refresh loop (Pod netns conntrack, bucketed by source IP, one
-batched transaction per tick). Only TCP sessions are renewed — UDP/QUIC
-(HTTP/3) relies on DNS lease TTLs; a sandbox-layer mirror miss marks the IPs
-pending and redelivers them on the next tick.
-
-Dispatch is a verdict map keyed by
-`ip saddr . iifname` (the host veth binding is defense in depth against UDP
-spoofing); the master chain defaults to **drop** so unregistered sources are
-denied before their slot is even observed.
+The authoritative enforcement layer is the Pod netns `forward` hook
+(`table opensandbox-fleet`, master chain policy **accept** with an
+unmarked-drop tail — the forward path never issues an explicit accept,
+because on the fast-sandbox Firecracker bridge topology
+(`bridge-nf-call-iptables=1`) an accept verdict returns the frame to the
+bridge L2 path and drops it before postrouting). Allowed destinations are
+marked in per-subject `hook prerouting` chains (`meta mark set 0x2` for
+allow/dyn set members, unconditional for default-allow policies); per-subject
+dispatch by `ip saddr` (the source IP is the only dispatch key — an iifname
+match would never fire on the bridge topology, where the IP hooks see
+skb->dev = the bridge) leads to subject chains whose deny sets drop explicitly,
+and the unmarked-drop tail denies everything else (unregistered sources,
+deny-first subjects). Intercepted MITM traffic is delivered locally (DNAT)
+and enforced by the dedicated INPUT chain on the conntrack original
+destination. DNS-learned leases are kept alive by the per-subject connection
+refresh loop (Pod netns conntrack, bucketed by source IP, one batched
+transaction per tick); only TCP sessions are renewed — UDP/QUIC (HTTP/3)
+relies on DNS lease TTLs.
 
 ```mermaid
 flowchart LR
@@ -87,17 +113,17 @@ flowchart LR
         APP --> TCP[TCP/UDP egress]
     end
 
-    DNSQ -->|resolv.conf rewritten to gateway| GW[gateway:53]
+    DNSQ -->|addressed to gateway:53| GW[gateway:53 - REDIRECT to :15353]
     GW --> DP[DNS proxy - per-query policy by source IP]
     DP -->|subject unknown / denied| NX[NXDOMAIN]
     DP -->|allowed| UP[Upstream resolver]
     UP -->|answer| DNSQ
     UP -->|resolved IPs with TTL| DYN[subject dynamic allow set - timeout lease]
 
-    TCP -->|via host veth| DISPATCH[dispatch chain - hook forward, policy DROP]
+    TCP -->|via host veth| DISPATCH[dispatch chain - hook forward, ACCEPT + unmarked-drop tail]
     DISPATCH -->|ct state established,related| ACC1[accept]
     DISPATCH -->|tcp/udp dport 853| DROP1[drop - DoT blocked]
-    DISPATCH -->|vmap: ip saddr . iifname| JUMP[jump subj_&lt;id&gt; chain]
+    DISPATCH -->|ip saddr| JUMP[jump subj_&lt;id&gt; chain]
 
     JUMP -->|deny_v4/v6 sets| DROP2[drop]
     JUMP -->|dyn_v4/v6 + allow_v4/v6 sets| ACC2[accept]
@@ -110,16 +136,12 @@ flowchart LR
 
 ## 4. Credential vault
 
-> Status: the **control plane** below (revision push + per-subject in-memory
-> store) is implemented. The **data plane** (shared mitmdump, REDIRECT with
-> preserved source IP, subject-aware active API, addon wiring) is **design
-> only — not yet implemented**. The diagram shows the target shape.
-
 Vault revisions are pushed over the proxy route and held **memory-only** per
 subject (OSEP-0012 model — no Secret volume, nothing written to egress disk).
 The shared mitmdump instance selects the subject's vault by the client's
-source IP (transparent REDIRECT preserves it); a rebind swaps the revision in
-memory and new flows pick up the new credentials.
+source IP (transparent REDIRECT/DNAT preserves it); a revision push rebinds
+in memory and new flows pick up the new credentials. See
+[fleet-mitm-data-plane](../../../docs/components/egress-fleet-mitm-data-plane.md).
 
 ```mermaid
 sequenceDiagram
@@ -134,45 +156,34 @@ sequenceDiagram
     S->>P: PUT /v1/sandboxfleets/{sid}/egress/credential-vault (full revision)
     P->>E: forward (UID header -> subject)
     E->>V: replace revision (memory-only, new flows rebind)
-    C->>M: HTTP(S) flow (REDIRECT preserves source IP)
+    C->>M: HTTP(S) flow (DNAT preserves source IP)
     M->>M: script: client source IP -> subject -> subject's vault
     M->>V: resolve credential/binding for the flow
     V-->>M: credential (active snapshot)
     M-->>C: proxied flow with credential applied
 ```
 
-Missing pieces for the data plane (all subject-aware extensions of the
-sidecar's single-vault mechanism — `mitmscripts/system.py` + the `/_active`
-unix socket):
-
-- shared mitmdump startup in the fleet assembly, and the per-sandbox netns
-  OUTPUT REDIRECT pairs installed from the host (`nsenter`), see
-  [fleet-mitm-data-plane](../../../docs/components/egress-fleet-mitm-data-plane.md)
-- subject dispatch inside the shared active socket: `/_active?clientIp=…`
-  → `registry.Resolve` → subject vault snapshot (one socket, no per-subject
-  sockets, no UID in the socket protocol)
-- addon wiring: `flow.client.peername[0]` -> active socket -> vault
-
 ## 5. Fail-closed invariants
 
 | Transition / event | Guarantee |
 |---|---|
-| Slot observed, no policy yet | deny-first: empty nft sets + drop chain, resolv.conf → gateway, DNS NXDOMAIN |
-| Policy push before slot | cached pending (TTL); applied on registration; generation mismatch discards it |
-| Policy push lands | one atomic `nft -f` transaction (chain + static sets); DNS selector switches per subject |
-| Rebind (new generation) | policy discarded in registry AND nft chain/sets/DNS leases force-reset |
-| Unload (slot deleted) | dispatch element + chain + all sets removed in one transaction |
-| Egress restart | stale rules wiped (ApplyReset), all subjects re-enter denying, server re-pushes |
-| Unregistered source | master chain policy drop — denied before the slot is ever observed |
-| Unparseable slot record | fail closed (event error logged, subject never activated) |
+| SET_BINDING, no data-plane-ready yet | deny-first: empty nft sets + drop chain, gateway DNS REDIRECT, DNS NXDOMAIN |
+| Vault push before SET_BINDING | cached pending (TTL); applied on registration; generation mismatch discards it |
+| data-plane-ready lands | one atomic `nft -f` transaction (chain + static sets); DNS selector switches per subject |
+| Rebind (new runtimeInstanceId/attachmentId) | policy discarded in registry AND nft chain/sets/DNS leases force-reset |
+| Unload (REMOVE_BINDING) | chain + all sets removed in one transaction; stale fence ignored |
+| Egress restart | stale rules wiped (ApplyReset); new instanceId triggers Fastlet replay of SET_BINDING + reached Hooks |
+| Unregistered source | unmarked -> master-chain tail drop — denied before the binding is ever observed |
+| Malformed action envelope | rejected (never silently ignored); the subject is never activated |
+| data-plane-ready without pending policy | failed (protocol violation) — the subject stays denying |
 
 ## Component map
 
 | Concern | Implementation |
 |---|---|
-| Subject state machine | `pkg/subject` (`MemoryRegistry`, `Controller`) |
-| Slot store (identity) | `pkg/slotsource` (`FileParser`, polling `FileSource`) |
-| Per-subject nft rules | `pkg/fleetnft` (verdict-map dispatch, atomic swap, reset) |
+| Actions wire model + validation | `pkg/actionhandler` (envelope, operations, Hooks) |
+| Subject state machine | `pkg/subject` (`MemoryRegistry`, lifecycle hooks) |
+| Per-subject nft rules | `pkg/fleetnft` (dispatch rules, atomic swap, reset) |
+| Actions endpoints + lifecycle mapping | `fleet_actions.go` (SET_BINDING / LIFECYCLE_HOOK / REMOVE_BINDING) |
 | Policy/vault HTTP surface | `fleet_server.go` (UID routing, pending cache, per-subject vault) |
 | DNS per-query dispatch | `pkg/dnsproxy` `SetQueryPolicySelector` |
-| resolv.conf rewrite | `pkg/resolvrewrite` |