feat(egress): fastlet Sandbox Actions handler protocol for fleet profile (OSEP-0022) - #1678
Conversation
jianpingpei
left a comment
There was a problem hiding this comment.
[P1] The fleet DNS interception does not match the resolver that FastSandbox currently installs. FastSandbox LinuxNetNSDriver copies the Fastlet resolver file into each slot (current test/example value is 10.96.0.10), while SetupGatewayDNSRedirect only redirects port 53 when the destination is attachment.network.gateway. Therefore real sandbox DNS traffic does not reach this proxy: default-deny can make allowed domains unresolvable, while default-allow can bypass domain deny and DNS-based IP learning. The fleet smoke queries 10.10.0.1 directly and the Firecracker E2E writes the gateway into resolv.conf, so they do not exercise the real integration. Please either make FastSandbox provide the gateway as the slot resolver when this handler is enabled, or intercept sandbox-veth DNS independently of the resolver destination, and cover it with an actual FastSandbox integration test.
…ndbox-group#1678) Addresses the four P1s and the P2 from the review: - /policy pushes for a StateDenying subject are stored as the pending policy instead of applying immediately: the lifecycle barrier (data-plane-ready) is the only activation signal. Cached pushes flushed at SET_BINDING now land AFTER the binding input's pending store, so the newer pushed policy wins and still waits for the Hook. - data-plane-ready peeks the pending policy instead of consuming it: a transient nft failure keeps it in place, so the Fastlet's retry of the same Hook succeeds instead of 409-ing forever. - LIFECYCLE_HOOK is fenced against the registered identity: a delayed Hook from a previous instance can no longer consume the replacement sandbox's pending policy or activate it before its own data plane is ready. - REMOVE_BINDING keeps the subject registered until terminal enforcement cleanup succeeds: a retried removal resumes cleanup instead of returning success with stale kernel rules. OnUnloaded removes nft first so a failure leaves every teardown step undone (no double gateway release). - gateway DNS redirect ownership is keyed per subject (not refcounted per gateway): duplicate SET_BINDING delivery is a no-op, a gateway move on rebind releases the old gateway, and one unload fully releases it. New tests: push-while-denying barrier, data-plane-ready retry after nft failure, stale-Hook fence rejection, REMOVE_BINDING retry after nft failure, duplicate-registration gateway idempotency, and gateway-move rebind.
…ile (OSEP-0022) The fleet profile's subject lifecycle switches from the fast-sandbox slot-store file observation (/run/fast-sandbox/network/*.json) to the public Sandbox Actions Handler protocol (sandbox.fast.io/actions/v1): the Fastlet delivers SET_BINDING / LIFECYCLE_HOOK / REMOVE_BINDING over Pod-loopback HTTP and the egress process is the Handler. - pkg/actionhandler: action envelope wire model, parsing and validation (apiVersion/operation/Hook checks, null-input vs ordinary string inputs) - fleet_actions.go: /_fastlet/v1/actions/status (instanceId incarnation probe) and /_fastlet/v1/actions; SET_BINDING registers deny-first and stores the policy pending, sandbox.data-plane-ready activates it, null input reverts to deny-first, REMOVE_BINDING cleans up (stale fence ignored) - subject: Fencing from revision.runtimeInstanceId/attachmentId, lifecycle hooks take the attachment network block, registry gains UnsetPolicy/Fence - fleetnft: attachment-based dispatch, ApplyDispatchUpdate removed - deleted: pkg/slotsource, pkg/subject/controller (file-driven lifecycle), pkg/sandboxnft (netns OUTPUT layer), pkg/resolvrewrite - proxy route /policy + /credential-vault unchanged (credentials stay memory-only, OSEP-0012; pending-push cache fenced on specGeneration) - OSEP-0022 updated: API-based integration, credential channel on the proxy route; docs updated docs(oseps): OSEP-0022 status draft -> implementing
…ions protocol The fleet profile no longer reads the slot store, so both test scripts register subjects through SET_BINDING / LIFECYCLE_HOOK / REMOVE_BINDING instead of writing slot files: - smoke-fleet.sh: set_binding/lifecycle_hook/remove_binding helpers posting action envelopes to /_fastlet/v1/actions; deny-first asserted at SET_BINDING, activation at data-plane-ready; rebind via a new identity fence; restart recovery via instanceId replay; dropped the resolv.conf and per-sandbox netns OUTPUT assertions (layers removed); MITM phase re-registers through actions - e2e-fleet-firecracker.sh: bind_subject/activate_subject replace write_slot; policy rides the binding input; removed the slot-store/resolv volumes and env from both containers - firecracker e2e doc: contract updated to the actions protocol fix(egress): encode binding input as a JSON string in the smoke/e2e actions envelopes fix(egress): inline curl in wait_for checks (shell functions are not visible in bash -c)
…ndbox-group#1678) Addresses the four P1s and the P2 from the review: - /policy pushes for a StateDenying subject are stored as the pending policy instead of applying immediately: the lifecycle barrier (data-plane-ready) is the only activation signal. Cached pushes flushed at SET_BINDING now land AFTER the binding input's pending store, so the newer pushed policy wins and still waits for the Hook. - data-plane-ready peeks the pending policy instead of consuming it: a transient nft failure keeps it in place, so the Fastlet's retry of the same Hook succeeds instead of 409-ing forever. - LIFECYCLE_HOOK is fenced against the registered identity: a delayed Hook from a previous instance can no longer consume the replacement sandbox's pending policy or activate it before its own data plane is ready. - REMOVE_BINDING keeps the subject registered until terminal enforcement cleanup succeeds: a retried removal resumes cleanup instead of returning success with stale kernel rules. OnUnloaded removes nft first so a failure leaves every teardown step undone (no double gateway release). - gateway DNS redirect ownership is keyed per subject (not refcounted per gateway): duplicate SET_BINDING delivery is a no-op, a gateway move on rebind releases the old gateway, and one unload fully releases it. New tests: push-while-denying barrier, data-plane-ready retry after nft failure, stale-Hook fence rejection, REMOVE_BINDING retry after nft failure, duplicate-registration gateway idempotency, and gateway-move rebind.
19c78d9 to
460b1cb
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 460b1cb324
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
460b1cb to
f5f3ed4
Compare
…ndbox-group#1699) With net.bridge.bridge-nf-call-iptables=1 (the fast-sandbox Firecracker bridge topology), an explicit accept verdict from the forward hook returns the frame to the bridge L2 path: a frame whose destination is the bridge itself is then treated as local delivery and dropped before postrouting, so IP-direct egress for allowed subjects never reaches SNAT. Only 'not hitting a drop rule' lets the frame continue IP routing. The forward path no longer accepts explicitly: - new per-subject 'hook prerouting' mark chains (shared mark hook chain + per-subject jump) set meta mark 0x2 for allow/dyn set members (default-deny) or unconditionally (default-allow) - the master dispatch chain flips to policy accept with an unmarked-drop tail (meta mark & 0x2 != 0x2 drop): unregistered sources and deny-first subjects carry no mark and are denied by the tail (fail-closed preserved) - per-subject forward chains keep only the deny-set drops (no accept verdicts); the INPUT (MITM) chain keeps its full verdicts unchanged - deny-first resets flush the mark chain; policy swaps rebuild it (a default-action change flips the mark strategy) - mark 0x2 is distinct from the DNS proxy's SO_MARK 0x1 bypass Unit tests: mark-based shapes (default-deny set marks, default-allow unconditional mark, deny-first resets, no forward accepts), v6 mark jump.
f5f3ed4 to
85ea85b
Compare
…word
The chain name 'mark' is rejected by nftables ('syntax error, unexpected
mark, expecting string or last') because 'mark' is a reserved keyword (meta
mark). Rename the shared prerouting hook chain to 'marking'; the per-subject
mark_<id> chains are unaffected. Verified the full generated script
(reset + deny-first + policy swap, DoH + MITM) against nftables 1.0.x.
…opology (opensandbox-group#1699) On the fast-sandbox Firecracker bridge topology (net.bridge.bridge-nf-call-iptables=1), frames entering the bridge destined to the bridge itself are pulled into the IP stack with skb->dev = the BRIDGE (not the pod-side veth). Every dispatch rule matched 'ip saddr <ip> iifname <hostVeth>', so NOTHING matched: no prerouting mark, no forward dispatch — all IP-direct traffic fell to the fail-closed tail and was dropped. DNS stayed fine (REDIRECT -> INPUT never touches forward) and the gateway was reachable (INPUT), exactly matching the reported symptoms. Verified on a real bridge topology in a container: with the veth's iifname the rules never fire (no mark -> tail drop); with the bridge name they do. Removing iifname entirely makes the mark-based fix work end to end. - fleetnft writeDispatchRule: all jumps/drops match the source IP only (forward dispatch, prerouting mark jump, MITM input dispatch, gateway drop). Source-IP unforgeability rests on IPAM + no NET_ADMIN/NET_RAW and Pod netns rp_filter strict mode. - iptables gateway MITM: per-subject DNAT/return rules drop the iifname (MitmRedirectEntry.HostVeth removed). - smoke-fleet.sh: shape assertions updated; the forged-IP test now asserts the observable consequence (not dispatched to the forging subject) instead of the removed iifname rejection. - docs/OSEP: dispatch-key notes updated.
The SET_BINDING input is the authoritative desired value: data-plane-ready must apply exactly what the binding carried. A runtime /policy push for a still-denying subject previously overwrote the pending policy (storePendingPolicy in resolvePolicyPush), so a create-with-allow binding could be silently replaced by a deny push and data-plane-ready would activate the wrong policy — DNS stayed NXDOMAIN while the nft allow set was correct only after a later update. resolvePolicyPush now accepts (202) pushes for denying subjects without touching the pending policy; pushes take effect only when the subject is active (in-place apply) or via the registration flush of an already-active subject. Tests updated: the flush no longer overrides the binding input.
…p#1704) serveDNS swallowed every WriteMsg error, so a query that was handled and decided (even forwarded and answered upstream) could have its reply die between the proxy socket and the client with zero signal — the silent no-answer windows reported for guest-originated DNS. Route all five reply sites through writeReply: failures now increment egress.dns.reply.failed_total {stage} (bounded: malformed/unknown_source/deny/upstream_error/answer) and log [dns] reply write failed with remote address and query name.
Background
The fleet profile's subject lifecycle switches from the fast-sandbox slot-store file observation (
/run/fast-sandbox/network/*.json) to the public Sandbox Actions Handler protocol (sandbox.fast.io/actions/v1): the Fastlet deliversSET_BINDING/LIFECYCLE_HOOK/REMOVE_BINDINGover Pod-loopback HTTP and the egress process is the Handler. The file-driven approach is removed entirely.Changes
New
pkg/actionhandler: action envelope wire model, parsing and validation (apiVersion/operation/Hook allowlists, null-input vs ordinary string inputs, required fencing fields)fleet_actions.go:GET /_fastlet/v1/actions/status(per-processinstanceId, triggers Fastlet replay after restart) +POST /_fastlet/v1/actionsOperation mapping
SET_BINDING: registers the subject deny-first (attachment provides IP/gateway/veth/CIDR), stores the policy pending; updates on an already-active subject apply in place; null input (binding removed) reverts to deny-firstLIFECYCLE_HOOK:sandbox.runtime-readyconfirms;sandbox.data-plane-readyapplies the policy -> active; no pending policy fails closed (409)REMOVE_BINDING: terminal cleanup; stale removals (fence mismatch) are ignored; missing state is successRemoved (-3133 lines)
pkg/slotsource(file observation),pkg/subject/controller.go(file-driven lifecycle),pkg/sandboxnft(netns OUTPUT defense-in-depth layer; the envelope has no netnsPath),pkg/resolvrewriteOPENSANDBOX_EGRESS_SLOT_STORE_DIR/_SLOT_POLL_INTERVALenv vars,fleetnft.ApplyDispatchUpdatePreserved semantics
SET_BINDINGuntildata-plane-ready/credential-vault(memory-only, OSEP-0012; binding input is persisted in the Sandbox CRD and is not a secret transport), pending cache fenced on specGenerationDocs
Notes
go.mod/vendor inconsistency is pre-existing in the repo (not introduced by this PR); verified locally with-mod=modactionHandlersdeclaring egress (targetHTTPPort 18080, hooks: runtime-ready + data-plane-ready); the four internal additions for the credential channel are detailed in OSEP-0022