kube-init: first-boot deploy and image-import fixes - #3
Closed
eriknordmark wants to merge 30 commits into
Closed
Conversation
Direct deps kube-init's typed client-go and containerd usage
relies on:
k8s.io/api v0.34.1
k8s.io/apimachinery v0.34.1
k8s.io/apiextensions-apiserver v0.34.1
k8s.io/client-go v12.0.0+incompatible (aliased to v0.34.1
via replace)
github.com/containerd/containerd/v2 v2.2.5
github.com/containerd/errdefs v1.0.0
k8s.io/kube-openapi's own go.mod aliases v0.31.0 as its module
version, but the repo never publishes that as a tag — the pseudo
below is what actually exists on the repo and is what our replace
directive points at.
Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
…eboot The primitives kube-init's FSM writes to persistent storage: - Marker: typed filesystem-path values (LonghornInitialized, NativeKubernetesMode, MultusInitialized, …). IsMarked / Mark / Unmark route through AtomicWriteFile so partial writes never leave a half-marker. - AtomicWriteFile: temp-file + rename atop the destination, with an fsync in between so the on-disk state matches the return value even across a crash. - SaveVarLib / RestoreVarLib: /var/lib snapshot beneath the vault. Supports the cluster→single-node conversion rollback: if the cluster mode transition can't complete, the /var/lib snapshot from the pre-transition boot is restored. - Reboot: the daemon's controlled reboot path. Writes a marker naming the reason before invoking the reboot binary so the next boot can consult the reason without racing the boot sequence. - K3sKubeconfig / ContainerdSocket / SealedDirName: canonical paths every subsystem shares. Constants (not vars) so a subsystem cannot accidentally rewrite them at test time. - ToK8sName: kubernetes-node-name transform (lowercased device UUID). Load-bearing for cross-subsystem consistency — the FSM's node-uuid label, monitor's reapply, tiebreaker's node-lookup, and Longhorn's node-object all agree on this spelling. Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
…ntainerd wait Everything that must be ready before k3s can start. The FSM sits in INSTALLING → STARTING_CTRD while this package runs; a failure here aborts the boot rather than starting k3s on a broken substrate. - Kernel modules loaded via modprobe (bridge, iptables_nat, iscsi_tcp, etc.). Absent kernel bits surface with the specific module name so operators can spot a kernel-config regression. - cpu_manager_state cleanup: kubelet sometimes leaves a stale cpu_manager_state under /var/lib/kubelet after a reboot with a changed --cpu-manager-policy. Deleting it on every boot before containerd starts is upstream-blessed cleanup. - Cgroup mount: /sys/fs/cgroup, kubepods.slice hierarchy. - iSCSI daemon start (needed for Longhorn's block-mode volumes). - Vault wait: pillar's vaultmgr signals "ready" via a fs marker; the daemon blocks here until the marker is present so k3s does not race the vault-mount. - EdgeNodeInfo wait: blocks until the pubsub subscription delivers the device identity (DeviceName / DeviceID). - Containerd start + reachability probe. Everything is idempotent — the FSM re-enters INSTALLING on recycle without expecting a clean state. Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
…password, install Everything about running the k3s process: - binary.go / install.go: k3s binary download with SHA-256 verify + symlink rebuild for the multi-call subcommands (kubectl / ctr / crictl → k3s). Idempotent — a re-install picks up unchanged bytes cheaply. - config.go: drop-in files under /etc/rancher/k3s/config.yaml.d with numeric prefixes deciding merge order (00-nodename, 01-clusterconfig, 03-enc-disable-local-path, 99-user-override). Slot 02 is reserved. Byte-stable rendering keeps k3s's drop-in-hash restart trigger quiet on no-op reconciles. - paths.go: absolute paths for the k3s binary and every state file the daemon writes. - supervisor.go: Setpgid + orphan-sweep supervisor for the k3s process. Ports its own PID stays alive until the child's process group is fully reaped; port-listening probe rules out an outer-shell-still-running scenario before signalling "started". - token.go: k3s node-token rotation on cluster-mode change. - node_password.go: FixNodePasswordSecret clears a stale node-password secret when the file marker StaleNodePasswdFlag is present. Delete via typed clientset (kube-system Secrets). - readiness.go: WaitKubeconfig (poll for /etc/rancher/k3s/k3s.yaml with ctx cancel), then WaitReady blocks until Node's Ready condition + kube-system pods are Ready + node-uuid label is applied. Uses a client-go client built from the kubeconfig file — dial errors during the boot window are tolerated as "keep polling", so no shell-out is needed for the chicken- and-egg case of probing an API that isn't up yet. Every step is bounded by the caller's ctx (INSTALLING / STARTING_K3S / WAIT_K3S_READY handlers set the deadline). Restart hooks live under /etc/k3s-supervisor/hooks.d and fire between stop and start on operator-driven restart (RT PerformanceProfile changes are the canonical consumer). Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
k3s version override + component-by-component upgrade driven by the KubeConfig pubsub subscription. Runs as a background loop during RUNNING; a version bump on the wire triggers download + SHA-verify + install of the new k3s binary followed by rolling component upgrades. - update.go: main loop. Pulls the desired k3s version from the KubeConfig subscription cache, downloads via curl into a /tmp path with a per-attempt timeout, SHA-256-verifies against the version-specific expected hash, atomically installs into /var/lib/k3s/bin/k3s. - cluster.go: checkClusterReady probes the API server via discovery.ServerVersion; componentIsInstalled probes each component's namespace via a typed Namespaces().Get, distinguishing "not installed" (NotFound → false, nil) from "cannot tell" (err → false, err) so a transient API outage isn't misread as "component gone". - k3s_update.go: k3s download / verify / install flow. - status.go: publishes update progress to zedagent via pubsub so the controller sees each component's version. Retry gate: the daemon consults the KubeClusterUpdateStatus subscription to skip retrying a version the controller has already reported as failed — matches the "one attempt per version" semantic pillar expects. Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
…ault() Holds the typed clientset, apiextensions clientset, dynamic client, discovery + deferred RESTMapper, and shared informer factories (typed + dynamic). Constructed once at daemon boot from state.K3sKubeconfig and exposed via a package-level singleton (SetDefault / Default) so subsystems don't have to thread a *Client parameter through every function. Default() panics if consulted before SetDefault — that path indicates a lifecycle bug and should crash loud rather than nil-deref deeper. ResetMapper() invalidates the discovery cache so kubectlx.applyOne picks up newly-installed CRDs on retry. WaitForKubeconfig polls for state.K3sKubeconfig with ctx cancellation, exposed for callers that need to gate on file existence explicitly. Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
…ce layer Every operation the daemon performs against Kubernetes or containerd goes through this package. No argv assembly, no stderr parsing, no subprocess exec. apply.go — Apply / ApplyFile / ApplyURL / Get + DeleteFile / DeleteURL / DeleteBytes. Server-side apply via the dynamic client, multi-document YAML parsing, mapper-reset + retry-once on meta.NoKindMatchError closes the CR-before-CRD race in one place. Retry via client-go/util/retry.OnError with an apierrors-based classifier (transient vs permanent). Force- conflict-override is on by default (ApplyOptions.NoForce inverts the flag) because kube-init owns the components it applies; context.Canceled short-circuits so peer-cancel doesn't drag 30s+ of backoff sleeps. wait.go — WaitCRDEstablished / WaitDeploymentReady / WaitDaemonSetReady / WaitJobComplete / WaitForCondition, all informer-driven via watchtools.UntilWithSync. WaitForCondition evaluates a JSONPath against the target object on every event — CR-defined phase fields (KubeVirt / CDI status.phase, etc.) fit this shape. containerd.go — ContainerdClient wraps containerd.Client (v2) with namespaces.WithNamespace(k8s.io) baked into every operation. ImportImage / ImageExists / DeleteImage / ListImages / TagImage. A health probe at construction turns "containerd is down" into one clear error instead of per-operation failure spam. No cri-api dependency — CRI's image operations are containerd image-service operations under the k8s.io namespace, which the Go client already exposes. Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
…ffort retry loop A component starts the moment its declared deps are Ready — no per-wave synchronization barrier. A fast component whose deps are satisfied never waits for a slow unrelated peer. Design lives at .claude/design/kube-init-client-go-and-dag.md §4.5. types.go — Component (Manifests + Apply + Ready + PolicyDeps + BestEffort + ReadyTimeout), Manifest (File / URL / Bytes), Edge (structural-derivation output tagged with rule: crd / namespace / webhook / sa / policy), RetryPolicy, RetryCallback. deploy.go — Graph + Run + Edges. plan() validates the graph and resolves edges. The scheduler runs a single-threaded coordinator loop that tracks launched/completed counters directly (a WaitGroup-based helper is a runtime panic when a completing goroutine's result enqueues children mid-drain). retry.go — Graph.RetryCtx is the daemon-scope context that keeps BestEffort background retries alive after Run's per-invocation ctx ends. Exponential backoff with jitter (defaults: 30s → 5min × 6 = ~30min budget). Re-arms Apply → Ready on Ready- timeout too, not just Apply-failure. Retries observe RetryCtx cancel (daemon shutdown). rules.go — placeholder for the four structural-dep rules (CRD→CR, Namespace→resource, webhook-Service→webhooked-resource, ServiceAccount→Pod-spec) that fire once callers populate Component.Manifests with real bytes. Today Manifests is empty across the shipped graph; ordering rides entirely on PolicyDeps. deploy_test.go — 25 tests covering plan validation, edge tagging, edge sorting, dependency-order execution, parallel- execution property (two-channel handshake, not sleep), error propagation with alphabetical head + rest, ctx cancellation, BestEffort semantics, Ready-only-on-Apply-success, MaxParallel concurrency cap, Apply=nil (Ready-only) shape, and four retry-loop tests (eventually-succeeds, exhausts, ctx-cancel-mid- loop, one-shot-without-RetryCtx). All pass -race across 50 iterations. Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
…nager Single-goroutine dispatch of pillar pubsub topics into per-topic subscriber caches. Modelled on pkg/pillar/cmd/monitor/subscriptions.go — one map of label → Subscription, deferred Activate, one goroutine pumping every topic via pubsub.MultiChannelWatch. Subscribers across kube-init packages (edgenodeinfo, encconfig, encstatus, kubeconfig, kcus) register their topics with the Manager during daemon boot; the Manager owns the connection and the pump goroutine. Registering after the pump has started is supported — new subscriptions activate lazily. Replaces file-polling under /run/zedagent/... / /persist/status/ zedkube/... — every subscriber owns its own decoded cache with the canonical pillar types (no local JSON re-declaration), so JSON-tag skew between pillar and kube-init cannot silently drop a field's value. Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
Six typed subscribers wired to the shared pubsubclient.Manager; one publisher for the daemon's own lifecycle status. - edgenodeinfo: EdgeNodeInfo subscription with WaitForFirst so the boot path can block on device identity (DeviceName, DeviceID). Exposes Get() for read-only consumers on the FSM loop. - encconfig: EdgeNodeClusterConfig subscription. Drives cluster shape (single vs HA), TieBreakerNodeID, cluster interface. - encstatus: EdgeNodeClusterStatus subscription. Present() folds both file deletion and the zero-UUID-delete sentinel into a single check so a cluster delete on the controller side is detected the same regardless of how zedkube signals it. - kubeconfig: KubeConfig subscription with K3sVersion accessor for the update loop. Tagged consistently with pillar's canonical json tags (capital K, no override). - kcus: KubeClusterUpdateStatus subscription gating upgrade retries. DestinationKubeUpdateVersion is uint32 (matching pillar), so a "cluster already tried and failed at version X" record correctly gates the next attempt. - kubeinitstatus: publishes the daemon's lifecycle events (transition-step, phase, etc.) so operators observing the controller see kube-init state transitions without needing to tail the kube container log. Every subscription decodes to pillar's canonical types (via pkg/pillar/types) — no local *JSON structs that can drift. Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
First-boot import of pre-packaged image tarballs (KubeVirt, CDI, Longhorn, Multus, SUC, the EVE-authored external-boot-image) into the user-side containerd's k8s.io namespace, so kubelet pods reference images without a first-boot registry pull. - Import via kubectlx.ContainerdClient (containerd v2 Go client under the k8s.io namespace). - ImportUpstreamImage is a no-op when the image is already present or the tarball is absent; kubelet falls back to a registry pull for anything missing. - ImportExternalBootImage re-tags the EVE-authored image (whose tarball name isn't the runtime tag) to the running EVE release's tag, then cleans up prior EVE-authored images for the same base name via ListImages + DeleteImage. - Manifest.json is parsed via `tar` (fatal-line-buffered) with the RepoTags extraction in Go — no jq dependency. Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
…ainerd env Wires kube-init into pillar's cost-aware management proxy so in-cluster traffic (CDI's image importer, containerd's image pulls) routes through pillar rather than out the WAN. - cni0.go: SetupCNI0ProxyIP assigns a well-known anchor IP on the flannel-created cni0 bridge — CDI pods reach the management proxy at that IP without a per-node hostname lookup. Enabled() only fires when pillar's mgmtproxy config file is present. - The CDI CR's spec.config.importProxy fields (HTTPSProxy / noProxy) are reconciled every steady-state tick: a dynamic Get reads the current values, and a merge-patch fixes any drift. In-Go JSONPath extraction reads the field values — no `-o jsonpath=…` shell-out. - containerd-launch env injection is a separate concern (pkg/kube/kube-init/prereqs) and pre-dates this package. Wildcard-webhook rules (rules[].apiGroups: ["*"]) are treated as "no derived edge from this webhook" in the deploy planner — a catch-all webhook shouldn't sequence the whole graph. Not the concern of this package but noted here so the wiring is consistent. Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
… startup stagger Every state change between "this node is a single-node cluster" and "this node is a member of an HA cluster" — including the recovery paths — lives here. - transition.go: the ordered step runner. Single→cluster rotates the k3s node-token, applies encstatus-derived configuration, and updates supervisor args. Cluster→single reverts. Progress callback surfaces the current step name to the FSM's control socket so `k3s-sctl status` reports `transition-step=<step>`. - masterleases.go: stale etcd master-lease cleanup. After a single→HA conversion, etcd carries a lease from the pre-conversion single-node identity; the deploy graph's Longhorn readiness stalls if that lease's expiry sits ahead of the new node's registration. Cleanup deletes the stale lease from etcd directly (etcdctl invocation — outside the client-go surface because etcd doesn't ship a stable Go SDK). - startup_rank.go: on HA startup, computes the local node's rank (0-based position in the sorted control-plane node list) and sleeps rank × 25s before launching k3s. Fans out simultaneous power-cycles into a staggered sequence so etcd quorum re-forms cleanly without three nodes fighting for the leader lease at t=0. Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
The steady-state loop the FSM enters after DEPLOYING. Every 15s health tick runs a fixed sequence of housekeeping + reconcile tasks; separate goroutines watch containerd exit, kubeconfig drift, log rotation, cluster-config-changed events, and the user-override drop-in for k3s config. Tasks the health tick runs (in order): - DHCP CNI daemon liveness (respawn on exit). - CNI plugin binary drift (copy from stage dir when needed). - Multus re-render + apply if the initialised marker is absent. - Node label + annotation reapply (node-uuid, longhorn disk config) when the labels-initialised marker is absent — one merge-patch does labels and annotations together, then a readback verifies both landed. - Debug-user RBAC re-application. - Kubeconfig sync (copy /etc/rancher/k3s/k3s.yaml to the kube-init-visible location, atomic). - Image reimport (external-boot-image via kubectlx. ContainerdClient — ImageExists check, tarball import only if the image is genuinely absent). - Longhorn post-install config (via components. LonghornPostInstallConfig). - Registration manifest reconcile. transition.go — countReadyNodes counts Nodes with Ready=True via typed clientset. Used by the cluster-mode monitor to detect "we're on the HA majority" transitions. parseTransitionMarker covers the FSM's cluster-transition marker file (unix-ts + reboot-count). Steady-state ticks are gated on state markers so a tick after a successful bootstrap is silent — no log line, no API call. Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
The tie-breaker is the third control-plane node in an EVE-K HA cluster whose only job is to give etcd a third vote so single- node loss doesn't strand the cluster. This package configures that role. ConfigApply is the phase entry point; it runs when the cluster first reaches three nodes AND the local node is the designated tie-breaker (per encconfig's TieBreakerNodeID). Steps: - nodesConfigApply — label every node with tie-breaker-node= true/false and cordon/uncordon accordingly. Merge patches via Nodes().Patch keep both label and spec.unschedulable updates atomic per node. - kubevirtConfig — scale virt-operator + KubeVirt CR to 2 replicas (tie-breaker excluded). Deployment scale via typed AppsV1 merge-patch; CR replica field via dynamic Resource Patch. - kubevirtTieBreakerConfigApply — patch every DaemonSet in the kubevirt namespace with a nodeSelector keeping pods off the tie-breaker. - cdiConfig — same nodeSelector patch on every Deployment in the cdi namespace. - longhornNodeSetSched — flip allowScheduling + evictionRequested on the tie-breaker's longhorn.io Node CR and every disk it owns. JSON-patch operations via dynamic Resource Patch. - longhornRescale — scale Longhorn CSI sidecar Deployments to 2 replicas; patch every longhorn-system DaemonSet with the nodeSelector (retries 5× because longhorn-manager occasionally races us during install). - drainNode — cordon + evict every non-DaemonSet pod on the tie-breaker via policy/v1.Eviction (matches kubectl drain --ignore-daemonsets --delete-emptydir-data --force semantics) + wait until pods leave the node. - StatusSet — stamp every node with tie-breaker-config-applied=1 as the "phase complete" marker so a subsequent reboot skips the whole flow. Every operation is idempotent — a partial ConfigApply that gets interrupted mid-way completes on the next FSM entry. Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
…eduler, RBAC, SR-IOV, kube-vip
Everything the deploy graph runs during DEPLOYING and everything
the monitor's steady-state ticks reconcile against.
components.go — DeployAll builds a deploy.Graph and runs it. The
graph has ~10 Components; only Longhorn declares a PolicyDep
(longhorn depends on manifests, which stages storage-classes.yaml
into the auto-deploy dir). Every Install* function:
- Longhorn: applyLonghornDiskConfig labels + annotates the node
via one merge-patch; the longhorn config YAML applies as a
drop-in.
- KubeVirt: InstallKubeVirt applies the operator manifest via
SSA, waits for virt-operator Ready, applies the CR via SSA,
applies feature-gate CR via SSA. kubeVirtConfigReplicas
writes .spec.replicas + .spec.infra.replicas via SSA with a
distinct field manager ("kube-init-replicas") so subsequent
applies of the operator manifest don't fight for ownership.
- CDI: apply operator, wait for cdi-operator Ready, apply CR.
WaitCDIReady goes through kubectlx.WaitForCondition on
.status.phase == "Deployed".
- Multus: template rendering (IP address substitution) then
applyMultus + linkMultusIntoK3s.
- descheduler / debug-rbac / manifests / kube-vip / SR-IOV
manifest staging / stale-mount-cleanup daemon launcher —
one Component each.
- WaitLonghornReady is informer-driven (via a wait-scoped
dynamic factory over longhorn.io/Node and typed DaemonSets in
longhorn-system). No polling; the compound predicate is
re-evaluated on every event.
kubevirt.go — MigrateKubeVirtFeatureGates reads the running CR
via dynamic Get, patches with dynamic Patch, marks the migration
done. buildFeatureGatesPatch is table-driven so a new gate is a
one-line addition.
longhorn.go — LonghornIsReady is the compound predicate the
monitor consults on every tick. Uses typed clientsets for
DaemonSet readiness and dynamic client for the Longhorn Node CR
and engine images. longhornEngineDeployedOnNode distinguishes
"status.nodeDeploymentMap absent" (fresh CR — not-ready-yet)
from "map present but this node is false" (recycle engine +
manager pods to force reconcile). CheckOverwriteNsmounter uses
client-go remotecommand.NewSPDYExecutor (via podExec helper) to
copy a fixed nsmounter binary into every longhorn-csi-plugin
pod that doesn't already carry the marker file.
registration.go — RegistrationApplyIfReady stages the
controller-supplied registration manifest into the k3s
server-manifests dir. RegistrationApplied checks the AddOn CR
via dynamic Get.
uninstall.go — the K3sBase conversion flow. Uninstalls
descheduler / Longhorn / CDI / KubeVirt / Multus in reverse
dependency order. Longhorn uninstall is bounded by a 1000-poll
budget (~83 min) covering the worst-case data-shred workload;
timeout aborts the conversion rather than declaring
NativeKubernetesMode with volumes half-shredded.
sriov.go / stale_mount_cleanup.go — hardware-conditional
manifest staging and background cleanup daemon launcher.
Every Install has an idempotence gate on a state.Marker; a
re-entry to DEPLOYING or a re-tick of the monitor is a cheap
noop when the marker is already set.
Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
Launches a virtctl vnc proxy on demand for a KubeVirt VMI and tears it down when the caller process exits. Caller-PID watchdog polls /proc/<pid> so an orphaned proxy from a disconnected debug session doesn't linger. Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
14 states, 27 events. The daemon boots into INIT, walks through INSTALLING → STARTING_CTRD → CONFIGURING → STARTING_K3S → IMPORTING (first-boot only) → WAIT_K3S_READY → DEPLOYING (first- boot only) → RUNNING, and handles restart / recycle / cluster- mode-transition events without dropping back into an ad-hoc respawn loop. State entries are `enterStateFn` dispatched from `transition`; async work runs on a per-attempt workCtx via startAsync so transition cancellation drains the work goroutine cleanly. The control socket at /run/k3s-supervisor.sock accepts `status`, `restart`, `stop`, `graph` commands; every operator-facing k3s- sctl invocation dispatches through the socket rather than touching flag files. initKubeclient wires kubeclient.Default at WAIT_K3S_READY's transition (test-safe when the kubeconfig file is absent — transition-table tests exploit this). components.DeployAll takes a retryCtx that main.go supplies as d.rootCtx, so BestEffort background retries in the deploy graph survive per-invocation workCtx while still dying on daemon shutdown. Restart-reason is a typed field carried through the STOPPING_K3S → RUNNING_HOOKS → STARTING_K3S transition so restart hooks (/etc/k3s-supervisor/hooks.d) fire once for operator-driven restarts and never for crash recovery. The RT operator's PerformanceProfile-change flow depends on this — CPU isolation / RDT partitioning need to be reconfigured between stop and start, not on every crash. README.md sketches the architecture, package boundaries, and the future-work list (bootstrap-cert fingerprint pinning, config-package retirement, and so on). Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
One-shot tool that dials /run/k3s-supervisor.sock and prints the
reply. Reads until EOF so multi-line responses (graph) print
naturally alongside single-line ones (status / restart / stop).
K3S_SUPERVISOR_SOCKET overrides the path for tests.
Exit code follows the last line: non-zero when it starts with
ERR.
Subcommands:
- restart — graceful k3s restart (pre-restart hooks fire).
- status — one-line status report (state, k3s pid, uptime,
backoff, restart count, phase, transition-step).
- stop — stop the kube-init daemon.
- graph — resolved deploy-graph edges (one per line, each
tagged with its rule source).
Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
…helper pkg/kube/Dockerfile — the kube linuxkit container's entrypoint is /usr/bin/kube-init. Everything else the container needs to ship (k3s multi-call binary, longhorn nsmounter, CNI plugin staging, longhorn/kubevirt/CDI/multus tarball drops under /images/) is installed via the existing build.yml conventions. pkg/kube/k3s-control.sh — small wrapper that dispatches to k3s-sctl. Kept so external tooling (eden / eve exec kube …) that used to invoke this path continues to work; every subcommand it takes now goes over the control socket rather than touching flag files. pkg/kube/kubevip-delete.sh — minimal helper that KubeVIPDelete calls to remove leftover kube-vip state files after component deletion (leftover static-pod manifests, arp cache). Not part of the deploy graph; runs from the uninstall path. Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
Eleven files under pkg/kube/ implemented the previous cluster-init flow in POSIX shell + kubectl. Every operation they exposed lives in pkg/kube/kube-init/ now — no external caller depends on them, and keeping them alongside the Go daemon would leave two divergent implementations of the same paths that operators could accidentally invoke. Deleted: cluster-init.sh cluster-update.sh cluster-utils.sh descheduler-utils.sh kubevirt-utils.sh longhorn-utils.sh pubsub.sh registration-utils.sh tie-breaker-utils.sh utils.sh vnc-proxy.sh lib/config.sh Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
…stDevices
The KubeVirt CRD requires spec.configuration.permittedHostDevices.
{pciHostDevices,mediatedDevices,usb} to be arrays. Bare keys
parse as `null` in YAML; server-side apply forwards `null` as an
explicit set-to-null, which the schema rejects with:
KubeVirt.kubevirt.io "kubevirt" is invalid:
spec.configuration.permittedHostDevices.mediatedDevices:
Invalid value: "null": ... must be of type array
Explicit `[]` satisfies the schema.
Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
The kube linuxkit package now ships a Go daemon in place of the shell path. Adds a kube-init build rule that DOCKER_GO-invokes `go build ./cmd/... .` inside pkg/kube/kube-init/, staged into pkg/kube/dist/ before the linuxkit pkg build. pillar-* targets are unchanged. Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
kube-init's vendor tree is upstream Go modules with well-known licenses; the SPDX generator's per-file scan blows out on the tree size without adding value. Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
EVE confines itself to a cpuset derived from eve_max_vcpus, which is 1 by
default, so kube-init and the whole k3s control plane — API server, etcd,
scheduler, kubelet — share a single core. Pods are unaffected: kubelet
places them under /sys/fs/cgroup/cpuset/kubepods, which keeps the full
host mask.
A first boot is the one time that matters and the one time nothing is
competing, so the confinement is lifted for the deploy and restored on
reaching RUNNING.
Widening an ancestor only raises its ceiling: 010-eve-cgroup writes
eve_max_vcpus into every eve/services/<srv> leaf individually, so each
leaf must be widened too. Freeing only eve/services/kube would leave
every sibling pinned, and pillar matters as much as kube here because it
downloads and hash-verifies the images a first boot needs. The leaves are
enumerated from the filesystem rather than hard-coded so they cannot
drift from EVESERVICES.
Order follows the cgroup v1 subset rule: ancestors first when widening
(and a failure there aborts, since a leaf cannot exceed its parent),
child-first when restoring. A single unreadable leaf is skipped rather
than abandoning its siblings. Only cpuset.cpus is touched; cpuset.mems is
left alone so NUMA locality is unchanged.
Note ctrd_max_vcpus is a dead knob: 010-eve-cgroup parses and defaults
ctrd_cgroup_cpus_limit but never writes it to any cpuset, so containerd is
unconfined regardless. The user containerd k3s talks to is still covered
here, because kube-init starts it and it inherits eve/services/kube.
The Apply concurrency cap reads the live cpuset rather than
runtime.NumCPU, which samples the mask at process start and so cannot
observe the widening.
Verified on an 8-CPU device: widening retargets the already-running k3s
and kube-init processes immediately (0 -> 0-7), and restore returns them
to 0.
Not enabled by default in the parent series. On the device measured, the
first-boot cost is dominated by Longhorn's sequential rollout (manager ->
driver-deployer -> csi-plugin -> engine-image) rather than by control
plane CPU. It is kept for workloads that do load the control plane — many
more CRDs, or the EROFS storage path — where the single core is the
constraint.
The benefit is not speculative: evetest raises these limits for exactly
this reason. tests/cluster/cluster_test.go and
tests/networking/{bootstrap,net_adapter}_test.go set dom0_max_vcpus=4,
eve_max_vcpus=3 with the comment "minimizing device onboarding time and
accelerating cluster formation". Those tests take the permanent trade
because application performance does not matter to them; this change is
the temporary form, which is what production needs.
Two known gaps to close before enabling:
- A crash between DEPLOYING and RUNNING leaves the mask wide until the
next reboot, when dom0-ztools' 010-eve-cgroup rewrites it from the
kernel cmdline. Restore is not wired to the BACKOFF or error paths.
- kube-init rewrites cpusets that dom0-ztools owns, and now for every
EVE service rather than just its own. That is deliberate — pillar's
image work is part of what a first boot waits on — but it means
kube-init briefly sets policy for services it does not manage, so the
ownership split is worth settling before this is enabled.
Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
|
Important Review skippedDraft detected. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
A first EVE-k boot whose Longhorn image pull outran the ten-minute readiness budget lost that pull entirely: the wait expired on elapsed time, the graph declared every component initialized, and the snapshot step sent SIGTERM to k3s and everything under it while a layer was still extracting. The retry meant to rescue the component then redid the interrupted work, costing minutes on the boots that were already slowest. A readiness budget now measures time without progress rather than time elapsed, so a component that is slow because it is moving hundreds of megabytes is left alone while a genuinely stuck one is still given up on and retried. Progress comes from containerd's in-flight ingest byte count, which advances during a pull and needs no heuristics; a separate ceiling bounds a component that reports progress forever without converging. Components with no progress source keep the wall clock. The guard owns the deadline outright: a component's own Ready wait now takes its bound from the caller's context, so an inner timeout can no longer expire while the guard still considers the component healthy. The snapshot additionally waits, bounded, for background retries to finish before stopping k3s. Reaching that point no longer implies convergence, only that waiting stopped, so the step that assumed nothing was in flight now checks. Signed-off-by: eriknordmark <erik@zededa.com> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A node reported cluster storage ready while it still could not attach a volume. Longhorn runs a volume's engine and replica processes inside the per-node instance-manager pod, and that pod is owned by an InstanceManager custom resource rather than a DaemonSet, so the daemonset sweep never observed it. Readiness was declared while the pod was still pulling a ~440 MB image, and the first app's volume attach then failed repeatedly with "not ready for workloads" until it happened to come up. Require a running InstanceManager for this node, and feed its state into the progress token so the guard does not read the resource's own convergence as a stall: while it settles, nothing else the probe watches changes, because the image is pulled and the pod already counts as ready. Longhorn creates the resource during node setup rather than on first volume request, so waiting on it cannot deadlock against a volume whose creation is itself gated on storage readiness. Signed-off-by: eriknordmark <erik@zededa.com> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A component that stopped converging left no record of what its progress probe had been watching, so a stall could only be inferred and the blocking signal had to be guessed at. Log the token as it changes, throttled, and once more when it has been still for half the deadline, naming the value it froze on. Diagnostic only; no behaviour change. Signed-off-by: eriknordmark <erik@zededa.com> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
eriknordmark
force-pushed
the
kubeinit-readiness-snapshot
branch
from
August 3, 2026 21:58
554a168 to
c7214e3
Compare
The multus manifest declares the NetworkAttachmentDefinition CRD and an instance of that kind, and both go to a single apply. The apiserver admits a custom resource only once its CRD is established, so the instance can lose that race and be rejected with 'no matches for kind "NetworkAttachmentDefinition"' while the CRD and the daemonset apply normally. The singleton attachment object is then missing and pods that attach to a network instance have nothing to reference. Tolerate the first apply failing, wait for the CRD to report Established, and apply again to create whatever lost the race; apply is idempotent, so a run that did not hit the race does no extra work. When the CRD never establishes, report the first apply's error alongside the wait failure — that error is the one naming the object that could not be created. This ports lf-edge#6242 to the Go daemon. That PR fixes the same race in cluster-init.sh, which this branch deletes, so without the port the race returns when kube-init replaces the script. Signed-off-by: eriknordmark <erik@zededa.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
On the first boot of a new EVE release the initialization markers are already in place, because they are restored from /persist, so kube-init takes its steady-state path and skips the per-release image import. The external-boot-image is then never re-tagged for the running release, and :latest keeps pointing at the previous one. That tag is what pillar references with imagePullPolicy Never when it launches a container app as a shim VMI, so after an upgrade every container app sits in BOOTING with ErrImageNeverPull while VM apps come up normally. Route the steady-state path through the import state as well. The import pre-checks each image in containerd, so a boot where they are all present costs a handful of lookups. Signed-off-by: eriknordmark <erik@zededa.com> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> (cherry picked from commit b27e809)
eriknordmark
changed the title
eriknordmark
added a commit
to eriknordmark/eve
that referenced
this pull request
Aug 4, 2026
# Conflicts: # pkg/kube/cluster-init.sh # pkg/kube/longhorn-utils.sh
7 tasks
rucoder
force-pushed
the
rucoder/kube-init-go
branch
from
August 5, 2026 17:23
7a50139 to
186e027
Compare
milan-zededa
force-pushed
the
rucoder/kube-init-go
branch
from
August 6, 2026 15:03
81fa284 to
ea4e1c4
Compare
7 tasks
eriknordmark
added a commit
to eriknordmark/eve
that referenced
this pull request
Aug 7, 2026
Replaces the three lf-edge#6271 kubevirt commits with lf-edge#6257 at df8ae7c plus the two of them that survive on top of it, since lf-edge#6257 fixes the VMIRS-delete logging the same way and rewrites Info to derive DomainId from a live Get rather than vmiList. Adds the proposed volumemgr readiness-publish follow-up to lf-edge#6240. Also records that lf-edge#5971 and rucoder#3 are held at the tips the 7/7 image used rather than caught up, so the bring-up baseline is not a variable in this run. Signed-off-by: eriknordmark <erik@zededa.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Author
|
Closing — everything here is now on The diff GitHub shows on this PR (+1.7M over 6113 files) is an artifact:
Thanks for taking the commits. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
An offer, not a request — take it, cherry-pick from it, or close it, whichever suits you. Opened as a draft against
kube-init-goso the code is reviewable rather than described in prose. It follows up on my comment on lf-edge#5971 asking whether you wanted these as a PR.Five commits on top of
7a501392a, covering three first-boot problems: aSNAPSHOT-cancels-a-pull interaction, a CRD ordering race, and a per-release image import that is skipped on the first boot of a new EVE release.Problem 1 —
SNAPSHOTdiscards an in-flight pull. On a first EVE-k boot,SNAPSHOTstops k3s by SIGTERMing its process group so/var/libis snapshotted at rest. If aBestEffortcomponent is still pulling an image at that moment, the pull dies with it and the work is thrown away. The Longhorn instance-manager image is ~440 MB and takes 8–20+ minutes depending on storage layout, so it is reliably the one in flight.Problem 2 — the multus NAD instance can be applied before its CRD is established, so a first boot can lose the NetworkAttachmentDefinition to a CRD that is present but not yet registered.
Problem 3 — the per-release image import is skipped after an EVE upgrade. The initialization markers are restored from
/persist, so the first boot of a new release takes the steady-state path and skipsStateImportingaltogether.ImportAllis per-release work: it re-tagseve-external-boot-imageto the running release, and pillar references that image as:latestwithimagePullPolicy: Neverwhen it launches a container app as a shim VMI. On an upgraded device:latesttherefore never exists, and every container app sits in BOOTING withErrImageNeverPullwhile VM apps come up normally.What the commits do.
kube-init: bound readiness by progress— a component's ready-wait is bounded by a no-progress deadline rather than pure wall-clock, so a slow-but-advancing pull is allowed to finish. Progress is read from containerd's active ingests, and the guard — not the individualReadyfunc — owns the timeout, so an inner wait cannot expire while the outer one still considers the component healthy.kube-init: wait for Longhorn's instance-manager— the node cannot serve a volume until that pod runs, and it is owned by anInstanceManagerCR rather than a DaemonSet, so the daemonset sweep never observed it. Its CR state transitions also feed the progress signal, so the guard does not declare a stall while the pod is legitimately starting.kube-init: log what the progress probe saw— during a quiet period, log the token the probe was watching. Added after I twice guessed wrong about what a stall was waiting on; the log now says.kube-init: establish the NAD CRD before its instance— wait for the multus NAD CRD to be established before applying an instance of it.kube-init: import images on every k3s start— routePhaseSteadythroughStateImportingas well, so the re-tag happens on the first boot of a new release. The import pre-checks each image, so a boot where they are all present costs a handful of containerd lookups.enterSnapshotadditionally waits for in-flight retries to quiesce (bounded) before stopping the supervisor, so the snapshot no longer races a pull.Withdrawn: the separate
InstallCDIgating concern I raised on lf-edge#5971. TheBestEffortretry loop emits a component's signals on a late success, so the CDI CR is not permanently skipped, and nothing here touches that path.PR dependencies
Based on
kube-init-go@7a501392a. Since that branch force-pushes as a whole series, this will need rebasing whenever it moves.How to test and validate this PR
Unit:
go vet ./...andgo test ./...underpkg/kube/kube-init— clean, with tests added for the no-progress guard, the retry-tracker quiescence, the progress-token logging, the context-timeout sentinel, the instance-manager state mapping, the NAD CRD ordering, and the STARTING_K3S transition table.On a device, the observable effect is during the first EVE-k boot on slow storage: watch
kubectl -n longhorn-system get pod -l longhorn.io/component=instance-managerwhile the ~440 MB image pulls. Before,SNAPSHOTcould tear that pull down mid-flight and the work restarted; after, the snapshot waits for it.The image-import change is confirmed on hardware: across three builds on the same device,
importing imageswas 0 on every boot of the two builds without it and the onlyeve-external-boot-imagetag present was the previous release's; with it, the current release tag and:latestare both present, the stale tag is pruned, and a container app that had failed 8/8 across those two builds reachedRUN_STATE_ONLINE.Measured across a 7-leg kvm→EVE-k conversion matrix: cancelled pulls went 3 → 0, and wall-clock give-ups 3 → 0. These commits were exercised as part of an integration build combining several in-flight PRs, not standalone on this branch — CI here would be the first isolated run.
Changelog notes
No user-facing changes on their own. Prevents a first-boot image pull from being discarded when the cluster snapshot is taken, and prevents the multus NAD from being lost to a not-yet-established CRD. Both shorten and de-flake first boot on slower storage. The import change fixes container apps failing to start after an EVE upgrade on EVE-k.
PR Backports
Not applicable — this targets a feature branch, not a release branch.
Checklist
And the last but not least:
Reasons for the unchecked boxes: no documentation change — this alters retry/deadline behaviour inside
kube-init, with no new knob or user-visible surface. The behaviour was exercised on amd64 as part of an integration build rather than standalone on this branch, so I have not ticked the device boxes; arm64 untested. Labels left alone since this targets a fork branch.