Cdi cr fix on 5971 - #2
Closed
eriknordmark wants to merge 29 commits into
Closed
Conversation
Direct deps kube-init's typed client-go and containerd usage
relies on:
k8s.io/api v0.34.1
k8s.io/apimachinery v0.34.1
k8s.io/apiextensions-apiserver v0.34.1
k8s.io/client-go v12.0.0+incompatible (aliased to v0.34.1
via replace)
github.com/containerd/containerd/v2 v2.2.5
github.com/containerd/errdefs v1.0.0
k8s.io/kube-openapi's own go.mod aliases v0.31.0 as its module
version, but the repo never publishes that as a tag — the pseudo
below is what actually exists on the repo and is what our replace
directive points at.
Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
…eboot The primitives kube-init's FSM writes to persistent storage: - Marker: typed filesystem-path values (LonghornInitialized, NativeKubernetesMode, MultusInitialized, …). IsMarked / Mark / Unmark route through AtomicWriteFile so partial writes never leave a half-marker. - AtomicWriteFile: temp-file + rename atop the destination, with an fsync in between so the on-disk state matches the return value even across a crash. - SaveVarLib / RestoreVarLib: /var/lib snapshot beneath the vault. Supports the cluster→single-node conversion rollback: if the cluster mode transition can't complete, the /var/lib snapshot from the pre-transition boot is restored. - Reboot: the daemon's controlled reboot path. Writes a marker naming the reason before invoking the reboot binary so the next boot can consult the reason without racing the boot sequence. - K3sKubeconfig / ContainerdSocket / SealedDirName: canonical paths every subsystem shares. Constants (not vars) so a subsystem cannot accidentally rewrite them at test time. - ToK8sName: kubernetes-node-name transform (lowercased device UUID). Load-bearing for cross-subsystem consistency — the FSM's node-uuid label, monitor's reapply, tiebreaker's node-lookup, and Longhorn's node-object all agree on this spelling. Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
…ntainerd wait Everything that must be ready before k3s can start. The FSM sits in INSTALLING → STARTING_CTRD while this package runs; a failure here aborts the boot rather than starting k3s on a broken substrate. - Kernel modules loaded via modprobe (bridge, iptables_nat, iscsi_tcp, etc.). Absent kernel bits surface with the specific module name so operators can spot a kernel-config regression. - cpu_manager_state cleanup: kubelet sometimes leaves a stale cpu_manager_state under /var/lib/kubelet after a reboot with a changed --cpu-manager-policy. Deleting it on every boot before containerd starts is upstream-blessed cleanup. - Cgroup mount: /sys/fs/cgroup, kubepods.slice hierarchy. - iSCSI daemon start (needed for Longhorn's block-mode volumes). - Vault wait: pillar's vaultmgr signals "ready" via a fs marker; the daemon blocks here until the marker is present so k3s does not race the vault-mount. - EdgeNodeInfo wait: blocks until the pubsub subscription delivers the device identity (DeviceName / DeviceID). - Containerd start + reachability probe. Everything is idempotent — the FSM re-enters INSTALLING on recycle without expecting a clean state. Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
…password, install Everything about running the k3s process: - binary.go / install.go: k3s binary download with SHA-256 verify + symlink rebuild for the multi-call subcommands (kubectl / ctr / crictl → k3s). Idempotent — a re-install picks up unchanged bytes cheaply. - config.go: drop-in files under /etc/rancher/k3s/config.yaml.d with numeric prefixes deciding merge order (00-nodename, 01-clusterconfig, 03-enc-disable-local-path, 99-user-override). Slot 02 is reserved. Byte-stable rendering keeps k3s's drop-in-hash restart trigger quiet on no-op reconciles. - paths.go: absolute paths for the k3s binary and every state file the daemon writes. - supervisor.go: Setpgid + orphan-sweep supervisor for the k3s process. Ports its own PID stays alive until the child's process group is fully reaped; port-listening probe rules out an outer-shell-still-running scenario before signalling "started". - token.go: k3s node-token rotation on cluster-mode change. - node_password.go: FixNodePasswordSecret clears a stale node-password secret when the file marker StaleNodePasswdFlag is present. Delete via typed clientset (kube-system Secrets). - readiness.go: WaitKubeconfig (poll for /etc/rancher/k3s/k3s.yaml with ctx cancel), then WaitReady blocks until Node's Ready condition + kube-system pods are Ready + node-uuid label is applied. Uses a client-go client built from the kubeconfig file — dial errors during the boot window are tolerated as "keep polling", so no shell-out is needed for the chicken- and-egg case of probing an API that isn't up yet. Every step is bounded by the caller's ctx (INSTALLING / STARTING_K3S / WAIT_K3S_READY handlers set the deadline). Restart hooks live under /etc/k3s-supervisor/hooks.d and fire between stop and start on operator-driven restart (RT PerformanceProfile changes are the canonical consumer). Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
k3s version override + component-by-component upgrade driven by the KubeConfig pubsub subscription. Runs as a background loop during RUNNING; a version bump on the wire triggers download + SHA-verify + install of the new k3s binary followed by rolling component upgrades. - update.go: main loop. Pulls the desired k3s version from the KubeConfig subscription cache, downloads via curl into a /tmp path with a per-attempt timeout, SHA-256-verifies against the version-specific expected hash, atomically installs into /var/lib/k3s/bin/k3s. - cluster.go: checkClusterReady probes the API server via discovery.ServerVersion; componentIsInstalled probes each component's namespace via a typed Namespaces().Get, distinguishing "not installed" (NotFound → false, nil) from "cannot tell" (err → false, err) so a transient API outage isn't misread as "component gone". - k3s_update.go: k3s download / verify / install flow. - status.go: publishes update progress to zedagent via pubsub so the controller sees each component's version. Retry gate: the daemon consults the KubeClusterUpdateStatus subscription to skip retrying a version the controller has already reported as failed — matches the "one attempt per version" semantic pillar expects. Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
…ault() Holds the typed clientset, apiextensions clientset, dynamic client, discovery + deferred RESTMapper, and shared informer factories (typed + dynamic). Constructed once at daemon boot from state.K3sKubeconfig and exposed via a package-level singleton (SetDefault / Default) so subsystems don't have to thread a *Client parameter through every function. Default() panics if consulted before SetDefault — that path indicates a lifecycle bug and should crash loud rather than nil-deref deeper. ResetMapper() invalidates the discovery cache so kubectlx.applyOne picks up newly-installed CRDs on retry. WaitForKubeconfig polls for state.K3sKubeconfig with ctx cancellation, exposed for callers that need to gate on file existence explicitly. Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
…ce layer Every operation the daemon performs against Kubernetes or containerd goes through this package. No argv assembly, no stderr parsing, no subprocess exec. apply.go — Apply / ApplyFile / ApplyURL / Get + DeleteFile / DeleteURL / DeleteBytes. Server-side apply via the dynamic client, multi-document YAML parsing, mapper-reset + retry-once on meta.NoKindMatchError closes the CR-before-CRD race in one place. Retry via client-go/util/retry.OnError with an apierrors-based classifier (transient vs permanent). Force- conflict-override is on by default (ApplyOptions.NoForce inverts the flag) because kube-init owns the components it applies; context.Canceled short-circuits so peer-cancel doesn't drag 30s+ of backoff sleeps. wait.go — WaitCRDEstablished / WaitDeploymentReady / WaitDaemonSetReady / WaitJobComplete / WaitForCondition, all informer-driven via watchtools.UntilWithSync. WaitForCondition evaluates a JSONPath against the target object on every event — CR-defined phase fields (KubeVirt / CDI status.phase, etc.) fit this shape. containerd.go — ContainerdClient wraps containerd.Client (v2) with namespaces.WithNamespace(k8s.io) baked into every operation. ImportImage / ImageExists / DeleteImage / ListImages / TagImage. A health probe at construction turns "containerd is down" into one clear error instead of per-operation failure spam. No cri-api dependency — CRI's image operations are containerd image-service operations under the k8s.io namespace, which the Go client already exposes. Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
…ffort retry loop A component starts the moment its declared deps are Ready — no per-wave synchronization barrier. A fast component whose deps are satisfied never waits for a slow unrelated peer. Design lives at .claude/design/kube-init-client-go-and-dag.md §4.5. types.go — Component (Manifests + Apply + Ready + PolicyDeps + BestEffort + ReadyTimeout), Manifest (File / URL / Bytes), Edge (structural-derivation output tagged with rule: crd / namespace / webhook / sa / policy), RetryPolicy, RetryCallback. deploy.go — Graph + Run + Edges. plan() validates the graph and resolves edges. The scheduler runs a single-threaded coordinator loop that tracks launched/completed counters directly (a WaitGroup-based helper is a runtime panic when a completing goroutine's result enqueues children mid-drain). retry.go — Graph.RetryCtx is the daemon-scope context that keeps BestEffort background retries alive after Run's per-invocation ctx ends. Exponential backoff with jitter (defaults: 30s → 5min × 6 = ~30min budget). Re-arms Apply → Ready on Ready- timeout too, not just Apply-failure. Retries observe RetryCtx cancel (daemon shutdown). rules.go — placeholder for the four structural-dep rules (CRD→CR, Namespace→resource, webhook-Service→webhooked-resource, ServiceAccount→Pod-spec) that fire once callers populate Component.Manifests with real bytes. Today Manifests is empty across the shipped graph; ordering rides entirely on PolicyDeps. deploy_test.go — 25 tests covering plan validation, edge tagging, edge sorting, dependency-order execution, parallel- execution property (two-channel handshake, not sleep), error propagation with alphabetical head + rest, ctx cancellation, BestEffort semantics, Ready-only-on-Apply-success, MaxParallel concurrency cap, Apply=nil (Ready-only) shape, and four retry-loop tests (eventually-succeeds, exhausts, ctx-cancel-mid- loop, one-shot-without-RetryCtx). All pass -race across 50 iterations. Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
…nager Single-goroutine dispatch of pillar pubsub topics into per-topic subscriber caches. Modelled on pkg/pillar/cmd/monitor/subscriptions.go — one map of label → Subscription, deferred Activate, one goroutine pumping every topic via pubsub.MultiChannelWatch. Subscribers across kube-init packages (edgenodeinfo, encconfig, encstatus, kubeconfig, kcus) register their topics with the Manager during daemon boot; the Manager owns the connection and the pump goroutine. Registering after the pump has started is supported — new subscriptions activate lazily. Replaces file-polling under /run/zedagent/... / /persist/status/ zedkube/... — every subscriber owns its own decoded cache with the canonical pillar types (no local JSON re-declaration), so JSON-tag skew between pillar and kube-init cannot silently drop a field's value. Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
Six typed subscribers wired to the shared pubsubclient.Manager; one publisher for the daemon's own lifecycle status. - edgenodeinfo: EdgeNodeInfo subscription with WaitForFirst so the boot path can block on device identity (DeviceName, DeviceID). Exposes Get() for read-only consumers on the FSM loop. - encconfig: EdgeNodeClusterConfig subscription. Drives cluster shape (single vs HA), TieBreakerNodeID, cluster interface. - encstatus: EdgeNodeClusterStatus subscription. Present() folds both file deletion and the zero-UUID-delete sentinel into a single check so a cluster delete on the controller side is detected the same regardless of how zedkube signals it. - kubeconfig: KubeConfig subscription with K3sVersion accessor for the update loop. Tagged consistently with pillar's canonical json tags (capital K, no override). - kcus: KubeClusterUpdateStatus subscription gating upgrade retries. DestinationKubeUpdateVersion is uint32 (matching pillar), so a "cluster already tried and failed at version X" record correctly gates the next attempt. - kubeinitstatus: publishes the daemon's lifecycle events (transition-step, phase, etc.) so operators observing the controller see kube-init state transitions without needing to tail the kube container log. Every subscription decodes to pillar's canonical types (via pkg/pillar/types) — no local *JSON structs that can drift. Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
First-boot import of pre-packaged image tarballs (KubeVirt, CDI, Longhorn, Multus, SUC, the EVE-authored external-boot-image) into the user-side containerd's k8s.io namespace, so kubelet pods reference images without a first-boot registry pull. - Import via kubectlx.ContainerdClient (containerd v2 Go client under the k8s.io namespace). - ImportUpstreamImage is a no-op when the image is already present or the tarball is absent; kubelet falls back to a registry pull for anything missing. - ImportExternalBootImage re-tags the EVE-authored image (whose tarball name isn't the runtime tag) to the running EVE release's tag, then cleans up prior EVE-authored images for the same base name via ListImages + DeleteImage. - Manifest.json is parsed via `tar` (fatal-line-buffered) with the RepoTags extraction in Go — no jq dependency. Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
…ainerd env Wires kube-init into pillar's cost-aware management proxy so in-cluster traffic (CDI's image importer, containerd's image pulls) routes through pillar rather than out the WAN. - cni0.go: SetupCNI0ProxyIP assigns a well-known anchor IP on the flannel-created cni0 bridge — CDI pods reach the management proxy at that IP without a per-node hostname lookup. Enabled() only fires when pillar's mgmtproxy config file is present. - The CDI CR's spec.config.importProxy fields (HTTPSProxy / noProxy) are reconciled every steady-state tick: a dynamic Get reads the current values, and a merge-patch fixes any drift. In-Go JSONPath extraction reads the field values — no `-o jsonpath=…` shell-out. - containerd-launch env injection is a separate concern (pkg/kube/kube-init/prereqs) and pre-dates this package. Wildcard-webhook rules (rules[].apiGroups: ["*"]) are treated as "no derived edge from this webhook" in the deploy planner — a catch-all webhook shouldn't sequence the whole graph. Not the concern of this package but noted here so the wiring is consistent. Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
… startup stagger Every state change between "this node is a single-node cluster" and "this node is a member of an HA cluster" — including the recovery paths — lives here. - transition.go: the ordered step runner. Single→cluster rotates the k3s node-token, applies encstatus-derived configuration, and updates supervisor args. Cluster→single reverts. Progress callback surfaces the current step name to the FSM's control socket so `k3s-sctl status` reports `transition-step=<step>`. - masterleases.go: stale etcd master-lease cleanup. After a single→HA conversion, etcd carries a lease from the pre-conversion single-node identity; the deploy graph's Longhorn readiness stalls if that lease's expiry sits ahead of the new node's registration. Cleanup deletes the stale lease from etcd directly (etcdctl invocation — outside the client-go surface because etcd doesn't ship a stable Go SDK). - startup_rank.go: on HA startup, computes the local node's rank (0-based position in the sorted control-plane node list) and sleeps rank × 25s before launching k3s. Fans out simultaneous power-cycles into a staggered sequence so etcd quorum re-forms cleanly without three nodes fighting for the leader lease at t=0. Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
The steady-state loop the FSM enters after DEPLOYING. Every 15s health tick runs a fixed sequence of housekeeping + reconcile tasks; separate goroutines watch containerd exit, kubeconfig drift, log rotation, cluster-config-changed events, and the user-override drop-in for k3s config. Tasks the health tick runs (in order): - DHCP CNI daemon liveness (respawn on exit). - CNI plugin binary drift (copy from stage dir when needed). - Multus re-render + apply if the initialised marker is absent. - Node label + annotation reapply (node-uuid, longhorn disk config) when the labels-initialised marker is absent — one merge-patch does labels and annotations together, then a readback verifies both landed. - Debug-user RBAC re-application. - Kubeconfig sync (copy /etc/rancher/k3s/k3s.yaml to the kube-init-visible location, atomic). - Image reimport (external-boot-image via kubectlx. ContainerdClient — ImageExists check, tarball import only if the image is genuinely absent). - Longhorn post-install config (via components. LonghornPostInstallConfig). - Registration manifest reconcile. transition.go — countReadyNodes counts Nodes with Ready=True via typed clientset. Used by the cluster-mode monitor to detect "we're on the HA majority" transitions. parseTransitionMarker covers the FSM's cluster-transition marker file (unix-ts + reboot-count). Steady-state ticks are gated on state markers so a tick after a successful bootstrap is silent — no log line, no API call. Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
The tie-breaker is the third control-plane node in an EVE-K HA cluster whose only job is to give etcd a third vote so single- node loss doesn't strand the cluster. This package configures that role. ConfigApply is the phase entry point; it runs when the cluster first reaches three nodes AND the local node is the designated tie-breaker (per encconfig's TieBreakerNodeID). Steps: - nodesConfigApply — label every node with tie-breaker-node= true/false and cordon/uncordon accordingly. Merge patches via Nodes().Patch keep both label and spec.unschedulable updates atomic per node. - kubevirtConfig — scale virt-operator + KubeVirt CR to 2 replicas (tie-breaker excluded). Deployment scale via typed AppsV1 merge-patch; CR replica field via dynamic Resource Patch. - kubevirtTieBreakerConfigApply — patch every DaemonSet in the kubevirt namespace with a nodeSelector keeping pods off the tie-breaker. - cdiConfig — same nodeSelector patch on every Deployment in the cdi namespace. - longhornNodeSetSched — flip allowScheduling + evictionRequested on the tie-breaker's longhorn.io Node CR and every disk it owns. JSON-patch operations via dynamic Resource Patch. - longhornRescale — scale Longhorn CSI sidecar Deployments to 2 replicas; patch every longhorn-system DaemonSet with the nodeSelector (retries 5× because longhorn-manager occasionally races us during install). - drainNode — cordon + evict every non-DaemonSet pod on the tie-breaker via policy/v1.Eviction (matches kubectl drain --ignore-daemonsets --delete-emptydir-data --force semantics) + wait until pods leave the node. - StatusSet — stamp every node with tie-breaker-config-applied=1 as the "phase complete" marker so a subsequent reboot skips the whole flow. Every operation is idempotent — a partial ConfigApply that gets interrupted mid-way completes on the next FSM entry. Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
…eduler, RBAC, SR-IOV, kube-vip
Everything the deploy graph runs during DEPLOYING and everything
the monitor's steady-state ticks reconcile against.
components.go — DeployAll builds a deploy.Graph and runs it. The
graph has ~10 Components; only Longhorn declares a PolicyDep
(longhorn depends on manifests, which stages storage-classes.yaml
into the auto-deploy dir). Every Install* function:
- Longhorn: applyLonghornDiskConfig labels + annotates the node
via one merge-patch; the longhorn config YAML applies as a
drop-in.
- KubeVirt: InstallKubeVirt applies the operator manifest via
SSA, waits for virt-operator Ready, applies the CR via SSA,
applies feature-gate CR via SSA. kubeVirtConfigReplicas
writes .spec.replicas + .spec.infra.replicas via SSA with a
distinct field manager ("kube-init-replicas") so subsequent
applies of the operator manifest don't fight for ownership.
- CDI: apply operator, wait for cdi-operator Ready, apply CR.
WaitCDIReady goes through kubectlx.WaitForCondition on
.status.phase == "Deployed".
- Multus: template rendering (IP address substitution) then
applyMultus + linkMultusIntoK3s.
- descheduler / debug-rbac / manifests / kube-vip / SR-IOV
manifest staging / stale-mount-cleanup daemon launcher —
one Component each.
- WaitLonghornReady is informer-driven (via a wait-scoped
dynamic factory over longhorn.io/Node and typed DaemonSets in
longhorn-system). No polling; the compound predicate is
re-evaluated on every event.
kubevirt.go — MigrateKubeVirtFeatureGates reads the running CR
via dynamic Get, patches with dynamic Patch, marks the migration
done. buildFeatureGatesPatch is table-driven so a new gate is a
one-line addition.
longhorn.go — LonghornIsReady is the compound predicate the
monitor consults on every tick. Uses typed clientsets for
DaemonSet readiness and dynamic client for the Longhorn Node CR
and engine images. longhornEngineDeployedOnNode distinguishes
"status.nodeDeploymentMap absent" (fresh CR — not-ready-yet)
from "map present but this node is false" (recycle engine +
manager pods to force reconcile). CheckOverwriteNsmounter uses
client-go remotecommand.NewSPDYExecutor (via podExec helper) to
copy a fixed nsmounter binary into every longhorn-csi-plugin
pod that doesn't already carry the marker file.
registration.go — RegistrationApplyIfReady stages the
controller-supplied registration manifest into the k3s
server-manifests dir. RegistrationApplied checks the AddOn CR
via dynamic Get.
uninstall.go — the K3sBase conversion flow. Uninstalls
descheduler / Longhorn / CDI / KubeVirt / Multus in reverse
dependency order. Longhorn uninstall is bounded by a 1000-poll
budget (~83 min) covering the worst-case data-shred workload;
timeout aborts the conversion rather than declaring
NativeKubernetesMode with volumes half-shredded.
sriov.go / stale_mount_cleanup.go — hardware-conditional
manifest staging and background cleanup daemon launcher.
Every Install has an idempotence gate on a state.Marker; a
re-entry to DEPLOYING or a re-tick of the monitor is a cheap
noop when the marker is already set.
Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
Launches a virtctl vnc proxy on demand for a KubeVirt VMI and tears it down when the caller process exits. Caller-PID watchdog polls /proc/<pid> so an orphaned proxy from a disconnected debug session doesn't linger. Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
14 states, 27 events. The daemon boots into INIT, walks through INSTALLING → STARTING_CTRD → CONFIGURING → STARTING_K3S → IMPORTING (first-boot only) → WAIT_K3S_READY → DEPLOYING (first- boot only) → RUNNING, and handles restart / recycle / cluster- mode-transition events without dropping back into an ad-hoc respawn loop. State entries are `enterStateFn` dispatched from `transition`; async work runs on a per-attempt workCtx via startAsync so transition cancellation drains the work goroutine cleanly. The control socket at /run/k3s-supervisor.sock accepts `status`, `restart`, `stop`, `graph` commands; every operator-facing k3s- sctl invocation dispatches through the socket rather than touching flag files. initKubeclient wires kubeclient.Default at WAIT_K3S_READY's transition (test-safe when the kubeconfig file is absent — transition-table tests exploit this). components.DeployAll takes a retryCtx that main.go supplies as d.rootCtx, so BestEffort background retries in the deploy graph survive per-invocation workCtx while still dying on daemon shutdown. Restart-reason is a typed field carried through the STOPPING_K3S → RUNNING_HOOKS → STARTING_K3S transition so restart hooks (/etc/k3s-supervisor/hooks.d) fire once for operator-driven restarts and never for crash recovery. The RT operator's PerformanceProfile-change flow depends on this — CPU isolation / RDT partitioning need to be reconfigured between stop and start, not on every crash. README.md sketches the architecture, package boundaries, and the future-work list (bootstrap-cert fingerprint pinning, config-package retirement, and so on). Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
One-shot tool that dials /run/k3s-supervisor.sock and prints the
reply. Reads until EOF so multi-line responses (graph) print
naturally alongside single-line ones (status / restart / stop).
K3S_SUPERVISOR_SOCKET overrides the path for tests.
Exit code follows the last line: non-zero when it starts with
ERR.
Subcommands:
- restart — graceful k3s restart (pre-restart hooks fire).
- status — one-line status report (state, k3s pid, uptime,
backoff, restart count, phase, transition-step).
- stop — stop the kube-init daemon.
- graph — resolved deploy-graph edges (one per line, each
tagged with its rule source).
Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
…helper pkg/kube/Dockerfile — the kube linuxkit container's entrypoint is /usr/bin/kube-init. Everything else the container needs to ship (k3s multi-call binary, longhorn nsmounter, CNI plugin staging, longhorn/kubevirt/CDI/multus tarball drops under /images/) is installed via the existing build.yml conventions. pkg/kube/k3s-control.sh — small wrapper that dispatches to k3s-sctl. Kept so external tooling (eden / eve exec kube …) that used to invoke this path continues to work; every subcommand it takes now goes over the control socket rather than touching flag files. pkg/kube/kubevip-delete.sh — minimal helper that KubeVIPDelete calls to remove leftover kube-vip state files after component deletion (leftover static-pod manifests, arp cache). Not part of the deploy graph; runs from the uninstall path. Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
Eleven files under pkg/kube/ implemented the previous cluster-init flow in POSIX shell + kubectl. Every operation they exposed lives in pkg/kube/kube-init/ now — no external caller depends on them, and keeping them alongside the Go daemon would leave two divergent implementations of the same paths that operators could accidentally invoke. Deleted: cluster-init.sh cluster-update.sh cluster-utils.sh descheduler-utils.sh kubevirt-utils.sh longhorn-utils.sh pubsub.sh registration-utils.sh tie-breaker-utils.sh utils.sh vnc-proxy.sh lib/config.sh Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
…stDevices
The KubeVirt CRD requires spec.configuration.permittedHostDevices.
{pciHostDevices,mediatedDevices,usb} to be arrays. Bare keys
parse as `null` in YAML; server-side apply forwards `null` as an
explicit set-to-null, which the schema rejects with:
KubeVirt.kubevirt.io "kubevirt" is invalid:
spec.configuration.permittedHostDevices.mediatedDevices:
Invalid value: "null": ... must be of type array
Explicit `[]` satisfies the schema.
Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
The kube linuxkit package now ships a Go daemon in place of the shell path. Adds a kube-init build rule that DOCKER_GO-invokes `go build ./cmd/... .` inside pkg/kube/kube-init/, staged into pkg/kube/dist/ before the linuxkit pkg build. pillar-* targets are unchanged. Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
kube-init's vendor tree is upstream Go modules with well-known licenses; the SPDX generator's per-file scan blows out on the tree size without adding value. Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
Bumps GO_VERSION in pkg/alpine/Dockerfile from 1.25.11 to 1.25.12, matching aligned defaults in the top-level Makefile (GOVER) and build-tools/src/scripts/Dockerfile (ARG GOVER). Clears the stdlib CVEs surfaced by OSV-scanner on downstream Go modules, including GO-2026-5856 which requires Go >= 1.25.12. Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
Propagates the new eve-alpine image hash produced by the Go toolchain bump in pkg/alpine to every consumer via tools/bump-and-commit.sh: pkg/*/Dockerfile, pkg/eve/Dockerfile.in, and eve-tools/bpftrace-compiler/root/Dockerfile. Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
A slow cdi-operator image pull permanently disables CDI on the node. The operator-readiness wait ahead of the CR apply returns an error on timeout, so the CR is never created and nothing revisits the step: the CRD and a healthy operator are present while `kubectl get cdi -A` stays empty, and with no CR there is no uploadproxy or importer, so every app volume that needs an upload parks in DELIVERED for the life of the boot. It surfaces as a repeating `kubectl patch cdi: cdis.cdi.kubevirt.io "cdi" not found`. Because the cdi component is BestEffort, the deploy treats the failure as success and continues, which hides it. Log a warning and apply the CR regardless. The wait itself is kept -- the operator owns the CR's admission webhook, so waiting first is still the right order -- and ApplyURL's backoff (10 attempts, 1-30s) absorbs a webhook that is not serving yet. Reproduced 3/3 on a kvm->EVE-k conversion test with ZFS /persist where the operator image took 5m43s-7m31s to pull, and verified fixed on a rebuilt image: the warning fired and the CR still reached phase=Deployed 3m17s later, with the app reaching RUNNING. Signed-off-by: eriknordmark <erik@zededa.com> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
An EVE-k node reports cluster storage as unready, and every app volume stays in CREATING_VOLUME, whenever any DaemonSet in the longhorn-system namespace lacks a Running-and-Ready pod on this node -- including DaemonSets that Longhorn does not own and that are never expected to become ready. The readiness check iterated over every DaemonSet in the namespace and required each one to be healthy, consulting its list of expected DaemonSets only afterwards to confirm those three exist. In practice this is reached through EVE's own collect-info, which leaves a SupportBundle agent DaemonSet behind; the node then refuses to serve volumes for as long as that object exists, with no way for an operator to tell why. Restrict the per-node health requirement to the DaemonSets Longhorn is expected to run, and skip anything else sharing the namespace. Restricting the loop also removes a second false failure: a DaemonSet whose node selector legitimately excludes this node reported zero pods here and was treated as missing. Signed-off-by: eriknordmark <erik@zededa.com> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> (cherry picked from commit b3f0bbd)
On an EVE-k node volumemgr waits up to 40 minutes for the cluster to be
able to serve a volume, and then reports Initialized regardless of how
that wait ended. A node whose Longhorn or CDI never came up is therefore
indistinguishable, from the outside, from a healthy one -- the only
difference is a line in volumemgr's own log. Anything consuming the
status, an operator inspecting it, or a test asserting on it is misled
in precisely the case that matters.
Report the outcome instead: Initialized now reflects whether cluster
storage became usable, and a new UnmetCondition carries the gate that
was still outstanding, reusing the sub-condition the kubernetes wait
already computes ("longhorn not ready: ...", "kubevirt not ready: ...").
Nodes that are not EVE-k have no such gate and are Initialized from the
start, as before.
Volumes are unaffected either way: they are gated separately and defer
and retry until storage appears.
Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
(cherry picked from commit 4c7b020)
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
rucoder
force-pushed
the
rucoder/kube-init-go
branch
from
July 30, 2026 00:21
d80f35d to
c581ba1
Compare
rucoder
force-pushed
the
rucoder/kube-init-go
branch
from
July 31, 2026 11:43
67accc1 to
7a50139
Compare
Author
|
Closing this — the rebase onto master superseded all three commits. The two non-CDI commits (kubeapi stray-daemonset, volumemgr storage readiness) are The CDI CR commit is no longer needed. It patched |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This is what I needed for testing to be robust on 5971. The first commit is to your changes and the others can be merged in master separately.
So please review the first commit.