pkg/kube: port cluster-init.sh to Go daemon - #5971
Conversation
9f4c996 to
cf7b773
Compare
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
be90fd2 to
7efadd7
Compare
andrewd-zededa
left a comment
andrewd-zededa
left a comment
There was a problem hiding this comment.
Just took a quick look, a few notes. I need to look closer at this sometime this week.
| return false, nil | ||
| } | ||
| if _, err := kubectl("get", "nodes.longhorn.io", nodeName, | ||
| "-n", "longhorn-system"); err != nil { |
There was a problem hiding this comment.
Can the repeated use of the longhorn-system namespace string be moved to a const?
There was a problem hiding this comment.
Done — this is now the single const kubectlx.LonghornNamespace (kubectlx/gvrs.go:39), and that definition is the only occurrence of the literal left in the package.
| return nil | ||
| } | ||
|
|
||
| out, err := kubectl("get", "kubevirt", "kubevirt", "-n", "kubevirt", |
There was a problem hiding this comment.
Can the "kubevirt" namespace string be moved to a const across the file?
There was a problem hiding this comment.
Done for the namespace — kubectlx.KubeVirtNamespace (kubectlx/gvrs.go:40).
The "kubevirt" literals that remain are the CR name, and kubevirtCRName (components.go:106) is referenced at only one of them: components.go:651/654, components/kubevirt.go:47/69 and tiebreaker/tiebreaker.go:220 still inline it. Worth a follow-up sweep.
| // RunDeschedulerOnBoot applies the descheduler job exactly once per | ||
| // boot. The descheduler reshuffles pods to honour spread policies | ||
| // that may have been violated while a node was offline; we run it | ||
| // after the node has finished joining and required DaemonSets are | ||
| // ready so it doesn't fight an in-progress rollout. | ||
| // | ||
| // Idempotent within a single boot cycle (guarded by | ||
| // deschedulerBootMarker, which lives in /tmp and clears on reboot). | ||
| // | ||
| // Errors wrapping ErrDeschedulerNotReady mean "try again later"; | ||
| // any other error is a hard failure. | ||
| func RunDeschedulerOnBoot(ctx context.Context) error { | ||
| if _, err := os.Stat(deschedulerBootMarker); err == nil { | ||
| log.Printf("update: descheduler already ran this boot, skipping") | ||
| return nil | ||
| } else if !errors.Is(err, os.ErrNotExist) { | ||
| return fmt.Errorf("stat descheduler marker: %w", err) | ||
| } | ||
|
|
||
| if ready, err := nodeReadyAndSchedulable(); err != nil { | ||
| return fmt.Errorf("check node ready: %w", err) | ||
| } else if !ready { | ||
| return fmt.Errorf("%w: node not Ready or scheduling disabled", | ||
| ErrDeschedulerNotReady) | ||
| } | ||
| if err := requireDaemonSetsReady("longhorn-system"); err != nil { | ||
| return err | ||
| } | ||
| if err := requireDaemonSetsReady("kubevirt"); err != nil { | ||
| return err | ||
| } | ||
|
|
||
| // A previous boot may have left a completed Job behind; the | ||
| // `apply -f` below would error on the immutable selector | ||
| // fields of the old Job. `--ignore-not-found` makes the call | ||
| // safe when nothing is there. We log but tolerate any other | ||
| // delete failure: the subsequent apply will surface a real | ||
| // problem with a clearer message. | ||
| if out, err := kubectlx.Run("delete", "job", "descheduler-job", | ||
| "-n", "kube-system", "--ignore-not-found"); err != nil { | ||
| log.Printf("update: delete stale descheduler job: %v (output: %s)", | ||
| err, truncateForLog(out, 1024)) | ||
| } | ||
|
|
||
| if out, err := kubectlx.Run("apply", "-f", deschedulerJobYAML); err != nil { | ||
| return fmt.Errorf("apply descheduler job: %w (output: %s)", | ||
| err, truncateForLog(out, 1024)) | ||
| } | ||
|
|
||
| if err := os.WriteFile(deschedulerBootMarker, []byte("1"), 0644); err != nil { | ||
| log.Printf("update: write boot marker %s: %v (descheduler will re-run on next call this boot)", | ||
| deschedulerBootMarker, err) | ||
| } | ||
| log.Printf("update: descheduler job applied") | ||
| return nil | ||
| } |
There was a problem hiding this comment.
Latest cleanup on master removed this from the kube container, its initiated from pillar now.
There was a problem hiding this comment.
Resolved — update/descheduler.go no longer exists. What remains is the RBAC + policy-configmap apply/delete (components.go:915, uninstall.go:101), which matches master's descheduler-utils.sh exactly; the CronJob side stays in pillar.
|
@andrewd-zededa , @eriknordmark had a cool idea to use actual PubSub. I'm going to implement it and rebase the PR on top of master and also integrate the latest scenarios/fixes that were implemented in shell scripts |
…obe TLS bypass The bootstrap-discovery probe in waitForBootstrapServer dials a k3s self-signed cert before we have the cluster CA in our trust store — we cannot get the cluster CA until we have joined the cluster, and we cannot join until this probe succeeds. So the probe runs with InsecureSkipVerify=true. That triggered a github-advanced-security finding on PR lf-edge#5971 ("InsecureSkipVerify should not be used in production code"). The bypass is intentional and the bot does not see why. This commit makes the justification load-bearing in the source: - The comment block above the http.Client construction now explains the chicken-and-egg explicitly, calls out the cluster-UUID match a few lines below as the actual authentication boundary, and notes that the bypass is scoped to this single probe. - The line itself carries both `//nolint:gosec` (existing — suppresses the gosec linter) and `lgtm[go/disabled- certificate-check]` (new — suppresses CodeQL inline). - The comment references the PR conversation so reviewers have one place to confirm the suppression is acceptable. No behaviour change. Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
7efadd7 to
a548717
Compare
…obe TLS bypass The bootstrap-discovery probe in waitForBootstrapServer dials a k3s self-signed cert before we have the cluster CA in our trust store — we cannot get the cluster CA until we have joined the cluster, and we cannot join until this probe succeeds. So the probe runs with InsecureSkipVerify=true. That triggered a github-advanced-security finding on PR lf-edge#5971 ("InsecureSkipVerify should not be used in production code"). The bypass is intentional and the bot does not see why. This commit makes the justification load-bearing in the source: - The comment block above the http.Client construction now explains the chicken-and-egg explicitly, calls out the cluster-UUID match a few lines below as the actual authentication boundary, and notes that the bypass is scoped to this single probe. - The line itself carries both `//nolint:gosec` (existing — suppresses the gosec linter) and `lgtm[go/disabled- certificate-check]` (new — suppresses CodeQL inline). - The comment references the PR conversation so reviewers have one place to confirm the suppression is acceptable. No behaviour change. Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
a548717 to
8454c9b
Compare
| // https://github.com/lf-edge/eve/pull/5971. | ||
| httpsClient := &http.Client{ | ||
| Transport: &http.Transport{ | ||
| TLSClientConfig: &tls.Config{InsecureSkipVerify: true}, //nolint:gosec // lgtm[go/disabled-certificate-check] |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #5971 +/- ##
==========================================
+ Coverage 24.12% 24.48% +0.35%
==========================================
Files 512 522 +10
Lines 93653 95369 +1716
==========================================
+ Hits 22598 23352 +754
- Misses 69267 70045 +778
- Partials 1788 1972 +184 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
@rucoder I'm working to test this on my local dev cluster, will report back any issues I see |
andrewd-zededa
left a comment
andrewd-zededa
left a comment
There was a problem hiding this comment.
Just a couple of quick finds, there are more repeated strings which should be consts for future maintenance.
| debugRoleBinding = "/etc/debuguser-role-binding.yaml" | ||
| kubevirtOperator = "/etc/kubevirt-operator.yaml" | ||
| kubevirtFeatures = "/etc/kubevirt-features.yaml" | ||
| longhornCfg = "/etc/lh-cfg-v1.9.1.yaml" |
There was a problem hiding this comment.
This string should be built off the longhorn version defined in uninstall.go
There was a problem hiding this comment.
Agreed. longhornCfg (components.go:61) bakes v1.9.1 into the filename while uninstall.go:29 carries it as a const, and images.go repeats it six more times.
Folding all of them into one versions package, with the path built as "/etc/lh-cfg-" + versions.Longhorn + ".yaml" — same treatment for kubevirtCRURL, cdiVersion, k3s.K3sVersion and the images.go tags. All const expressions, so nothing moves to runtime.
| // app instance workloads and is referenced by the multus | ||
| // NetworkAttachmentDefinition and the debug-user RoleBinding. | ||
| func ensureEveKubeAppNamespace() error { | ||
| if _, err := kubectl("get", "namespace", "eve-kube-app"); err == nil { |
There was a problem hiding this comment.
eve-kube-app should be a const
There was a problem hiding this comment.
Done here — eveKubeAppNamespace (components.go:102). Two stragglers elsewhere: monitor/monitor.go:887/892 still use the literal, and vnc/vnc.go:40 defines a separate vncNamespace const for the same namespace.
|
@andrewd-zededa I think I did not push the latest version |
|
@rucoder Are you actively working on this? If not, I'd be interested in picking it up, if you're okay with that. I would cherry-pick your commits and continue the work on my fork. My plan is to finish the 2-node HA solution that @zedi-pramodh started. I'd also like to implement it in Go rather than adding more shell script complexity. And more broadly, I'm planning to become more involved in the Kubernetes side of the EVE project. Regarding this:
Have you already implemented this locally and just haven't opened the PR yet, or is it still in the planning stage? |
@rucoder Any updates? I'm still waiting to hear back on whether you're actively working on this. If not, I'd like to pick it up, if you're okay with that.
Even if you can't get to it right away, it'd help if you could at least open a draft PR for the follow-up part so the work in progress is visible. |
|
@rucoder Can you rebase this on master? I'd like to take it for a spin on the soak test setup I have which exercises the first-boot of EVE-k a lot. |
The steady-state loop the FSM enters after DEPLOYING. Every 15s health tick runs a fixed sequence of housekeeping + reconcile tasks; separate goroutines watch containerd exit, kubeconfig drift, log rotation, cluster-config-changed events, and the user-override drop-in for k3s config. Tasks the health tick runs (in order): - DHCP CNI daemon liveness (respawn on exit). - CNI plugin binary drift (copy from stage dir when needed). - Multus re-render + apply if the initialised marker is absent. - Node label + annotation reapply (node-uuid, longhorn disk config) when the labels-initialised marker is absent — one merge-patch does labels and annotations together, then a readback verifies both landed. - Debug-user RBAC re-application. - Kubeconfig sync (copy /etc/rancher/k3s/k3s.yaml to the kube-init-visible location, atomic). - Image reimport (external-boot-image via kubectlx. ContainerdClient — ImageExists check, tarball import only if the image is genuinely absent). - Longhorn post-install config (via components. LonghornPostInstallConfig). - Registration manifest reconcile. transition.go — countReadyNodes counts Nodes with Ready=True via typed clientset. Used by the cluster-mode monitor to detect "we're on the HA majority" transitions. parseTransitionMarker covers the FSM's cluster-transition marker file (unix-ts + reboot-count). Steady-state ticks are gated on state markers so a tick after a successful bootstrap is silent — no log line, no API call. The join watchdog is driven by progress, not by a deadline. How long a join takes is unknowable — it covers a k3s restart, an image import, registration and every kube-system pod — so any budget for it is a guess that some slower device will fail. How long a join may make no progress at all is a stable quantity, so every advance to a stage this join has not reached before kicks the timer: a slow-but-advancing join is never rebooted, while a wedged one trips promptly. A crash loop does not count as progress, because it revisits stages rather than advancing. The cluster-config watch runs in every state, not only in RUNNING. A controller can withdraw a node's cluster configuration at any moment, including seconds after a join completed while the node is still working through readiness. A zero-UUID or otherwise empty status is treated as "no live cluster" only when it really is one, so a status that arrives late or half-written cannot be mistaken for a withdrawal. Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
The tie-breaker is the third control-plane node in an EVE-K HA cluster whose only job is to give etcd a third vote so single- node loss doesn't strand the cluster. This package configures that role. ConfigApply is the phase entry point; it runs when the cluster first reaches three nodes AND the local node is the designated tie-breaker (per encconfig's TieBreakerNodeID). Steps: - nodesConfigApply — label every node with tie-breaker-node= true/false and cordon/uncordon accordingly. Merge patches via Nodes().Patch keep both label and spec.unschedulable updates atomic per node. - kubevirtConfig — scale virt-operator + KubeVirt CR to 2 replicas (tie-breaker excluded). Deployment scale via typed AppsV1 merge-patch; CR replica field via dynamic Resource Patch. - kubevirtTieBreakerConfigApply — patch every DaemonSet in the kubevirt namespace with a nodeSelector keeping pods off the tie-breaker. - cdiConfig — same nodeSelector patch on every Deployment in the cdi namespace. - longhornNodeSetSched — flip allowScheduling + evictionRequested on the tie-breaker's longhorn.io Node CR and every disk it owns. JSON-patch operations via dynamic Resource Patch. - longhornRescale — scale Longhorn CSI sidecar Deployments to 2 replicas; patch every longhorn-system DaemonSet with the nodeSelector (retries 5× because longhorn-manager occasionally races us during install). - drainNode — cordon + evict every non-DaemonSet pod on the tie-breaker via policy/v1.Eviction (matches kubectl drain --ignore-daemonsets --delete-emptydir-data --force semantics) + wait until pods leave the node. - StatusSet — stamp every node with tie-breaker-config-applied=1 as the "phase complete" marker so a subsequent reboot skips the whole flow. Every operation is idempotent — a partial ConfigApply that gets interrupted mid-way completes on the next FSM entry. Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
…eduler, RBAC, SR-IOV, kube-vip
Everything the deploy graph runs during DEPLOYING and everything
the monitor's steady-state ticks reconcile against.
components.go — DeployAll builds a deploy.Graph and runs it. The
graph has ~10 Components; only Longhorn declares a PolicyDep
(longhorn depends on manifests, which stages storage-classes.yaml
into the auto-deploy dir). Every Install* function:
- Longhorn: applyLonghornDiskConfig labels + annotates the node
via one merge-patch; the longhorn config YAML applies as a
drop-in.
- KubeVirt: InstallKubeVirt applies the operator manifest via
SSA, waits for virt-operator Ready, applies the CR via SSA,
applies feature-gate CR via SSA. kubeVirtConfigReplicas
writes .spec.replicas + .spec.infra.replicas via SSA with a
distinct field manager ("kube-init-replicas") so subsequent
applies of the operator manifest don't fight for ownership.
- CDI: apply operator, wait for cdi-operator Ready, apply CR.
WaitCDIReady goes through kubectlx.WaitForCondition on
.status.phase == "Deployed".
- Multus: template rendering (IP address substitution) then
applyMultus + linkMultusIntoK3s.
- descheduler / debug-rbac / manifests / kube-vip / SR-IOV
manifest staging / stale-mount-cleanup daemon launcher —
one Component each.
- WaitLonghornReady is informer-driven (via a wait-scoped
dynamic factory over longhorn.io/Node and typed DaemonSets in
longhorn-system). No polling; the compound predicate is
re-evaluated on every event.
kubevirt.go — MigrateKubeVirtFeatureGates reads the running CR
via dynamic Get, patches with dynamic Patch, marks the migration
done. buildFeatureGatesPatch is table-driven so a new gate is a
one-line addition.
longhorn.go — LonghornIsReady is the compound predicate the
monitor consults on every tick. Uses typed clientsets for
DaemonSet readiness and dynamic client for the Longhorn Node CR
and engine images. longhornEngineDeployedOnNode distinguishes
"status.nodeDeploymentMap absent" (fresh CR — not-ready-yet)
from "map present but this node is false" (recycle engine +
manager pods to force reconcile). CheckOverwriteNsmounter uses
client-go remotecommand.NewSPDYExecutor (via podExec helper) to
copy a fixed nsmounter binary into every longhorn-csi-plugin
pod that doesn't already carry the marker file.
registration.go — RegistrationApplyIfReady stages the
controller-supplied registration manifest into the k3s
server-manifests dir. RegistrationApplied checks the AddOn CR
via dynamic Get.
uninstall.go — the K3sBase conversion flow. Uninstalls
descheduler / Longhorn / CDI / KubeVirt / Multus in reverse
dependency order. Longhorn uninstall is bounded by a 1000-poll
budget (~83 min) covering the worst-case data-shred workload;
timeout aborts the conversion rather than declaring
NativeKubernetesMode with volumes half-shredded.
sriov.go / stale_mount_cleanup.go — hardware-conditional
manifest staging and background cleanup daemon launcher.
Every Install has an idempotence gate on a state.Marker; a
re-entry to DEPLOYING or a re-tick of the monitor is a cheap
noop when the marker is already set.
Longhorn's readiness is taken from its instance managers rather than
from the deployment alone: the CSI plugin can report available while the
per-node instance managers are still coming up, and a volume attach in
that window fails. Multus is restarted through its DaemonSet, so a
re-applied configuration is actually picked up by the pods already
running.
Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
Launches a virtctl vnc proxy on demand for a KubeVirt VMI and tears it down when the caller process exits. Caller-PID watchdog polls /proc/<pid> so an orphaned proxy from a disconnected debug session doesn't linger. Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
14 states, 27 events. The daemon boots into INIT, walks through INSTALLING → STARTING_CTRD → CONFIGURING → STARTING_K3S → IMPORTING (first-boot only) → WAIT_K3S_READY → DEPLOYING (first- boot only) → RUNNING, and handles restart / recycle / cluster- mode-transition events without dropping back into an ad-hoc respawn loop. State entries are `enterStateFn` dispatched from `transition`; async work runs on a per-attempt workCtx via startAsync so transition cancellation drains the work goroutine cleanly. The control socket at /run/k3s-supervisor.sock accepts `status`, `restart`, `stop`, `graph` commands; every operator-facing k3s- sctl invocation dispatches through the socket rather than touching flag files. initKubeclient wires kubeclient.Default at WAIT_K3S_READY's transition (test-safe when the kubeconfig file is absent — transition-table tests exploit this). components.DeployAll takes a retryCtx that main.go supplies as d.rootCtx, so BestEffort background retries in the deploy graph survive per-invocation workCtx while still dying on daemon shutdown. Restart-reason is a typed field carried through the STOPPING_K3S → RUNNING_HOOKS → STARTING_K3S transition so restart hooks (/etc/k3s-supervisor/hooks.d) fire once for operator-driven restarts and never for crash recovery. The RT operator's PerformanceProfile-change flow depends on this — CPU isolation / RDT partitioning need to be reconfigured between stop and start, not on every crash. README.md sketches the architecture, package boundaries, and the future-work list (bootstrap-cert fingerprint pinning, config-package retirement, and so on). Two behaviours in the loop are worth calling out. A cluster transition that fails retries in place instead of falling through to a state that assumes it succeeded — falling through leaves a node half-transitioned, which is not recoverable by pushing new configuration. And images are imported on every k3s start rather than only on first boot, because a restart can find the content store missing entries that the running cluster still references. The operator breakpoints are plumbed through here: a wait item placed in the FSM's path lets a test or an engineer hold a node at a chosen point and release it, which is how the multi-node transitions are inspected on real hardware. Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
One-shot tool that dials /run/k3s-supervisor.sock and prints the
reply. Reads until EOF so multi-line responses (graph) print
naturally alongside single-line ones (status / restart / stop).
K3S_SUPERVISOR_SOCKET overrides the path for tests.
Exit code follows the last line: non-zero when it starts with
ERR.
Subcommands:
- restart — graceful k3s restart (pre-restart hooks fire).
- status — one-line status report (state, k3s pid, uptime,
backoff, restart count, phase, transition-step).
- stop — stop the kube-init daemon.
- graph — resolved deploy-graph edges (one per line, each
tagged with its rule source).
Breakpoints are managed from the same CLI (break set / clear / list), so
holding a node at a chosen point does not require reaching into
/persist by hand.
Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
…helper pkg/kube/Dockerfile — the kube linuxkit container's entrypoint is /usr/bin/kube-init. Everything else the container needs to ship (k3s multi-call binary, longhorn nsmounter, CNI plugin staging, longhorn/kubevirt/CDI/multus tarball drops under /images/) is installed via the existing build.yml conventions. pkg/kube/k3s-control.sh — small wrapper that dispatches to k3s-sctl. Kept so external tooling (eden / eve exec kube …) that used to invoke this path continues to work; every subcommand it takes now goes over the control socket rather than touching flag files. pkg/kube/kubevip-delete.sh — minimal helper that KubeVIPDelete calls to remove leftover kube-vip state files after component deletion (leftover static-pod manifests, arp cache). Not part of the deploy graph; runs from the uninstall path. sqlite is included in the image because the datastore snapshot is taken with SQLite's VACUUM INTO, which is what makes an online-consistent copy possible without stopping k3s. Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
Eleven files under pkg/kube/ implemented the previous cluster-init flow in POSIX shell + kubectl. Every operation they exposed lives in pkg/kube/kube-init/ now — no external caller depends on them, and keeping them alongside the Go daemon would leave two divergent implementations of the same paths that operators could accidentally invoke. Deleted: cluster-init.sh cluster-update.sh cluster-utils.sh descheduler-utils.sh kubevirt-utils.sh longhorn-utils.sh pubsub.sh registration-utils.sh tie-breaker-utils.sh utils.sh vnc-proxy.sh lib/config.sh Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
…stDevices
The KubeVirt CRD requires spec.configuration.permittedHostDevices.
{pciHostDevices,mediatedDevices,usb} to be arrays. Bare keys
parse as `null` in YAML; server-side apply forwards `null` as an
explicit set-to-null, which the schema rejects with:
KubeVirt.kubevirt.io "kubevirt" is invalid:
spec.configuration.permittedHostDevices.mediatedDevices:
Invalid value: "null": ... must be of type array
Explicit `[]` satisfies the schema.
Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
The kube linuxkit package now ships a Go daemon in place of the shell path. Adds a kube-init build rule that DOCKER_GO-invokes `go build ./cmd/... .` inside pkg/kube/kube-init/, staged into pkg/kube/dist/ before the linuxkit pkg build. pillar-* targets are unchanged. Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
kube-init's vendor tree is upstream Go modules with well-known licenses; the SPDX generator's per-file scan blows out on the tree size without adding value. Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
Covers the rollback path behind the /var/lib snapshot: a node that has joined an HA cluster is converted back to standalone K3s by the controller withdrawing its EdgeNodeCluster config. Joining overwrites /var/lib with cluster-mode state, so converting back is a restore rather than a reconfiguration -- without a usable snapshot the node boots K3s against cluster-mode state and crash-loops, which is what this catches. Only one of the three nodes is converted, so the test also covers the cluster surviving a member leaving. Placed after TestThreeNodesCluster in the suite: it shares that test's device and network requirements so the VMs are reused, and it is ordered last because it destructively changes the topology the preceding test relies on. (cherry picked from commit e46c515) Phase 4 — that removing a member leaves the survivors a working cluster — is asserted from the node that actually reports it. Cluster-wide info is published by whichever node holds the eve-kube-stats-leader lease and actively unpublished by the others, so a non-leader survivor never reports a node list at all. It is also polled from the last reported state rather than awaited as a fresh message: publication is change-driven, so the removal is reported once, while the converted node is still rebooting, and then the survivors fall silent because nothing further changes. The conversion reboots the node by design — the pre-cluster /var/lib has to be restored before k3s starts, which cannot be done underneath a running k3s — so the test declares that reboot. EdgeDevice.ExpectReboots exists for exactly this: a reboot EVE performs as a required consequence of a configuration change, which the close-time audit would otherwise report as a crash. Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
andrewd-zededa
left a comment
andrewd-zededa
left a comment
There was a problem hiding this comment.
I have a few concerns and of varying priority. The kubevirt version mismatch needs to be fixed, the others also appear to be regressions.
| // KubeVirt v1.6.0 (5 images — operator + the 4 pods it spawns). | ||
| {Tarball: "/images/virt-operator.tar", Name: "quay.io/kubevirt/virt-operator", Tag: "v1.6.0"}, | ||
| {Tarball: "/images/virt-api.tar", Name: "quay.io/kubevirt/virt-api", Tag: "v1.6.0"}, | ||
| {Tarball: "/images/virt-controller.tar", Name: "quay.io/kubevirt/virt-controller", Tag: "v1.6.0"}, | ||
| {Tarball: "/images/virt-handler.tar", Name: "quay.io/kubevirt/virt-handler", Tag: "v1.6.0"}, | ||
| {Tarball: "/images/virt-launcher.tar", Name: "quay.io/kubevirt/virt-launcher", Tag: "v1.6.0"}, |
There was a problem hiding this comment.
images/images.go:88-92 imports and re-tags virt-operator/virt-api/virt-controller/virt-handler/virt-launcher as v1.6.0, but components/components.go:77 applies the v1.7.3 CR,
There was a problem hiding this comment.
Confirmed, and there is a third witness: update-component/expected_versions.yaml declares kubevirt: "v1.7.3", agreeing with kubevirt-operator.yaml (v1.7.3 at both the KUBEVIRT_VERSION env and the virt-operator image) and with components.go's kubevirtCRURL. Only images.go says v1.6.0.
Worth recording that it is inert today: nothing in the tree produces /images/*.tar — grepping the Makefile and all of pkg/ for virt-operator.tar hits only images.go:88 — and importUpstreamImageWith no-ops on a missing tarball, so every UpstreamImages entry short-circuits. The upstream-image-tar-rule that the doc comment at images.go:49 points at does not exist either. @rucoder — I assume the tarballs arrive with the follow-on PR that bundles images via EROFS? That is when this starts to bite: a v1.6.0 pre-load against a v1.7.3 operator is never hit, so the node pulls from quay anyway and the pre-load buys nothing.
Fixing it by hoisting the versions into a single versions package, so images.go, components.go and uninstall.go cannot drift again. One thing that shapes that: only k3s has a runtime override (k3s.version → KubeConfig.K3sVersion → getDesiredK3sVersion, which falls back to the compile-time const). kubevirt/cdi/longhorn/multus have no config item, so for those the const is the sole authority and a cross-check against expected_versions.yaml is the only guard available.
| // KubeVIPApply applies the Kube-VIP service account, configmap, and | ||
| // daemonset. Order matters: SA + CM before DS so pod startup finds | ||
| // its config. | ||
| func KubeVIPApply(ctx context.Context) error { |
There was a problem hiding this comment.
It looks like nothing calls this. The shell ran check_kubevip_lb once per main-loop iteration (cluster-init.sh:1088), diffing LBInterfaces[0] against /var/lib/kubevip-applied — that marker is also absent from the new code.
There was a problem hiding this comment.
Confirmed, for both this and KubeVIPDelete below: neither has a caller anywhere in the tree.
The reconciliation around them is missing too — grepping the PR's pkg/kube for kubevip-applied and LBInterfaces returns nothing, so neither the marker (cluster-init.sh:1080) nor the per-iteration diff (check_kubevip_lb, called at cluster-init.sh:1809) has a Go counterpart. And the shell invoked kubevip-apply.sh <iface> <cidr>, templated per interface, whereas KubeVIPApply applies three static files with no interface or CIDR input.
That makes it a feature port rather than a fix, so I am leaving it out of the batch I am pushing.
| // (daemonset first so pods drain before the SA/CM disappear). | ||
| // Per-file delete failures are warnings, not errors — uninstall | ||
| // proceeds across stale state. | ||
| func KubeVIPDelete(ctx context.Context) error { |
There was a problem hiding this comment.
It looks like nothing calls this. The shell ran check_kubevip_lb once per main-loop iteration (cluster-init.sh:1088), diffing LBInterfaces[0] against /var/lib/kubevip-applied
| // code would treat an already-converted device as un-converted | ||
| // (re-enabling KubeVirt install). Must run before any consumer | ||
| // of state.NativeKubernetesMode reads the marker. | ||
| if err := state.MigrateLegacyBaseK3sMode(); err != nil { |
There was a problem hiding this comment.
MigrateLegacyBaseK3sMode runs before /var/lib is mounted:
- the base-k3s-mode check always reads empty and the rename never happens
- workInit then reads NativeKubernetesMode (main.go:1559) as false and leaves installKubevirt = true, re-installing KubeVirt/CDI on a node that deliberately had them removed by the K3S_BASE conversion.
There was a problem hiding this comment.
Confirmed, both halves. state.MigrateLegacyBaseK3sMode() runs from main(), but /var/lib is bind-mounted by prereqs.MountKubeRoot (prereqs.go:502), reached only via RunAll (prereqs.go:155) from workInit (main.go:1541) — and both marker paths are under /var/lib (state/markers.go:58, :73). So the legacy check reads the container's own empty /var/lib and the rename never happens.
The consequence you describe follows: workInit reads the marker at main.go:1559, sees false, and leaves installKubevirt = true on a node the K3S_BASE conversion deliberately stripped.
Fixing by moving the migrate call into workInit, immediately after RunAll returns.
| return nil | ||
| } | ||
|
|
||
| func upgradeComponent(ctx context.Context, comp string) error { |
There was a problem hiding this comment.
CheckClusterComponents dropped the pre-upgrade readiness gate. The shell bailed out of the whole update pass if a component wasn't healthy on its current version.
The failure mode here can lead to components stuck mid version and then the health worker may be parked forever blocking k3s drift checks, node-label reapply, multus reapply, SR-IOV staging, CDI proxy patching.
There was a problem hiding this comment.
Confirmed. cluster-update.sh:196-200 gates each component on update_Component_CheckReady against its current version and bails the whole pass if it fails; upgradeComponent goes straight from componentIsInstalled to the version compare, so an unhealthy component gets upgraded on top of a bad state.
compCheckReady already exists, so it is a short change, but it introduces a new abort path and I would rather not guess at the shape: the shell returned non-zero specifically so the main loop would restart k3s. @rucoder — should the Go version return an error (matching that), or log and skip the component and carry on? I will add it once you have a preference.
| if appliedVersionGEQ(appliedVersion, KubeVersion) { | ||
| log.Printf("update: cluster components at applied=%s, target=%d — no update", | ||
| appliedVersion, KubeVersion) | ||
| return nil | ||
| } |
There was a problem hiding this comment.
The existing cluster-update.sh path had a "push KCUS status forward" path here, this can now lead to a system reporting BaseOsUpdating forever.
There was a problem hiding this comment.
Confirmed. cluster-update.sh:170-177 publishes longhorn completed when the existing KCUS reports Component==2 && Status==5; the Go early return does not, so a node that took a k3s update and rebooted reports BaseOsUpdating indefinitely.
The pieces are already there — kcus.Get(), and types.CompK3s/types.CompStatusCompleted (values 2 and 5, clusterupdatetypes.go:66, :20), used in exactly this shape by updateFailed() at version.go:76-82. Fixing it in the batch I am pushing.
|
I've pushed five commits to this branch (maintainer edit — nothing of @rucoder's history was rewritten, they append to
The last one is worth separate attention: a full Marking this draft while re-testing is in progress. The changes above want a full pass before this goes back in the merge queue, and I'd rather the draft state say so than have it look ready. The code itself is ready to look at. @andrewd-zededa — could I get a re-review? I've replied to all of your threads individually. Summarising where each landed:
|
|
Pushed
The test reads the manifest and asserts both are present, so a future manifest bump that removes either one fails the suite and the ordering assumption gets revisited deliberately instead of changing by accident. It only reads a file — no API server, no cluster. |
The KubeVirt image catalog pinned the five virt-* images at v1.6.0 while every other declaration of the version says v1.7.3: the CR URL applied by InstallKubeVirt, the operator manifest that names the image and sets KUBEVIRT_VERSION, and expected_versions.yaml, which drives the update-component drift check. A node pre-loading v1.6.0 would never satisfy a v1.7.3 pod spec, so the operator's pods would pull from quay anyway and the pre-load would buy nothing. The mismatch is currently latent: no rule produces the /images/*.tar tarballs yet, and a missing tarball is a logged no-op that defers to kubelet's pull-on-first-use. Correcting it now keeps the catalog honest for whenever the tarballs start shipping. Signed-off-by: eriknordmark <erik@zededa.com> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Component versions were spread across five files: the Longhorn config path baked the version into a filename, the KubeVirt CR URL inlined it, CDI and k3s each had their own const, and the pre-loaded image catalog repeated all of them once per image. Nothing tied those copies together, which is how the KubeVirt tags came to disagree with the operator manifest. Introduce a leaf versions package holding one const per component and derive every other site from it. All const expressions, so this is a compile-time rearrangement with no runtime change. The k3s const stays a default rather than an authority: the controller's k3s.version config item still overrides it at runtime. Two declarations remain outside the Go tree and cannot reference the package — expected_versions.yaml and the vendored kubevirt-operator manifest — so the package doc names them as sites a bump must also touch. Signed-off-by: eriknordmark <erik@zededa.com> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A device converted to native Kubernetes under an older EVE image records that in a marker whose name has since changed. The rename ran from main(), before the prereqs pass bind-mounts /var/lib, so it inspected the container's own empty directory, found nothing, and returned successfully having done nothing. The real marker was still under its legacy name once the mount appeared, so the daemon read the device as un-converted and reinstalled KubeVirt and CDI on a node that had deliberately had them removed. Run the migration inside the init work step instead, right after the prereqs pass returns and well before the marker is read. A failure now returns rather than calling log.Fatalf: at this point the daemon has a state machine that can back off and retry, which it did not when the call ran from main(). Signed-off-by: eriknordmark <erik@zededa.com> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
zedagent holds the device in BaseOsUpdating for as long as the cluster update status names an unfinished component. A k3s update reports its own step complete and then reboots; on the way back up the component pass finds everything already converged, returns early, and publishes nothing. The status stays parked on the k3s step and the device reports BaseOsUpdating indefinitely, which the shell avoided by advancing the status itself on that path. Advance it the same way: on the converged path, a status sitting on a completed k3s step is retroactively finished. Both halves of that condition matter — a status naming another component, or naming k3s at any stage short of completed, belongs to a pass still in flight and is left alone. Signed-off-by: eriknordmark <erik@zededa.com> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The kube-init daemon is its own Go module, and make test enumerates its modules explicitly, so none of kube-init's tests have ever been executed by the target — the package ships a test suite that no CI path runs. The container build only compiles the binary, and no workflow names the module either, so the gap is invisible short of noticing the absence in the log. Add it alongside the other per-module invocations. The suite needs ./... because kube-init's tests live in its subpackages rather than at the module root, unlike the newlog and edgeview entries above. Runs green under -race in about 75 seconds. Signed-off-by: eriknordmark <erik@zededa.com> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The multus manifest declares the NetworkAttachmentDefinition CRD and an instance of that kind in the same file, so applying it depends on ApplyFile waiting for establishment between documents. Nothing checked that the manifest still ships both halves: drop the instance or rename the CRD and the ordering silently stops mattering, or silently starts failing. Read the manifest and assert both. Signed-off-by: eriknordmark <erik@zededa.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Codespell rejects "unparseable" and "COPYs"; spell the first the way the rest of the module already does and the second as plain English. Revive's exported rule wants a doc comment to open with the identifier name followed by a verb, which the WaitFromContext and OnProgress comments did not. The waitSystemPodsReady paragraph had also drifted onto var OnProgress, so move it back to the function it describes. Signed-off-by: eriknordmark <erik@zededa.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A container app migrated to EVE-k never leaves BOOTING: its virt-launcher pod fails with ErrImageNeverPull for eve-external-boot-image. The import that should place that image into containerd looks for the tarball under /images/, but the kube package ships it at /etc/ (Dockerfile COPY), which is also where the shell cluster-init.sh this daemon replaces read it from. A missing tarball is treated as a silent no-op, so the import reports success, the image is never tagged, and since pillar references it with PullNever and it exists on no registry, the device cannot recover. Point the constant at /etc/, and have the restart-time re-import use the images package constants rather than repeating the literals, so the two sites cannot drift apart again. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Signed-off-by: eriknordmark <erik@zededa.com>
|
The external-boot-image import is a no-op as written: Since Seen on a kvm→k conversion run today. Everything else comes up healthy, which is what makes it easy to misread: k3s Ready, KubeVirt and Longhorn Running, StorageClass ready — only the app is stuck. while There are two sites, from two commits:
A fix on top of the current PR head, pointing the constant at One caveat for whoever applies it: the ~30 |
|
Pushed to this branch as |
|
@eriknordmark yes, I forgot to fix ExternalBootImage bug. it is fixed on a follow-up branch with EROFS. thanks for finding this! |
6 participants
Description
This is the first of two PRs that lay the groundwork for proper Real-Time workload support in EVE-K. It replaces
pkg/kube/cluster-init.sh(and its eleven shell library files) with a Go daemon atpkg/kube/kube-init/that supervises the k3s lifecycle through an event-driven finite state machine. The follow-up PR will add a new facility for managing the pre-built upstream image tarballs that live under
pkg/kube-images/— splitting that change out keeps each PR independently reviewable.Why now — the RT motivation
We could not integrate RT support cleanly on top of
cluster-init.shbecause the shell had no proper state machine for the k3s lifecycle. The shell loop body was a sequence of conditional touchpoints driven byflag files; it had no notion of states, no atomic transitions, and no hookable pre-restart phase.
That breaks the RT operator's restart flow specifically. RT support requires
plan-k3s-rt-restart.yaml(a Rancher SUC plan) to drive a clean k3s restart whenever aPerformanceProfileCR changes — for example, when CPU isolation, RDT cache partitioning, or kubelet RT scheduling arguments need new values. The restart MUST run pre-restart hooks between stop and start (write the new isolated-CPU set, repartition L3 cache, install the new kubelet args, etc.) before k3s comes back. In the shell, there was nowhere to plug those hooks in: the supervisor loopjust respawned k3s whenever it died, and SUC's force-kill was indistinguishable from a crash. We could not reliably distinguish "SUC asked us to cycle k3s for an RT config change" from "k3s segfaulted", which meant the RT-specific pre-restart work either got skipped or got duplicated across every crash recovery.
The Go FSM in this PR fixes that at the foundation. The state graph distinguishes crash recovery (
BACKOFF— no hooks) from operator-driven restart (STOPPING_K3S → RUNNING_HOOKS → STARTING_K3S— hooks fire). A typedrestartReasoncarries the cause through the transition so the right hooks run for the right reason./etc/k3s-supervisor/hooks.d/is owned by the FSM, and the RT operator can drop scripts in that directory knowing they will fire exactly once between stop and start. The control socket at/run/k3s-supervisor.socklets the RT operatortrigger a restart synchronously and observe progress (e.g.,
transition-step=…).This PR does not ship the RT operator manifests or any RT code path — those land in a later PR once the FSM foundation is in place. What this PR delivers is the foundation that unblocks the RT integration.
What's in this PR
The stack is 26 incremental commits scoped one Go package per commit, followed by 16 backport commits that bring across master-side bug fixes that landed after our branch point. Each commit builds, vets, and tests
green in isolation. Architectural overview and package map live in
pkg/kube/kube-init/README.md.Port (26 commits):
state— marker primitives, atomic-write, var-lib save/restore, rebootkubectlx— kubectl/ctr/crictl wrappers + apply-with-backoff classifierk3s— config rendering, token rotation, install + symlink rebuild,supervisor (Setpgid + orphan sweep + port wait), readiness probes,
node-password persist/restore across reboots
prereqs— kernel modules, cgroup, iSCSI, vault wait, EdgeNodeInfo wait,containerd
images— pre-packaged tarball import with kubelet pull-on-first-usefallback
vnc— VNC proxy + caller-PID watchdogtiebreaker— HA tie-breaker configurationdeploy— declarative deploy DAG runner withBestEffort+WaitReadycomponents— Multus, KubeVirt, CDI, Longhorn (deployedBestEffortso the FSM enters RUNNING without waiting up to 10 minutes for CR
convergence; steady-state reconciler picks up the final settling),
descheduler, debug RBAC, kube-vip, storage classes, SR-IOV manifest
mgmtproxy— cni0 anchor IP + CDI ImportProxy patch + containerd-launchenv injection so CRI image pulls route through pillar's cost-aware proxy
monitor— running-state watchers (containerd, kubeconfig sync, logrotation, cluster-config watch, user-override watch)
clustermode— single↔HA transitionsupdate— k3s + cluster-component upgrade flow (k3s download withSHA-256 verify, atomic install)
pubsubclient— Manager modelled onpkg/pillar/cmd/monitor/subscriptions.go;single map of label → Subscription, deferred Activate, one goroutine
pumping all topics through
pubsub.MultiChannelWatchedgenodeinfo—EdgeNodeInfosubscription (DeviceName/DeviceID),blocking
WaitForFirstfor the boot-time identity readkubeconfig—KubeConfigsubscription with K3sVersion accessorkcus—KubeClusterUpdateStatussubscription gating upgrade retriesencconfig—EdgeNodeClusterConfigsubscription (cluster shape,TieBreakerNodeID)
encstatus—EdgeNodeClusterStatussubscription withPresent()folding file-deletion and zero-UUID-delete-sentinel into a single check
main— FSM entry point (14 states, 27 events, control socket on/run/k3s-supervisor.sock)k3s-sctl— operator CLI client for the control socket (replaces theflag-file shell wrappers)
removes the 12 shell scripts the daemon replaces
The pubsub subscriber packages replace the JSON-file polling the original port used against
/run/zedagent/...and/persist/status/zedkube/..., eliminating ~400 lines of local JSON-shape duplication. The migration incidentally closes three latent bugs that came from JSON-tag mismatches the local*JSONstructs had against the canonical pillartypes: (1)KubeConfigwas taggedjson:"k3sVersion"while pillar publishesK3sVersion(no tag, capital K), so every device was silently using the compile-time k3s default; (2)KubeClusterUpdateStatus.DestinationKubeUpdateVersionwas locally typedstringwhile pillar usesuint32, so the failed-upgrade retry gate never fired; (3)EdgeNodeClusterStatuszero-UUID delete sentinel was missed byos.Stat-based detection inwaitForBootstrapServer.config.go:waitForBootstrapServerdials the bootstrap node's self-signed cert withInsecureSkipVerify: true; the cluster-UUID equality check below it is the actual authentication boundary, because we cannot have the cluster CA in the trust store before we have joined the cluster. The architectural justification is now load-bearing in the comment block; the line carries//nolint:gosecandlgtm[go/disabled-certificate-check]. A long-term fix pinning the bootstrap cert fingerprint viaEdgeNodeClusterStatusis noted inREADME.md's future-work list (requires pillar'szedkubeto publish the fingerprint, out of scope here).Master backports (16 commits, each cross-references its upstream SHA):
monitor: restart containerd if it dies—ea71f1b76update: bump KubeVersion + block downgrades—a67d55ce9k3s,monitor: treat zero-UUID ENC status as no-cluster—158981334prereqs: clean stale cpu_manager_state on boot—1927e2f28+6719f918ck3s: remove stale flannel.1 around supervisor cycles—2c417d5fek3s: symlink host-local into /usr/bin for k3s v1.34+—75fe3cd94state: relocate kube-save-var-lib under vault—647a03b2dcomponents: bump KubeVirt CR URL to v1.7.3—849b4cd7e+5041cb83cclustermode: stagger k3s startup by control-plane rank—be1537e68clustermode: clean stale etcd masterleases post single→cluster—d5664c079k3s: persist + restore node-password across reboots—91b9589c1mgmtproxy: containerd CRI env injection—7ec6f2a64mgmtproxy: cni0 anchor IP + CDI ImportProxy patch—7ec6f2a64components: SR-IOV manifest + binary staging—a2bb3a52ccomponents: stale-mount-cleanup daemon launcher—b036179damonitor,update: route component-presence checks through kubectlx— the kube container ships only the k3s multi-call binary; barekubectlis not on PATH, soupdate.checkComponentInstalledandmonitor.countReadyNodeswere silently failing on every tickTest coverage: the FSM transition graph is covered by a 60-row table-driven test in
pkg/kube/kube-init/main_test.go; per-package unit tests cover the non-trivial parsers, state machines, and recovery paths(supervisor process tree handling, deploy DAG wave ordering, hash-file parsing, version comparison, transition-marker corruption tolerance, pubsub-subscriber cache state machines, mgmtproxy env builder and NO_PROXY assembly, k3s node-password brownfield-first-boot/steady-state/missing-runtime/permissions, …). All packages pass
go test -race.v1 → v2: from shell-outs to typed Kubernetes APIs
Two rounds of code-review feedback drove architectural rewrites of two subsystems on top of the initial port. What ships here is v2.
Kubernetes and containerd in-process, no more shelling out. The v1 daemon still called
k3s kubectl,k3s ctr -n k8s.io, andk3s crictland parsed their stdout — the shell control-flow was rewritten in Go but the outward interface was unchanged. v2 replaces every one of those with a typed API call behindpkg/kube/kube-init/kubectlx/:Apply/ApplyFile/ApplyURL— dynamic client + server-side apply, with aRESTMapperdiscovery-cache reset + retry onNoKindMatchError(closes the CRD-then-CR race manifests that bundle a CRD and a CR of that kind in one file hit).WaitCRDEstablished,WaitDeploymentReady,WaitDaemonSetReady,WaitJobComplete,WaitForCondition—watchtools.UntilWithSyncon typed and dynamic informers.ContainerdClient— containerd v2 Go client (github.com/containerd/containerd/v2/client) for image import / exists / tag / list; thek8s.ionamespace is baked in.remotecommand.NewSPDYExecutorfor the one remainingkubectl execcallsite (the Longhornnsmountermarker).policy/v1.Eviction; cordon via node patch.Stdout-parsing helpers and their regexes are gone. Every operation is now a typed call with structured errors, so silent stdout-parse failures like the
checkComponentInstalledandcountReadyNodesregressions we backported (see "What's in this PR") cannot recur.Event-driven deploy DAG. v1 ran the deploy graph as a wave-based scheduler: every component in wave N had to complete before any wave N+1 started, so a fast component was blocked by an unrelated slow peer, and a
BestEffortcomponent that failed apply was never revisited. v2 replaces this with a work-queue scheduler inpkg/kube/kube-init/deploy/: components declare typed dependencies and start as soon as their predecessors are done.BestEffortcomponents (KubeVirt, CDI, Longhorn) that fail apply are handed to a background retry loop with exponential backoff, scoped to the daemon's root context — a transient failure at deploy time no longer leaves the component permanently un-reconciled.Informer-driven readiness.
WaitLonghornReadyuses per-call filtered informer factories on DaemonSets inlonghorn-systemand thelonghorn.io/NodeCR — no periodic polling. Times out cleanly after 10 minutes.Consolidation. Duplicated
schema.GroupVersionResourceliterals, namespace strings, and error/patch idioms (IgnoreNotFound,BuildMergeLabelPatch,IsNodeReady) live inpkg/kube/kube-init/kubectlx/gvrs.go.images.ImportAllshares one containerd client across the batch; the monitor caches its containerd client across health-tick invocations rather than re-dialling.v2 → v3: validated on a three-node cluster
v3 is what came back from running the daemon on real hardware. The full
evetestnode-cluster suite — single node, three-node HA, and a new cluster→single conversion test — passes on a three-node libvirt cluster (TestSingleNodeCluster911s,TestThreeNodesCluster1060s,TestClusterToSingleConversion984s). Everything below is a behaviour the suite forced out; each is folded into the subsystem commit it belongs to rather than appearing as a fix on top.A kubeconfig must not outlive the CA it describes. Joining a cluster retires the node's whole PKI, but the kubeconfigs derived from it were being left on disk. Since the readiness wait only ever waited for the file to be present, a leftover satisfied it instantly and the client was built against an authority that no longer existed — so every call failed the TLS handshake against a perfectly healthy apiserver. The symptom is indistinguishable from an unresponsive node: on
edge-dev2the pre-join kubeconfig was adopted at 12:48:55, k3s wrote the real one at 12:48:57, and the join then burned 5m13s before the watchdog rebooted a node that had joined correctly. The same stale copy is what pillar reads (kubeapi.EVEkubeConfigFileis/run/.kube/k3s/k3s.yaml), sozedkube's node drain-and-delete failed the same handshake and left the departed node in the survivors' cluster — one cause behind two unrelated-looking failures. The kubeconfigs are now retired together with the TLS material, and the readiness wait rebuilds its client if the file rotates underneath it anyway.Readiness says why, not just "not yet". Every distinct cause of "no answer from the apiserver" previously presented as the same hanging call. The wait now records what the client is aimed at and the fingerprint of the CA it trusts, then separates the layers on each failure: whether anything is accepting on the API port, what
/readyz?verbosesays is outstanding (etcd, informer sync, a poststarthook), and whether the kubeconfig moved. Every call is individually bounded —rest.Config.Timeoutis deliberately left unset because it would cut informer watches, so a per-call context is what stops one unanswered request from consuming the whole readiness budget. Pod readiness is counted with a field selector for the local node, so an unhealthy or still-joining peer cannot hold up a node that is itself ready.Cluster-scoped objects are not collateral for a single node's reconfiguration.
NetworkAttachmentDefinitionis a cluster-scoped CRD, so tearing it down to re-address one node takes the definition away from every other node in the cluster; the node re-applies its own configuration and restarts the Multus DaemonSet instead, and a failure there is logged rather than failing a transition that has already passed the point of no return. CRs are also gated on their CRD being genuinely established — a terminating CRD still reportsEstablished=True, so a deletion timestamp disqualifies it.The join watchdog is driven by progress, not by a deadline. How long a join takes is unknowable: it covers a k3s restart, an image import, registration and every kube-system pod, so any fixed budget is a guess that some slower device will fail. How long a join may make no progress at all is stable, so every advance to a stage this join has not reached before kicks the timer — a slow-but-advancing join is never rebooted, while a wedged one trips promptly. A crash loop does not count as progress, because it revisits stages rather than advancing them.
A withdrawal is observed in every state. A controller can withdraw a node's cluster configuration at any moment, including seconds after a join completes while the node is still working through readiness, so the cluster-config watch is no longer confined to
RUNNING. A zero-UUID or half-written status is treated as "no live cluster" only when it really is one, so a status that arrives late cannot be mistaken for a withdrawal.The
/var/libsnapshot and its rollback are now their own commit, with the reasoning in one place: what is captured is node identity rather than the ~419MB of re-derivable binaries, the datastore is captured with SQLite'sVACUUM INTOso an online-consistent copy needs nothing stopped, and because the restore is an overlay copy the cluster-mode leftovers have to be removed explicitly —server/db/etcd, thestate.db-wal/-shmsidecars (SQLite replays a stray-walas the restored database's own journal and k3s then dies with "database disk image is malformed" on every start), and Longhorn replicas belonging to volumes the restored cluster knows nothing about.The node publishes what it is actually doing. FSM state and phase, derived cluster membership, the current transition step, whether the join watchdog is armed, and the last error, so pillar and the controller can read the daemon's own view instead of inferring it from files.
Commit history. The port is now one commit per subsystem, each carrying its final content, so there are no fix-ups to read through — 27 commits. The pre-packaged-image (EROFS) work that builds on top of this stays on a separate branch and will come as its own PR.
How to test and validate this PR
PR was validated e2e against evetest
make evetest NAME=TestNodeClusterSuitemake run-live HV=kor flash to hardware/persist/kubelog/k3s-install.log— the daemon's structuredlog replaces the shell's noisier output
INIT → INSTALLING → STARTING_CTRD → CONFIGURING → STARTING_K3S → IMPORTING → WAIT_K3S_READY → DEPLOYING → RUNNINGkubectl get nodesshows the node Readyregression in pillar's GlobalConfig handling)
clustermoderunner steps fire in order (visible viak3s-sctl statustransition-step=…)cleanup_stale_masterleasesremoves the pre-conversionsingle-node entry from etcd:
rank * 25 sbefore launching k3s; checkkube-init log for
STARTING_K3S: applying staggered delaysignals cluster→single (the regression
158981334addressed)KubeConfigwith a non-defaultK3sVersionupdate.getDesiredK3sVersionpicks up the override(previously silently ignored due to the JSON-tag mismatch above)
Changelog notes
pkg/kube/cluster-init.shand its eleven shell library files arereplaced by a Go daemon (
/usr/bin/kube-init) that supervises the k3slifecycle through an event-driven finite state machine. The control
socket at
/run/k3s-supervisor.sockexposesstatus/restart/stopcommands; the operator-facing
k3s-start/k3s-stop/k3s-statusaliases now dispatch through this socket rather than touching flag
files. No user-visible behaviour change in the steady state. Recovery
from k3s crashes, containerd death, single↔HA cluster transitions, and
controller-driven cluster delete is now bounded, instrumented, and
unblocks RT support landing in a later PR. The k3s version override and
failed-upgrade retry gate now actually fire — both were silently broken
in the file-polling path the pubsub migration replaces.
PR Backports
Checklist
I've provided a proper description
I've added the proper documentation (
pkg/kube/kube-init/README.md)I've tested my PR on amd64 device
I've tested my PR on arm64 device
I've written the test verification instructions
I've set the proper labels to this PR
I've checked the boxes above, or I've provided a good reason why I didn't
check them.