zedkube: detect and recover stuck kubelet volume mounts by eriknordmark · Pull Request #6197 · lf-edge/eve

eriknordmark · 2026-07-22T01:48:49Z

Description

In cluster mode a Longhorn volume can be attached to a node (block device
present under /dev/longhorn, its VolumeAttachment reporting Attached)
while kubelet never issues NodeStage for it. The consuming pod — a CDI
importer or a virt-launcher — then sits in ContainerCreating/Init
indefinitely with no FailedMount event, and the app never runs. The stall
is in kubelet's volume manager (shipped via k3s) — not Longhorn, CDI or EVE —
and only a fresh kubelet clears it. Nothing on the device detected or recovered
from this before.

This adds a zedkube check (checkStuckVolumeMount) that flags a pod left
Pending past a threshold on this node whose Longhorn PVC is
attached-but-unmounted and shows no container/init image-pull or crash error
(those are excluded as different failure modes). When it finds one it restarts
k3s so kubelet comes back with a fresh volume manager: it resets
cluster-init.sh's restart backoff (K3S_MANUAL_START_FLAG) and sends SIGTERM
to the k3s server process, which the supervisor then relaunches. This is
possible because pillar runs in the host PID namespace and shares the /run
bind with the kube container.

Recovery is rate-limited per episode — node-scoped bounded attempts
(stuckMountMaxRecover) plus a cooldown (stuckMountSuppressWindow) — so the
detector cannot thrash k3s restarts faster than kubelet can recover. Every
restart logs a distinctive, greppable MOUNT-WEDGE-RECOVERY marker at Warn
level with the attempt count, the signaled PIDs, and the wedged pod/PV. A
build-time flag (stuckMountDryRun) disables the action and leaves only
detection logging.

The dry-run form of this check was field-validated first: on a lab EVE-k device
it fired on a real CDI upload-pod wedge (attach succeeded, device present, no
FailedMount, ContainerCreating ~10m) with no false positives, confirming the
detection signature before the disruptive restart was enabled here.

How to test and validate this PR

On an EVE-k device, when a pod is stuck ContainerCreating on an
attached-but-unmounted Longhorn volume for more than ~5 minutes, zedkube
restarts k3s and logs:

MOUNT-WEDGE-RECOVERY: restarting kubelet/k3s to clear the volume-mount wedge (attempt N/3): pod=<name> pv=<pv> attached+device-present but unmounted, Pending <dur>
MOUNT-WEDGE-RECOVERY: sent SIGTERM to k3s server pid(s) [<pids>]; cluster-init.sh will relaunch. attempt N/3, wedge: ...

k3s/kubelet come back within ~1 minute and the previously wedged pod proceeds
to mount and start. With no wedge present the check logs nothing and resets its
per-episode counter. Grep the device newlog for MOUNT-WEDGE-RECOVERY to see
every restart.

Changelog notes

No user-facing changes.

PR Backports

16.0-stable: No (revisit once soaked on master).
14.5-stable: No.
13.4-stable: No.

Checklist

I've provided a proper description
I've added the proper documentation
I've tested my PR on amd64 device (dry-run detection field-validated; recovery build-verified)
I've tested my PR on arm64 device
I've written the test verification instructions
I've set the proper labels to this PR

codecov · 2026-07-22T05:07:19Z

Codecov Report

❌ Patch coverage is 0% with 117 lines in your changes missing coverage. Please review.
✅ Project coverage is 23.10%. Comparing base (485098c) to head (53333f5).
⚠️ Report is 72 commits behind head on master.

Files with missing lines	Patch %	Lines
pkg/pillar/cmd/zedkube/stuckmount.go	0.00%	115 Missing ⚠️
pkg/pillar/cmd/zedkube/pendingvmi.go	0.00%	1 Missing ⚠️
pkg/pillar/cmd/zedkube/zedkube.go	0.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #6197      +/-   ##
==========================================
+ Coverage   22.76%   23.10%   +0.34%     
==========================================
  Files         509      520      +11     
  Lines       93274    95106    +1832     
==========================================
+ Hits        21230    21977     +747     
- Misses      70313    71214     +901     
- Partials     1731     1915     +184

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

naiming-zededa · 2026-07-22T22:46:06Z

+	now := time.Now()
+	var wedged []string
+	for i := range pods.Items {
+		if desc, ok := z.podMountWedge(pods.Items[i], now); ok {


Should we here check only if the pod is scheduled on this node?

It is filtered, just not here — podMountWedge early-returns unless p.Spec.NodeName == z.nodeName, so pods scheduled elsewhere are skipped. But you're right that it belongs on the LIST rather than in the loop; I'll fold it into the field selector below so we stop pulling every pod in the namespace.

Done in bff6975 — the node filter is now on the LIST itself, so pods on other nodes never come back over the API. The p.Spec.NodeName != z.nodeName check stays in podMountWedge as a guard.

naiming-zededa · 2026-07-22T22:59:44Z

+	}
+	ctx, cancel := context.WithTimeout(context.Background(), kubeAPITimeout)
+	defer cancel()
+	pods, err := clientset.CoreV1().Pods(kubeapi.EVEKubeNameSpace).List(ctx, metav1.ListOptions{})


maybe using:

podList, err := clientset.CoreV1().Pods(kubeapi.EVEKubeNameSpace).List(ctx, metav1.ListOptions{ FieldSelector: "status.phase!=Running", })

Agreed, and this matches what drain.go and sriov_devplugin.go already do (FieldSelector: "spec.nodeName=" + nodeName) — this LIST is the outlier. I'll use spec.nodeName=<node>,status.phase=Pending: status.phase=Pending rather than !=Running, since the detector only ever considers Pending pods. The client-side phase/node checks stay in podMountWedge as a guard.

Done in bff6975:

pods, err := clientset.CoreV1().Pods(kubeapi.EVEKubeNameSpace).List(ctx, metav1.ListOptions{ FieldSelector: "spec.nodeName=" + z.nodeName + ",status.phase=" + string(corev1.PodPending), })

gofmt, go build/go vet -tags kubevirt,k and the zedkube package tests pass.

In cluster mode a Longhorn volume can be attached to the node (block device present, VolumeAttachment reporting Attached) while kubelet never issues NodeStage for it, so the consuming pod sits in ContainerCreating /Init forever with no FailedMount event and the app never runs. The stall is in kubelet's volume manager (shipped via k3s), not Longhorn, CDI or EVE, and only a fresh kubelet clears it. Nothing detected or recovered from this before. Add a zedkube check that flags a pod left Pending past a threshold on this node whose Longhorn PVC is attached-but-unmounted and shows no container/init startup error (image pull, crash, or missing secret/config). When one is found it restarts k3s: it resets cluster-init.sh's restart backoff and sends SIGTERM to the k3s server process, which the supervisor then relaunches, so kubelet returns with a fresh volume manager. Recovery is rate-limited per episode (bounded attempts plus a cooldown) and every restart logs a distinctive MOUNT-WEDGE-RECOVERY marker. A build-time flag disables the action and leaves only detection logging. Signed-off-by: eriknordmark <erik@zededa.com> Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

This branch merged lf-edge#6197 at head 7d3a45b; the PR has since been amended to 53333f5, adding CreateContainerConfigError to the container-error exclusion the mount-wedge detector consults. Without it a CDI upload pod stalled on a missing per-upload secret is misread as a kubelet mount wedge and the detector restarts k3s three times to no effect - observed in resize soak 20260724-163249 iter-012. The README SHA is updated to match. Signed-off-by: eriknordmark <erik@zededa.com> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

eriknordmark · 2026-07-28T19:41:13Z

Field data: 12 recoveries over ~2.5 days, 10 of them on orphaned CDI upload pods

Ran this branch's detector on a single-node EVE-k lab device (sc-sm-e300-10, 1 core) under a
continuous ztest suite from 07-26 to 07-28, on an integration image that carries this PR at its
current head 53333f56b (so including the CreateContainerConfigError exclusion). Every fire
below sent SIGTERM successfully and cluster-init relaunched k3s — the recovery path itself works,
and no test failure was attributable to any of them; the ~50–60 s of apiserver unavailability
per restart never broke an overlapping test.

The interesting result is what it fired on:

#	time (PDT)	pod	attempt	class
1	07-26 04:33	`cdi-upload-d9e60f0b-…-pvc-0`	1/3	orphan
2	07-26 21:44	`cdi-upload-e92c8e9f-…-pvc-0`	1/3	orphan
3	07-27 07:12	`cdi-upload-e92c8e9f-…-pvc-0`	1/3	orphan
4	07-27 09:28	`cdi-upload-a6f587c5-…-pvc-0`	1/3	live in-test CDI pod
5	07-27 21:58	`cdi-upload-d9e60f0b-…-pvc-0`	1/3	orphan
6	07-28 01:30	`cdi-upload-e92c8e9f-…-pvc-0`	1/3	orphan
7	07-28 04:42	`virt-launcher-ztest-…t04-31-404-…`	1/3	live app VM launcher
8	07-28 06:38	`cdi-upload-d9e60f0b-…-pvc-0`	1/3	orphan
9	07-28 06:53	`cdi-upload-d9e60f0b-…-pvc-0`	2/3	orphan
10	07-28 07:08	`cdi-upload-d9e60f0b-…-pvc-0`	3/3	orphan → cap reached
11	07-28 08:05	`cdi-upload-d9e60f0b-…-pvc-0`	1/3	orphan, counter reset by reboot
12	07-28 10:58	`cdi-upload-e92c8e9f-…-pvc-0`	1/3	orphan

2 of 12 were doing real work. #7 is the best evidence for the detector's value: a
virt-launcher pod — a live app's VM blocked on its volume, not a CDI importer and not an
orphan. #4 is a live in-test CDI upload pod. Both cleared after the restart and both spanning
tests passed.

10 of 12 were futile. Two PVCs (d9e60f0b-…-pvc-0, e92c8e9f-…-pvc-0) had been Bound for
4 d 23 h and 5 d 22 h with no owning DataVolume (kubectl get dv -n eve-kube-app empty) —
the CDI teardown-race orphans. A k3s restart cannot help these: nothing recreates a deleted
DataVolume's per-upload secret. They kept qualifying because their upload pods are recreated on
every boot and pass through ContainerCreating/Pending before reaching
CreateContainerConfigError, so the exclusion added in this PR doesn't catch them in that
window.

The per-episode cap does not bound total restarts

Rows 8–11 are the part worth acting on. The orphan walked 1/3 → 2/3 → 3/3 fifteen minutes apart,
correctly hit stuckMountMaxRecover and logged

checkStuckVolumeMount: 1 wedged pod(s) after 3 recovery attempts; giving up until they clear

— and then a device reboot reset stuckMountRecoverCount (it is in-memory zedkube state) and it
fired again at 1/3. This ztest suite issues controller-commanded reboots every ~2 h, so the
steady state was 3 gratuitous k3s restarts per boot, indefinitely. The safeguards behave
exactly as designed within an episode; they just can't see across pillar restarts.

Suggestion

This is direct support for the orphan-reaping follow-up already noted on this PR. Concretely,
before restarting k3s in recoverKubeletMountWedge, distinguish the two cases:

Orphan — the pod is a cdi-upload-* pod whose PVC has no owning DataVolume (or whose
per-upload Secret is gone). Remedy is to delete the pod + PVC (and clear the scratch PVC
finalizer), not to restart k3s. Restarting is guaranteed not to fix it and costs a
control-plane outage each time.
True kubelet wedge — as in Fixing broken links - hopefully these are the right #4 and remove adapters which are not present on device #7. Restart k3s as now.

A cheap alternative that would have suppressed all 10 futile restarts without new logic: treat
"the same pod+PV has already consumed a full episode" as sticky across pillar restarts (persist
stuckMountRecoverCount keyed by pod UID, or refuse to re-arm for a pod whose PVC has no
DataVolume).

Verification

Deleting the two orphan PVCs and their upload pods on 07-28 12:35 PDT left eve-kube-app with
only the live workload's PVC. If the fires stop after that, it confirms the 10 orphan-driven
recoveries were entirely attributable to those two stale objects — I'll follow up with the
observed rate. (Separately, this device also had a leaked Longhorn support-bundle DaemonSet
gating all volume creation; that is unrelated to this PR and is filed as #6230.)

eriknordmark · 2026-07-28T21:47:30Z

Follow-up: confirmed — the 10 futile recoveries were entirely the two stale PVCs

Deleted both orphan PVCs (+ their scratch PVCs and upload pods) at 07-28 12:35 PDT, leaving
eve-kube-app with only live workload objects. Since then, with the same suite running on the
same image:

Zero MOUNT-WEDGE-RECOVERY fires in the 2h20m since deletion (last was 10:58, i.e. ~1.6 h
before it), versus a prior steady state of one every 15 min inside an episode and a fresh
episode after every boot.
Crucially, a controller reboot has occurred since (20:16:26Z = 13:16 PDT; device now up
1 h 29 m). That reboot is precisely what used to re-arm the orphan — it reset the in-memory
stuckMountRecoverCount and produced a fresh 1/3 within 19–50 min, every time. This boot is
1.5 h old with nothing.
7 further ztest invocations completed in that window, all PASS.

So the 10/12 attribution holds: those recoveries were caused solely by two Bound PVCs with no
owning DataVolume, and nothing about the wedge detection logic itself. The 2 remaining fires
(cdi-upload-a6f587c5, and the virt-launcher pod) stand as genuine post-attach wedges that the
restart cleared.

This doesn't change the suggestion — it strengthens it. The detector's recovery is sound; what it
lacks is a way to tell "volume is attached but the consumer can never start because its
DataVolume/Secret is gone" from "kubelet failed to stage an otherwise-healthy volume". On a
long-lived device a single stale CDI object is enough to generate unbounded control-plane
restarts, and an operator has no signal pointing at the real cause.

This branch merged lf-edge#6197 at head 7d3a45b; the PR has since been amended to 53333f5, adding CreateContainerConfigError to the container-error exclusion the mount-wedge detector consults. Without it a CDI upload pod stalled on a missing per-upload secret is misread as a kubelet mount wedge and the detector restarts k3s three times to no effect - observed in resize soak 20260724-163249 iter-012. The README SHA is updated to match. Signed-off-by: eriknordmark <erik@zededa.com> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> (cherry picked from commit d9bb1f8)

This branch merged lf-edge#6197 at head 7d3a45b; the PR has since been amended to 53333f5, adding CreateContainerConfigError to the container-error exclusion the mount-wedge detector consults. Without it a CDI upload pod stalled on a missing per-upload secret is misread as a kubelet mount wedge and the detector restarts k3s three times to no effect - observed in resize soak 20260724-163249 iter-012. The README SHA is updated to match. Signed-off-by: eriknordmark <erik@zededa.com> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> (cherry picked from commit 4219af7)

The mount-wedge detector listed every pod in the EVE namespace and discarded the irrelevant ones in the loop, so on a multi-node cluster it pulled other nodes' pods over the API on every tick. Only a Pending pod scheduled on this node can exhibit the wedge, so ask the apiserver for exactly that set, matching how the drain and SR-IOV device-plugin paths already restrict their LISTs. The per-pod node and phase checks stay as a guard. Signed-off-by: eriknordmark <erik@zededa.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: eriknordmark <erik@zededa.com>

github-actions Bot requested review from andrewd-zededa, naiming-zededa and zedi-pramodh July 22, 2026 01:49

eriknordmark mentioned this pull request Jul 22, 2026

pillar: diagnose stuck EVE-k app bring-up (CDI + readiness gate) #6200

Merged

6 tasks

eriknordmark force-pushed the kubelet-mount-wedge-detector branch from a366dea to bd56b88 Compare July 22, 2026 20:31

eriknordmark changed the title ~~zedkube: detect stuck kubelet volume mounts~~ zedkube: detect and recover stuck kubelet volume mounts Jul 22, 2026

naiming-zededa reviewed Jul 22, 2026

View reviewed changes

eriknordmark force-pushed the kubelet-mount-wedge-detector branch from bd56b88 to 2deedcc Compare July 23, 2026 17:21

github-actions Bot requested a review from naiming-zededa July 23, 2026 17:22

eriknordmark force-pushed the kubelet-mount-wedge-detector branch from 2deedcc to 7d3a45b Compare July 23, 2026 22:18

eriknordmark force-pushed the kubelet-mount-wedge-detector branch from 7d3a45b to 53333f5 Compare July 24, 2026 06:27

eriknordmark marked this pull request as ready for review August 4, 2026 12:38

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

zedkube: detect and recover stuck kubelet volume mounts - #6197

zedkube: detect and recover stuck kubelet volume mounts#6197
eriknordmark wants to merge 2 commits into
lf-edge:masterfrom
eriknordmark:kubelet-mount-wedge-detector

eriknordmark commented Jul 22, 2026 •

edited

Loading

Uh oh!

codecov Bot commented Jul 22, 2026 •

edited

Loading

Uh oh!

naiming-zededa Jul 22, 2026

Uh oh!

eriknordmark Aug 4, 2026

Uh oh!

eriknordmark Aug 4, 2026

Uh oh!

naiming-zededa Jul 22, 2026

Uh oh!

eriknordmark Aug 4, 2026

Uh oh!

eriknordmark Aug 4, 2026

Uh oh!

eriknordmark commented Jul 28, 2026

Uh oh!

eriknordmark commented Jul 28, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

eriknordmark commented Jul 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

How to test and validate this PR

Changelog notes

PR Backports

Checklist

Uh oh!

codecov Bot commented Jul 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

naiming-zededa Jul 22, 2026

Choose a reason for hiding this comment

Uh oh!

eriknordmark Aug 4, 2026

Choose a reason for hiding this comment

Uh oh!

eriknordmark Aug 4, 2026

Choose a reason for hiding this comment

Uh oh!

naiming-zededa Jul 22, 2026

Choose a reason for hiding this comment

Uh oh!

eriknordmark Aug 4, 2026

Choose a reason for hiding this comment

Uh oh!

eriknordmark Aug 4, 2026

Choose a reason for hiding this comment

Uh oh!

eriknordmark commented Jul 28, 2026

Field data: 12 recoveries over ~2.5 days, 10 of them on orphaned CDI upload pods

The per-episode cap does not bound total restarts

Suggestion

Verification

Uh oh!

eriknordmark commented Jul 28, 2026

Follow-up: confirmed — the 10 futile recoveries were entirely the two stale PVCs

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

eriknordmark commented Jul 22, 2026 •

edited

Loading

codecov Bot commented Jul 22, 2026 •

edited

Loading