EVE-k: fix stray-daemonset storage block and report readiness truthfully

[eriknordmark]

Description

Two independent robustness fixes for how an EVE-k node decides, and reports, whether cluster storage is usable. Both found while running kvm→EVE-k conversion tests.

A stray DaemonSet in longhorn-system blocks every volume on the node. checkLonghornReady required every DaemonSet in that namespace to have a Running-and-Ready pod here, using its list of expected ones (longhorn-manager, longhorn-csi-plugin, engine-image) only to confirm those three exist. Anything else sharing the namespace that never becomes ready therefore gates storage permanently, with app volumes stuck in CREATING_VOLUME as the only visible symptom. EVE's own collect-info reaches this by leaving a SupportBundle DaemonSet behind — see #6230, where it caused 7/7 test failures on a device and 16 consecutive passes once the object was removed. Now only Longhorn's own DaemonSets gate readiness, which also drops a latent false failure where a DaemonSet whose node selector excludes this node counted as missing.

VolumeMgrStatus.Initialized was set regardless of whether storage came up. volumemgr waits up to 40 minutes for cluster storage and then published success unconditionally, so a node whose Longhorn or CDI never converged was indistinguishable from a healthy one outside volumemgr's own log. Initialized now reflects the outcome, and a new UnmetCondition carries the outstanding gate — reusing the sub-condition the kubernetes wait already computes and currently discards. Non-EVE-k nodes have no such gate and are Initialized from the start, as before. Volumes are unaffected either way: they are gated separately and retry until storage appears.

The point of reporting the outcome is access from outside volumemgr. The status lands in /run/volumemgr/VolumeMgrStatus/volumemgr.json, so a test can ssh in and assert on it and collect-info captures it in the bundle. diag now subscribes as well and prints a warning naming the outstanding gate, so an operator at the console can see why applications are waiting for storage rather than having to correlate agent logs.

Deliberately not fixed here: volumemgr publishes nothing at all during those 40 minutes, because pubVolumeMgrStatus and the inputs to getRemainingDiskSpace are created after the waits — so "converging" cannot be told from "wedged". That needs a startup reorder, and any early publish must carry a real RemainingSpace, since nodeagent treats 0 as grounds for MaintenanceMode. Separate PR.

How to test and validate this PR

Every change here has a unit test that fails without it, each confirmed by reverting the change and re-running:

go test -tags k -run TestCheckLonghornReady ./kubeapi/
go test -tags k -run TestVolumeMgrStatus ./cmd/volumemgr/
go test -run TestDiag ./cmd/diag/

TestCheckLonghornReadyIgnoresStrayDaemonset fails with the production symptom (longhorn daemonset:longhorn-support-bundle-agent missing on this node) while the other three keep passing. TestVolumeMgrStatusReportsUnmetCondition fails if the reported readiness goes back to being a constant. TestDiagReportsUnmetStorageCondition fails without the console warning; the two tests beside it pin the quiet cases — storage usable, and volumemgr not having published yet.

Testing the reported readiness needed one production change: generateAndPublishVolumeMgrStatus computes the remaining disk space from /persist before publishing, which no test environment has, so the function returned before publishing anything. The status construction is now a separate method and the publishing path is otherwise untouched.

On an EVE-k device, first fix: with Longhorn healthy, create any DaemonSet in longhorn-system that cannot get a ready pod, then deploy an app with a volume. Before this change the app sits in CREATING_VOLUME and volumemgr logs longhorn not ready: longhorn daemonset:<name> missing on this node; after it, the app deploys. Through the real path instead: run collect-info, confirm the leftover SupportBundle DaemonSet, then deploy.

Second fix: on a node where cluster storage never comes up, /run/volumemgr/VolumeMgrStatus/volumemgr.json should show "Initialized": false with "UnmetCondition" naming the gate, where it previously showed true. On a healthy node, and on any non-EVE-k device, true with an empty UnmetCondition. In the failing case the console — and /run/diag.out — additionally carries WARNING: cluster storage did not become ready at startup: <condition>, and nothing extra in the healthy case.

A new evetest package covers the healthy path:

make evetest NAME=TestDiagOutput

Nothing tested diag's output before — it reaches no EVE API message, so a lost section on a healthy device went unnoticed. The test deploys an application and reads the summary the way a consumer does, through GET /eve/v1/diag on the metadata server, requiring the device to report itself online and connected, its management port up, and the deployed application running. It is also the first coverage of msrv's diag handler. Unlike the unit tests above it is general coverage rather than a regression test for this change: it asserts the healthy path, which looks the same before and after.

Changelog notes

Fixed a leftover DaemonSet in the longhorn-system namespace — for example one left behind by collect-info — preventing a Kubernetes-enabled node from creating any application volume, leaving applications waiting for storage. The device now also reports accurately whether its cluster storage became usable, instead of always reporting success.

PR Backports

The template's list does not yet include 17.0-stable, which is where these belong.

• 17.0-stable: To be backported.
• 16.0-stable: No.
• 14.5-stable: No.
• 13.4-stable: No.

Checklist

• I've provided a proper description
• I've added the proper documentation
• I've tested my PR on amd64 device
• I've tested my PR on arm64 device
• I've written the test verification instructions
• I've set the proper labels to this PR

And the last but not least:

• I've checked the boxes above, or I've provided a good reason why I didn't
  check them.

pkg/pillar/docs/volumemgr.md now describes VolumeMgrStatus and its three fields,
which it did not mention at all; the fields themselves are documented in the type.

The second fix is verified on an amd64 EVE-k device (QEMU under Eden) in both
directions across a seven-topology conversion sweep. Where cluster storage came up,
volumemgr published "Initialized":true with an empty UnmetCondition; on a run
where it did not, the same status carried
"UnmetCondition":"timed out waiting for the condition (last unmet condition: longhorn not ready: daemonset:longhorn-manager not running on node)" with
"Initialized":false — a state that was reported as success before this change, and
that took device-log archaeology to identify. The first fix is covered by the unit
tests; its device reproduction needs a stray DaemonSet planted by hand, and the steps
are above. TestDiagOutput passes against an amd64 kvm device built from this branch,
retrieving the summary below through the metadata endpoint — note the header, which shows
the new subscription driving the print, and the absence of any cluster-storage warning on
a node whose storage is usable from the start:

INFO: updated diag information at 2026-08-02T12:16:28.911588547Z due to Appinstance
WARNING: device: online attest: Complete vault: DISABLED unlock:no-tpm pcr: DISABLED
INFO: applications: 0 starting, 1 running
INFO: Summary: Connected to EV Controller and onboarded
INFO: Using highest priority DevicePortConfig key zedagent
INFO: Have 1 total ports. 1 ports should be connected to EV controller
INFO: Port eth0: Mac: da:39:78:d0:19:f2 link: up use: mgmt 172.20.20.123
INFO: App diag-reader uuid 945c1a47-1abc-4915-bdd2-707612475b06 state RUNNING

Not run on arm64: everything here is architecture-independent Go in pillar.

[codecov]

Codecov Report

❌ Patch coverage is 25.00000% with 33 lines in your changes missing coverage. Please review.
✅ Project coverage is 23.36%. Comparing base (5d69266) to head (dc8490c).
⚠️ Report is 26 commits behind head on master.

Files with missing lines	Patch %	Lines
pkg/pillar/cmd/diag/diag.go	10.71%	25 Missing ⚠️
pkg/pillar/cmd/volumemgr/volumemgr.go	0.00%	6 Missing ⚠️
pkg/pillar/cmd/volumemgr/handlediskmetrics.go	66.66%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #6240      +/-   ##
==========================================
+ Coverage   22.93%   23.36%   +0.43%     
==========================================
  Files         510      520      +10     
  Lines       93473    95230    +1757     
==========================================
+ Hits        21440    22254     +814     
- Misses      70292    71039     +747     
- Partials     1741     1937     +196     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[andrewd-zededa]

Can we subscribe to longhorn readiness in the future and republish VolumeMgrStatus if there are future longhorn control plane issues?

[andrewd-zededa]

I don't see anyone using this field yet, is there a consumer coming in another PR?

[eriknordmark]

No consumer — and Initialized right above it has never had one either; nodeagent is the only subscriber and reads just RemainingSpace. The motivation is making the state available to tests and diagnosis: the status lands in /run/volumemgr/VolumeMgrStatus/volumemgr.json, so a test can ssh in and assert on it, and collect-info picks it up in the bundle.

Adding diag as a consumer was easy, so I've done it here — it prints a warning naming the outstanding gate when cluster storage didn't become usable.

Nothing reports this to the controller today: ZInfoClusterNode carries only the node_status enum and pillar doesn't populate it, so that's an eve-api change and a separate PR. Let's discuss what else we'd want in the API in this area. One thing to design around: these two fields are a one-shot startup outcome, not a live condition — volumemgr decides once after its wait and republishes the same value.

[eriknordmark]

Heads-up on an interaction with 6259 that neither PR sees on its own.

#6240 adds pkg/pillar/kubeapi/longhornready_test.go, whose
TestCheckLonghornReadyHealthy and TestCheckLonghornReadyIgnoresStrayDaemonset
call checkLonghornReady with a fake.NewSimpleClientset. #6259 makes
checkLonghornReady end with instanceManagerReady(ctx, nodeName), and the real
gate builds its Longhorn client from the on-device kubeconfig. Under a fake
clientset that file does not exist, so both tests fail with:

longhorn instance-manager: kubeconfig: stat /run/.kube/k3s/k3s.yaml: no such file or directory

No merge order is required — both are based on master and touch disjoint
lines, so they merge cleanly in either order. But whichever lands second must
carry the test fix, or master goes red. instanceManagerReady is a package-level
var, so stubbing it is enough:

// stubInstanceManagerGate satisfies the instance-manager check for tests that
// exercise the daemonset logic: the real gate builds a Longhorn client from the
// on-device kubeconfig, which a fake clientset cannot supply.
func stubInstanceManagerGate(t *testing.T) {
	saved := instanceManagerReady
	t.Cleanup(func() { instanceManagerReady = saved })
	instanceManagerReady = func(context.Context, string) error { return nil }
}

called at the top of both tests (plus a "context" import).

Found on an integration branch carrying both PRs; with the stub applied,
go test -tags k ./kubeapi/... passes.