pillar: diagnose stuck EVE-k app bring-up (CDI + readiness gate) - #6200
Merged
Conversation
When RolloutDiskToPVC exhausts its upload retries the app volume stays in CREATING_VOLUME, and the existing failure diagnosis walks only the data PVC, its PV, and the Longhorn engine. That path does not surface the case where the CDI upload pod reaches Ready but is then torn down — the data PVC's CDI annotations show pod.phase Failed / ContainerStatusUnknown — while its local-path scratch PVC is left stuck Terminating under the pvc-protection finalizer, so the upload never completes and the volume never leaves CREATING_VOLUME. Log the upload pod phase, the CDI teardown annotations on the data PVC, and the scratch PVC's phase / deletion-pending / storage class / finalizers before the existing checks, so this wedge is diagnosable from the device's own logs. Diagnostic only; no behavior change. Signed-off-by: eriknordmark <erik@zededa.com> Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
github-actions
Bot
requested review from
andrewd-zededa,
naiming-zededa and
zedi-pramodh
July 22, 2026 06:09
On EVE-k the boot-time WaitForKubernetes poll gates on node-Ready, KubeVirt-CR-Available, and (optionally) Longhorn. On the 20-minute timeout it returned only "timed out waiting for the condition", and domainmgr then logged a hardcoded "kubevirt not ready" regardless of which check was actually failing — so an app wedged at INSTALLED while the node or Longhorn (not kubevirt) was the laggard was mislabeled and undiagnosable from the log alone. Record the last unmet sub-check and attach it to the returned error, and drop domainmgr's fixed label for a neutral "not satisfied", so the specific gate rides in the logged error. Diagnostic only: gating and retry behavior are unchanged, and all callers (volumemgr, domainmgr, zedkube) only log the error. Signed-off-by: eriknordmark <erik@zededa.com> Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
github-actions
Bot
requested review from
OhmSpectator,
jsfakian and
milan-zededa
July 22, 2026 18:19
eriknordmark
changed the title
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## master #6200 +/- ##
==========================================
+ Coverage 22.75% 23.12% +0.36%
==========================================
Files 509 519 +10
Lines 93285 95023 +1738
==========================================
+ Hits 21231 21977 +746
- Misses 70324 71132 +808
- Partials 1730 1914 +184 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
eriknordmark
removed request for
OhmSpectator,
jsfakian,
milan-zededa and
zedi-pramodh
July 22, 2026 23:37
eriknordmark
marked this pull request as ready for review
July 22, 2026 23:38
eriknordmark
added
stable
milan-zededa
approved these changes
Jul 23, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
EVE-k app bring-up after a volume-create can stall in two spots that leave no clear signal in the device's own logs. This adds diagnostics for both; no behavior change. This is useful for troubleshooting issues in the field and also when running regression and stress tests.
CDI upload-pod teardown wedge (
RolloutDiskToPVC, follow-up to volumemgr,kubeapi: defer+retry app volumes until EVE-k cluster storage is ready #6121). When the CDI upload pod reaches Ready and is then torn down — the data PVC showspod.phase=Failed/ContainerStatusUnknownand its local-path scratch PVC is left stuckTerminatingunder thepvc-protectionfinalizer — the upload never completes and the volume staysCREATING_VOLUME. The existing failure walk inspects only the data PVC/PV/engine and misses this, so log the upload-pod phase, the CDI teardown annotations on the data PVC, and the scratch PVC's phase / deletion-pending / storage class / finalizers.WaitForKubernetesreadiness gate. The boot-time gate polls node-Ready, KubeVirt-CR-Available, and (optionally) Longhorn; on the 20-minute timeout it returned only"timed out waiting for the condition", and domainmgr then logged a hardcoded"kubevirt not ready"regardless of which check actually failed. That mislabels a stuck node/Longhorn check as a kubevirt problem and leaves an app wedged atINSTALLEDundiagnosable from the log alone. Now the returned error names the last unmet sub-check (node / kubevirt / longhorn) and domainmgr logs it verbatim.Related: #6197 (kubelet mount-wedge detector) covers a distinct wedge — a
Pendingpod whose Longhorn volume is attached but neverNodeStaged.How to test and validate this PR
EVE-k only, and diagnostic-only (readiness gating and volume-create success/failure are unchanged; only log content differs). On an EVE-k device:
RolloutDiskToPVCexhausts its retries →volumemgrlogs the upload-pod / scratch-PVC teardown state in/persist/newlog.domainmgrlogsWaitForKubernetes not satisfied: … (last unmet condition: kubevirt not ready: …).Not covered by an automated test (log output only).
Changelog notes
No user-facing changes.
PR Backports
Part of this extends the #6121 diagnostic block, which is not on the current LTS branches.
-17.0: Yes
Checklist
gofmt,go vet -tags k) and ships in the resize-allprs-stress soak build; not independently device-verified as a standalone at open time🤖 Generated with Claude Code