volumemgr: build-tagged fault injection for volume delete - #6190
Draft
eriknordmark wants to merge 1 commit into
Draft
volumemgr: build-tagged fault injection for volume delete#6190eriknordmark wants to merge 1 commit into
eriknordmark wants to merge 1 commit into
Conversation
Testing the delete-retry loop requires making a volume destroy fail on demand and then recover, which is hard to arrange on a real kubevirt/longhorn cluster. Add two file-existence fault-injection gates on the delete path, using the same marker-file idiom as kubeapi.DrainStatusFaultInjectionWait but compiled in only under a new faultinjection build tag (FAULT_INJECTION=y), so production images never carry them: - VolumeDestroyFaultInjected gates volumeHandlerCSI.DestroyVolume before the replicated-volume skip, so it fails the destroy for any volume and drives volumemgr's retry loop directly. - DeletePVCFaultInjected gates kubeapi.DeletePVC, reproducing a k8s PVC-delete failure on the non-replicated path. Two separate markers let a test target a specific layer since which operation fails in the field is not known in advance. Without the build tag both gates resolve to a no-op returning false, so the fault branches are dead code the compiler drops from the shipped binary. Signed-off-by: eriknordmark <erik@zededa.com> Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
github-actions
Bot
requested review from
OhmSpectator,
andrewd-zededa,
naiming-zededa and
zedi-pramodh
July 21, 2026 04:28
8 tasks
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## master #6190 +/- ##
==========================================
+ Coverage 22.64% 23.01% +0.37%
==========================================
Files 507 518 +11
Lines 93195 94921 +1726
==========================================
+ Hits 21100 21848 +748
- Misses 70369 71163 +794
- Partials 1726 1910 +184 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Makes a volume destroy / PVC delete fail on demand so the volume-delete retry path (#6176) can be exercised end to end on a kubevirt/longhorn node — a scenario that is otherwise hard to arrange on a real cluster.
Two file-existence fault gates on the delete path, using the same marker idiom as
kubeapi.DrainStatusFaultInjectionWait:/tmp/VolumeDestroy_FaultInjection_FailfailsvolumeHandlerCSI.DestroyVolumebefore the replicated-volume skip, so it drives the retry loop for replicated Longhorn volumes too./tmp/DeletePVC_FaultInjection_Failfailskubeapi.DeletePVCon the non-replicated path.Both are compiled in only under a new
faultinjectionbuild tag (FAULT_INJECTION=y); without it they resolve to no-op stubs returning false, so shipped images carry none of it. Wired through the linuxkit build-arg plumbing (topMakefile, pillarMakefile/Dockerfile).This is test-only tooling that complements #6176; it is not required for the fix itself.
PR dependencies
None. Pairs with the companion eden e2e test (lf-edge/eden#1217), which consumes these markers.
How to test and validate this PR
Build an EVE-k image with the tag enabled:
On a running kubevirt/longhorn EVE-k node, deploy an app with a volume and wait for the volume to reach the Created sub-state, then arm a fault and delete the app:
The volume destroy fails; with #6176 present the
VolumeStatusstays published in the Deleting sub-state with an error and is re-driven off the gc tick. Removing the marker lets the next retry succeed and the volume is unpublished.Automated: covered end to end by the companion eden suite (lf-edge/eden#1217). Validated on host EVE-k (master base): the retain→retry→recover and bounded give-up paths pass on an image built with this and #6176, and the recovery test fails on an image built with this but without #6176 — i.e. the test catches the pre-fix leak.
Without the
faultinjectionbuild tag the gates are no-op stubs, so a normal build is unaffected.Changelog notes
No user-facing changes (test-only tooling, compiled out of shipped images).
PR Backports
Checklist
🤖 Generated with Claude Code