hypervisor: recover a VMIRS stranded at Spec.Replicas == 0 - #6269
Open
andrewd-zededa wants to merge 1 commit into
Open
hypervisor: recover a VMIRS stranded at Spec.Replicas == 0#6269andrewd-zededa wants to merge 1 commit into
andrewd-zededa wants to merge 1 commit into
Conversation
github-actions
Bot
requested review from
OhmSpectator,
eriknordmark,
naiming-zededa,
rene,
rucoder,
shjala and
zedi-pramodh
August 3, 2026 22:29
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #6269 +/- ##
==========================================
+ Coverage 22.93% 23.33% +0.39%
==========================================
Files 510 520 +10
Lines 93473 95248 +1775
==========================================
+ Hits 21440 22223 +783
- Misses 70292 71100 +808
- Partials 1741 1925 +184 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
andrewd-zededa
force-pushed
the
eve-k-vmirs-replicas-reconcile
branch
from
August 4, 2026 00:45
80ed190 to
e73b0f9
Compare
Contributor
Author
|
evetest run of latest: |
andrewd-zededa
added
stable
|
I am already outdated :) AFAIR, we never want to bring the replica set to 0, we only delete the VMIRS. The idea is not to loose the vmirs and when all nodes are rebooted at same time, atleast one of the nodes will start it. What changed now ? |
DetachUtilVmirsReplicaReset scales a VMIRS 0 then 1 under one bounded context; if the deadline expires, the API server flaps, or the process dies between the two writes, Replicas stays 0 forever. Nothing else in pillar ever reads or repairs Spec.Replicas, and StopReplicaVMI deletes the VMIRS rather than scaling it, so zero replicas is never a legitimate steady state. Two compounding gaps let this go unnoticed and unrepaired: Start() treats IsAlreadyExists on Create() as success without checking the existing object's replica count, and Info() reports a zero-replica VMIRS (no VMI, no virt-launcher pod) as UNKNOWN with a nil error, before scheduledOnMe's !onMe short-circuit is ever reached. That left domainmgr reporting the app RUNNING to the controller indefinitely while the cluster ran it with zero replicas. - Info() now checks desired replicas for IsMetaReplicaVMI domains before the scheduledOnMe short-circuit, and reports HALTED with an error on a stranded VMIRS. This is deliberately HALTED, not BROKEN, so verifyStatus's recovery branch sets BootFailed and (for kube) publishes BOOTING while skipping Delete/Cleanup, letting maybeRetryBoot drive recovery through the existing retry loop. - Start()'s IsAlreadyExists branch now calls ensureVmirsReplicas, which raises Spec.Replicas to at least 1 with a bounded Get/mutate/Update retry on conflicting writers. The repair only ever writes upward on observing zero, so concurrent callers converge and it can never produce the stranded state itself. - Both new k8s API calls are bounded by kubeapi.KubeAPITimeout() (exported getter over kubeapi's existing private constant, to avoid a third redeclaration of the same budget) rather than context.Background(), since Info()/Start() run on domainmgr's watchdog-timed verifyStatus tick and an unbounded call against a degraded apiserver would otherwise risk a watchdog reboot. - DetachOldWorkload no longer discards DetachUtilVmirsReplicaReset's return value, so a stranded VMIRS is attributable in logs. - StopReplicaVMI no longer logs "Stop VMI Replicaset error <nil>" on every successful stop, which polluted the logs used to verify this fix. Adds an evetest chaos test (evetest/tests/cluster) that deploys an app on a single-node EVE-K cluster, scales its VMIRS to 0 replicas directly over SSH to reproduce the stranded state without needing a real failover, and asserts via the EVE API that the app surfaces a retryable warning and then self-heals back to RUNNING. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Signed-off-by: Andrew Durbin <andrewd@zededa.com>
andrewd-zededa
force-pushed
the
eve-k-vmirs-replicas-reconcile
branch
from
August 4, 2026 01:57
e73b0f9 to
601a7fa
Compare
Contributor
Author
This is just error recovery if the scale up here was unable to occur: eve/pkg/pillar/kubeapi/kubeapi.go Line 650 in 798bca4 |
I see, so did we add this to quickly terminate the stuck VMI and restart it again ? Was this code part of 17.0.0-lts ? |
Contributor
Author
This is to handle an issue where the new vmi is not getting scheduled, and yes part of the lts. |
Contributor
Author
|
Latest evetest run: |
andrewd-zededa
marked this pull request as ready for review
August 4, 2026 02:33
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
DetachUtilVmirsReplicaReset scales a VMIRS 0 then 1 under one bounded context; if the deadline expires, the API server flaps, or the process dies between the two writes, Replicas stays 0 forever. Nothing else in pillar ever reads or repairs Spec.Replicas, and StopReplicaVMI deletes the VMIRS rather than scaling it, so zero replicas is never a legitimate steady state.
Two compounding gaps let this go unnoticed and unrepaired: Start() treats IsAlreadyExists on Create() as success without checking the existing object's replica count, and Info() reports a zero-replica VMIRS (no VMI, no virt-launcher pod) as UNKNOWN with a nil error, before scheduledOnMe's !onMe short-circuit is ever reached. That left domainmgr reporting the app RUNNING to the controller indefinitely while the cluster ran it with zero replicas.
Adds an evetest chaos test (evetest/tests/cluster) that deploys an app on a single-node EVE-K cluster, scales its VMIRS to 0 replicas directly over SSH to reproduce the stranded state without needing a real failover, and asserts via the EVE API that the app surfaces a retryable warning and then self-heals back to RUNNING.
PR dependencies
None
How to test and validate this PR
Unit tests (
pkg/pillar/hypervisor/kubevirt_test.go): coversvmirsStranded,vmirsDesiredReplicas,ensureVmirsReplicas(includingconflict-retry).
make -C pkg/pillar testpasses, no regressions.evetest (
TestVMIRSStrandedReplicasRecovery,tests/cluster): deploysan app on eve-k, scales its VMIRS to 0 via SSH, asserts self-recovery via
the EVE API. Build/vet-clean; not yet run against real hardware — run with
make evetest NAME=TestVMIRSStrandedReplicasRecovery.Manual repro:
Changelog notes
Enhanced error recovery on failover of app instances during periods of kube-apiserver instability.
PR Backports
Checklist
And the last but not least:
check them.
Please, check the boxes above after submitting the PR in interactive mode.