eve-k: add cluster tie-breaker support to evetest, fix pkg/kube kubeclient regression - #6309
Merged
Merged
Conversation
andrewd-zededa
added
stable
andrewd-zededa
changed the title
Contributor
Author
|
Fixed a regression introduced in #5971. Tie-breaker config not applied due to stale kubeclient blocking api access. |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #6309 +/- ##
==========================================
+ Coverage 24.31% 24.66% +0.34%
==========================================
Files 512 522 +10
Lines 93853 95569 +1716
==========================================
+ Hits 22820 23571 +751
- Misses 69219 70000 +781
- Partials 1814 1998 +184 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
| // out of the cluster, which is a destructive change to the topology the | ||
| // preceding test relies on. | ||
| // Every cluster subtest re-creates its devices, because | ||
| // clusterDeviceRequirements sets CreateFromScratchWithLiveImage, which |
Contributor
There was a problem hiding this comment.
I think we should replace CreateFromScratchWithLiveImage with something like ResetDeviceConfig to maximize device reuse and reduce the runtime of this already fairly long test suite.
Contributor
Author
There was a problem hiding this comment.
I think I agree, although I'd like to avoid pulling in that work to this pr.
andrewd-zededa
marked this pull request as ready for review
August 14, 2026 15:17
7 tasks
ClusterNode gets a TieBreaker field. NewEdgeClusterConfig finds the onboarded UUID of that node and writes it to EdgeNodeCluster.TieBreakerNodeId. It writes the UUID to the configuration of all three nodes, because each node compares that UUID with its own to know if the role applies to itself. A new TestHarness.deviceUUID helper does the name-to-UUID lookup. A maximum of one node can be the tie-breaker. TestTieBreakerCluster starts a three-node cluster with a tie-breaker. It first makes sure that all three nodes report the same designation. It then waits for the tie-breaker-config-applied label, which pkg/kube writes last. The test then examines the node labels, the cordon, the KubeVirt and Longhorn replica counts, the CDI and longhorn-manager node selectors, and the drained node. It last starts an application, and makes sure that no VMI and no Longhorn replica goes to the tie-breaker, and that the application volume uses the two-replica storage class. The test warns, and does not fail, about the Longhorn node specification, the KubeVirt CR infra replicas, and the node selectors of virt-handler and longhorn-csi-plugin. pkg/kube applies these one time at cluster creation and does not reconcile them, thus other controllers set them back. The warnings show if the values hold at first, or never apply. The test runs in TestNodeClusterSuite after TestThreeNodesCluster. The comment about device reuse in that suite was not correct, because clusterDeviceRequirements prevents reuse. The comment is now correct. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Andrew Durbin <andrewd@zededa.com>
initKubeclient builds the process-wide client-go client one time, at
EvK3sReady, from /etc/rancher/k3s/k3s.yaml. No code ever rebuilds it
after that.
A node that joins a multi-node cluster can have its local k3s server
change its TLS material after EvK3sReady fires. The cached client
then fails every API call with "certificate signed by unknown
authority", and it never recovers. reapplyNodeLabels uses this same
client, so state.NodeLabelsInitialized never gets set, and the
tie-breaker phase skips forever on its own first check ("node labels
not yet initialized").
The old shell implementation did not have this problem. It ran a new
kubectl process for each call, and each process read the current
kubeconfig file fresh from disk.
kubeconfigSyncLoop already reads the kubeconfig file on every tick,
to mirror it for other consumers. This change adds a second check on
the same tick: rebuild kubeclient.Default() when the file content
differs from what the current client was built from, and stop the
informer goroutines of the client it replaces.
The very first tick always rebuilds the client one time, even when
nothing changed, because there is no known content to compare
against yet. The client that initKubeclient built has no stop
function to call, so its informer goroutines stay until the process
ends. Both costs happen at most one time per node boot.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Andrew Durbin <andrewd@zededa.com>
andrewd-zededa
force-pushed
the
eve-k-evetest-tb-support
branch
from
August 14, 2026 23:40
960269f to
6de65f1
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
ClusterNode gets a TieBreaker field. NewEdgeClusterConfig finds the onboarded UUID of that node and writes it to
EdgeNodeCluster.TieBreakerNodeId. It writes the UUID to the configuration of all three nodes, because each node compares that UUID with its own to know if the role applies to itself. A new TestHarness.deviceUUID helper does the name-to-UUID lookup. A maximum of one node can be the tie-breaker.
TestTieBreakerCluster starts a three-node cluster with a tie-breaker. It first makes sure that all three nodes report the same designation. It then waits for the tie-breaker-config-applied label, which pkg/kube writes last. The test then examines the node labels, the cordon, the KubeVirt and Longhorn replica counts, the CDI and longhorn-manager node selectors, and the drained node. It last starts an application, and makes sure that no VMI and no Longhorn replica goes to the tie-breaker, and that the application volume uses the two-replica storage class.
The test warns, and does not fail, about the Longhorn node specification, the KubeVirt CR infra replicas, and the node selectors of virt-handler and longhorn-csi-plugin. pkg/kube applies these one time at cluster creation and does not reconcile them, thus other controllers set them back. The warnings show if the values hold at first, or never apply.
The test runs in TestNodeClusterSuite after TestThreeNodesCluster. The comment about device reuse in that suite was not correct, because clusterDeviceRequirements prevents reuse. The comment is now correct.
Second commit — a real regression found while validating the test above,
not a test-only change.**
kube-init's Kubernetes client is built once, atk3s-ready time, and never rebuilt. A node joining a multi-node cluster can
have its local k3s TLS material change afterward; every subsequent API call
from that node then fails permanently with "certificate signed by unknown
authority", which blocks node-label initialization and, transitively, the
tie-breaker phase (and CDI config, the kubevirt feature-gate migration, and
anything else that tick performs). The fix rebuilds the client whenever the
kubeconfig file's content changes, on the same tick that already mirrors
that file for other consumers. See the commit message for the full
mechanism. Reviewers: please give this one the same scrutiny as a
standalone
pkg/kubePR — it changes daemon behavior on real devices, notjust test coverage.
PR dependencies
None
How to test and validate this PR
Run the new test
Parameters:
TPM(bool),FILESYSTEM(defaults to ext4).Or the whole suite, which places it after
TestThreeNodesCluster:This brings up three kubevirt devices from scratch and deploys an app, so
budget accordingly — it is the slowest suite in the tree, and this adds a
fourth full cluster formation to it.
Validating the kubeclient fix specifically
Watch the kube-init log during cluster formation. Before this fix, a node
whose TLS material changed after joining would log the same
"certificate signed by unknown authority" error on every subsequent tick,
forever. After this fix, look for:
within one healthCheckInterval (15s) of that error appearing, followed by
the errors clearing on the next tick.
Changelog notes
Fixed a bug where a node joining a multi-node EVE-K cluster could get
permanently stuck partway through initialization — node labels, the
tie-breaker role, and cluster component configuration — if its local
Kubernetes TLS material changed during the join. No other user-facing
changes; the evetest additions are test-framework only.
PR Backports
Checklist
And the last but not least:
check them.
Please, check the boxes above after submitting the PR in interactive mode.