fix: do not fatally abort install on nm-online connectivity check - #1313
Open
rustydb wants to merge 1 commit into
Open
fix: do not fatally abort install on nm-online connectivity check#1313rustydb wants to merge 1 commit into
rustydb wants to merge 1 commit into
Conversation
The management-network readiness check ran `nm-online -x`, which exits immediately instead of waiting, and treated any non-zero result as fatal. On the already-installed first-boot path this check runs before the RancherD config is written, so a management link that is briefly offline while the bond re-converges aborts the whole install and leaves the node with no role until manual repair. Two changes: - Wait for the network instead of sampling once: use `nm-online -s -t 45` instead of `-x`. `-s` waits for NetworkManager startup to complete (interfaces finished activating) rather than for global connectivity, which an isolated management network may never reach. The 45s timeout gives a slow-rate LACP bond headroom: `lacp_rate=slow` emits an LACPDU only every 30s, so a single missed interval can cost ~30s before the bond syncs. Because `-s` returns as soon as startup completes, this budget only costs time on a genuinely slow or degraded link. - Make the check non-fatal: log a warning and continue instead of returning an error, so a slow or briefly-degraded management link no longer aborts the install (and, on the installed-node path, no longer strands the node without a RancherD config). This is observed on both LACP/802.3ad and single-NIC active-backup bonds, and across install methods, so it is not specific to any one bond mode. Refs: harvester/harvester#10885 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Russell Bunch <rusty@4lambda.io>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem:
The management-network readiness check ran
nm-online -x, which exits immediately instead of waiting, and treated any non-zero result as fatal. On the already-installed first-boot path this check runs before the RancherD config is written, so a management link that is briefly offline while the bond re-converges aborts the whole install and leaves the node with no role until manual repair.Solution:
Two changes:
nm-online -s -t 45instead of-x.-swaits for NetworkManager startup to complete (interfaces finished activating) rather than for global connectivity, which an isolated management network may never reach. The 45s timeout gives a slow-rate LACP bond headroom:lacp_rate=slowemits an LACPDU only every 30s, so a single missed interval can cost ~30s before the bond syncs. Because-sreturns as soon as startup completes, this budget only costs time on a genuinely slow or degraded link.This is observed on both LACP/802.3ad and single-NIC active-backup bonds, and across install methods, so it is not specific to any one bond mode.
Related Issue(s):
Refs: harvester/harvester#10885
Test plan:
Additional documentation or context