Skip to content

fix: do not fatally abort install on nm-online connectivity check - #1313

Open
rustydb wants to merge 1 commit into
harvester:v1.8from
rustydb:rustydb/fix-nm-online-fatal-abort
Open

fix: do not fatally abort install on nm-online connectivity check#1313
rustydb wants to merge 1 commit into
harvester:v1.8from
rustydb:rustydb/fix-nm-online-fatal-abort

Conversation

@rustydb

@rustydb rustydb commented Jul 10, 2026

Copy link
Copy Markdown

Problem:

The management-network readiness check ran nm-online -x, which exits immediately instead of waiting, and treated any non-zero result as fatal. On the already-installed first-boot path this check runs before the RancherD config is written, so a management link that is briefly offline while the bond re-converges aborts the whole install and leaves the node with no role until manual repair.

Solution:

Two changes:

  • Wait for the network instead of sampling once: use nm-online -s -t 45 instead of -x. -s waits for NetworkManager startup to complete (interfaces finished activating) rather than for global connectivity, which an isolated management network may never reach. The 45s timeout gives a slow-rate LACP bond headroom: lacp_rate=slow emits an LACPDU only every 30s, so a single missed interval can cost ~30s before the bond syncs. Because -s returns as soon as startup completes, this budget only costs time on a genuinely slow or degraded link.
  • Make the check non-fatal: log a warning and continue instead of returning an error, so a slow or briefly-degraded management link no longer aborts the install (and, on the installed-node path, no longer strands the node without a RancherD config).

This is observed on both LACP/802.3ad and single-NIC active-backup bonds, and across install methods, so it is not specific to any one bond mode.

Related Issue(s):

Refs: harvester/harvester#10885

Test plan:

Additional documentation or context

The management-network readiness check ran `nm-online -x`, which exits
immediately instead of waiting, and treated any non-zero result as fatal.
On the already-installed first-boot path this check runs before the RancherD
config is written, so a management link that is briefly offline while the bond
re-converges aborts the whole install and leaves the node with no role until
manual repair.

Two changes:

- Wait for the network instead of sampling once: use `nm-online -s -t 45`
  instead of `-x`. `-s` waits for NetworkManager startup to complete (interfaces
  finished activating) rather than for global connectivity, which an isolated
  management network may never reach. The 45s timeout gives a slow-rate LACP
  bond headroom: `lacp_rate=slow` emits an LACPDU only every 30s, so a single
  missed interval can cost ~30s before the bond syncs. Because `-s` returns as
  soon as startup completes, this budget only costs time on a genuinely slow or
  degraded link.
- Make the check non-fatal: log a warning and continue instead of returning an
  error, so a slow or briefly-degraded management link no longer aborts the
  install (and, on the installed-node path, no longer strands the node without a
  RancherD config).

This is observed on both LACP/802.3ad and single-NIC active-backup bonds, and
across install methods, so it is not specific to any one bond mode.

Refs: harvester/harvester#10885

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: Russell Bunch <rusty@4lambda.io>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant