diff --git a/docs/host/host.md b/docs/host/host.md index 85922e5a14..094246ff69 100644 --- a/docs/host/host.md +++ b/docs/host/host.md @@ -142,26 +142,106 @@ Again, removing a control plane node in this situation is **not recommended** be ### 6. Delete RKE2 services and shut down the node. +:::warning + +**Critical Warnings Before Running the Uninstall Script `/opt/rke2/bin/rke2-uninstall.sh`** + +- You must run the script **only on the target node**. Always verify your current hostname (using `hostname`) before running the command. + +- The script runs immediately **without asking for confirmation** (`y/n`). Once invoked, it stops services and permanently destroys node configurations and local data. + +::: + 1. Log in to the node using the root account. 1. Run the script `/opt/rke2/bin/rke2-uninstall.sh` to delete RKE2 services running on the node. 1. Shut down the node. +### 7. Remove the node. + +1. On the Harvester UI, go to the **Hosts** screen. + +1. Locate the node that you want to remove, and then click **⋮ > Delete**. + +![delete.png](/img/v1.2/host/delete-node.png) + +This deletion process typically completes within one minute. If the UI status does not update automatically, refresh your browser to reload the cluster state. + +:::important + +The process deletes cluster components and configurations without wiping local disks and bootloader partitions. + +Transfer the decommissioned host to your IT infrastructure or system administration team for drive sanitization procedures compliant with your organization's data retention and hardware lifecycle policies. + +::: + +### Known Issue: Node Stuck in "Draining" State During Removal + +**Symptom** + +During node deletion, the host status displayed on the UI typically is `Kubelet stopped posting node status`. + +![uninstalled-node.png](/img/v1.9/host/node-removal-normally-uninstalled.png) + +If the UI displays `Node is draining due to kubelet/node not ready`, or toggles between `Kubelet stopped posting node status` and `Node is draining due to kubelet/node not ready` for more than 5 minutes without removing the host, you have encountered this issue. In the backend, `fleet-agent` pods are repeatedly scheduled onto the target node, creating a continuous `Terminating` loop that prevents node deletion. + +![re-drain-node.png](/img/v1.9/host/node-removal-node-draining.png) + +```sh +$ kubectl get pods -A -owide | grep fleet +cattle-fleet-local-system fleet-agent-6dfc5584bf-4jm7r 0/1 Terminating 0 46s harv42 +cattle-fleet-local-system fleet-agent-747c7b5f98-44lq5 0/1 Pending 0 46s harv42 +cattle-fleet-system fleet-controller-6c5d89f545-q925w 3/3 Running 3 (126m ago) 3h10m 10.52.0.8 harv31 +``` + +**Cause** + +During node deletion, the changing cluster topology prompts `fleet-controller` to roll out new revisions of the `fleet-agent` deployment. Because `fleet-agent` includes broad tolerations (such as `node.kubernetes.io/unreachable:NoSchedule` and `node.kubernetes.io/unschedulable:NoSchedule`), `kube-scheduler` may continue to select the draining node as a valid target and redeploy agent pods onto it. + :::note -There's a [known issue](https://github.com/harvester/harvester/issues/1497) about node hard delete. -Once resolved, you can skip this step. +This is an intermittent issue caused by a scheduling race condition. It occurs most frequently on clusters with **three or more management nodes** when one of those management nodes is being deleted, as `kube-scheduler` attempts to schedule replacement `fleet-agent` pods onto it. ::: -### 7. Remove the node. +**Workaround** -1. On the Harvester UI, go to the **Hosts** screen. +The `fleet-agent` deployment template includes a soft node affinity rule, which you can use to break the scheduling loop. By default, Harvester nodes do not carry the `fleet.cattle.io/agent` label. -1. Locate the node that you want to remove, and then click **⋮ > Delete**. +```yaml +spec: + affinity: + nodeAffinity: + preferredDuringSchedulingIgnoredDuringExecution: + - preference: + matchExpressions: + - key: fleet.cattle.io/agent + operator: In + values: + - "true" + weight: 1 +``` -![delete.png](/img/v1.2/host/delete-node.png) +Temporarily assigning `fleet.cattle.io/agent=true` to a healthy node (other than the node being deleted) increases its scheduling weight. This forces `kube-scheduler` to steer replacement `fleet-agent` pods away from the draining node. + +1. Temporarily pin fleet-agent to a healthy active node: + + ```sh + kubectl label node fleet.cattle.io/agent=true --overwrite + ``` + +1. Monitor the node deletion status on the Harvester UI until the process completes. + +1. Remove the temporary label after the node is successfully deleted from the cluster. + + ```sh + kubectl label node fleet.cattle.io/agent- + ``` + +**Related Issue** + +[#11349](https://github.com/harvester/harvester/issues/11349) ## Role Management diff --git a/static/img/v1.9/host/node-removal-node-draining.png b/static/img/v1.9/host/node-removal-node-draining.png new file mode 100644 index 0000000000..4105fe8e94 Binary files /dev/null and b/static/img/v1.9/host/node-removal-node-draining.png differ diff --git a/static/img/v1.9/host/node-removal-normally-uninstalled.png b/static/img/v1.9/host/node-removal-normally-uninstalled.png new file mode 100644 index 0000000000..ee6757b422 Binary files /dev/null and b/static/img/v1.9/host/node-removal-normally-uninstalled.png differ diff --git a/versioned_docs/version-v1.7/host/host.md b/versioned_docs/version-v1.7/host/host.md index 31aa0b6229..88ddeefd0a 100644 --- a/versioned_docs/version-v1.7/host/host.md +++ b/versioned_docs/version-v1.7/host/host.md @@ -142,26 +142,106 @@ Again, removing a control plane node in this situation is **not recommended** be ### 6. Delete RKE2 services and shut down the node. +:::warning + +**Critical Warnings Before Running the Uninstall Script `/opt/rke2/bin/rke2-uninstall.sh`** + +- You must run the script **only on the target node**. Always verify your current hostname (using `hostname`) before running the command. + +- The script runs immediately **without asking for confirmation** (`y/n`). Once invoked, it stops services and permanently destroys node configurations and local data. + +::: + 1. Log in to the node using the root account. 1. Run the script `/opt/rke2/bin/rke2-uninstall.sh` to delete RKE2 services running on the node. 1. Shut down the node. +### 7. Remove the node. + +1. On the Harvester UI, go to the **Hosts** screen. + +1. Locate the node that you want to remove, and then click **⋮ > Delete**. + +![delete.png](/img/v1.2/host/delete-node.png) + +This deletion process typically completes within one minute. If the UI status does not update automatically, refresh your browser to reload the cluster state. + +:::important + +The process deletes cluster components and configurations without wiping local disks and bootloader partitions. + +Transfer the decommissioned host to your IT infrastructure or system administration team for drive sanitization procedures compliant with your organization's data retention and hardware lifecycle policies. + +::: + +### Known Issue: Node Stuck in "Draining" State During Removal + +**Symptom** + +During node deletion, the host status displayed on the UI typically is `Kubelet stopped posting node status`. + +![uninstalled-node.png](/img/v1.9/host/node-removal-normally-uninstalled.png) + +If the UI displays `Node is draining due to kubelet/node not ready`, or toggles between `Kubelet stopped posting node status` and `Node is draining due to kubelet/node not ready` for more than 5 minutes without removing the host, you have encountered this issue. In the backend, `fleet-agent` pods are repeatedly scheduled onto the target node, creating a continuous `Terminating` loop that prevents node deletion. + +![re-drain-node.png](/img/v1.9/host/node-removal-node-draining.png) + +```sh +$ kubectl get pods -A -owide | grep fleet +cattle-fleet-local-system fleet-agent-6dfc5584bf-4jm7r 0/1 Terminating 0 46s harv42 +cattle-fleet-local-system fleet-agent-747c7b5f98-44lq5 0/1 Pending 0 46s harv42 +cattle-fleet-system fleet-controller-6c5d89f545-q925w 3/3 Running 3 (126m ago) 3h10m 10.52.0.8 harv31 +``` + +**Cause** + +During node deletion, the changing cluster topology prompts `fleet-controller` to roll out new revisions of the `fleet-agent` deployment. Because `fleet-agent` includes broad tolerations (such as `node.kubernetes.io/unreachable:NoSchedule` and `node.kubernetes.io/unschedulable:NoSchedule`), `kube-scheduler` may continue to select the draining node as a valid target and redeploy agent pods onto it. + :::note -There's a [known issue](https://github.com/harvester/harvester/issues/1497) about node hard delete. -Once resolved, you can skip this step. +This is an intermittent issue caused by a scheduling race condition. It occurs most frequently on clusters with **three or more management nodes** when one of those management nodes is being deleted, as `kube-scheduler` attempts to schedule replacement `fleet-agent` pods onto it. ::: -### 7. Remove the node. +**Workaround** -1. On the Harvester UI, go to the **Hosts** screen. +The `fleet-agent` deployment template includes a soft node affinity rule, which you can use to break the scheduling loop. By default, Harvester nodes do not carry the `fleet.cattle.io/agent` label. -1. Locate the node that you want to remove, and then click **⋮ > Delete**. +```yaml +spec: + affinity: + nodeAffinity: + preferredDuringSchedulingIgnoredDuringExecution: + - preference: + matchExpressions: + - key: fleet.cattle.io/agent + operator: In + values: + - "true" + weight: 1 +``` -![delete.png](/img/v1.2/host/delete-node.png) +Temporarily assigning `fleet.cattle.io/agent=true` to a healthy node (other than the node being deleted) increases its scheduling weight. This forces `kube-scheduler` to steer replacement `fleet-agent` pods away from the draining node. + +1. Temporarily pin fleet-agent to a healthy active node: + + ```sh + kubectl label node fleet.cattle.io/agent=true --overwrite + ``` + +1. Monitor the node deletion status on the Harvester UI until the process completes. + +1. Remove the temporary label after the node is successfully deleted from the cluster. + + ```sh + kubectl label node fleet.cattle.io/agent- + ``` + +**Related Issue** + +[#11349](https://github.com/harvester/harvester/issues/11349) ## Role Management diff --git a/versioned_docs/version-v1.8/host/host.md b/versioned_docs/version-v1.8/host/host.md index 5b7d8ddf1f..52920b2842 100644 --- a/versioned_docs/version-v1.8/host/host.md +++ b/versioned_docs/version-v1.8/host/host.md @@ -142,26 +142,106 @@ Again, removing a control plane node in this situation is **not recommended** be ### 6. Delete RKE2 services and shut down the node. +:::warning + +**Critical Warnings Before Running the Uninstall Script `/opt/rke2/bin/rke2-uninstall.sh`** + +- You must run the script **only on the target node**. Always verify your current hostname (using `hostname`) before running the command. + +- The script runs immediately **without asking for confirmation** (`y/n`). Once invoked, it stops services and permanently destroys node configurations and local data. + +::: + 1. Log in to the node using the root account. 1. Run the script `/opt/rke2/bin/rke2-uninstall.sh` to delete RKE2 services running on the node. 1. Shut down the node. +### 7. Remove the node. + +1. On the Harvester UI, go to the **Hosts** screen. + +1. Locate the node that you want to remove, and then click **⋮ > Delete**. + +![delete.png](/img/v1.2/host/delete-node.png) + +This deletion process typically completes within one minute. If the UI status does not update automatically, refresh your browser to reload the cluster state. + +:::important + +The process deletes cluster components and configurations without wiping local disks and bootloader partitions. + +Transfer the decommissioned host to your IT infrastructure or system administration team for drive sanitization procedures compliant with your organization's data retention and hardware lifecycle policies. + +::: + +### Known Issue: Node Stuck in "Draining" State During Removal + +**Symptom** + +During node deletion, the host status displayed on the UI typically is `Kubelet stopped posting node status`. + +![uninstalled-node.png](/img/v1.9/host/node-removal-normally-uninstalled.png) + +If the UI displays `Node is draining due to kubelet/node not ready`, or toggles between `Kubelet stopped posting node status` and `Node is draining due to kubelet/node not ready` for more than 5 minutes without removing the host, you have encountered this issue. In the backend, `fleet-agent` pods are repeatedly scheduled onto the target node, creating a continuous `Terminating` loop that prevents node deletion. + +![re-drain-node.png](/img/v1.9/host/node-removal-node-draining.png) + +```sh +$ kubectl get pods -A -owide | grep fleet +cattle-fleet-local-system fleet-agent-6dfc5584bf-4jm7r 0/1 Terminating 0 46s harv42 +cattle-fleet-local-system fleet-agent-747c7b5f98-44lq5 0/1 Pending 0 46s harv42 +cattle-fleet-system fleet-controller-6c5d89f545-q925w 3/3 Running 3 (126m ago) 3h10m 10.52.0.8 harv31 +``` + +**Cause** + +During node deletion, the changing cluster topology prompts `fleet-controller` to roll out new revisions of the `fleet-agent` deployment. Because `fleet-agent` includes broad tolerations (such as `node.kubernetes.io/unreachable:NoSchedule` and `node.kubernetes.io/unschedulable:NoSchedule`), `kube-scheduler` may continue to select the draining node as a valid target and redeploy agent pods onto it. + :::note -There's a [known issue](https://github.com/harvester/harvester/issues/1497) about node hard delete. -Once resolved, you can skip this step. +This is an intermittent issue caused by a scheduling race condition. It occurs most frequently on clusters with **three or more management nodes** when one of those management nodes is being deleted, as `kube-scheduler` attempts to schedule replacement `fleet-agent` pods onto it. ::: -### 7. Remove the node. +**Workaround** -1. On the Harvester UI, go to the **Hosts** screen. +The `fleet-agent` deployment template includes a soft node affinity rule, which you can use to break the scheduling loop. By default, Harvester nodes do not carry the `fleet.cattle.io/agent` label. -1. Locate the node that you want to remove, and then click **⋮ > Delete**. +```yaml +spec: + affinity: + nodeAffinity: + preferredDuringSchedulingIgnoredDuringExecution: + - preference: + matchExpressions: + - key: fleet.cattle.io/agent + operator: In + values: + - "true" + weight: 1 +``` -![delete.png](/img/v1.2/host/delete-node.png) +Temporarily assigning `fleet.cattle.io/agent=true` to a healthy node (other than the node being deleted) increases its scheduling weight. This forces `kube-scheduler` to steer replacement `fleet-agent` pods away from the draining node. + +1. Temporarily pin fleet-agent to a healthy active node: + + ```sh + kubectl label node fleet.cattle.io/agent=true --overwrite + ``` + +1. Monitor the node deletion status on the Harvester UI until the process completes. + +1. Remove the temporary label after the node is successfully deleted from the cluster. + + ```sh + kubectl label node fleet.cattle.io/agent- + ``` + +**Related Issue** + +[#11349](https://github.com/harvester/harvester/issues/11349) ## Role Management