Skip to content

Update the node removal with one known issue #1108

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
w13915984028 merged 2 commits into from
Aug 20, 2026
Merged

Update the node removal with one known issue #1108

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
92f1031
Update the node removal with one known issue
w13915984028 Aug 19, 2026
56c6fa7
Update per review comments
w13915984028 Aug 20, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
94 changes: 88 additions & 6 deletions docs/host/host.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -142,26 +142,108 @@ Again, removing a control plane node in this situation is **not recommended** be

### 6. Delete RKE2 services and shut down the node.

:::warning

1. **Target Node Only**
The uninstall script `/opt/rke2/bin/rke2-uninstall.sh` **MUST ONLY BE RUN DIRECTLY ON THE TARGET NODE BEING REMOVED**. Double-check your active hostname (`hostname`) before invoking it.

1. **Immediate Destruction (No Confirmation Prompt):**
The uninstall script `/opt/rke2/bin/rke2-uninstall.sh` **does NOT ask for double confirmation or prompt `y/n`** before execution. Once invoked, it immediately stops services and tears down the node environment.
Comment thread
w13915984028 marked this conversation as resolved.
Outdated

:::

1. Log in to the node using the root account.

1. Run the script `/opt/rke2/bin/rke2-uninstall.sh` to delete RKE2 services running on the node.

1. Shut down the node.

### 7. Remove the node.

1. On the Harvester UI, go to the **Hosts** screen.

1. Locate the node that you want to remove, and then click **⋮ > Delete**.

![delete.png](/img/v1.2/host/delete-node.png)

This removal process typically completes within one minute. If the UI status does not update automatically, force-refresh the browser page to reload the cluster state.
Comment thread
w13915984028 marked this conversation as resolved.
Outdated

:::important

Removing a node tears down the Harvester control plane components, container runtimes, and Kubernetes configurations, but it does not wipe local disks and bootloader partitions.

Forward the decommissioned host to your IT infrastructure or system administration team to perform drive sanitization according to your organization's data retention and hardware lifecycle policies.
Comment thread
w13915984028 marked this conversation as resolved.
Outdated

:::

### 8. Known Issue: Node stuck in "Draining" state during removal

**Symptom**

During node deletion, the host status on the UI normally shows `Kubelet stopped posting node status`.

![uninstalled-node.png](/img/v1.9/host/node-removal-normally-uninstalled.png)

When it shows `Node is draining due to kubelet/node not ready`, you have encountered this issue..

![re-drain-node.png](/img/v1.9/host/node-removal-node-draining.png)

In the backend, `fleet-agent` pods are repeatedly scheduled onto the target node, triggering a continuous `Terminating` loop that halts node removal.
Comment thread
w13915984028 marked this conversation as resolved.
Outdated
Comment thread
w13915984028 marked this conversation as resolved.
Outdated

```sh
$ kubectl get pods -A -owide | grep fleet
cattle-fleet-local-system fleet-agent-6dfc5584bf-4jm7r 0/1 Terminating 0 46s <none> harv42 <none> <none>
cattle-fleet-local-system fleet-agent-747c7b5f98-44lq5 0/1 Pending 0 46s <none> harv42 <none> <none>
cattle-fleet-system fleet-controller-6c5d89f545-q925w 3/3 Running 3 (126m ago) 3h10m 10.52.0.8 harv31 <none> <none>
```

**Cause**

During node removal the cluster topology changes, and it triggers the `fleet-controller` to roll out new revisions of the `fleet-agent` deployment. Because `fleet-agent` carries broad tolerations (`node.kubernetes.io/unreachable:NoSchedule`, `node.kubernetes.io/unschedulable:NoSchedule` and more), `kube-scheduler` may continue to select the draining node as a valid target and redeploy agent pods onto it.
Comment thread
w13915984028 marked this conversation as resolved.
Outdated

:::note

There's a [known issue](https://github.com/harvester/harvester/issues/1497) about node hard delete.
Once resolved, you can skip this step.
This is an intermittent issue caused by a scheduling race condition. It occurs most frequently on clusters with **3 or more control-plane nodes** when the node being deleted is a **control-plane node**, as `kube-scheduler` assigns replacement `fleet-agent` pods to it.
Comment thread
w13915984028 marked this conversation as resolved.
Outdated

:::

### 7. Remove the node.
**Workaround**

1. On the Harvester UI, go to the **Hosts** screen.
The `fleet-agent` deployment template includes a preferred node affinity rule. By default, nodes in Harvester do not carry the `fleet.cattle.io/agent` label.
Comment thread
w13915984028 marked this conversation as resolved.
Outdated

1. Locate the node that you want to remove, and then click **⋮ > Delete**.
```yaml
spec:
affinity:
nodeAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- preference:
matchExpressions:
- key: fleet.cattle.io/agent
operator: In
values:
- "true"
weight: 1
```

![delete.png](/img/v1.2/host/delete-node.png)
You can use this soft affinity mechanism to break the scheduling loop. Temporarily assigning `fleet.cattle.io/agent=true` to a healthy active node (other than the node being removed) increases its scheduling weight, forcing `kube-scheduler` to steer replacement fleet-agent pods away from the draining node.
Comment thread
w13915984028 marked this conversation as resolved.
Outdated

1. Temporarily pin fleet-agent to a healthy active node:

```sh
kubectl label node <healthy-active-node> fleet.cattle.io/agent=true --overwrite
```

1. Wait some time and check the node deletion status on the Harvester UI.

1. Remove the temporary label after the node is successfully removed from the cluster:
Comment thread
w13915984028 marked this conversation as resolved.
Outdated

```sh
kubectl label node <healthy-active-node> fleet.cattle.io/agent-
```

**Related Issue**

[#11349](https://github.com/harvester/harvester/issues/11349)

## Role Management

Expand Down
Binary file added static/img/v1.9/host/node-removal-node-draining.png
View file
Open in desktop
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added static/img/v1.9/host/node-removal-normally-uninstalled.png
View file
Open in desktop
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
94 changes: 88 additions & 6 deletions versioned_docs/version-v1.7/host/host.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -142,26 +142,108 @@ Again, removing a control plane node in this situation is **not recommended** be

### 6. Delete RKE2 services and shut down the node.

:::warning

1. **Target Node Only**
The uninstall script `/opt/rke2/bin/rke2-uninstall.sh` **MUST ONLY BE RUN DIRECTLY ON THE TARGET NODE BEING REMOVED**. Double-check your active hostname (`hostname`) before invoking it.

1. **Immediate Destruction (No Confirmation Prompt):**
The uninstall script `/opt/rke2/bin/rke2-uninstall.sh` **does NOT ask for double confirmation or prompt `y/n`** before execution. Once invoked, it immediately stops services and tears down the node environment.

:::

1. Log in to the node using the root account.

1. Run the script `/opt/rke2/bin/rke2-uninstall.sh` to delete RKE2 services running on the node.

1. Shut down the node.

### 7. Remove the node.

1. On the Harvester UI, go to the **Hosts** screen.

1. Locate the node that you want to remove, and then click **⋮ > Delete**.

![delete.png](/img/v1.2/host/delete-node.png)

This removal process typically completes within one minute. If the UI status does not update automatically, force-refresh the browser page to reload the cluster state.

:::important

Removing a node tears down the Harvester control plane components, container runtimes, and Kubernetes configurations, but it does not wipe local disks and bootloader partitions.

Forward the decommissioned host to your IT infrastructure or system administration team to perform drive sanitization according to your organization's data retention and hardware lifecycle policies.

:::

### 8. Known Issue: Node stuck in "Draining" state during removal

**Symptom**

During node deletion, the host status on the UI normally shows `Kubelet stopped posting node status`.

![uninstalled-node.png](/img/v1.9/host/node-removal-normally-uninstalled.png)

When it shows `Node is draining due to kubelet/node not ready`, you have encountered this issue..

![re-drain-node.png](/img/v1.9/host/node-removal-node-draining.png)

In the backend, `fleet-agent` pods are repeatedly scheduled onto the target node, triggering a continuous `Terminating` loop that halts node removal.

```sh
$ kubectl get pods -A -owide | grep fleet
cattle-fleet-local-system fleet-agent-6dfc5584bf-4jm7r 0/1 Terminating 0 46s <none> harv42 <none> <none>
cattle-fleet-local-system fleet-agent-747c7b5f98-44lq5 0/1 Pending 0 46s <none> harv42 <none> <none>
cattle-fleet-system fleet-controller-6c5d89f545-q925w 3/3 Running 3 (126m ago) 3h10m 10.52.0.8 harv31 <none> <none>
```

**Cause**

During node removal the cluster topology changes, and it triggers the `fleet-controller` to roll out new revisions of the `fleet-agent` deployment. Because `fleet-agent` carries broad tolerations (`node.kubernetes.io/unreachable:NoSchedule`, `node.kubernetes.io/unschedulable:NoSchedule` and more), `kube-scheduler` may continue to select the draining node as a valid target and redeploy agent pods onto it.

:::note

There's a [known issue](https://github.com/harvester/harvester/issues/1497) about node hard delete.
Once resolved, you can skip this step.
This is an intermittent issue caused by a scheduling race condition. It occurs most frequently on clusters with **3 or more control-plane nodes** when the node being deleted is a **control-plane node**, as `kube-scheduler` assigns replacement `fleet-agent` pods to it.

:::

### 7. Remove the node.
**Workaround**

1. On the Harvester UI, go to the **Hosts** screen.
The `fleet-agent` deployment template includes a preferred node affinity rule. By default, nodes in Harvester do not carry the `fleet.cattle.io/agent` label.

1. Locate the node that you want to remove, and then click **⋮ > Delete**.
```yaml
spec:
affinity:
nodeAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- preference:
matchExpressions:
- key: fleet.cattle.io/agent
operator: In
values:
- "true"
weight: 1
```

![delete.png](/img/v1.2/host/delete-node.png)
You can use this soft affinity mechanism to break the scheduling loop. Temporarily assigning `fleet.cattle.io/agent=true` to a healthy active node (other than the node being removed) increases its scheduling weight, forcing `kube-scheduler` to steer replacement fleet-agent pods away from the draining node.

1. Temporarily pin fleet-agent to a healthy active node:

```sh
kubectl label node <healthy-active-node> fleet.cattle.io/agent=true --overwrite
```

1. Wait some time and check the node deletion status on the Harvester UI.

1. Remove the temporary label after the node is successfully removed from the cluster:

```sh
kubectl label node <healthy-active-node> fleet.cattle.io/agent-
```

**Related Issue**

[#11349](https://github.com/harvester/harvester/issues/11349)

## Role Management

Expand Down
94 changes: 88 additions & 6 deletions versioned_docs/version-v1.8/host/host.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -142,26 +142,108 @@ Again, removing a control plane node in this situation is **not recommended** be

### 6. Delete RKE2 services and shut down the node.

:::warning

1. **Target Node Only**
The uninstall script `/opt/rke2/bin/rke2-uninstall.sh` **MUST ONLY BE RUN DIRECTLY ON THE TARGET NODE BEING REMOVED**. Double-check your active hostname (`hostname`) before invoking it.

1. **Immediate Destruction (No Confirmation Prompt):**
The uninstall script `/opt/rke2/bin/rke2-uninstall.sh` **does NOT ask for double confirmation or prompt `y/n`** before execution. Once invoked, it immediately stops services and tears down the node environment.

:::

1. Log in to the node using the root account.

1. Run the script `/opt/rke2/bin/rke2-uninstall.sh` to delete RKE2 services running on the node.

1. Shut down the node.

### 7. Remove the node.

1. On the Harvester UI, go to the **Hosts** screen.

1. Locate the node that you want to remove, and then click **⋮ > Delete**.

![delete.png](/img/v1.2/host/delete-node.png)

This removal process typically completes within one minute. If the UI status does not update automatically, force-refresh the browser page to reload the cluster state.

:::important

Removing a node tears down the Harvester control plane components, container runtimes, and Kubernetes configurations, but it does not wipe local disks and bootloader partitions.

Forward the decommissioned host to your IT infrastructure or system administration team to perform drive sanitization according to your organization's data retention and hardware lifecycle policies.

:::

### 8. Known Issue: Node stuck in "Draining" state during removal

**Symptom**

During node deletion, the host status on the UI normally shows `Kubelet stopped posting node status`.

![uninstalled-node.png](/img/v1.9/host/node-removal-normally-uninstalled.png)

When it shows `Node is draining due to kubelet/node not ready`, you have encountered this issue..

![re-drain-node.png](/img/v1.9/host/node-removal-node-draining.png)

In the backend, `fleet-agent` pods are repeatedly scheduled onto the target node, triggering a continuous `Terminating` loop that halts node removal.

```sh
$ kubectl get pods -A -owide | grep fleet
cattle-fleet-local-system fleet-agent-6dfc5584bf-4jm7r 0/1 Terminating 0 46s <none> harv42 <none> <none>
cattle-fleet-local-system fleet-agent-747c7b5f98-44lq5 0/1 Pending 0 46s <none> harv42 <none> <none>
cattle-fleet-system fleet-controller-6c5d89f545-q925w 3/3 Running 3 (126m ago) 3h10m 10.52.0.8 harv31 <none> <none>
```

**Cause**

During node removal the cluster topology changes, and it triggers the `fleet-controller` to roll out new revisions of the `fleet-agent` deployment. Because `fleet-agent` carries broad tolerations (`node.kubernetes.io/unreachable:NoSchedule`, `node.kubernetes.io/unschedulable:NoSchedule` and more), `kube-scheduler` may continue to select the draining node as a valid target and redeploy agent pods onto it.

:::note

There's a [known issue](https://github.com/harvester/harvester/issues/1497) about node hard delete.
Once resolved, you can skip this step.
This is an intermittent issue caused by a scheduling race condition. It occurs most frequently on clusters with **3 or more control-plane nodes** when the node being deleted is a **control-plane node**, as `kube-scheduler` assigns replacement `fleet-agent` pods to it.

:::

### 7. Remove the node.
**Workaround**

1. On the Harvester UI, go to the **Hosts** screen.
The `fleet-agent` deployment template includes a preferred node affinity rule. By default, nodes in Harvester do not carry the `fleet.cattle.io/agent` label.

1. Locate the node that you want to remove, and then click **⋮ > Delete**.
```yaml
spec:
affinity:
nodeAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- preference:
matchExpressions:
- key: fleet.cattle.io/agent
operator: In
values:
- "true"
weight: 1
```

![delete.png](/img/v1.2/host/delete-node.png)
You can use this soft affinity mechanism to break the scheduling loop. Temporarily assigning `fleet.cattle.io/agent=true` to a healthy active node (other than the node being removed) increases its scheduling weight, forcing `kube-scheduler` to steer replacement fleet-agent pods away from the draining node.

1. Temporarily pin fleet-agent to a healthy active node:

```sh
kubectl label node <healthy-active-node> fleet.cattle.io/agent=true --overwrite
```

1. Wait some time and check the node deletion status on the Harvester UI.

1. Remove the temporary label after the node is successfully removed from the cluster:

```sh
kubectl label node <healthy-active-node> fleet.cattle.io/agent-
```

**Related Issue**

[#11349](https://github.com/harvester/harvester/issues/11349)

## Role Management

Expand Down
Loading