Could fleet exclude `node.kubernetes.io/unschedulable:NoSchedule` from the automatic generation of `tolerations`?

[masap]

Is there an existing issue for this?

• I have searched the existing issues

Current Behavior

According to the Fleet documentation [1], Fleet automatically generates tolerations based on the taints present on nodes. This allows Fleet pods to be deployed even on nodes with taints.

The cattle-cluster-agent uses either a fixed set of tolerations, or dynamically-added tolerations based on taints applied to the control plane nodes. This structure allows Taint based Evictions to work properly for cattle-cluster-agent.

However, this same behavior also occurs for the taint node.kubernetes.io/unschedulable:NoSchedule on nodes being deleted.

This behavior caused an issue in a Harvester user's environment. In Harvester, the user followed the official node deletion procedure [2] to delete a node, but the node deletion got stuck in the Maintenance state. A Harvester node is essentially a Kubernetes node. The deletion procedure involves Drain -> RKE2 uninstall -> Delete from Harvester UI [1]. In the user's environment, Fleet pods were continuously assigned to the target node, resulting in a state where draining could not complete.

Normally, nodes being deleted have the taint node.kubernetes.io/unschedulable:NoSchedule applied, so they are excluded from the Kubernetes scheduler. However, due to Fleet's aforementioned behavior, they become eligible for pod assignment. Additionally, because most pods are removed by the drain operation, the node appears to the scheduler as having the most available capacity. In fact, in that user's environment, the scheduler repeatedly assigned pods to that node.

To avoid this issue, would it be possible to exclude node.kubernetes.io/unschedulable:NoSchedule?

SURE-11664

[1] https://ranchermanager.docs.rancher.com/how-to-guides/new-user-guides/launch-kubernetes-with-rancher/about-rancher-agents#_scheduling_rules
[2] https://docs.harvesterhci.io/v1.8/host/#deleting-a-node

Expected Behavior

fleet exclude node.kubernetes.io/unschedulable:NoSchedule from the automatic generation of tolerations.

Steps To Reproduce

Remove Harvester nodes according to https://ranchermanager.docs.rancher.com/how-to-guides/new-user-guides/launch-kubernetes-with-rancher/about-rancher-agents#_scheduling_rules.

Environment

- Architecture: amd64
- Fleet Version: v0.14.0
- Cluster:
  - Provider: Harvester 1.7.0 (Bare metal)
  - Options: 
  - Kubernetes Version: v1.34.2+rke2r1

Logs

Anything else?

No response

[w13915984028]

Link with harvester/harvester#11349 (comment), it shows how the issue happens, please refer the case B2.

As fleet-agent has a massive deployments, not sure node.kubernetes.io/unschedulable:NoSchedule could be removed without causing other issues, Harvester books this as a known issue and had a workaround: harvester/docs#1108 . Thanks.