feat(autoscaler): add scale-down eligibility knobs (ignore-daemonsets-utilization, skip-nodes-with-*)

charts/ksail-operator/crds/ksail.io_clusters.yaml

@@ -112,6 +112,16 @@ spec:
                                 strategy for the cluster autoscaler.
                               type: string
                             type: array
+                          ignoreDaemonsetsUtilization:
+                            description: |-
+                              IgnoreDaemonsetsUtilization excludes DaemonSet pods from a node's
+                              resource-utilization calculation when the autoscaler decides whether a node
+                              is unneeded (upstream --ignore-daemonsets-utilization, off by default). Set
+                              this when every DaemonSet is a system component whose per-node overhead
+                              should not, on its own, keep an otherwise-empty node above the scale-down
+                              threshold — otherwise heavy node agents (CNI, storage, observability) can
+                              pin every autoscaler node as "utilized" and prevent scale-down entirely.
+                            type: boolean
                           maxNodesTotal:
                             description: |-
                               MaxNodesTotal caps the total number of nodes in the cluster
@@ -190,6 +200,28 @@ spec:
                               ScaleDownUnneededTime is how long a node must be unneeded before it is
                               eligible for scale down (e.g. "10m").
                             type: string
+                          skipNodesWithLocalStorage:
+                            description: |-
+                              SkipNodesWithLocalStorage controls whether the Cluster Autoscaler refuses to
+                              scale down a node that runs a pod with local storage (emptyDir, hostPath, or
+                              a local PersistentVolume). Upstream defaults to true (never remove such
+                              nodes). Set false to allow scale-down of nodes whose only local storage is
+                              ephemeral scratch — required for overflow nodes to ever drain, since
+                              emptyDir is pervasive. A pointer so an explicit false is preserved and only
+                              an unset value inherits the upstream default. Ignored unless the node
+                              autoscaler is installed (Talos on Hetzner with enabled: true).
+                            type: boolean
+                          skipNodesWithSystemPods:
+                            description: |-
+                              SkipNodesWithSystemPods controls whether the Cluster Autoscaler refuses to
+                              scale down a node that runs a non-DaemonSet kube-system pod without a
+                              controlling PodDisruptionBudget. Upstream defaults to true. Set false to let
+                              overflow nodes hosting movable system Deployments (metrics-server, CSI
+                              controllers, relays) drain — verify those components tolerate eviction and
+                              carry PDBs. A pointer so an explicit false is preserved and only an unset
+                              value inherits the upstream default. Ignored unless the node autoscaler is
+                              installed (Talos on Hetzner with enabled: true).
+                            type: boolean
                         type: object
                       pod:
                         description: Pod configures pod-level autoscaling (horizontal

docs/src/content/docs/configuration/declarative-configuration.mdx

@@ -414,6 +414,9 @@ NodeAutoscalerConfig defines configuration for node-level autoscaling. When Enab
 | `expander` | []enum | – | Node expander strategy for the cluster autoscaler. Accepts either a single value (e.g. LeastWaste) or an ordered priority list (e.g. [LeastNodes |
 | `scaleDownUnneededTime` | string | – | How long a node should be unneeded before it is eligible for scale down (e.g. 10m) |
 | `capacityBuffers` | boolean | – | Enable the Cluster Autoscaler capacity-buffers feature: KSail installs the CapacityBuffer CRD (capacitybuffers.autoscaling.x-k8s.io) and enables the buffer controller and pod-injection flags. CapacityBuffer resources then reserve scale-up headroom as virtual (pod-less) chunks simulated in autoscaler memory — a native replacement for low-priority balloon-pod overprovisioning. Ignored unless the node autoscaler is installed (Talos on Hetzner with enabled: true) |
+| `ignoreDaemonsetsUtilization` | boolean | – | Exclude DaemonSet pods from a node's resource-utilization calculation when the Cluster Autoscaler decides whether a node is unneeded (upstream --ignore-daemonsets-utilization |
+| `skipNodesWithLocalStorage` | boolean | – | Whether the Cluster Autoscaler refuses to scale down a node running a pod with local storage (emptyDir, hostPath, or a local PersistentVolume). Upstream --skip-nodes-with-local-storage defaults to true. Set false to let nodes whose only local storage is ephemeral scratch (emptyDir) be removed — required for overflow nodes to drain, since emptyDir is pervasive. Ensure durable data lives on real PVCs first. Ignored unless the node autoscaler is installed (Talos on Hetzner with enabled: true). |
+| `skipNodesWithSystemPods` | boolean | – | Whether the Cluster Autoscaler refuses to scale down a node running a non-DaemonSet kube-system pod that has no controlling PodDisruptionBudget. Upstream --skip-nodes-with-system-pods defaults to true. Set false to let overflow nodes hosting movable system Deployments drain — confirm those components tolerate eviction and carry PDBs first. Ignored unless the node autoscaler is installed (Talos on Hetzner with enabled: true). |
 
 ### spec.cluster.autoscaler.node.pools[] (NodePool)
 

pkg/apis/cluster/v1alpha1/autoscaler.go

@@ -44,6 +44,32 @@ type NodeAutoscalerConfig struct {
 	// headroom as virtual (pod-less) chunks — a native replacement for
 	// low-priority balloon-pod overprovisioning.
 	CapacityBuffers bool `json:"capacityBuffers,omitzero" jsonschema:"description=Enable the Cluster Autoscaler capacity-buffers feature: KSail installs the CapacityBuffer CRD (capacitybuffers.autoscaling.x-k8s.io) and enables the buffer controller and pod-injection flags. CapacityBuffer resources then reserve scale-up headroom as virtual (pod-less) chunks simulated in autoscaler memory — a native replacement for low-priority balloon-pod overprovisioning. Ignored unless the node autoscaler is installed (Talos on Hetzner with enabled: true)"` //nolint:lll
+	// IgnoreDaemonsetsUtilization excludes DaemonSet pods from a node's
+	// resource-utilization calculation when the autoscaler decides whether a node
+	// is unneeded (upstream --ignore-daemonsets-utilization, off by default). Set
+	// this when every DaemonSet is a system component whose per-node overhead
+	// should not, on its own, keep an otherwise-empty node above the scale-down
+	// threshold — otherwise heavy node agents (CNI, storage, observability) can
+	// pin every autoscaler node as "utilized" and prevent scale-down entirely.
+	IgnoreDaemonsetsUtilization bool `json:"ignoreDaemonsetsUtilization,omitzero" jsonschema:"description=Exclude DaemonSet pods from a node's resource-utilization calculation when the Cluster Autoscaler decides whether a node is unneeded (upstream --ignore-daemonsets-utilization, off by default). Enable this when DaemonSets are system components (CNI, CSI, observability, security agents) whose per-node overhead should not keep an otherwise-empty node above the scale-down utilization threshold. Ignored unless the node autoscaler is installed (Talos on Hetzner with enabled: true)"` //nolint:lll
+	// SkipNodesWithLocalStorage controls whether the Cluster Autoscaler refuses to
+	// scale down a node that runs a pod with local storage (emptyDir, hostPath, or
+	// a local PersistentVolume). Upstream defaults to true (never remove such
+	// nodes). Set false to allow scale-down of nodes whose only local storage is
+	// ephemeral scratch — required for overflow nodes to ever drain, since
+	// emptyDir is pervasive. A pointer so an explicit false is preserved and only
+	// an unset value inherits the upstream default. Ignored unless the node
+	// autoscaler is installed (Talos on Hetzner with enabled: true).
+	SkipNodesWithLocalStorage *bool `json:"skipNodesWithLocalStorage,omitzero" jsonschema_description:"Whether the Cluster Autoscaler refuses to scale down a node running a pod with local storage (emptyDir, hostPath, or a local PersistentVolume). Upstream --skip-nodes-with-local-storage defaults to true. Set false to let nodes whose only local storage is ephemeral scratch (emptyDir) be removed — required for overflow nodes to drain, since emptyDir is pervasive. Ensure durable data lives on real PVCs first. Ignored unless the node autoscaler is installed (Talos on Hetzner with enabled: true)."` //nolint:lll
+	// SkipNodesWithSystemPods controls whether the Cluster Autoscaler refuses to
+	// scale down a node that runs a non-DaemonSet kube-system pod without a
+	// controlling PodDisruptionBudget. Upstream defaults to true. Set false to let
+	// overflow nodes hosting movable system Deployments (metrics-server, CSI
+	// controllers, relays) drain — verify those components tolerate eviction and
+	// carry PDBs. A pointer so an explicit false is preserved and only an unset
+	// value inherits the upstream default. Ignored unless the node autoscaler is
+	// installed (Talos on Hetzner with enabled: true).
+	SkipNodesWithSystemPods *bool `json:"skipNodesWithSystemPods,omitzero" jsonschema_description:"Whether the Cluster Autoscaler refuses to scale down a node running a non-DaemonSet kube-system pod that has no controlling PodDisruptionBudget. Upstream --skip-nodes-with-system-pods defaults to true. Set false to let overflow nodes hosting movable system Deployments drain — confirm those components tolerate eviction and carry PDBs first. Ignored unless the node autoscaler is installed (Talos on Hetzner with enabled: true)."` //nolint:lll
 }
 
 // NodePool defines a Hetzner node pool managed by the cluster autoscaler.

pkg/svc/installer/clusterautoscaler/installer.go

@@ -148,6 +148,18 @@ type chartExtraArgs struct {
 	// see no values drift.
 	CapacityBufferControllerEnabled   bool `json:"capacity-buffer-controller-enabled,omitempty"`
 	CapacityBufferPodInjectionEnabled bool `json:"capacity-buffer-pod-injection-enabled,omitempty"`
+	// IgnoreDaemonsetsUtilization drops DaemonSet pods from a node's utilization
+	// when the autoscaler evaluates it for scale-down (upstream
+	// --ignore-daemonsets-utilization). omitempty keeps it out of the rendered
+	// values unless enabled, so existing releases see no values drift.
+	IgnoreDaemonsetsUtilization bool `json:"ignore-daemonsets-utilization,omitempty"`
+	// SkipNodesWithLocalStorage and SkipNodesWithSystemPods mirror the upstream
+	// flags of the same name (both default true). They are pointers so a nil
+	// (unset) value leaves the flag out of the rendered values — inheriting the
+	// autoscaler default — while an explicit true/false is rendered verbatim
+	// (omitempty drops only the nil pointer, never a non-nil *bool to false).
+	SkipNodesWithLocalStorage *bool `json:"skip-nodes-with-local-storage,omitempty"`
+	SkipNodesWithSystemPods   *bool `json:"skip-nodes-with-system-pods,omitempty"`
 	// KubeAPIContentType forces the autoscaler's Kubernetes client to negotiate the
 	// given content type. capacity-buffers require application/json: the CapacityBuffer
 	// controller's client would otherwise negotiate protobuf (the autoscaler default for
@@ -281,13 +293,16 @@ func buildChartValues(
 		CloudProvider:     "hetzner",
 		AutoscalingGroups: groups,
 		ExtraArgs: chartExtraArgs{
-			Expander:              expandersToHelmValue(cfg.Expander),
-			ScaleDownUnneededTime: scaleDownTime,
-			MaxNodesTotal:         cfg.MaxNodesTotal,
-			ScaleDownAfterAdd:     "5m",
-			ScaleDownAfterDelete:  "2m",
-			OkTotalUnreadyCount:   defaultOkTotalUnreadyCount,
-			V:                     "4",
+			Expander:                    expandersToHelmValue(cfg.Expander),
+			ScaleDownUnneededTime:       scaleDownTime,
+			MaxNodesTotal:               cfg.MaxNodesTotal,
+			ScaleDownAfterAdd:           "5m",
+			ScaleDownAfterDelete:        "2m",
+			OkTotalUnreadyCount:         defaultOkTotalUnreadyCount,
+			V:                           "4",
+			IgnoreDaemonsetsUtilization: cfg.IgnoreDaemonsetsUtilization,
+			SkipNodesWithLocalStorage:   cfg.SkipNodesWithLocalStorage,
+			SkipNodesWithSystemPods:     cfg.SkipNodesWithSystemPods,
 		},
 		ExtraEnv:        extraEnv,
 		ExtraEnvSecrets: buildExtraEnvSecrets(),

pkg/svc/installer/clusterautoscaler/installer_test.go

@@ -649,6 +649,113 @@ func TestClusterAutoscalerInstaller_ValuesYaml_MaxNodesTotalOmittedWhenZero(t *t
 	require.NoError(t, err)
 }
 
+// TestClusterAutoscalerInstaller_ValuesYaml_IgnoreDaemonsetsUtilization verifies
+// that the ignore-daemonsets-utilization extraArg is rendered only when the knob
+// is enabled, and omitted (upstream default) otherwise so existing releases see
+// no values drift.
+func TestClusterAutoscalerInstaller_ValuesYaml_IgnoreDaemonsetsUtilization(t *testing.T) {
+	t.Parallel()
+
+	t.Run("enabled renders the flag", func(t *testing.T) {
+		t.Parallel()
+		assertAutoscalerValuesYaml(t,
+			v1alpha1.NodeAutoscalerConfig{IgnoreDaemonsetsUtilization: true},
+			[]string{"ignore-daemonsets-utilization: true"}, nil)
+	})
+
+	t.Run("disabled omits the flag", func(t *testing.T) {
+		t.Parallel()
+		assertAutoscalerValuesYaml(t,
+			v1alpha1.NodeAutoscalerConfig{IgnoreDaemonsetsUtilization: false},
+			nil, []string{"ignore-daemonsets-utilization"})
+	})
+}
+
+// assertAutoscalerValuesYaml installs with cfg and asserts the rendered chart
+// ValuesYaml contains every wantContain substring and none of the wantOmit ones.
+func assertAutoscalerValuesYaml(
+	t *testing.T,
+	cfg v1alpha1.NodeAutoscalerConfig,
+	wantContain, wantOmit []string,
+) {
+	t.Helper()
+
+	client := helm.NewMockInterface(t)
+	client.EXPECT().
+		GetReleaseStorageLabels(mock.Anything, mock.Anything, mock.Anything).
+		Return(nil, nil)
+	expectAddRepository(t, client, nil)
+	client.EXPECT().
+		InstallOrUpgradeChart(mock.Anything, mock.MatchedBy(func(spec *helm.ChartSpec) bool {
+			for _, want := range wantContain {
+				assert.Contains(t, spec.ValuesYaml, want)
+			}
+
+			for _, omit := range wantOmit {
+				assert.NotContains(t, spec.ValuesYaml, omit)
+			}
+
+			return true
+		})).
+		Return(nil, nil)
+
+	installer, err := clusterautoscalerinstaller.NewInstaller(
+		client, 5*time.Second, cfg, false, true, true,
+	)
+	require.NoError(t, err)
+	require.NoError(t, installer.Install(context.Background()))
+}
+
+// TestClusterAutoscalerInstaller_ValuesYaml_SkipNodesFlags verifies that the
+// skip-nodes-with-local-storage / skip-nodes-with-system-pods pointer knobs are
+// omitted when nil (inheriting the upstream default true) and rendered verbatim
+// when set to an explicit true or false.
+func TestClusterAutoscalerInstaller_ValuesYaml_SkipNodesFlags(t *testing.T) {
+	t.Parallel()
+
+	ptr := func(b bool) *bool { return &b }
+
+	tests := []struct {
+		name                   string
+		localStore, systemPods *bool
+		wantContain, wantOmit  []string
+	}{
+		{
+			name:     "nil omits both flags",
+			wantOmit: []string{"skip-nodes-with-local-storage", "skip-nodes-with-system-pods"},
+		},
+		{
+			name:       "explicit false renders both",
+			localStore: ptr(false),
+			systemPods: ptr(false),
+			wantContain: []string{
+				"skip-nodes-with-local-storage: false",
+				"skip-nodes-with-system-pods: false",
+			},
+		},
+		{
+			name:       "explicit true renders both",
+			localStore: ptr(true),
+			systemPods: ptr(true),
+			wantContain: []string{
+				"skip-nodes-with-local-storage: true",
+				"skip-nodes-with-system-pods: true",
+			},
+		},
+	}
+
+	for _, test := range tests {
+		t.Run(test.name, func(t *testing.T) {
+			t.Parallel()
+
+			assertAutoscalerValuesYaml(t, v1alpha1.NodeAutoscalerConfig{
+				SkipNodesWithLocalStorage: test.localStore,
+				SkipNodesWithSystemPods:   test.systemPods,
+			}, test.wantContain, test.wantOmit)
+		})
+	}
+}
+
 // TestClusterAutoscalerInstaller_ValuesYaml_CapacityBuffers verifies that
 // enabling capacityBuffers renders the two feature flags, the buffer
 // controller's RBAC rules (CapacityBuffer CRs plus the Deployment and

schemas/ksail-config.schema.json

@@ -535,6 +535,18 @@
                     "capacityBuffers": {
                       "type": "boolean",
                       "description": "Enable the Cluster Autoscaler capacity-buffers feature: KSail installs the CapacityBuffer CRD (capacitybuffers.autoscaling.x-k8s.io) and enables the buffer controller and pod-injection flags. CapacityBuffer resources then reserve scale-up headroom as virtual (pod-less) chunks simulated in autoscaler memory — a native replacement for low-priority balloon-pod overprovisioning. Ignored unless the node autoscaler is installed (Talos on Hetzner with enabled: true)"
+                    },
+                    "ignoreDaemonsetsUtilization": {
+                      "type": "boolean",
+                      "description": "Exclude DaemonSet pods from a node's resource-utilization calculation when the Cluster Autoscaler decides whether a node is unneeded (upstream --ignore-daemonsets-utilization"
+                    },
+                    "skipNodesWithLocalStorage": {
+                      "type": "boolean",
+                      "description": "Whether the Cluster Autoscaler refuses to scale down a node running a pod with local storage (emptyDir, hostPath, or a local PersistentVolume). Upstream --skip-nodes-with-local-storage defaults to true. Set false to let nodes whose only local storage is ephemeral scratch (emptyDir) be removed — required for overflow nodes to drain, since emptyDir is pervasive. Ensure durable data lives on real PVCs first. Ignored unless the node autoscaler is installed (Talos on Hetzner with enabled: true)."
+                    },
+                    "skipNodesWithSystemPods": {
+                      "type": "boolean",
+                      "description": "Whether the Cluster Autoscaler refuses to scale down a node running a non-DaemonSet kube-system pod that has no controlling PodDisruptionBudget. Upstream --skip-nodes-with-system-pods defaults to true. Set false to let overflow nodes hosting movable system Deployments drain — confirm those components tolerate eviction and carry PDBs first. Ignored unless the node autoscaler is installed (Talos on Hetzner with enabled: true)."
                     }
                   },
                   "additionalProperties": false,