target: cache compiled LabelSelector per ClusterGroup to reduce CPU

[himabindugit]

#5444
Refers to #5444

Problem

clusterGroupsForCluster calls metav1.LabelSelectorAsSelector on every reconcile loop for every ClusterGroup, with no caching. At scale this causes an allocation storm.

With ~3900 clusters × 10 ClusterGroups: 39,130 LabelSelectorAsSelector compilations per OnBundleChange event. pprof showed LabelSelectorAsSelector consuming 61% CPU on the fleet-controller, leading to OOM and lease-lock failures.

Reported in: #5444

Fix

Cache the compiled labels.Selector in a sync.Map on the Manager, keyed by namespace/name@resourceVersion. The ResourceVersion ensures the selector is recompiled when a ClusterGroup is updated.

ClusterGroups change infrequently and their count is small, so stale entries from old ResourceVersions are acceptable and not evicted.

Added unit tests covering:

• Matching and non-matching selectors
• Nil selector skipped
• Invalid selector not cached
• Cache populated after first call
• New ResourceVersion gets a new cache entry

Additional Information

AI-Assisted Code Review

This change was analyzed using Claude Code (Anthropic, claude-sonnet-4-6) as part of a broader investigation into fleet-controller CPU instability (production incident IM-890045).

What the analysis found:

• LabelSelectorAsSelector is called inside nested per-cluster × per-bundle loops in target.go, recompiling selectors on every reconcile cycle
• At production scale (~3,900 clusters, ~4,000 bundles, ~50 ClusterGroups), this results in ~600M recompilations per cycle — the root cause of the sustained CPU pin
• The memoization fix (cache keyed by ClusterGroup UID + resourceVersion)
  reduces this to O(ClusterGroups) ≈ 50 unique compilations

Evidence:

• Production pprof CPU trace (30-second capture, port-forwarded to the live pod) confirmed LabelSelectorAsSelector as the dominant hot path
• Fleet-controller logs from the July 7 incident showed a 3,990-line reconciliation burst within one minute of lease acquisition after restart

Correctness conclusion:
The AI analysis concluded that correctness is maintained — cache entries are invalidated on resourceVersion change, ensuring stale selectors are never used.
This was independently verified by Stephen O'Neal (sonealv), and GitHub Copilot.

Checklist

• I have updated the documentation via a pull request in the fleet-product-docs repository.

[Copilot]

Pull request overview

This PR optimizes target selection by caching compiled Kubernetes LabelSelector objects per ClusterGroup (keyed by namespace/name@resourceVersion) to reduce repeated metav1.LabelSelectorAsSelector compilations during reconcile loops.

Changes:

• Implement per-Manager selector compilation caching via sync.Map in clusterGroupsForCluster.
• Add unit tests for selector matching behavior and caching behavior.
• Extend the target.Manager struct to hold the selector cache.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

File	Description
internal/cmd/controller/target/query.go	Adds selector caching in clusterGroupsForCluster to reduce CPU/allocations during target queries.
internal/cmd/controller/target/query_test.go	Adds unit tests for selector matching/caching behavior.
internal/cmd/controller/target/builder.go	Adds a sync.Map field to Manager to store the selector cache.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[himabindugit]

Hi @aruiz14 — following up on this PR.
We've been running this in production with ~3,900 clusters and consistently seeing 90%+ CPU on the fleet-controller node. The fix addresses the LabelSelectorAsSelector recompile storm we identified via pprof, bringing operations from ~716M per cycle down dramatically.

Note:

1. This PR change was analyzed using Claude Code, PR description was updated saying this.
2. We have a load test running at production scale to collect metrics and will share results here once available.

Could you let us know if there's any additional information needed to move this forward, or an expected timeline for review?
Thanks

[himabindugit]

Load test + production incident evidence

Environment: Load test (2026-07-28):
Sandbox cluster — 3,500+ downstream clusters, fleet-controller at 97% CPU on a 32-core node.

30-second CPU profile (before fix):

Duration: 30.20s, Total samples = 401.12s (1328.14%)

  flat  flat%   cum    cum%
 5.10s  1.27%  222.79s 55.54%  k8s.io/apimachinery/pkg/apis/meta/v1.LabelSelectorAsSelector

LabelSelectorAsSelector accounts for 55.54% of all CPU (222.79s out of 401.12s sampled) — consistent with the original production profile where it was 61.49%.

Production incident (2026-07-27):

A post-incident investigation on our production cluster (3,932 clusters / 4,153 bundles / 25,865 bundledeployments) found fleet-controller listing all bundles every ~1.5–1.9 seconds with no resourceVersion and no limit — ~40 full-namespace etcd scans per minute. This drove the etcd node's EBS volume to its 3,000 IOPS ceiling (9.6 GB read from disk in one minute from a 1 GB database), freezing etcd and crashing fleet-controller.

The excessive LIST frequency is a direct consequence of the O(N³) reconciliation loop. The caching fix in this PR reduces reconciliation frequency dramatically, which in turn reduces LIST pressure on etcd.

[d3flex]

Can you please squash some commits in the history into a couple of logical commits? Right now the history has a lot of fixup/WIP ones. It would make easier the review and keep the history tidy. And while fixing that, two other things missing: commit body and signatures. add a few words on the commits . Mostly are empty (especially the caching change)

[himabindugit]

• clusterGroupsForCluster

Thanks for the feedback! Addressed all three points:

• Squashed commits — consolidated into 2 logical commits: one for the LabelSelector caching fix and one for the loop hoist.
• Commit bodies — added descriptions explaining the problem, the approach, and the performance impact to both commits.
• Signed-off-by — added signatures to both commits.

[himabindugit]

Pull request overview

This PR optimizes target selection by caching compiled Kubernetes LabelSelector objects per ClusterGroup (keyed by namespace/name@resourceVersion) to reduce repeated metav1.LabelSelectorAsSelector compilations during reconcile loops.

Changes:

* Implement per-`Manager` selector compilation caching via `sync.Map` in `clusterGroupsForCluster`.

* Add unit tests for selector matching behavior and caching behavior.

* Extend the `target.Manager` struct to hold the selector cache.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.
File Description
internal/cmd/controller/target/query.go Adds selector caching in clusterGroupsForCluster to reduce CPU/allocations during target queries.
internal/cmd/controller/target/query_test.go Adds unit tests for selector matching/caching behavior.
internal/cmd/controller/target/builder.go Adds a sync.Map field to Manager to store the selector cache.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

• Fixed — using cgCopy := cg before appending &cgCopy to avoid the range-variable alias bug.

• Added in TestClusterGroupsForCluster_NewResourceVersionCreatesNewCacheEntry — swaps to a second fake client returning rv=2 to verify a new cache entry is created.