fix: only count topology domains the pod can actually reach - #57
Open
lukeraphael wants to merge 3 commits into
Open
fix: only count topology domains the pod can actually reach#57lukeraphael wants to merge 3 commits into
lukeraphael wants to merge 3 commits into
Conversation
Topology spread groups were seeded with every domain any NodePool could supply, so a pod pinned to one NodePool counted zero-pod domains that only other NodePools offer. That holds the group minimum at zero and puts every reachable domain outside maxSkew, making a DoNotSchedule spread permanently unsatisfiable and blocking drift/consolidation replacement. Track the NodePool requirements alongside the taints for each domain and skip domains no NodePool can supply to the pod when its NodeAffinityPolicy is honor.
🤖 Devin AI EngineerI'll be helping with this pull request! Here's what you should know: ✅ I will automatically:
Note: I can only respond to comments from users who have write access to this repository. ⚙️ Control Options:
|
There was a problem hiding this comment.
🔍 Devin Review: 1 flag
Not posted on this PR by your GitHub settings — view it in Devin Review. (Configure)
There was a problem hiding this comment.
Heron review
global-review-orchestrator-guardian
Verdict: ✅ Approve · 0 findings · head b90646ba14 · openai/gpt-5.6-luna · 3m 37s · session
Last updated 2026-08-28 08:02 UTC · reply @heron dismiss <reason> to dismiss on its comment thread
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
A topology spread group seeded its domains from every NodePool's instance-type offerings, even domains no node the pod could land on can supply. In a cluster where some NodePools span another region and a pod is pinned to one pool (
karpenter.sh/nodepool=monitoring), those foreign zones stay at count 0, sodomainMinCountis pinned at 0 and no domain is withinmaxSkew: 1of it — aDoNotSchedulezone spread becomes permanently unsatisfiable and drift/consolidation never converges (unsatisfiable topology constraint for topology spread, key=topology.kubernetes.io/zone). kube-scheduler doesn't hit this because it only counts zones of nodes matching the pod.TopologyDomainGroupnow remembers which NodePool contributed each domain, along with that pool's taints and requirements, andForEachDomaintakes the whole pod-awareTopologyNodeFilterinstead of just the taint policy:The affinity check is deliberately conflict-only (
Requirements.Intersects, notCompatible): a NodePool's stored requirements are the ones common to every node it can launch, so a label the pool leaves to the instance type is undefined there and must not count as a mismatch. Only an outright value conflict (e.g. pod wantsnodepool=monitoring, source isnodepool=accelerators) drops a domain, so the filter can never hide a domain that a node from that pool could actually provide.Domain groups stay candidate-invariant and cacheable — the pod-dependent filtering happens at
NewTopologyGrouptime.Testing
topologydomaingroup_internal_test.go: node selector, required node affinity,NodeAffinityPolicy: Ignore, taints/tolerations, and a pod selecting an instance-type-only label (domains kept).topology_test.go: end-to-end spread of 4 pods pinned to a 2-zone NodePool while another pool offers a third zone — expects skew2,2. Reverting the affinity condition reproduces the exact production error and fails this test.pkg/controllers/provisioning/scheduling,provisioning, anddisruptionsuites pass.Link to Devin session: https://app.devin.ai/sessions/e2c5be51371140be96fe3a13f15e0260
Open in Devin Desktop: https://app.devin.ai/desktop/session/e2c5be51371140be96fe3a13f15e0260?variant=devin
Requested by: @lukeraphael