OCPBUGS-92836: fix bootstrap deadlock when ManagementCPUsOverride rejects pods with 0 nodes

[bhuvi-op]

Bug

https://redhat.atlassian.net/browse/OCPBUGS-92836

Summary

During Baremetal IPI bootstrap (specifically IBM Fusion HCI / Cistern environments), the
ManagementCPUsOverride admission plugin hard-rejects pod creation when len(nodes) == 0.
This blocks the machine-approver from starting, which prevents nodes from joining
(CSRs cannot be approved), creating a circular deadlock and stalling the installation
indefinitely.

Root Cause

In openshift-kube-apiserver/admission/autoscaling/managementcpusoverride/admission.go,
the Admit function returned admission.NewForbidden(...) when the node list was empty.
During initial bootstrap, no nodes exist yet, so every pod creation — including critical
infrastructure like the machine-approver — was rejected with:

pods "openshift-apiserver-operator-765bd75869-" is forbidden:
autoscaling.openshift.io/ManagementCPUsOverride the cluster does not have any nodes

Fix

Return nil (allow without mutation) when len(nodes) == 0. This is safe because:

• The plugin cannot determine CPU partitioning status without nodes
• Pods are allowed through unmodified (no incorrect mutations applied)
• Once nodes join, the plugin resumes normal operation for subsequent pod creations
• The customer-verified workaround (disabling the plugin entirely) confirms this code path is correct

Testing

• Unit tests pass: go test ./openshift-kube-apiserver/admission/autoscaling/managementcpusoverride/
• Existing test case updated to validate the new behavior (no error, pod passes through unmodified)
• Customer workaround (manually disabling the plugin on bootstrap) confirms installations succeed once the admission rejection is removed

/cc @ehila

[openshift-ci-robot]

@bhuvi-op: This pull request references Jira Issue OCPBUGS-92836, which is invalid:

• expected the bug to target the "4.22.0" version, but no target version was set
• release note text must be set and not match the template OR release note type must be set to "Release Note Not Required". For more information you can reference the OpenShift Bug Process.
• expected Jira Issue OCPBUGS-92836 to depend on a bug targeting a version in 5.0.0 and in one of the following states: MODIFIED, ON_QA, VERIFIED, but no dependents were found

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

Details

In response to this:

Summary

• Fix installation deadlock where ManagementCPUsOverride admission plugin hard-rejects pods when the cluster has 0 nodes during bootstrap
• Allow pods through without mutation when no nodes exist, breaking the circular dependency that prevents machine-approver from starting

Bug

https://redhat.atlassian.net/browse/OCPBUGS-92836

Root Cause

The plugin returned admission.NewForbidden when len(nodes) == 0, blocking all pod creation (including machine-approver) during bootstrap before any nodes join.

Fix

Return nil (allow without mutation) when len(nodes) == 0. This is safe because the plugin cannot determine CPU partitioning status without nodes anyway, and will function normally once nodes join.

Test plan

• Unit tests pass (go test ./openshift-kube-apiserver/admission/autoscaling/managementcpusoverride/)
• Verify Baremetal IPI installation completes on IBM Fusion HCI / Cistern setup

Made with Cursor

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Repository: openshift/coderabbit/.coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: f5f8cb36-1e91-4c69-a397-e40e3e6108de

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[openshift-ci-robot]

@bhuvi-op: the contents of this pull request could not be automatically validated.

The following commits could not be validated and must be approved by a top-level approver:

• 4dc4e98|UPSTREAM: : OCPBUGS-92836: fix bootstrap deadlock when ManagementCPUsOverride rejects pods with 0 nodes: does not specify an upstream backport in the commit message

Comment /validate-backports to re-evaluate validity of the upstream PRs, for example when they are merged upstream.

[openshift-ci]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: bhuvi-op
Once this PR has been reviewed and has the lgtm label, please assign jubittajohn for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• DOWNSTREAM_OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[openshift-ci]

Hi @bhuvi-op. Thanks for your PR.

I'm waiting for a openshift member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work.

Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[openshift-ci-robot]

@bhuvi-op: the contents of this pull request could not be automatically validated.

The following commits could not be validated and must be approved by a top-level approver:

• e0cf699|UPSTREAM: : OCPBUGS-92836: fix bootstrap deadlock when ManagementCPUsOverride rejects pods with 0 nodes: does not specify an upstream backport in the commit message

Comment /validate-backports to re-evaluate validity of the upstream PRs, for example when they are merged upstream.

[openshift-ci-robot]

@bhuvi-op: This pull request references Jira Issue OCPBUGS-92836, which is invalid:

• expected the bug to target the "4.22.0" version, but no target version was set
• release note text must be set and not match the template OR release note type must be set to "Release Note Not Required". For more information you can reference the OpenShift Bug Process.
• expected dependent Jira Issue OCPBUGS-95612 to be in one of the following states: MODIFIED, ON_QA, VERIFIED, but it is New instead
• expected dependent Jira Issue OCPBUGS-95612 to target a version in 5.0.0, but no target version was set

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

Details

In response to this:

Bug

https://redhat.atlassian.net/browse/OCPBUGS-92836

Summary

During Baremetal IPI bootstrap (specifically IBM Fusion HCI / Cistern environments), the
ManagementCPUsOverride admission plugin hard-rejects pod creation when len(nodes) == 0.
This blocks the machine-approver from starting, which prevents nodes from joining
(CSRs cannot be approved), creating a circular deadlock and stalling the installation
indefinitely.

Root Cause

In openshift-kube-apiserver/admission/autoscaling/managementcpusoverride/admission.go,
the Admit function returned admission.NewForbidden(...) when the node list was empty.
During initial bootstrap, no nodes exist yet, so every pod creation — including critical
infrastructure like the machine-approver — was rejected with:

pods "openshift-apiserver-operator-765bd75869-" is forbidden:
autoscaling.openshift.io/ManagementCPUsOverride the cluster does not have any nodes

Fix

Return nil (allow without mutation) when len(nodes) == 0. This is safe because:

• The plugin cannot determine CPU partitioning status without nodes
• Pods are allowed through unmodified (no incorrect mutations applied)
• Once nodes join, the plugin resumes normal operation for subsequent pod creations
• The customer-verified workaround (disabling the plugin entirely) confirms this code path is correct

Testing

• Unit tests pass: go test ./openshift-kube-apiserver/admission/autoscaling/managementcpusoverride/
• Existing test case updated to validate the new behavior (no error, pod passes through unmodified)
• Customer workaround (manually disabling the plugin on bootstrap) confirms installations succeed once the admission rejection is removed

/cc @ehila

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[bhuvi-op]

/assign ehila

[openshift-ci]

@bhuvi-op: GitHub didn't allow me to assign the following users: ehila.

Note that only openshift members with read permissions, repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time.
For more information please see the contributor guide

Details

In response to this:

/assign ehila

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[bhuvi-op]

/jira refresh

[openshift-ci-robot]

@bhuvi-op: This pull request references Jira Issue OCPBUGS-92836, which is invalid:

• expected the bug to target the "4.22.0" version, but no target version was set
• release note text must be set and not match the template OR release note type must be set to "Release Note Not Required". For more information you can reference the OpenShift Bug Process.
• expected dependent Jira Issue OCPBUGS-95612 to be in one of the following states: MODIFIED, ON_QA, VERIFIED, but it is New instead
• expected dependent Jira Issue OCPBUGS-95612 to target a version in 5.0.0, but no target version was set

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

Details

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[eggfoobar]

/close

Closing in favor of #2713

[openshift-ci]

@eggfoobar: Closed this PR.

Details

In response to this:

/close

Closing in favor of #2713

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[openshift-ci-robot]

@bhuvi-op: This pull request references Jira Issue OCPBUGS-92836. The bug has been updated to no longer refer to the pull request using the external bug tracker.

Details

In response to this:

Bug

https://redhat.atlassian.net/browse/OCPBUGS-92836

Summary

During Baremetal IPI bootstrap (specifically IBM Fusion HCI / Cistern environments), the
ManagementCPUsOverride admission plugin hard-rejects pod creation when len(nodes) == 0.
This blocks the machine-approver from starting, which prevents nodes from joining
(CSRs cannot be approved), creating a circular deadlock and stalling the installation
indefinitely.

Root Cause

In openshift-kube-apiserver/admission/autoscaling/managementcpusoverride/admission.go,
the Admit function returned admission.NewForbidden(...) when the node list was empty.
During initial bootstrap, no nodes exist yet, so every pod creation — including critical
infrastructure like the machine-approver — was rejected with:

pods "openshift-apiserver-operator-765bd75869-" is forbidden:
autoscaling.openshift.io/ManagementCPUsOverride the cluster does not have any nodes

Fix

Return nil (allow without mutation) when len(nodes) == 0. This is safe because:

• The plugin cannot determine CPU partitioning status without nodes
• Pods are allowed through unmodified (no incorrect mutations applied)
• Once nodes join, the plugin resumes normal operation for subsequent pod creations
• The customer-verified workaround (disabling the plugin entirely) confirms this code path is correct

Testing

• Unit tests pass: go test ./openshift-kube-apiserver/admission/autoscaling/managementcpusoverride/
• Existing test case updated to validate the new behavior (no error, pod passes through unmodified)
• Customer workaround (manually disabling the plugin on bootstrap) confirms installations succeed once the admission rejection is removed

/cc @ehila

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.