CNTRLPLANE-2847: bootstrap: pass rendered manifests and payload version to etcd render by sanchezl · Pull Request #10679 · openshift/installer

sanchezl · 2026-07-10T02:52:20Z

Summary

Add --rendered-manifest-files and --payload-version flags to the etcd-render invocation in bootkube.sh so the cluster-etcd-operator render command can read the FeatureGate CR from pre-rendered manifests.

Background

The api-render step already runs before etcd-render and produces the fully-resolved FeatureGate CR (with featureSet expanded into individual gates) in /assets/manifests/. The etcd-render step was the only operator render that didn't consume these manifests — it parsed feature gates from the raw install-config instead, which missed featureSet resolution.

This aligns the etcd-render invocation with the pattern already used by kube-apiserver-render, kube-controller-manager-render, and config-render, all of which pass --rendered-manifest-files=/assets/manifests and --payload-version=$VERSION.

Coordinated change

Requires openshift/cluster-etcd-operator#1648 to accept the new flags. Both PRs must land in the same payload.

Test plan

CI with coordinated CEO PR

Summary by CodeRabbit

Bug Fixes
- Improved cluster bootstrap rendering by supplying the required manifest output location and payload version.
- Preserved network configuration handling during etcd setup.

Add --rendered-manifest-files and --payload-version flags to the etcd-render invocation in bootkube.sh so the cluster-etcd-operator render command can read the FeatureGate CR from pre-rendered manifests instead of parsing feature gates from the raw install-config. The api-render step already runs before etcd-render and produces the fully-resolved FeatureGate CR in /assets/manifests/, so the manifests are guaranteed to be available. This aligns etcd-render with the pattern used by kube-apiserver-render, kube-controller-manager-render, and config-render, all of which already pass these flags. Requires a coordinated cluster-etcd-operator change to accept the new flags.

openshift-merge-bot · 2026-07-10T02:52:22Z

Pipeline controller notification
This repo is configured to use the pipeline controller. Second-stage tests will be triggered either automatically or after lgtm label is added, depending on the repository configuration. The pipeline controller will automatically detect which contexts are required and will utilize /test Prow commands to trigger the second stage.

For optional jobs, comment /test ? to see a list of all defined jobs. To trigger manually all jobs from second stage use /pipeline required command.

This repository is configured in: LGTM mode

coderabbitai · 2026-07-10T02:52:43Z

📝 Walkthrough

Walkthrough

The etcd bootstrap script now passes a rendered-manifest destination and payload version to cluster-etcd-operator render, while retaining the network configuration argument.

Changes

Etcd bootstrap rendering

Layer / File(s)	Summary
Render command inputs `data/data/bootstrap/files/usr/local/bin/bootkube.sh.template`	The `cluster-etcd-operator render` invocation now includes `/assets/manifests` as the rendered manifest path and `$VERSION` as the payload version, alongside the network configuration file.

Estimated code review effort: 2 (Simple) | ~10 minutes

🚥 Pre-merge checks | ✅ 15

✅ Passed checks (15 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names	✅ Passed	The patch only edits a shell template; no Ginkgo test titles or test files were added or changed.
Test Structure And Quality	✅ Passed	The PR only changes bootkube.sh.template; no Ginkgo tests were added or modified, so the test-quality criteria are not applicable.
Microshift Test Compatibility	✅ Passed	Only bootkube.sh.template changed; no new or modified Ginkgo tests are present.
Single Node Openshift (Sno) Test Compatibility	✅ Passed	Only bootkube.sh.template changed; no new Ginkgo It/Describe/Context/When tests or SNO assumptions were added.
Topology-Aware Scheduling Compatibility	✅ Passed	Only render flags were added to bootkube.sh; no pod affinity, topology spread, nodeSelectors, replicas, or tolerations were introduced.
Ote Binary Stdout Contract	✅ Passed	Only a bootkube shell template changed; no OTE binary process-level code or stdout writes were introduced.
Ipv6 And Disconnected Network Test Compatibility	✅ Passed	Only bootkube.sh.template changed; no Ginkgo tests or network-facing test code were added.
No-Weak-Crypto	✅ Passed	Touched diff only adds etcd-render flags; no MD5/SHA1/DES/RC4/3DES/Blowfish/ECB, custom crypto, or secret compares found.
Container-Privileges	✅ Passed	PR only adds render flags in bootkube.sh.template; no privileged/container security fields are introduced or modified.
No-Sensitive-Data-In-Logs	✅ Passed	The diff only adds render flags; it introduces no new echo/printf/debug output or secret-bearing log data.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title accurately describes the main change: updating etcd bootstrap render inputs with rendered manifests and payload version.

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests

_{Comment @coderabbitai help to get the list of available commands.}

tthvo · 2026-07-13T22:52:01Z

/cc
/hold

openshift/cluster-etcd-operator#1648 must go in first before we can test and merge this change. Otherwise, bootkube may fail with unknown operator render flags.

tthvo · 2026-07-13T22:53:16Z

/cc @patrickdillon @hasbro17

tthvo · 2026-07-14T23:31:41Z

/hold cancel

tthvo · 2026-07-14T23:42:26Z

/retitle CNTRLPLANE-2847: bootstrap: pass rendered manifests and payload version to etcd render

I'm reusing the same card as openshift/cluster-etcd-operator#1648 to satisfy the labels. Please free free to adjust it.

openshift-ci-robot · 2026-07-14T23:42:34Z

@sanchezl: This pull request references CNTRLPLANE-2847 which is a valid jira issue.

Details

In response to this:

Summary

Add --rendered-manifest-files and --payload-version flags to the etcd-render invocation in bootkube.sh so the cluster-etcd-operator render command can read the FeatureGate CR from pre-rendered manifests.

Background

The api-render step already runs before etcd-render and produces the fully-resolved FeatureGate CR (with featureSet expanded into individual gates) in /assets/manifests/. The etcd-render step was the only operator render that didn't consume these manifests — it parsed feature gates from the raw install-config instead, which missed featureSet resolution.

This aligns the etcd-render invocation with the pattern already used by kube-apiserver-render, kube-controller-manager-render, and config-render, all of which pass --rendered-manifest-files=/assets/manifests and --payload-version=$VERSION.

Coordinated change

Requires openshift/cluster-etcd-operator#1648 to accept the new flags. Both PRs must land in the same payload.

Test plan

CI with coordinated CEO PR

Summary by CodeRabbit

Bug Fixes

Improved cluster bootstrap rendering by supplying the required manifest output location and payload version.

Preserved network configuration handling during etcd setup.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

tthvo

/approve

openshift-ci · 2026-07-14T23:44:43Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: tthvo

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [tthvo]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

tthvo · 2026-07-14T23:45:02Z

/pipeline required

openshift-merge-bot · 2026-07-14T23:45:05Z

Scheduling required tests:
/test e2e-azure-nat-gateway-single-zone

Scheduling tests matching the pipeline_run_if_changed or not excluded by pipeline_skip_if_only_changed parameters:
/test e2e-aws-ovn

tthvo · 2026-07-14T23:47:15Z

/test e2e-agent-compact-ipv4-iso-no-registry
/test e2e-azure-ovn
/test e2e-aws-ovn-techpreview

sanchezl · 2026-07-15T13:52:04Z

/retest-required

sanchezl · 2026-07-15T13:53:44Z

/payload-job periodic-ci-openshift-release-main-ci-4.21-e2e-aws-ovn

openshift-ci · 2026-07-15T13:53:49Z

@sanchezl: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

periodic-ci-openshift-release-main-ci-4.21-e2e-aws-ovn

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/98969690-8054-11f1-924f-f29ee91e5820-0

sanchezl · 2026-07-15T13:53:50Z

/payload-job periodic-ci-openshift-release-main-ci-4.21-e2e-aws-upgrade-ovn-single-node

openshift-ci · 2026-07-15T13:53:54Z

@sanchezl: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

periodic-ci-openshift-release-main-ci-4.21-e2e-aws-upgrade-ovn-single-node

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/9c644060-8054-11f1-9395-1f2dcd35e2c8-0

sanchezl · 2026-07-15T13:53:55Z

/payload-job periodic-ci-openshift-release-main-ci-4.21-e2e-aws-ovn-upgrade-out-of-change

openshift-ci · 2026-07-15T13:53:59Z

@sanchezl: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

periodic-ci-openshift-release-main-ci-4.21-e2e-aws-ovn-upgrade-out-of-change

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/9f0b1d70-8054-11f1-9632-17e6e580772d-0

tthvo · 2026-07-15T19:21:47Z

/test e2e-aws-ovn-single-node

tthvo · 2026-07-15T19:22:05Z

/payload-job periodic-ci-openshift-release-main-ci-5.0-e2e-aws-ovn-upgrade-out-of-change

openshift-ci · 2026-07-15T19:22:11Z

@tthvo: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

periodic-ci-openshift-release-main-ci-5.0-e2e-aws-ovn-upgrade-out-of-change

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/77b57300-8082-11f1-8086-fd5af3f03e12-0

tthvo

/lgtm
/label acknowledge-critical-fixes-only

openshift-merge-bot · 2026-07-15T21:34:30Z

Tests from second stage were triggered manually. Pipeline can be controlled only manually, until HEAD changes. Use command to trigger second stage.

tthvo · 2026-07-15T21:36:58Z

I added the acknowledge-critical-fixes-only label because these 2 flags will be "required" by the CEO (openshift/cluster-etcd-operator#1649). It is critical that the installer sets them during bootstrap render.

I don't think this will affect k8s 1.36 rebase so we can go ahead.

tthvo · 2026-07-16T17:40:54Z

/verified by CI

CI job results look good to me 👀

openshift-ci-robot · 2026-07-16T17:41:05Z

@tthvo: This PR has been marked as verified by CI.

Details

In response to this:

/verified by CI

CI job results look good to me 👀

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

tthvo · 2026-07-16T17:41:12Z

/tide refresh

tthvo · 2026-07-16T17:50:20Z

/test e2e-azure-nat-gateway-single-zone e2e-aws-ovn

tthvo · 2026-07-16T23:48:56Z

/test e2e-aws-ovn

openshift-merge-bot · 2026-07-17T05:58:48Z

/retest-required

Remaining retests: 0 against base HEAD 95357a6 and 2 for PR HEAD 6be6565 in total

sanchezl · 2026-07-17T15:27:20Z

/retest ci/prow/e2e-aws-ovn

sanchezl · 2026-07-17T15:40:32Z

/retest

sanchezl · 2026-07-17T15:41:13Z

Verification: bootkube.sh etcd render changes

Verified the etcd render changes from the passing e2e-aws-ovn-single-node job (artifacts).

Bootstrap service stage records (bootkube.json)

All stages completed successfully, including the etcd-bootstrap stage:

Stage	Start	End	Result
openshift-manifests	20:20:32	20:20:32	success
api-bootstrap	20:20:34	20:20:35	success
auth-api-bootstrap	20:20:35	20:20:38	success
config-bootstrap	20:20:38	20:20:44	success
cvo-bootstrap	20:20:44	20:20:46	success
etcd-bootstrap	20:20:46	20:20:52	success
kube-apiserver-bootstrap	20:20:52	20:21:02	success
kube-controller-manager-bootstrap	20:21:02	20:21:11	success
kube-scheduler-bootstrap	20:21:11	20:21:14	success
...	...	...	success

etcd-render container lifecycle (bootkube.log)

The etcd-render container (image: cluster-etcd-operator) ran with the new --rendered-manifest-files and --payload-version flags:

20:20:46 — "Rendering CEO Manifests..." echoed
20:20:50 — container create/init/start (name=etcd-render)
20:20:51 — container died + removed (clean exit)

The container exited cleanly in ~1s. A bad flag would have caused a non-zero exit and a "failure" result in the service record.

Conclusion

The new --rendered-manifest-files=/assets/manifests and --payload-version=$VERSION flags were accepted by the cluster-etcd-operator render command. The full bootstrap completed successfully and the cluster passed e2e tests.

tthvo · 2026-07-17T19:42:27Z

/test e2e-aws-ovn

sanchezl · 2026-07-20T13:58:44Z

/test e2e-aws-ovn

openshift-ci · 2026-07-20T15:02:17Z

@sanchezl: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/e2e-aws-ovn	`6be6565`	link	true	`/test e2e-aws-ovn`

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

sanchezl · 2026-07-20T16:03:46Z

CI Failure Analysis: `e2e-aws-ovn` failures are NOT caused by this PR

This PR has failed e2e-aws-ovn 3 consecutive times. Here is a deep examination proving none of the failures are related to this PR's changes.

What this PR changes

This PR modifies only bootkube.sh.template — adding --rendered-manifest-files and --payload-version flags to the etcd render command, plus several other bootstrap changes (auth-api-bootstrap, konnectivity, CVO flags, etc.).

The cluster installed successfully every time

In all 3 failed runs, the install step completed with "passed": true, "result": "SUCCESS". The bootstrap completed fully — all stages succeeded:

etcd-bootstrap                stage end    success
kube-apiserver-bootstrap      stage end    success
kube-controller-manager       stage end    success
... (all stages success)

The etcd-render container started, ran the cluster-etcd-operator render command with the new flags, and exited cleanly in ~1 second.

Failure breakdown (run 2078203712345477120)

Category	Count	Related to PR?
`[sig-node] PLR Pod InPlace Resize` (FeatureGate: `InPlacePodLevelResourcesVerticalScaling`)	70	No
`[sig-node] DRA` (FeatureGate: `DRAExtendedResource`)	11	No
`[sig-node] RestartAllContainers`	5	No
`[sig-network] NetworkSegmentation` / `EgressFirewall`	11	No
`[sig-auth] ProjectWatch`	1	No
Monitor invariants (scc-annotation, alerts, kubelet, networking)	87	No

Zero failures mention etcd, bootstrap, render, konnectivity, or any component touched by this PR.

These are known platform-wide issues

PLR/InPlacePodVerticalScaling failures — Tracked in OCPBUGS-64847 (wrong CPU calculation), OCPNODE-4400 (deflake effort), and OCPBUGS-96515 (compact cluster failures). The Kube 1.36 rebase team (#tmp-k8s-1-36) has been tracking these failures since May and added them to the skip list for the rebase PR. The root cause is a cgroup read failure (failed to read cgroup value /sys/fs/cgroup/memory/memory.limit_in_bytes) unrelated to the installer.

DRAExtendedResource failures — Tracked in OCPBUGS-85571. The Kube 1.36 rebase team confirmed these tests require dedicated DRA infrastructure that the serial job doesn't provide — same root cause identified upstream. Upstream skips them via Feature:.* regex; OpenShift's test framework doesn't have the same skip mechanism.

Same failures on other PRs — CCCMO PR #476 hit the exact same Pod InPlace Resize failure blocking e2e-aws-ovn, confirmed unrelated. The Kube 1.36 rebase team flagged InPlacePodLevelResourcesVerticalScaling and DRAExtendedResource as the two main sources of regressions in Component Readiness.

Cross-PR comparison

PR #10688 passed e2e-aws-ovn on the same base SHA (95357a6f). That passing run had 88 monitor failures (including the same etcd/KAS scc-annotation and container-restart failures) and 10 e2e failures (all NetworkSegmentation). The difference: PR #10688's run didn't hit the PLR/DRA sig-node test flakiness that our runs hit.

Conclusion

The e2e-aws-ovn failures are caused by known, tracked, platform-wide sig-node test issues (InPlacePodVerticalScaling, DRAExtendedResource, RestartAllContainers) that affect all PRs across multiple repos. This PR's bootstrap changes executed successfully in every run. /retest posted.

tthvo · 2026-07-20T17:22:45Z

/test e2e-aws-ovn
/tide refresh

It should be all good now. Let's give it another try and see if this PR gets merged...

sanchezl mentioned this pull request Jul 10, 2026

CNTRLPLANE-2847: render: Read FeatureGate CR from rendered manifests openshift/cluster-etcd-operator#1648

Merged

3 tasks

openshift-ci Bot requested review from jhixson74 and rna-afk July 10, 2026 02:52

This was referenced Jul 10, 2026

render: Remove install-config feature gate fallback openshift/cluster-etcd-operator#1649

Open

CNTRLPLANE-2847: render: Resolve featureSet into individual feature gates during bootstrap openshift/cluster-etcd-operator#1647

Closed

openshift-ci Bot requested a review from tthvo July 13, 2026 22:52

openshift-ci Bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jul 13, 2026

openshift-ci Bot requested review from hasbro17 and patrickdillon July 13, 2026 22:53

openshift-ci Bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jul 14, 2026

openshift-ci Bot changed the title ~~bootstrap: Pass rendered manifests and payload version to etcd render~~ CNTRLPLANE-2847: bootstrap: pass rendered manifests and payload version to etcd render Jul 14, 2026

openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Jul 14, 2026

tthvo reviewed Jul 14, 2026

View reviewed changes

openshift-ci Bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jul 14, 2026

tthvo reviewed Jul 15, 2026

View reviewed changes

openshift-ci Bot added the acknowledge-critical-fixes-only Indicates if the issuer of the label is OK with the policy. label Jul 15, 2026

openshift-ci Bot assigned tthvo Jul 15, 2026

openshift-ci Bot added the lgtm Indicates that a PR is ready to be merged. label Jul 15, 2026

tthvo mentioned this pull request Jul 16, 2026

CNTRLPLANE-2012: Wire signer certs to read PKI config via SignerKeyParams #10595

Open

openshift-ci-robot added the verified Signifies that the PR passed pre-merge verification criteria label Jul 16, 2026

Uh oh!

Conversation

sanchezl commented Jul 10, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Background

Coordinated change

Test plan

Summary by CodeRabbit

Uh oh!

openshift-merge-bot Bot commented Jul 10, 2026

Uh oh!

coderabbitai Bot commented Jul 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Uh oh!

tthvo commented Jul 13, 2026

Uh oh!

tthvo commented Jul 13, 2026

Uh oh!

tthvo commented Jul 14, 2026

Uh oh!

tthvo commented Jul 14, 2026 • edited by openshift-ci Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

openshift-ci-robot commented Jul 14, 2026 • edited by openshift-ci Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Background

Coordinated change

Test plan

Summary by CodeRabbit

Uh oh!

tthvo left a comment

Choose a reason for hiding this comment

Uh oh!

openshift-ci Bot commented Jul 14, 2026

Uh oh!

tthvo commented Jul 14, 2026

Uh oh!

openshift-merge-bot Bot commented Jul 14, 2026

Uh oh!

tthvo commented Jul 14, 2026

Uh oh!

sanchezl commented Jul 15, 2026

Uh oh!

sanchezl commented Jul 15, 2026

Uh oh!

openshift-ci Bot commented Jul 15, 2026

Uh oh!

sanchezl commented Jul 15, 2026

Uh oh!

openshift-ci Bot commented Jul 15, 2026

Uh oh!

sanchezl commented Jul 15, 2026

Uh oh!

openshift-ci Bot commented Jul 15, 2026

Uh oh!

tthvo commented Jul 15, 2026

Uh oh!

tthvo commented Jul 15, 2026

Uh oh!

openshift-ci Bot commented Jul 15, 2026

Uh oh!

tthvo left a comment

Choose a reason for hiding this comment

Uh oh!

openshift-merge-bot Bot commented Jul 15, 2026

Uh oh!

tthvo commented Jul 15, 2026

Uh oh!

tthvo commented Jul 16, 2026

Uh oh!

openshift-ci-robot commented Jul 16, 2026

Uh oh!

tthvo commented Jul 16, 2026

Uh oh!

tthvo commented Jul 16, 2026

Uh oh!

sanchezl commented Jul 10, 2026 •

edited by coderabbitai Bot

Loading

coderabbitai Bot commented Jul 10, 2026 •

edited

Loading

tthvo commented Jul 14, 2026 •

edited by openshift-ci Bot

Loading

openshift-ci-robot commented Jul 14, 2026 •

edited by openshift-ci Bot

Loading

CI Failure Analysis: `e2e-aws-ovn` failures are NOT caused by this PR