Skip to content

CNTRLPLANE-2847: render: Read FeatureGate CR from rendered manifests#1648

Merged
openshift-merge-bot[bot] merged 1 commit into
openshift:mainfrom
sanchezl:render-featuregate-cr
Jul 14, 2026
Merged

CNTRLPLANE-2847: render: Read FeatureGate CR from rendered manifests#1648
openshift-merge-bot[bot] merged 1 commit into
openshift:mainfrom
sanchezl:render-featuregate-cr

Conversation

@sanchezl

@sanchezl sanchezl commented Jul 10, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Replace manual install-config feature gate parsing with the standard operator render pattern: read the pre-resolved FeatureGate CR from rendered manifests via --rendered-manifest-files and --payload-version
  • Include a fallback that resolves featureSet and featureGates from the install-config when the new flags are not yet available
  • Fix the root cause of the featureSet resolution gap that PR #1647 addresses

Background

The CEO render was the only operator that parsed feature gates directly from the raw install-config (featureGates: [Key=Value] strings). Every other operator render (CKASO, CKCMO, MCO) reads the pre-resolved FeatureGate CR from /assets/manifests/ — the api-render step in bootkube.sh runs first and resolves featureSet (e.g. TechPreviewNoUpgrade) into individual gates, writing the result into the CR status.

Because the CEO was parsing the raw install-config, it missed featureSet resolution entirely, causing ConfigurablePKI to not be detected when using featureSet: TechPreviewNoUpgrade instead of explicit featureGates entries.

Approach

Primary path (when --rendered-manifest-files and --payload-version are provided): Read the pre-resolved FeatureGate CR from rendered manifests, matching the pattern used by CKASO and CKCMO. This eliminates the entire class of feature-gate-resolution bugs.

Fallback path (when the flags are not provided): Resolve featureSet into individual gates via features.FeatureSets() from openshift/api, then layer explicit featureGates overrides on top. This fallback uses strings.Cut for safe parsing and handles the featureSetfeatureGates override correctly. It should be removed once the installer carries the coordinated change.

Changes

pkg/cmd/render/render.go:

  • Add optional --rendered-manifest-files and --payload-version flags
  • Add getFeatureGates() dispatcher that tries the CR path, falls back to install-config
  • Add getFeatureGatesFromManifests() — reads FeatureGate CR, returns enabled/disabled sets
  • Add getFeatureGatesFromInstallConfig() — resolves featureSet + featureGates from install-config
  • Remove the old getFeatureGatesStatus() function

pkg/cmd/render/render_test.go:

  • Add writeFeatureGateManifest() test helper for the CR path
  • Add Test_getFeatureGatesFromInstallConfig testing featureSet resolution, overrides
  • Update test infrastructure to provide rendered manifest directories
  • Update TestTemplateDataWithCustomPKI to provide ConfigurablePKI via FeatureGate CR

Coordinated change

openshift/installer#10679 passes --rendered-manifest-files and --payload-version to the etcd-render invocation. This CEO PR can land independently — the fallback path handles the transition. Once the installer PR merges, a follow-up can remove the fallback.

Test plan

  • go test ./pkg/cmd/render/ — all tests pass (both primary and fallback paths)
  • go vet ./pkg/cmd/render/ — clean
  • CI

Summary by CodeRabbit

  • New Features
    • Rendering can now determine feature gate enablement from pre-rendered FeatureGate manifests supplied by the user.
    • Added CLI flags to specify rendered manifest locations and the payload version used during rendering.
  • Bug Fixes
    • Bootstrap certificate secrets now consistently reflect enabled/disabled feature gates derived from the rendered manifests or the install configuration (including correct handling of configurable PKI selection).
  • Tests
    • Updated the rendering/template test harness to generate a FeatureGate manifest at runtime and extended coverage for parsing feature gates from install configuration.

@coderabbitai

coderabbitai Bot commented Jul 10, 2026

Copy link
Copy Markdown

Walkthrough

The render command accepts optional rendered-manifest paths and a payload version, resolves FeatureGate configuration from manifests when both are available, and falls back to install-config parsing otherwise. Tests now generate FeatureGate manifests and pass these options.

Changes

FeatureGate rendering

Layer / File(s) Summary
Render inputs and option wiring
pkg/cmd/render/render.go
Render options and CLI flags carry rendered manifest paths and payload version into template data creation.
FeatureGate resolution and gate sets
pkg/cmd/render/render.go
FeatureGate manifests are parsed into enabled and disabled sets, with install-config featureSet and featureGates used as fallback inputs.
Manifest-backed render tests
pkg/cmd/render/render_test.go
Test helpers write FeatureGate manifests, pass the new options, configure custom PKI gates explicitly, and validate install-config fallback parsing.

Estimated code review effort: 4 (Complex) | ~45 minutes

Sequence Diagram(s)

sequenceDiagram
  participant RenderCommand
  participant renderOpts
  participant FeatureGateManifest
  participant InstallConfig
  participant TemplateData
  RenderCommand->>renderOpts: pass manifest paths and payload version
  renderOpts->>FeatureGateManifest: scan and parse FeatureGate resource
  FeatureGateManifest-->>renderOpts: return enabled and disabled feature sets
  renderOpts->>InstallConfig: parse fallback featureSet and featureGates
  renderOpts->>TemplateData: provide resolved feature gate state
  TemplateData-->>RenderCommand: create template and PKI data
Loading

Suggested reviewers: ingvagabund, everettraven

🚥 Pre-merge checks | ✅ 14 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (14 passed)
Check name Status Explanation
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names ✅ Passed All test titles in the changed file are fixed strings; no dynamic values, timestamps, UUIDs, or generated identifiers appear in titles.
Test Structure And Quality ✅ Passed PASS: pkg/cmd/render has no Ginkgo tests; the touched tests are standard table-driven testing cases with t.TempDir and no Eventually/cluster cleanup concerns.
Microshift Test Compatibility ✅ Passed No new Ginkgo e2e tests were added; the touched tests are standard Go unit tests using testing.T, so MicroShift API restrictions don’t apply.
Single Node Openshift (Sno) Test Compatibility ✅ Passed No new Ginkgo e2e tests were added; the PR only changes pkg/cmd/render unit tests and contains no multi-node or SNO-sensitive assumptions.
Topology-Aware Scheduling Compatibility ✅ Passed The PR only changes feature-gate parsing/test fixtures; no new affinity, topology spread, nodeSelector, PDB, or replica-scheduling logic was added.
Ote Binary Stdout Contract ✅ Passed No process-level stdout writes were added; render.go only logs via klog/errOut, render_test.go has no TestMain/init, and vendored klog defaults to stderr.
Ipv6 And Disconnected Network Test Compatibility ✅ Passed Touched tests are unit tests only; no It/Describe/Context/When constructs or external-network calls were added, so the IPv6/disconnected e2e check is not applicable.
No-Weak-Crypto ✅ Passed Changed code only adds feature-gate parsing/tests; no MD5/SHA1/DES/RC4/3DES/Blowfish/ECB, custom crypto, or secret/token comparisons found.
Container-Privileges ✅ Passed PR only changes feature-gate parsing/test helpers; no modified manifests or securityContext fields add privileged/root/host* settings.
No-Sensitive-Data-In-Logs ✅ Passed No new log statements expose secrets or PII; the added code only logs bootstrap IP/strategy and a missing-flags warning.
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly matches the main change: render now reads the FeatureGate CR from rendered manifests.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@pkg/cmd/render/render.go`:
- Around line 863-874: In the FeatureGate manifest loop, replace each silent
continue for os.ReadFile, yaml.YAMLToJSON, and json.Unmarshal failures with
contextual error returns that identify the affected file and wrap the original
error; update the enclosing function as needed to propagate these errors instead
of reporting that no FeatureGate exists. Use the loop and FeatureGate decoding
logic around os.ReadFile, YAMLToJSON, and json.Unmarshal as the implementation
reference.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository: openshift/coderabbit/.coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 7e583b38-9d3d-4186-bf0c-2dc2791d0d0e

📥 Commits

Reviewing files that changed from the base of the PR and between 831d5e6 and 05a3ad1.

📒 Files selected for processing (2)
  • pkg/cmd/render/render.go
  • pkg/cmd/render/render_test.go

Comment thread pkg/cmd/render/render.go
Comment on lines +863 to +874
for _, file := range files {
data, err := os.ReadFile(file)
if err != nil {
continue
}
jsonBytes, err := yaml.YAMLToJSON(data)
if err != nil {
continue
}
fg := &configv1.FeatureGate{}
if err := json.Unmarshal(jsonBytes, fg); err != nil {
continue

@coderabbitai coderabbitai Bot Jul 10, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🩺 Stability & Availability | 🟡 Minor | ⚡ Quick win

Preserve manifest read and decoding errors.

These continue branches hide the actual failing file and can misleadingly report that no FeatureGate exists. Return contextual errors instead. As per path instructions, Go code must never ignore error returns.

Proposed fix
 data, err := os.ReadFile(file)
 if err != nil {
-	continue
+	return nil, fmt.Errorf("failed to read manifest %s: %w", file, err)
 }
 jsonBytes, err := yaml.YAMLToJSON(data)
 if err != nil {
-	continue
+	return nil, fmt.Errorf("failed to convert manifest %s to JSON: %w", file, err)
 }
 fg := &configv1.FeatureGate{}
 if err := json.Unmarshal(jsonBytes, fg); err != nil {
-	continue
+	return nil, fmt.Errorf("failed to decode manifest %s: %w", file, err)
 }
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
for _, file := range files {
data, err := os.ReadFile(file)
if err != nil {
continue
}
jsonBytes, err := yaml.YAMLToJSON(data)
if err != nil {
continue
}
fg := &configv1.FeatureGate{}
if err := json.Unmarshal(jsonBytes, fg); err != nil {
continue
for _, file := range files {
data, err := os.ReadFile(file)
if err != nil {
return nil, fmt.Errorf("failed to read manifest %s: %w", file, err)
}
jsonBytes, err := yaml.YAMLToJSON(data)
if err != nil {
return nil, fmt.Errorf("failed to convert manifest %s to JSON: %w", file, err)
}
fg := &configv1.FeatureGate{}
if err := json.Unmarshal(jsonBytes, fg); err != nil {
return nil, fmt.Errorf("failed to decode manifest %s: %w", file, err)
}
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@pkg/cmd/render/render.go` around lines 863 - 874, In the FeatureGate manifest
loop, replace each silent continue for os.ReadFile, yaml.YAMLToJSON, and
json.Unmarshal failures with contextual error returns that identify the affected
file and wrap the original error; update the enclosing function as needed to
propagate these errors instead of reporting that no FeatureGate exists. Use the
loop and FeatureGate decoding logic around os.ReadFile, YAMLToJSON, and
json.Unmarshal as the implementation reference.

Source: Path instructions

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The continue is intentional — this function scans a directory of mixed manifests (e.g. Deployment, ConfigMap, FeatureGate). Files that are not valid YAML or do not unmarshal into a FeatureGate are expected and must be skipped. Returning an error here would break on the first non-FeatureGate manifest.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@sanchezl That makes sense — since readFeatureGate scans a directory that can contain arbitrary manifest kinds (Deployment, ConfigMap, FeatureGate, etc.), skipping files that aren't valid YAML or don't unmarshal into a FeatureGate is the correct behavior, not an ignored error. Thanks for the clarification, withdrawing this suggestion.


✏️ Learnings added
Learnt from: sanchezl
Repo: openshift/cluster-etcd-operator PR: 1648
File: pkg/cmd/render/render.go:877-888
Timestamp: 2026-07-13T16:41:46.375Z
Learning: In `pkg/cmd/render/render.go` (cluster-etcd-operator), the `readFeatureGate` function on `renderOpts` scans directories of pre-rendered manifests that contain mixed resource kinds (Deployment, ConfigMap, FeatureGate, etc.). The `continue` statements on `os.ReadFile`, `yaml.YAMLToJSON`, and `json.Unmarshal` failures inside this loop are intentional: non-FeatureGate or non-YAML files are expected and must be skipped rather than treated as errors.

Note: Learnings are effective only in the context of similar code segments. To apply general coding guidelines, please configure review instructions. You can manage existing learnings in the app.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🧹 Nitpick comments (1)
pkg/cmd/render/render_test.go (1)

650-658: 🎯 Functional Correctness | 🔵 Trivial | 💤 Low value

Keep the feature-gate fixture consistent. defaultFeatureGates() already puts ConfigurablePKI in disabled, so appending it to enabled leaves the generated manifest listing the gate in both sections. Move it from disabled to enabled for this test fixture.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@pkg/cmd/render/render_test.go` around lines 650 - 658, Update the test
fixture setup around defaultFeatureGates() so FeatureGateConfigurablePKI is
moved from the disabled slice to the enabled slice rather than appended to both;
remove it from disabled before assigning enabledFeatureGates and
disabledFeatureGates in testConfig.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@pkg/cmd/render/render.go`:
- Around line 922-935: The fallback feature-gate parser uses unchecked
assertions that can panic on malformed install-config data. In the parser
containing featureGatesRaw and enabled/disabled updates, validate that
featureGatesRaw is a []any and each entry is a string, returning contextual
errors for invalid types; propagate these errors through newTemplateData so
parsing fails cleanly.
- Around line 817-823: Update renderOpts.getFeatureGates to distinguish both
flags being absent from only one being provided: fall back to install-config
parsing only when renderedManifestFiles and payloadVersion are both empty; if
exactly one is set, return a configuration error instead of silently falling
back, and continue using getFeatureGatesFromManifests when both are present.

---

Nitpick comments:
In `@pkg/cmd/render/render_test.go`:
- Around line 650-658: Update the test fixture setup around
defaultFeatureGates() so FeatureGateConfigurablePKI is moved from the disabled
slice to the enabled slice rather than appended to both; remove it from disabled
before assigning enabledFeatureGates and disabledFeatureGates in testConfig.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository: openshift/coderabbit/.coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: b49c7794-19d5-4e54-bb09-cfc5fe96ab1d

📥 Commits

Reviewing files that changed from the base of the PR and between 05a3ad1 and 74d2dc0.

📒 Files selected for processing (2)
  • pkg/cmd/render/render.go
  • pkg/cmd/render/render_test.go

Comment thread pkg/cmd/render/render.go
Comment thread pkg/cmd/render/render.go

@hasbro17 hasbro17 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See: https://github.com/openshift/cluster-etcd-operator/pull/1648/changes#r3561973119
TLDR: The fallback getFeatureGatesFromInstallConfig must resolve configv1.Default when featureSet is absent from install-config, otherwise the empty gate sets cause a panic at
featureGate.Enabled(ShortCertRotation) during bootstrap. See this job's
bootkube.log
.

Comment thread pkg/cmd/render/render.go
enabled := sets.New[configv1.FeatureGateName]()
disabled := sets.New[configv1.FeatureGateName]()

if featureSetRaw, found := installConfig["featureSet"]; found {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this fallback path is still causing issues on non-techpreview installs. I'm seeing repeated panics in the bootkube logs for etcd-render.

Jul 10 03:42:15 ip-10-0-97-204 bootkube.sh[5696]: W0710 03:42:15.634081       1 render.go:821] --rendered-manifest-files or --payload-version not provided, falling back to install-config feature gate parsing
Jul 10 03:42:15 ip-10-0-97-204 bootkube.sh[5696]: panic: feature "ShortCertRotation" is not registered in FeatureGates []
...
Jul 10 03:42:15 ip-10-0-97-204 systemd[1]: bootkube.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Jul 10 03:42:15 ip-10-0-97-204 systemd[1]: bootkube.service: Failed with result 'exit-code'.
Jul 10 03:42:15 ip-10-0-97-204 systemd[1]: bootkube.service: Consumed 20.897s CPU time, 1.3G memory peak.
Jul 10 03:42:20 ip-10-0-97-204 systemd[1]: bootkube.service: Scheduled restart job, restart counter is at 1.

Will share the review bot analysis below for more details.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖
The fallback path in getFeatureGatesFromInstallConfig panics on Default installs where featureSet is absent from the install-config (the standard case — configv1.Default is "" and omitempty causes the
installer to omit it entirely).

When neither featureSet nor featureGates is present, the function returns empty enabled/disabled sets. These flow through createBootstrapCertSecretscreateCertSecrets (certs.go:58) →
NewEtcdCertSignerController (etcdcertsignercontroller.go:159) → NewHardcodedFeatureGateAccess([], []). The controller then calls featureGates.Enabled(features.FeatureShortCertRotation), and since
ShortCertRotation is in neither set, featuregate.go:39 panics:

panic(fmt.Errorf("feature %q is not registered in FeatureGates %v", key, f.KnownFeatures()))

The old getFeatureGatesStatus prevented this with disabled.Insert(necessaryFeatureGates...) where necessaryFeatureGates = []configv1.FeatureGateName{"ShortCertRotation"}. The new code drops that safety
net without a complete replacement for the absent-featureSet case.

CI evidence: The bootkube.log from this job's artifacts
shows the etcd-render container crashing twice with the panic stack trace:

Jul 10 03:42:15 bootkube.sh[5696]: etcdcertsignercontroller.go:159 +0x528
Jul 10 03:42:15 bootkube.sh[5696]: certs.go:58 +0xb49
Jul 10 03:42:15 bootkube.sh[5696]: render.go:301 +0xbdb
...
Jul 10 03:42:15 systemd[1]: bootkube.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Jul 10 03:42:15 systemd[1]: bootkube.service: Failed with result 'exit-code'.
Jul 10 03:42:20 systemd[1]: bootkube.service: Scheduled restart job, restart counter is at 1.

Then the same panic repeats at 03:42:26. The job eventually passes because bootkube.service keeps restarting and on a later attempt the api-render step has already produced the FeatureGate CR manifest, so
the primary getFeatureGatesFromManifests path succeeds. But the fallback panics every time it's hit, adding fragility and latency to bootstrap.

The install-config for this job (install-config.yaml) confirms no featureSet field is present.

Suggested fix: Always resolve the featureSet, defaulting to configv1.Default when absent:

featureSet := configv1.Default
if featureSetRaw, found := installConfig["featureSet"]; found {
    if featureSetStr, ok := featureSetRaw.(string); ok {
        featureSet = configv1.FeatureSet(featureSetStr)
    }
}
if resolved := features.FeatureSets(0, features.SelfManaged, featureSet); resolved != nil {
    // populate enabled/disabled from resolved
}

This populates all known gates (including ShortCertRotation in disabled) for every featureSet, making the old hardcode genuinely unnecessary.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I know this goes away when openshift/installer#10679 lands but this has to come first I think.

@sanchezl
sanchezl force-pushed the render-featuregate-cr branch from d5bbe4f to ad61905 Compare July 13, 2026 14:24

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@pkg/cmd/render/render_test.go`:
- Around line 1057-1067: Extend the test cases around the existing
ConfigurablePKI feature-gate scenarios in the render test table with a Default
featureSet configuration that explicitly sets featureGates to
ConfigurablePKI=true and expects ConfigurablePKI enabled. This should exercise
the fallback parser’s explicit-enable branch without changing the existing
TechPreview or false-override cases.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository: openshift/coderabbit/.coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 473854b1-b33d-4caa-9078-8020dc0593db

📥 Commits

Reviewing files that changed from the base of the PR and between d5bbe4f and ad61905.

📒 Files selected for processing (2)
  • pkg/cmd/render/render.go
  • pkg/cmd/render/render_test.go
🚧 Files skipped from review as they are similar to previous changes (1)
  • pkg/cmd/render/render.go

Comment on lines +1057 to 1067
"featureGates override disables ConfigurablePKI even with TechPreview": {
installConfig: `
apiVersion: v1
metadata:
name: my-cluster
featureGates: [ShortCertRotation=true, UpgradeStatus=false]
featureSet: TechPreviewNoUpgrade
featureGates: [ConfigurablePKI=false]
`,
expectedEnabled: sets.New(features.FeatureShortCertRotation),
expectedDisabled: sets.New(features.FeatureGateUpgradeStatus),
},
"unexpected data": {
installConfig: `
apiVersion: v1
metadata:
name: my-cluster
featureGates: [ShortCertRotation=true, UpgradeStatus=foobar]
`,
expectedEnabled: sets.New(features.FeatureShortCertRotation),
expectedDisabled: sets.New[configv1.FeatureGateName](),
expectConfigurablePKI: false,
},
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

Cover explicit fallback enablement.

Add a Default-featureSet case with featureGates: [ConfigurablePKI=true]. The suite covers FeatureSet enablement and the false override, but not the fallback parser’s explicit-enable branch; the Custom PKI integration test uses manifest-backed resolution instead. As per coding guidelines, “All changes must include unit test additions/changes.”

Suggested test case
+       "featureGates override enables ConfigurablePKI with default featureSet": {
+           installConfig: `
+apiVersion: v1
+metadata:
+  name: my-cluster
+featureGates: [ConfigurablePKI=true]
+`,
+           expectConfigurablePKI: true,
+       },
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
"featureGates override disables ConfigurablePKI even with TechPreview": {
installConfig: `
apiVersion: v1
metadata:
name: my-cluster
featureGates: [ShortCertRotation=true, UpgradeStatus=false]
featureSet: TechPreviewNoUpgrade
featureGates: [ConfigurablePKI=false]
`,
expectedEnabled: sets.New(features.FeatureShortCertRotation),
expectedDisabled: sets.New(features.FeatureGateUpgradeStatus),
},
"unexpected data": {
installConfig: `
apiVersion: v1
metadata:
name: my-cluster
featureGates: [ShortCertRotation=true, UpgradeStatus=foobar]
`,
expectedEnabled: sets.New(features.FeatureShortCertRotation),
expectedDisabled: sets.New[configv1.FeatureGateName](),
expectConfigurablePKI: false,
},
}
"featureGates override disables ConfigurablePKI even with TechPreview": {
installConfig: `
apiVersion: v1
metadata:
name: my-cluster
featureSet: TechPreviewNoUpgrade
featureGates: [ConfigurablePKI=false]
`,
expectConfigurablePKI: false,
},
"featureGates override enables ConfigurablePKI with default featureSet": {
installConfig: `
apiVersion: v1
metadata:
name: my-cluster
featureGates: [ConfigurablePKI=true]
`,
expectConfigurablePKI: true,
},
}
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@pkg/cmd/render/render_test.go` around lines 1057 - 1067, Extend the test
cases around the existing ConfigurablePKI feature-gate scenarios in the render
test table with a Default featureSet configuration that explicitly sets
featureGates to ConfigurablePKI=true and expects ConfigurablePKI enabled. This
should exercise the fallback parser’s explicit-enable branch without changing
the existing TechPreview or false-override cases.

Source: Coding guidelines

@sanchezl

Copy link
Copy Markdown
Contributor Author

/retest

1 similar comment
@sanchezl

Copy link
Copy Markdown
Contributor Author

/retest

@hasbro17

Copy link
Copy Markdown
Contributor

/testwith openshift/cluster-etcd-operator/main/e2e-aws-ovn-pki-rsa-techpreview openshift/installer#10595

@hasbro17

Copy link
Copy Markdown
Contributor

/retest-required

Comment thread pkg/cmd/render/render.go
Comment on lines +93 to +94
fs.StringArrayVar(&r.renderedManifestFiles, "rendered-manifest-files", nil, "Files or directories containing pre-rendered manifests, used to determine FeatureGate status.")
fs.StringVar(&r.payloadVersion, "payload-version", "", "Version that will eventually be placed into ClusterOperator.status. This normally comes from the CVO set via env var: OPERATOR_IMAGE_VERSION.")

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These flags rendered-manifest-files and payload-version are already defined as generic render flags in openshift/library-go@pkg/operator/render/options/generic.go#L57-L59, right? I'm referencing kube-controller-render, which also accepts the 2 flags.

We may already have existing feature-gate funcs to help us here too, for example 👀

@sanchezl sanchezl Jul 14, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, CKCMO and CKASO embed genericrenderoptions.GenericOptions which provides rendered-manifest-files, payload-version, and FeatureGates() out of the box.

The etcd render command does not use library-go's generic render options infrastructure at all, it has its own renderOpts struct with manually-defined flags. Refactoring it to embed GenericOptions would be a much larger change (the entire render pipeline would need to be restructured to match the pattern CKCMO uses with ManifestOptions + GenericOptions).

This PR takes the narrower approach doing the minimum needed to unblock the configurable PKI work. A full refactor to align with the generic render framework is worth doing but can be a followup effort.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The etcd render command does not use library-go's generic render options infrastructure at all

Oh whoops, my bad, I didn't realize that at all! Thanks for the details 🙏

@sanchezl

Copy link
Copy Markdown
Contributor Author

/retest

@sanchezl

Copy link
Copy Markdown
Contributor Author

/testwith openshift/cluster-etcd-operator/main/e2e-aws-ovn-pki-rsa-techpreview openshift/installer#10595

@sanchezl

Copy link
Copy Markdown
Contributor Author

/test e2e-agnostic-ovn

@sanchezl

Copy link
Copy Markdown
Contributor Author

/test e2e-gcp-operator

@sanchezl

Copy link
Copy Markdown
Contributor Author

/test e2e-aws-ovn-single-node

@hasbro17

Copy link
Copy Markdown
Contributor

Alright so this works now looking at the etcd-signer that's correctly configured per the PKI config to be RSA 4096 as the test does it.
https://prow.ci.openshift.org/view/gs/test-platform-results/logs/multi-pr-openshift-cluster-etcd-operator-1648-openshift-installer-10595-e2e-aws-ovn-pki-rsa-techpreview/2077020289362825216

--- Checking: etcd-signer (openshift-etcd/etcd-signer) ---
...
        Subject: CN=openshift-etcd_etcd-signer@1784038543
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (4096 bit)

There's a typo bug in my test. The verification script checks for secret etcd-metrics-signer but the operator creates it as etcd-metric-signer (no trailing 's'):

--- Checking: etcd-metrics-signer (openshift-etcd/etcd-metrics-signer) ---
Could not find certificate from <stdin>

  FAIL: Expected algorithm 'rsaEncryption', got ''
  FAIL: Expected 'Public-Key: (4096 bit)', got 'not found'

Will fix that separately. But this should be good to move forward.

@hasbro17 hasbro17 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/approve

Just one nit

Comment on lines +650 to +651
enabled, disabled := defaultFeatureGates()
enabled = append(enabled, features.FeatureGateConfigurablePKI)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The test appends ConfigurablePKI to the enabled list but doesn't remove it from disabled (which defaultFeatureGates() populates from the Default feature set). The resulting FeatureGate manifest has
ConfigurablePKI in both sections.
Works only because Enabled() checks the enabled slice first.
Fix: filter ConfigurablePKI out of the disabled slice when adding it to enabled.

@openshift-ci

openshift-ci Bot commented Jul 14, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: hasbro17

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci openshift-ci Bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jul 14, 2026
@sanchezl

Copy link
Copy Markdown
Contributor Author

Verification Analysis

What changed

  1. Primary path: Added --rendered-manifest-files and --payload-version flags to the render command. When both are provided, reads the pre-resolved FeatureGate CR from rendered manifests (matching the CKASO/CKCMO pattern).
  2. Fallback path: When flags are not provided, resolves feature gates from install-config's featureSet field, defaulting to configv1.Default when absent.
  3. Panic fix: The fallback now always resolves all known feature gates for the featureSet. Previously, an absent featureSet (the standard case for Default installs) returned empty gate sets, causing featureGate.Enabled(ShortCertRotation) to panic.

CI Results (9/9 passing)

Check Status Build
unit PASS 2076838282171256832
verify PASS 2076838282200616960
verify-deps PASS 2076674200155721728
images PASS 2076674189242142720
okd-scos-images PASS 2076756075503161344
configmap-scale PASS 2076674188441030656
e2e-agnostic-ovn PASS 2077020709137158144
e2e-gcp-operator PASS 2077020724853215232
e2e-aws-ovn-single-node PASS 2077020725683687424

Artifact-level evidence: Panic fix confirmed

The fallback path (--rendered-manifest-files not provided) was exercised during bootstrap in all three e2e jobs. Bootkube logs confirm the fix:

Job Before fix (build 2075418...) After fix (build 2077020...)
e2e-agnostic-ovn 122 panicsfeature "ShortCertRotation" is not registered in FeatureGates [] 0 panics
e2e-aws-ovn-single-node panics (cluster never came up) 0 panics

Both jobs logged falling back to install-config feature gate parsing confirming the fallback was hit, and bootstrap completed successfully without panics.

Multi-PR payload job

The /testwith job testing this PR + installer#10595 (e2e-aws-ovn-pki-rsa-techpreview) failed due to a bug in the installer PR's verification script — it checks for a secret named etcd-metrics-signer (with 's') but the actual secret has always been named etcd-metric-signer (without 's') since its introduction in 2020. The etcd-signer check (same code path, same PKI provider) passed. Not caused by this PR.

Caveats

  • 6 e2e jobs are still pending (label-gated, not code issues)
  • The primary manifest-reading path cannot be verified until installer#10595 lands and passes the new flags

@sanchezl

Copy link
Copy Markdown
Contributor Author

/verified by pull-ci-openshift-cluster-etcd-operator-main-e2e-agnostic-ovn,pull-ci-openshift-cluster-etcd-operator-main-e2e-aws-ovn-single-node,pull-ci-openshift-cluster-etcd-operator-main-e2e-gcp-operator

@openshift-ci-robot openshift-ci-robot added the verified Signifies that the PR passed pre-merge verification criteria label Jul 14, 2026
@openshift-ci-robot

Copy link
Copy Markdown

@sanchezl: This PR has been marked as verified by pull-ci-openshift-cluster-etcd-operator-main-e2e-agnostic-ovn,pull-ci-openshift-cluster-etcd-operator-main-e2e-aws-ovn-single-node,pull-ci-openshift-cluster-etcd-operator-main-e2e-gcp-operator.

Details

In response to this:

/verified by pull-ci-openshift-cluster-etcd-operator-main-e2e-agnostic-ovn,pull-ci-openshift-cluster-etcd-operator-main-e2e-aws-ovn-single-node,pull-ci-openshift-cluster-etcd-operator-main-e2e-gcp-operator

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@sanchezl

Copy link
Copy Markdown
Contributor Author

/label acknowledge-critical-fixes-only

@openshift-ci openshift-ci Bot added the acknowledge-critical-fixes-only Indicates if the issuer of the label is OK with the policy. label Jul 14, 2026
@sanchezl sanchezl changed the title render: Read FeatureGate CR from rendered manifests CNTRLPLANE-2847: render: Read FeatureGate CR from rendered manifests Jul 14, 2026
@openshift-ci-robot openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Jul 14, 2026
@openshift-ci-robot

openshift-ci-robot commented Jul 14, 2026

Copy link
Copy Markdown

@sanchezl: This pull request references CNTRLPLANE-2847 which is a valid jira issue.

Details

In response to this:

Summary

  • Replace manual install-config feature gate parsing with the standard operator render pattern: read the pre-resolved FeatureGate CR from rendered manifests via --rendered-manifest-files and --payload-version
  • Include a fallback that resolves featureSet and featureGates from the install-config when the new flags are not yet available
  • Fix the root cause of the featureSet resolution gap that PR #1647 addresses

Background

The CEO render was the only operator that parsed feature gates directly from the raw install-config (featureGates: [Key=Value] strings). Every other operator render (CKASO, CKCMO, MCO) reads the pre-resolved FeatureGate CR from /assets/manifests/ — the api-render step in bootkube.sh runs first and resolves featureSet (e.g. TechPreviewNoUpgrade) into individual gates, writing the result into the CR status.

Because the CEO was parsing the raw install-config, it missed featureSet resolution entirely, causing ConfigurablePKI to not be detected when using featureSet: TechPreviewNoUpgrade instead of explicit featureGates entries.

Approach

Primary path (when --rendered-manifest-files and --payload-version are provided): Read the pre-resolved FeatureGate CR from rendered manifests, matching the pattern used by CKASO and CKCMO. This eliminates the entire class of feature-gate-resolution bugs.

Fallback path (when the flags are not provided): Resolve featureSet into individual gates via features.FeatureSets() from openshift/api, then layer explicit featureGates overrides on top. This fallback uses strings.Cut for safe parsing and handles the featureSetfeatureGates override correctly. It should be removed once the installer carries the coordinated change.

Changes

pkg/cmd/render/render.go:

  • Add optional --rendered-manifest-files and --payload-version flags
  • Add getFeatureGates() dispatcher that tries the CR path, falls back to install-config
  • Add getFeatureGatesFromManifests() — reads FeatureGate CR, returns enabled/disabled sets
  • Add getFeatureGatesFromInstallConfig() — resolves featureSet + featureGates from install-config
  • Remove the old getFeatureGatesStatus() function

pkg/cmd/render/render_test.go:

  • Add writeFeatureGateManifest() test helper for the CR path
  • Add Test_getFeatureGatesFromInstallConfig testing featureSet resolution, overrides
  • Update test infrastructure to provide rendered manifest directories
  • Update TestTemplateDataWithCustomPKI to provide ConfigurablePKI via FeatureGate CR

Coordinated change

openshift/installer#10679 passes --rendered-manifest-files and --payload-version to the etcd-render invocation. This CEO PR can land independently — the fallback path handles the transition. Once the installer PR merges, a follow-up can remove the fallback.

Test plan

  • go test ./pkg/cmd/render/ — all tests pass (both primary and fallback paths)
  • go vet ./pkg/cmd/render/ — clean
  • CI

Summary by CodeRabbit

  • New Features
  • Rendering can now determine feature gate enablement from pre-rendered FeatureGate manifests supplied by the user.
  • Added CLI flags to specify rendered manifest locations and the payload version used during rendering.
  • Bug Fixes
  • Bootstrap certificate secrets now consistently reflect enabled/disabled feature gates derived from the rendered manifests or the install configuration (including correct handling of configurable PKI selection).
  • Tests
  • Updated the rendering/template test harness to generate a FeatureGate manifest at runtime and extended coverage for parsing feature gates from install configuration.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@hasbro17

Copy link
Copy Markdown
Contributor

/lgtm

@openshift-ci openshift-ci Bot added the lgtm Indicates that a PR is ready to be merged. label Jul 14, 2026
@openshift-merge-bot

Copy link
Copy Markdown
Contributor

Scheduling required tests:
/test e2e-agnostic-ovn
/test e2e-agnostic-ovn-upgrade
/test e2e-aws-ovn-serial-1of2
/test e2e-aws-ovn-serial-2of2
/test e2e-aws-ovn-single-node
/test e2e-gcp-operator
/test e2e-gcp-operator-disruptive
/test e2e-metal-ipi-ovn-ipv6
/test e2e-operator

@openshift-ci

openshift-ci Bot commented Jul 14, 2026

Copy link
Copy Markdown
Contributor

@sanchezl: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

@openshift-merge-bot
openshift-merge-bot Bot merged commit 0185d88 into openshift:main Jul 14, 2026
17 of 18 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

acknowledge-critical-fixes-only Indicates if the issuer of the label is OK with the policy. approved Indicates a PR has been approved by an approver from all required OWNERS files. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. lgtm Indicates that a PR is ready to be merged. verified Signifies that the PR passed pre-merge verification criteria

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants