OCPEDGE-2094: Add console notification for pacemaker podman-etcd failures

[lucaconsalvi]

Summary

• Add a new ConsoleNotificationController that watches the PacemakerCluster CR and manages a ConsoleNotification banner in the OpenShift web console when pacemaker health problems are detected
• The controller detects: unhealthy etcd resources, fencing unavailability, maintenance mode, and stale pacemaker status
• The banner is automatically removed when health is restored
• Follows existing patterns: shares the PacemakerCluster informer with healthcheck and metrics controllers, registered alongside them in runPacemakerControllers()

Files Changed

File	Change
pkg/tnf/pkg/pacemaker/consolenotification.go	New controller + health evaluation logic
pkg/tnf/pkg/pacemaker/consolenotification_test.go	9 table-driven tests covering all failure modes
pkg/tnf/operator/starter.go	Register the new controller in runPacemakerControllers()

Test plan

• Unit tests pass for all health evaluation scenarios (healthy, stale, maintenance, node offline, etcd unhealthy, fencing unavailable, multiple problems, nil nodes, offline-skip)
• All existing TNF package tests pass
• Full binary builds cleanly
• Manual verification on a TNF cluster: disable etcd resource, verify banner appears, re-enable, verify banner removed

Jira: https://issues.redhat.com/browse/OCPEDGE-2094

🤖 Generated with Claude Code

Summary by CodeRabbit

• New Features

  • Added Pacemaker health console notifications that surface detected issues on affected clusters.
  • Notifications include a troubleshooting link and update their text when health changes.
  • Notifications are removed when the cluster returns to a healthy state.

• Bug Fixes

  • Enhanced Pacemaker monitoring for stale status, maintenance mode, offline nodes, unhealthy node-local etcd, and fencing availability.

• Tests

  • Added unit test coverage for health evaluation across multiple scenarios (including nil/healthy inputs).

[openshift-ci-robot]

@lucaconsalvi: This pull request references OCPEDGE-2094 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "5.0.0" version, but no target version was set.

Details

In response to this:

Summary

• Add a new ConsoleNotificationController that watches the PacemakerCluster CR and manages a ConsoleNotification banner in the OpenShift web console when pacemaker health problems are detected
• The controller detects: unhealthy etcd resources, fencing unavailability, maintenance mode, and stale pacemaker status
• The banner is automatically removed when health is restored
• Follows existing patterns: shares the PacemakerCluster informer with healthcheck and metrics controllers, registered alongside them in runPacemakerControllers()

Files Changed

File	Change
pkg/tnf/pkg/pacemaker/consolenotification.go	New controller + health evaluation logic
pkg/tnf/pkg/pacemaker/consolenotification_test.go	9 table-driven tests covering all failure modes
pkg/tnf/operator/starter.go	Register the new controller in runPacemakerControllers()

Test plan

• Unit tests pass for all health evaluation scenarios (healthy, stale, maintenance, node offline, etcd unhealthy, fencing unavailable, multiple problems, nil nodes, offline-skip)
• All existing TNF package tests pass
• Full binary builds cleanly
• Manual verification on a TNF cluster: disable etcd resource, verify banner appears, re-enable, verify banner removed

Jira: https://issues.redhat.com/browse/OCPEDGE-2094

🤖 Generated with Claude Code

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[openshift-ci]

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

[openshift-merge-bot]

Pipeline controller notification
This repo is configured to use the pipeline controller. Second-stage tests will be triggered either automatically or after lgtm label is added, depending on the repository configuration. The pipeline controller will automatically detect which contexts are required and will utilize /test Prow commands to trigger the second stage.

For optional jobs, comment /test ? to see a list of all defined jobs. To trigger manually all jobs from second stage use /pipeline required command.

This repository is configured in: LGTM mode

[coderabbitai]

Walkthrough

Changes

Pacemaker controller startup now includes a console notification controller. It evaluates PacemakerCluster health and creates, updates, or deletes an OpenShift ConsoleNotification based on detected issues. Tests cover health evaluation scenarios.

Pacemaker console notification flow

Layer / File(s)	Summary
Pacemaker health evaluation pkg/tnf/pkg/pacemaker/consolenotification.go, pkg/tnf/pkg/pacemaker/consolenotification_test.go	Health checks detect stale status, maintenance mode, offline nodes, unhealthy etcd resources, and unavailable fencing, with table-driven coverage.
ConsoleNotification reconciliation pkg/tnf/pkg/pacemaker/consolenotification.go	The controller deletes notifications for absent or healthy clusters and creates or updates banners containing detected problems and troubleshooting details.
Controller startup wiring pkg/tnf/operator/starter.go	The dynamic client is passed into Pacemaker controller startup, which launches the console notification controller with the shared informer and event recorder.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Sequence Diagram(s)

sequenceDiagram
  participant PacemakerInformer
  participant ConsoleNotificationController
  participant DynamicClient
  PacemakerInformer->>ConsoleNotificationController: Observe PacemakerCluster
  ConsoleNotificationController->>ConsoleNotificationController: Evaluate health
  ConsoleNotificationController->>DynamicClient: Create or update ConsoleNotification
  ConsoleNotificationController->>DynamicClient: Delete notification when healthy

Loading

---

Important

Pre-merge checks failed

Please resolve all errors before merging. Addressing warnings is optional.

❌ Failed checks (1 error)

Check name	Status	Explanation	Resolution
No-Sensitive-Data-In-Logs	❌ Error	ConsoleNotification logs print the full banner text, which includes node names like 'Node %s is offline' and other cluster health details.	Stop logging the rendered notification text; log only a fixed status message or redact node names and other cluster-specific details.

✅ Passed checks (14 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	Docstring coverage is 83.33% which is sufficient. The required threshold is 80.00%.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names	✅ Passed	The new pacemaker tests use fixed table-driven names only; no Ginkgo titles or interpolated values appear in the changed test file.
Test Structure And Quality	✅ Passed	The new tests are pure table-driven unit tests: no cluster resources or waits, each case targets one health behavior, and failure messages are descriptive.
Microshift Test Compatibility	✅ Passed	The only added test is a standard Go unit test (TestEvaluateHealth), not a Ginkgo e2e test, and it uses no MicroShift-unsupported OpenShift APIs.
Single Node Openshift (Sno) Test Compatibility	✅ Passed	The added test is a plain Go unit test using testing.T, with no Ginkgo e2e specs or multi-node/SNO-sensitive assumptions.
Topology-Aware Scheduling Compatibility	✅ Passed	No new pod scheduling constraints were added; the change only manages a ConsoleNotification via the dynamic client and informer.
Ote Binary Stdout Contract	✅ Passed	PASS: the PR adds only klog logging in non-entrypoint code; no fmt.Print/stdout writes were introduced, and cli.Run/klog route logs to stderr.
Ipv6 And Disconnected Network Test Compatibility	✅ Passed	The added tests are plain Go unit tests, not Ginkgo e2e tests, and the new controller/test code has no external connectivity requirements.
No-Weak-Crypto	✅ Passed	Only a test file changed; it adds health-evaluation fixtures/assertions and contains no crypto primitives, custom crypto, or secret comparisons.
Container-Privileges	✅ Passed	The PR only changes Go controller/test code; no diff adds container manifests or securityContext fields like privileged, hostPID, or allowPrivilegeEscalation.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly matches the main change: adding a pacemaker console notification for etcd-related failures.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[openshift-ci]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign mshitrit for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• pkg/tnf/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[coderabbitai]

Actionable comments posted: 4

🧹 Nitpick comments (1)

pkg/tnf/pkg/pacemaker/consolenotification_test.go (1)

38-196: 📐 Maintainability & Code Quality | 🔵 Trivial | 🏗️ Heavy lift

Add tests for the ConsoleNotification lifecycle.

Health evaluation is covered, but creation, deletion, update drift, conflict/error handling, and preservation of unmanaged metadata are not. Exercise sync with a fake dynamic client before relying on this controller in production.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@pkg/tnf/pkg/pacemaker/consolenotification_test.go` around lines 38 - 196, Add
lifecycle-focused tests for the ConsoleNotification controller’s sync flow,
using a fake dynamic client to cover creation, deletion, update drift, resource
conflicts/errors, and preservation of unmanaged metadata. Keep the existing
evaluateHealth tests unchanged, and assert the resulting resource state and
propagated handling through the relevant sync/controller methods.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@pkg/tnf/pkg/pacemaker/consolenotification_test.go`:
- Around line 104-120: Update the “offline node skips resource and fencing
checks” test setup around makeNodeWithConditions to explicitly set the node’s
Etcd Healthy condition to False before evaluation, while preserving the offline
and fencing-unavailable conditions and the expected offline-only notification.

In `@pkg/tnf/pkg/pacemaker/consolenotification.go`:
- Line 180: Update the logging statements in the console notification creation
and related paths around klog.Infof so they do not include the notification
text; instead log the operation and notification problem count, avoiding
exposure of node hostnames while retaining useful summary information.
- Around line 102-112: Update the Pacemaker status checks around
getConditionStatus and the cr.Status.Nodes guard so uninitialized status is
reported as unhealthy: flag zero LastUpdated, treat missing or
metav1.ConditionUnknown conditions as problems, and report nil or empty Nodes
according to the CRD’s catastrophic-state guidance. Preserve existing stale,
maintenance, and healthy-status behavior, and add coverage for uninitialized and
Unknown statuses.
- Around line 187-195: Update the reconciliation logic around existingText and
the ConsoleNotification Update call to compare and reconcile every
controller-owned field, including backgroundColor, color, link, location, and
text. Mutate a copy of existing rather than updating u directly, preserve its
annotations, finalizers, labels, and other metadata, set the desired owned
fields on the copy, retain the existing resource version, and submit that copy
through the dynamic client.

---

Nitpick comments:
In `@pkg/tnf/pkg/pacemaker/consolenotification_test.go`:
- Around line 38-196: Add lifecycle-focused tests for the ConsoleNotification
controller’s sync flow, using a fake dynamic client to cover creation, deletion,
update drift, resource conflicts/errors, and preservation of unmanaged metadata.
Keep the existing evaluateHealth tests unchanged, and assert the resulting
resource state and propagated handling through the relevant sync/controller
methods.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Repository: openshift/coderabbit/.coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: d68afcfd-0760-4e1e-83e7-699379c44f13

📥 Commits

Reviewing files that changed from the base of the PR and between 1fe5873 and caead3c.

📒 Files selected for processing (3)

• pkg/tnf/operator/starter.go
• pkg/tnf/pkg/pacemaker/consolenotification.go
• pkg/tnf/pkg/pacemaker/consolenotification_test.go

[lucaconsalvi]

Manual Testing on TNF Cluster

Tested the console notification on a live TNF baremetal cluster deployed via cluster-bot with a custom CEO image built from this PR.

Cluster: workflow-launch baremetalds-two-node-fencing with release image containing PR #1654

Test 1 — Banner appears on pacemaker failure

Triggered maintenance mode via pcs property set maintenance-mode=true on master-0. After the CronJob updated the PacemakerCluster CR (~90s), the controller created the ConsoleNotification:

$ oc get consolenotification pacemaker-etcd-failure -o jsonpath='{.spec.text}'
Pacemaker cluster is in maintenance mode. Etcd resource is unhealthy on node master-0. Etcd resource is unhealthy on node master-1. Check pacemaker status for details.

Red banner visible at the top of the OpenShift web console with the "Troubleshooting guide" link.

Test 2 — Banner disappears when health is restored

Disabled maintenance mode via pcs property set maintenance-mode=false. After the CronJob updated the CR:

$ oc get consolenotifications
No resources found

PacemakerCluster CR returned to Healthy=True, InService=True. Banner disappeared from the console.

Result

PASS — Full lifecycle verified: banner creation, content accuracy, troubleshooting link, and automatic cleanup on health restoration.

[openshift-ci]

@lucaconsalvi: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.