Skip to content

OCPEDGE-2118: Record Kubernetes events for etcd transition lifecycle#1653

Open
lucaconsalvi wants to merge 3 commits into
openshift:mainfrom
lucaconsalvi:ocpedge-2118-etcd-transition-events
Open

OCPEDGE-2118: Record Kubernetes events for etcd transition lifecycle#1653
lucaconsalvi wants to merge 3 commits into
openshift:mainfrom
lucaconsalvi:ocpedge-2118-etcd-transition-events

Conversation

@lucaconsalvi

@lucaconsalvi lucaconsalvi commented Jul 16, 2026

Copy link
Copy Markdown

Add event recording to the TNF setup runner and etcd transition code so the full CEO-to-pacemaker handoff is observable via kubectl get events -n openshift-etcd.

Nine events are emitted across the transition lifecycle:

  • runner.go: auth, cluster, fencing, etcd resource, constraints
  • etcd.go: transition started, waiting, static pod removed, completed

Events use the direct K8s API (not library-go Recorder) since the setup runner is a short-lived Job. Failures are logged but never block the transition.

Summary by CodeRabbit

  • New Features
    • Added Kubernetes Normal events to surface TNF setup milestones and the etcd static-container transition lifecycle, including key configuration steps and the static etcd removal phase.
    • Added a verification script that generates mock etcd transition events and validates they appear as expected.
  • Bug Fixes
    • Setup progress event recording is best-effort; failures no longer block or interrupt the setup flow.

Add event recording to the TNF setup runner and etcd transition
code so the full CEO-to-pacemaker handoff is observable via
`kubectl get events -n openshift-etcd`.

Nine events are emitted across the transition lifecycle:
- runner.go: auth, cluster, fencing, etcd resource, constraints
- etcd.go: transition started, waiting, static pod removed, completed

Events use the direct K8s API (not library-go Recorder) since the
setup runner is a short-lived Job. Failures are logged but never
block the transition.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@openshift-merge-bot

Copy link
Copy Markdown
Contributor

Pipeline controller notification
This repo is configured to use the pipeline controller. Second-stage tests will be triggered either automatically or after lgtm label is added, depending on the repository configuration. The pipeline controller will automatically detect which contexts are required and will utilize /test Prow commands to trigger the second stage.

For optional jobs, comment /test ? to see a list of all defined jobs. To trigger manually all jobs from second stage use /pipeline required command.

This repository is configured in: LGTM mode

@openshift-ci

openshift-ci Bot commented Jul 16, 2026

Copy link
Copy Markdown
Contributor

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

@openshift-ci-robot

openshift-ci-robot commented Jul 16, 2026

Copy link
Copy Markdown

@lucaconsalvi: This pull request references OCPEDGE-2118 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "5.0.0" version, but no target version was set.

Details

In response to this:

Add event recording to the TNF setup runner and etcd transition code so the full CEO-to-pacemaker handoff is observable via kubectl get events -n openshift-etcd.

Nine events are emitted across the transition lifecycle:

  • runner.go: auth, cluster, fencing, etcd resource, constraints
  • etcd.go: transition started, waiting, static pod removed, completed

Events use the direct K8s API (not library-go Recorder) since the setup runner is a short-lived Job. Failures are logged but never block the transition.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci-robot openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Jul 16, 2026
@openshift-ci openshift-ci Bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jul 16, 2026
@coderabbitai

coderabbitai Bot commented Jul 16, 2026

Copy link
Copy Markdown

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Repository: openshift/coderabbit/.coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 3be330da-e053-49ac-81f2-cf3c0baf54a2

📥 Commits

Reviewing files that changed from the base of the PR and between 4530ded and d75bdc7.

📒 Files selected for processing (1)
  • pkg/tnf/pkg/tools/events.go

Walkthrough

TNF setup and etcd transition flows now emit Kubernetes Event objects for lifecycle milestones. The etcd handover accepts a Kubernetes client, while unit and shell tests validate event reasons, types, sources, and visibility.

Changes

TNF transition events

Layer / File(s) Summary
Event recording and setup milestone wiring
pkg/tnf/pkg/tools/events.go, pkg/tnf/setup/runner.go
Adds RecordSetupEvent for Normal events in openshift-etcd and records milestones after authentication, PCS cluster, fencing, etcd resource-agent, and constraint configuration.
Etcd handover event flow
pkg/tnf/pkg/etcd/etcd.go, pkg/tnf/setup/runner.go
Passes the Kubernetes client through static etcd removal and records transition start, waiting, removal, stable revision, and completion events.
Event validation
pkg/tnf/pkg/etcd/etcd_test.go, hack/test-transition-events.sh
Uses fake clients and an oc script to validate transition event reasons, types, sources, and visibility.

Estimated code review effort: 3 (Moderate) | ~25 minutes

Sequence Diagram(s)

sequenceDiagram
  participant RunTnfSetup
  participant RecordSetupEvent
  participant KubernetesEventsAPI
  participant RemoveStaticContainer
  participant waitForStaticContainerRemoved
  RunTnfSetup->>RecordSetupEvent: record setup milestones
  RecordSetupEvent->>KubernetesEventsAPI: create Normal events
  RunTnfSetup->>RemoveStaticContainer: pass kubeClient during handover
  RemoveStaticContainer->>RecordSetupEvent: record transition start
  RemoveStaticContainer->>waitForStaticContainerRemoved: wait for static etcd removal
  waitForStaticContainerRemoved->>RecordSetupEvent: record removal milestone
  RemoveStaticContainer->>RecordSetupEvent: record transition completion
  RecordSetupEvent->>KubernetesEventsAPI: create Normal events
Loading

Suggested reviewers: jaypoulz, kaleemsiddiqu

🚥 Pre-merge checks | ✅ 14 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Microshift Test Compatibility ⚠️ Warning FAIL: hack/test-transition-events.sh hard-codes openshift-etcd, which MicroShift lacks, and it has no [Skipped:MicroShift]/runtime guard. Either skip this script in MicroShift CI or add a runtime MicroShift check before using openshift-etcd.
✅ Passed checks (14 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly matches the main change: adding Kubernetes event recording for the etcd transition lifecycle.
Docstring Coverage ✅ Passed Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names ✅ Passed No Ginkgo titles were added; the changed test file uses a single static Go test name and contains no It/Describe/Context/When calls.
Test Structure And Quality ✅ Passed The added test is a focused unit test with fake clients; it has no cluster waits or cleanup needs, and assertion style matches nearby repo tests. No Ginkgo anti-patterns found.
Single Node Openshift (Sno) Test Compatibility ✅ Passed No new Ginkgo e2e tests were added; the PR only touches unit tests, Go helpers, and a shell script, so SNO-specific Ginkgo compatibility checks don’t apply.
Topology-Aware Scheduling Compatibility ✅ Passed PASS: The PR only adds event recording around TNF/etcd transitions; touched files contain no node selectors, affinity, topology spread, replica, or taint logic.
Ote Binary Stdout Contract ✅ Passed No OTE process-level stdout writes were added; changes are limited to TNF runtime libraries and a shell script, with no main/TestMain/BeforeSuite stdout output introduced.
Ipv6 And Disconnected Network Test Compatibility ✅ Passed PASS: No new Ginkgo e2e tests were added; the new shell harness and unit tests contain no IPv4-only literals, public hosts, or external downloads.
No-Weak-Crypto ✅ Passed Touched files contain no MD5/SHA1/DES/RC4/3DES/Blowfish/ECB, custom crypto, or secret/token comparisons.
Container-Privileges ✅ Passed No changed manifest or code file introduces privileged=true, hostPID/hostNetwork/hostIPC, SYS_ADMIN, or allowPrivilegeEscalation=true.
No-Sensitive-Data-In-Logs ✅ Passed New logs only emit fixed event reasons/messages; no passwords, tokens, PII, hostnames, or customer data are logged.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Warning

There were issues while running some tools. Please review the errors and either fix the tool's configuration or disable the tool if it's a critical failure.

🔧 golangci-lint (2.12.2)

Error: can't load config: unsupported version of the configuration: "" See https://golangci-lint.run/docs/product/migration-guide for migration instructions
The command is terminated due to an error: can't load config: unsupported version of the configuration: "" See https://golangci-lint.run/docs/product/migration-guide for migration instructions


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@pkg/tnf/pkg/tools/events.go`:
- Around line 3-28: Update RecordSetupEvent so the reason component used in the
event metadata.name is converted to lowercase before constructing the name.
Preserve the existing event naming format and reason value used elsewhere in the
event.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository: openshift/coderabbit/.coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 11fb1836-7157-46ed-9989-87c3820a66e7

📥 Commits

Reviewing files that changed from the base of the PR and between 1fe5873 and 4bd68bb.

📒 Files selected for processing (5)
  • hack/test-transition-events.sh
  • pkg/tnf/pkg/etcd/etcd.go
  • pkg/tnf/pkg/etcd/etcd_test.go
  • pkg/tnf/pkg/tools/events.go
  • pkg/tnf/setup/runner.go

Comment thread pkg/tnf/pkg/tools/events.go
@lucaconsalvi
lucaconsalvi marked this pull request as ready for review July 16, 2026 12:23
@openshift-ci openshift-ci Bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jul 16, 2026
The event reason (e.g. EtcdTransitionAuthCompleted) was used directly
in metadata.name which should be a valid lowercase DNS subdomain.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@openshift-ci
openshift-ci Bot requested review from jaypoulz and kaleemsiddiqu July 16, 2026 12:28
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

@fonta-rh fonta-rh left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ahhh, I'm amazed, I have nothing to nitpick. This does what it should minimally, good test coverage! Good work!

@fonta-rh

Copy link
Copy Markdown
Contributor

/lgtm

@openshift-ci openshift-ci Bot added the lgtm Indicates that a PR is ready to be merged. label Jul 17, 2026
@openshift-merge-bot

Copy link
Copy Markdown
Contributor

Scheduling required tests:
/test e2e-agnostic-ovn
/test e2e-agnostic-ovn-upgrade
/test e2e-aws-ovn-serial-1of2
/test e2e-aws-ovn-serial-2of2
/test e2e-aws-ovn-single-node
/test e2e-gcp-operator
/test e2e-gcp-operator-disruptive
/test e2e-metal-ipi-ovn-ipv6
/test e2e-operator

@lucaconsalvi

Copy link
Copy Markdown
Author

E2E Verification on Live TNF Cluster

Cluster: Deployed via cluster-bot (workflow-launch baremetalds-two-node-fencing) with the CEO image built from this PR.

Method: Verified all 9 transition events are emitted during the CEO-to-Pacemaker etcd handoff using oc get events -n openshift-etcd.

Results

All 9 events recorded successfully in openshift-etcd namespace:

# Reason Source Status
1 EtcdTransitionAuthCompleted tnf-setup-runner PASS
2 EtcdTransitionClusterConfigured tnf-setup-runner PASS
3 EtcdTransitionFencingConfigured tnf-setup-runner PASS
4 EtcdTransitionEtcdResourceCreated tnf-setup-runner PASS
5 EtcdTransitionConstraintsConfigured tnf-setup-runner PASS
6 EtcdTransitionStarted tnf-setup-runner PASS
7 EtcdTransitionWaitingForRemoval tnf-setup-runner PASS
8 EtcdTransitionStaticContainerRemoved tnf-setup-runner PASS
9 EtcdTransitionCompleted tnf-setup-runner PASS
  • All events are Normal type with InvolvedObject: Job/tnf-setup-job
  • Event names comply with RFC 1123 subdomain rules (lowercase)
  • Events are visible in both the OpenShift console (Events tab) and CLI
  • Event recording failures are logged but do not block the setup — non-blocking by design

/verified by @lucaconsalvi

@openshift-ci-robot openshift-ci-robot added the verified Signifies that the PR passed pre-merge verification criteria label Jul 17, 2026
@openshift-ci-robot

Copy link
Copy Markdown

@lucaconsalvi: This PR has been marked as verified by @lucaconsalvi.

Details

In response to this:

E2E Verification on Live TNF Cluster

Cluster: Deployed via cluster-bot (workflow-launch baremetalds-two-node-fencing) with the CEO image built from this PR.

Method: Verified all 9 transition events are emitted during the CEO-to-Pacemaker etcd handoff using oc get events -n openshift-etcd.

Results

All 9 events recorded successfully in openshift-etcd namespace:

# Reason Source Status
1 EtcdTransitionAuthCompleted tnf-setup-runner PASS
2 EtcdTransitionClusterConfigured tnf-setup-runner PASS
3 EtcdTransitionFencingConfigured tnf-setup-runner PASS
4 EtcdTransitionEtcdResourceCreated tnf-setup-runner PASS
5 EtcdTransitionConstraintsConfigured tnf-setup-runner PASS
6 EtcdTransitionStarted tnf-setup-runner PASS
7 EtcdTransitionWaitingForRemoval tnf-setup-runner PASS
8 EtcdTransitionStaticContainerRemoved tnf-setup-runner PASS
9 EtcdTransitionCompleted tnf-setup-runner PASS
  • All events are Normal type with InvolvedObject: Job/tnf-setup-job
  • Event names comply with RFC 1123 subdomain rules (lowercase)
  • Events are visible in both the OpenShift console (Events tab) and CLI
  • Event recording failures are logged but do not block the setup — non-blocking by design

/verified by @lucaconsalvi

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@lucaconsalvi

Copy link
Copy Markdown
Author

/retest

@openshift-ci

openshift-ci Bot commented Jul 17, 2026

Copy link
Copy Markdown
Contributor

@lucaconsalvi: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/prow/e2e-aws-ovn-serial-1of2 d75bdc7 link true /test e2e-aws-ovn-serial-1of2
ci/prow/e2e-agnostic-ovn d75bdc7 link true /test e2e-agnostic-ovn
ci/prow/e2e-aws-ovn-serial-2of2 d75bdc7 link true /test e2e-aws-ovn-serial-2of2
ci/prow/e2e-aws-ovn-single-node d75bdc7 link true /test e2e-aws-ovn-single-node
ci/prow/e2e-metal-ipi-ovn-ipv6 d75bdc7 link true /test e2e-metal-ipi-ovn-ipv6

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

@lucaconsalvi

Copy link
Copy Markdown
Author

/retest

@fonta-rh

Copy link
Copy Markdown
Contributor

/approve

@openshift-ci

openshift-ci Bot commented Jul 20, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: fonta-rh
Once this PR has been reviewed and has the lgtm label, please assign p0lyn0mial for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. lgtm Indicates that a PR is ready to be merged. verified Signifies that the PR passed pre-merge verification criteria

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants