OCPBUGS-84695: feat(tnf): multi-node job controller framework#1655
OCPBUGS-84695: feat(tnf): multi-node job controller framework#1655jaypoulz wants to merge 1 commit into
Conversation
|
Pipeline controller notification For optional jobs, comment This repository is configured in: LGTM mode |
openshift-ci-robot
added
jira/severity-important
|
@jaypoulz: This pull request references Jira Issue OCPBUGS-84695, which is valid. 3 validation(s) were run on this bug
The bug has been updated to refer to the pull request using the external bug tracker. DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
openshift-ci-robot
added
the
jira/valid-bug
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
WalkthroughAdds a TNF Job-controller lifecycle framework for node-scoped and cluster-wide Jobs, including readiness conditions, retry orchestration, restart helpers, stuck-Job recovery, operator wiring, utilities, tests, and documentation. ChangesTNF Job lifecycle
Estimated code review effort: 4 (Complex) | ~60 minutes Sequence Diagram(s)sequenceDiagram
participant TNFOperator
participant JobController
participant KubernetesAPI
participant OperatorStatus
TNFOperator->>JobController: start or restart TNF Job controller
JobController->>KubernetesAPI: inspect and configure Job
JobController->>OperatorStatus: set Blocked or Degraded condition
KubernetesAPI-->>JobController: report Job state
JobController->>KubernetesAPI: delete or recreate Job
Possibly related PRs
Suggested reviewers: Important Pre-merge checks failedPlease resolve all errors before merging. Addressing warnings is optional. ❌ Failed checks (1 error, 2 warnings)
✅ Passed checks (12 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (2)
pkg/tnf/pkg/jobs/lifecycle_test.go (2)
691-691: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick winDon't ignore the
Createreturn values in the retry-progression setup.The fake-client
Createcalls at Lines 691, 700, 710, 714, and 731 discard the returned error. If aCreatesilently fails, the subsequentsyncMultiNodeJobStatewould find no job (NotFound → no-op), the retry state wouldn't advance, and the test could pass or fail for the wrong reason. Assertrequire.NoError(...)on each so setup failures surface immediately.As per path instructions: "Never ignore error returns".
💡 Example fix for one call
- fakeKubeClient.BatchV1().Jobs(operatorclient.TargetNamespace).Create(ctx, failedJob, metav1.CreateOptions{}) + _, err = fakeKubeClient.BatchV1().Jobs(operatorclient.TargetNamespace).Create(ctx, failedJob, metav1.CreateOptions{}) + require.NoError(t, err)🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@pkg/tnf/pkg/jobs/lifecycle_test.go` at line 691, Update the retry-progression test setup to check every fake-client Job Create call with require.NoError, including the calls near lines 691, 700, 710, 714, and 731. Pass the test context and each returned error so setup failures stop the test immediately before syncMultiNodeJobState runs.Source: Path instructions
211-223: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value
expectControllerRundoesn't affect the assertion — the "skip when already running" case isn't actually verified.Both branches assert
require.True(t, isRunning), so the field never changes what is checked. For "Skip starting controller when already running",runningControllers[key]is pre-seeded totrue, so the assertion passes whether or not a duplicate start was skipped. Consider distinguishing an actual skip (e.g., assert no new goroutine/controller was created, or that the controller count didn't change) so the dedup path is meaningfully exercised.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@pkg/tnf/pkg/jobs/lifecycle_test.go` around lines 211 - 223, The lifecycle test’s expectControllerRun branch does not distinguish starting a controller from skipping an already-running one. Update the verification around runningControllers and the “Skip starting controller when already running” case to assert an observable deduplication outcome, such as unchanged controller/goroutine count, while retaining the running-state assertion for newly started controllers.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@docs/tnf/job-controllers.md`:
- Around line 11-15: Update the Blocked Condition section in job-controllers.md
to remove the exhausted-retries bullet, keeping only the node-not-ready timeout
behavior. Add that exhausted retries set the <JobName>Degraded condition to the
Degraded description, using the existing condition naming and format.
---
Nitpick comments:
In `@pkg/tnf/pkg/jobs/lifecycle_test.go`:
- Line 691: Update the retry-progression test setup to check every fake-client
Job Create call with require.NoError, including the calls near lines 691, 700,
710, 714, and 731. Pass the test context and each returned error so setup
failures stop the test immediately before syncMultiNodeJobState runs.
- Around line 211-223: The lifecycle test’s expectControllerRun branch does not
distinguish starting a controller from skipping an already-running one. Update
the verification around runningControllers and the “Skip starting controller
when already running” case to assert an observable deduplication outcome, such
as unchanged controller/goroutine count, while retaining the running-state
assertion for newly started controllers.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository: openshift/coderabbit/.coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 0c16170f-d107-4fba-bcb4-ba3d437bdd0c
📒 Files selected for processing (12)
docs/tnf/job-controllers.mdpkg/tnf/operator/nodehandler.gopkg/tnf/operator/starter.gopkg/tnf/pkg/jobs/jobcontroller.gopkg/tnf/pkg/jobs/jobcontroller_test.gopkg/tnf/pkg/jobs/lifecycle.gopkg/tnf/pkg/jobs/lifecycle_test.gopkg/tnf/pkg/jobs/tnf.gopkg/tnf/pkg/jobs/tnf_test.gopkg/tnf/pkg/jobs/utils.gopkg/tnf/pkg/tools/jobs.gopkg/tnf/pkg/tools/nodes.go
💤 Files with no reviewable changes (2)
- pkg/tnf/pkg/jobs/tnf_test.go
- pkg/tnf/pkg/jobs/tnf.go
708a96d to
37a5d9f
Compare
There was a problem hiding this comment.
Actionable comments posted: 2
🧹 Nitpick comments (2)
pkg/tnf/pkg/jobs/lifecycle.go (1)
90-153: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winDuplicated blocked-condition tracking logic.
checkNodeReadinessAndSetCondition(single-node path) and the affected-nodes block insidesyncMultiNodeJobState(multi-node path) both independently implement "track first-blocked timestamp, set/clearTNF<JobName>Blockedafter 10 minutes" against the sharedjobBlockedSincemap. The logic has already started to diverge (message format differs:[%s]for a single node vs%vfor a slice). Extracting a shared helper (e.g.updateBlockedCondition(ctx, jobName, notReadyNodeNames []string, ...)) would remove this duplication and prevent future drift.Also applies to: 164-231
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@pkg/tnf/pkg/jobs/lifecycle.go` around lines 90 - 153, The blocked-condition tracking is duplicated between checkNodeReadinessAndSetCondition and the affected-nodes handling in syncMultiNodeJobState. Extract the shared timestamp tracking and TNF<JobName>Blocked condition set/clear behavior into a helper such as updateBlockedCondition, pass the relevant not-ready node names, and update both call sites to use it with consistent message formatting while preserving existing readiness error behavior.pkg/operator/ceohelpers/node_helpers.go (1)
52-62: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winReuse
tools.IsNodeReadyhere. This duplicates the Ready-condition check already inpkg/tnf/pkg/tools/nodes.goand can be replaced directly.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@pkg/operator/ceohelpers/node_helpers.go` around lines 52 - 62, Update the Ready-node filtering loop to call the existing tools.IsNodeReady helper for each node instead of manually iterating node.Status.Conditions and checking condition type/status. Preserve appending each ready node once and retain the existing filtering behavior.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@pkg/tnf/pkg/jobs/lifecycle.go`:
- Around line 164-231: Prevent the blocked-node path in syncMultiNodeJobState
from reaching configureMultiNodeJob with an unset retryState[jobName].
Initialize the JobRetryState before returning from a blocked sync, or have
configureMultiNodeJob detect a nil state after its syncMultiNodeJobState call
and return without locking it, while preserving the existing job-blocking
behavior.
- Around line 360-403: Update the failed-job handling flow around
IsFailed/IsStopped so the next node and attempt values are calculated without
mutating state, delete the failed job with DeleteAndWait first, and only commit
NodeIndex and AttemptNumber after deletion succeeds. Preserve degraded-condition
updates and retry logging, while ensuring a delete error returns without
advancing retry state.
---
Nitpick comments:
In `@pkg/operator/ceohelpers/node_helpers.go`:
- Around line 52-62: Update the Ready-node filtering loop to call the existing
tools.IsNodeReady helper for each node instead of manually iterating
node.Status.Conditions and checking condition type/status. Preserve appending
each ready node once and retain the existing filtering behavior.
In `@pkg/tnf/pkg/jobs/lifecycle.go`:
- Around line 90-153: The blocked-condition tracking is duplicated between
checkNodeReadinessAndSetCondition and the affected-nodes handling in
syncMultiNodeJobState. Extract the shared timestamp tracking and
TNF<JobName>Blocked condition set/clear behavior into a helper such as
updateBlockedCondition, pass the relevant not-ready node names, and update both
call sites to use it with consistent message formatting while preserving
existing readiness error behavior.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository: openshift/coderabbit/.coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 484b8923-171e-4cb0-9423-d32c859e15c7
📒 Files selected for processing (10)
bindata/tnfdeployment/role.yamldocs/tnf/job-controllers.mdpkg/operator/ceohelpers/node_helpers.gopkg/tnf/pkg/jobs/jobcontroller.gopkg/tnf/pkg/jobs/lifecycle.gopkg/tnf/pkg/jobs/lifecycle_test.gopkg/tnf/pkg/jobs/tnf.gopkg/tnf/pkg/jobs/tnf_test.gopkg/tnf/pkg/jobs/utils.gopkg/tnf/pkg/tools/nodes.go
💤 Files with no reviewable changes (2)
- pkg/tnf/pkg/jobs/tnf.go
- pkg/tnf/pkg/jobs/tnf_test.go
🚧 Files skipped from review as they are similar to previous changes (2)
- pkg/tnf/pkg/jobs/utils.go
- pkg/tnf/pkg/jobs/lifecycle_test.go
37a5d9f to
b09bb41
Compare
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (1)
pkg/tnf/pkg/jobs/lifecycle.go (1)
262-263: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick winCompare target nodes independent of order.
handleFencingSecretChangere-lists control-plane nodes on every sync,GetNodeNamespreserves that list order, andslicesEqualtreats different permutations as drift. Sorting or set-comparing here avoids deleting/recreating the Job and resetting retries when the API returns the same nodes in a different order.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@pkg/tnf/pkg/jobs/lifecycle.go` around lines 262 - 263, Update the target-node comparison in handleFencingSecretChange so state.TargetNodes and the current GetNodeNames result are compared independent of ordering, using sorting or set equality before slicesEqual. Preserve the existing nodesChanged behavior for actual membership changes and avoid treating permutations as drift.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@docs/tnf/job-controllers.md`:
- Around line 18-23: Update the Degraded Condition documentation in
docs/tnf/job-controllers.md to clarify that retry cycles continue after the
retry budget is exhausted: describe the limit as 6 tries per cycle, with cycles
continuing until recovery. Remove “permanent failure” and any wording implying
Degraded is terminal, while retaining the existing condition name, reason, and
active-until-success behavior.
---
Nitpick comments:
In `@pkg/tnf/pkg/jobs/lifecycle.go`:
- Around line 262-263: Update the target-node comparison in
handleFencingSecretChange so state.TargetNodes and the current GetNodeNames
result are compared independent of ordering, using sorting or set equality
before slicesEqual. Preserve the existing nodesChanged behavior for actual
membership changes and avoid treating permutations as drift.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository: openshift/coderabbit/.coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 9b247758-ff1a-4c98-98c7-ab9a47cd52d2
📒 Files selected for processing (12)
docs/tnf/job-controllers.mdpkg/tnf/operator/nodehandler.gopkg/tnf/operator/starter.gopkg/tnf/pkg/jobs/jobcontroller.gopkg/tnf/pkg/jobs/jobcontroller_test.gopkg/tnf/pkg/jobs/lifecycle.gopkg/tnf/pkg/jobs/lifecycle_test.gopkg/tnf/pkg/jobs/tnf.gopkg/tnf/pkg/jobs/tnf_test.gopkg/tnf/pkg/jobs/utils.gopkg/tnf/pkg/tools/jobs.gopkg/tnf/pkg/tools/nodes.go
💤 Files with no reviewable changes (2)
- pkg/tnf/pkg/jobs/tnf_test.go
- pkg/tnf/pkg/jobs/tnf.go
🚧 Files skipped from review as they are similar to previous changes (3)
- pkg/tnf/pkg/jobs/jobcontroller.go
- pkg/tnf/pkg/jobs/utils.go
- pkg/tnf/pkg/jobs/lifecycle_test.go
b09bb41 to
c3f8662
Compare
There was a problem hiding this comment.
Actionable comments posted: 2
♻️ Duplicate comments (1)
pkg/tnf/pkg/jobs/lifecycle.go (1)
336-391: 🎯 Functional Correctness | 🟠 Major | ⚡ Quick winRetry state is still advanced before the failed job's deletion is confirmed.
state.NodeIndex/state.AttemptNumberare mutated at Lines 379-380 beforeDeleteAndWaitruns at Lines 383-386. If deletion fails, the error is returned but the state mutation already happened; the next sync re-detects the same still-existing failed Job and advances the index/attempt again, skipping a node/attempt due to a transient delete error. This is the same issue flagged in a prior review on this segment and does not appear to have been addressed yet (unlike the sibling nil-pointer fix, which was).🐛 Proposed fix: only commit state after a successful delete
- // Apply state changes - state.NodeIndex = nextNodeIndex - state.AttemptNumber = nextAttemptNumber - // Delete the failed job so it can be recreated on next node klog.Infof("Deleting failed job %s to retry on node index %d", jobName, nextNodeIndex) if err := DeleteAndWait(ctx, kubeClient, jobName, operatorclient.TargetNamespace); err != nil { return fmt.Errorf("failed to delete failed job: %w", err) } + + // Apply state changes only after successful deletion, so a delete + // failure doesn't cause a node/attempt to be skipped on retry. + state.NodeIndex = nextNodeIndex + state.AttemptNumber = nextAttemptNumber }🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@pkg/tnf/pkg/jobs/lifecycle.go` around lines 336 - 391, Move the state assignments in the failed-job retry flow so state.NodeIndex and state.AttemptNumber are committed only after DeleteAndWait succeeds. Keep the calculated nextNodeIndex and nextAttemptNumber as local values during deletion, and return the deletion error without mutating retry state if deletion fails.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@pkg/tnf/operator/starter.go`:
- Around line 429-434: Update schedulableNodesFunc to exclude control-plane
nodes carrying the arbiter:NoSchedule taint before returning the node slice.
Preserve the existing control-plane label selection and ensure only nodes
eligible for scheduling are available to the lifecycle’s NodeName assignment.
In `@pkg/tnf/pkg/jobs/lifecycle.go`:
- Around line 3-25: Add the standard library "slices" import to the import block
in lifecycle.go so the slicesEqual function can resolve its calls to slices.Sort
and compile successfully.
---
Duplicate comments:
In `@pkg/tnf/pkg/jobs/lifecycle.go`:
- Around line 336-391: Move the state assignments in the failed-job retry flow
so state.NodeIndex and state.AttemptNumber are committed only after
DeleteAndWait succeeds. Keep the calculated nextNodeIndex and nextAttemptNumber
as local values during deletion, and return the deletion error without mutating
retry state if deletion fails.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository: openshift/coderabbit/.coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: bd8a999a-cd72-4a2e-9365-f75ad6b94e7d
📒 Files selected for processing (12)
docs/tnf/job-controllers.mdpkg/tnf/operator/nodehandler.gopkg/tnf/operator/starter.gopkg/tnf/pkg/jobs/jobcontroller.gopkg/tnf/pkg/jobs/jobcontroller_test.gopkg/tnf/pkg/jobs/lifecycle.gopkg/tnf/pkg/jobs/lifecycle_test.gopkg/tnf/pkg/jobs/tnf.gopkg/tnf/pkg/jobs/tnf_test.gopkg/tnf/pkg/jobs/utils.gopkg/tnf/pkg/tools/jobs.gopkg/tnf/pkg/tools/nodes.go
💤 Files with no reviewable changes (2)
- pkg/tnf/pkg/jobs/tnf_test.go
- pkg/tnf/pkg/jobs/tnf.go
🚧 Files skipped from review as they are similar to previous changes (7)
- pkg/tnf/pkg/tools/nodes.go
- pkg/tnf/pkg/jobs/jobcontroller.go
- pkg/tnf/pkg/tools/jobs.go
- pkg/tnf/operator/nodehandler.go
- pkg/tnf/pkg/jobs/jobcontroller_test.go
- pkg/tnf/pkg/jobs/utils.go
- pkg/tnf/pkg/jobs/lifecycle_test.go
c3f8662 to
d052f7b
Compare
| if err != nil { | ||
| klog.Infof("Cluster not running (err: %v), skipping update-setup on this node", err) | ||
| return nil | ||
| return fmt.Errorf("pacemaker is not running on this node %s: %w", currentNodeName, err) |
There was a problem hiding this comment.
This is necessary because we need the update-setup job to round-robin now.
There was a problem hiding this comment.
Actionable comments posted: 6
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@pkg/tnf/pkg/jobs/lifecycle_test.go`:
- Around line 72-148: Reconcile the node field naming and types in both
table-driven suites in pkg/tnf/pkg/jobs/lifecycle_test.go:72-148 and
pkg/tnf/pkg/jobs/lifecycle_test.go:237-254. In TestRunTNFJobController, make
struct literals match the declared field and existing tt.node reads. In the
restart/run test table, unify the declaration, literals, and loop reads with the
node parameter type expected by RestartNodeJobOrRunController and
RunNodeJobController, passing the correct node value or pointer.
- Around line 747-760: Remove the orphaned duplicate import block appearing
after TestSyncMultiNodeJobState_RetryProgression; retain only the valid
top-of-file imports and leave the test implementation unchanged.
In `@pkg/tnf/pkg/jobs/lifecycle.go`:
- Around line 284-312: In the nodesChanged/configChanged branch of the lifecycle
logic, defer updating state.AttemptNumber, state.NodeIndex, state.TargetNodes,
and state.LastJobConfig until after the existing job lookup and any
DeleteAndWait call complete successfully. Preserve the current error returns for
lookup or deletion failures so retries still see the original state; commit the
new retry and configuration state immediately before returning nil on success.
- Around line 644-705: Fix the undefined nodeTarget reference in
RestartNodeJobOrRunController by passing the existing node parameter to
RunNodeJobController. Preserve the surrounding controller invocation and restart
handling unchanged.
- Around line 519-543: The job hook’s nil return for an unready node does not
prevent syncManaged from reaching ApplyJob. Update the syncManaged flow around
the hook execution to explicitly stop and skip ApplyJob when admission is
blocked, while preserving normal application for ready jobs and propagating
actual hook errors.
In `@pkg/tnf/pkg/tools/nodes.go`:
- Around line 54-66: The HasArbiterNoScheduleTaint function uses the wrong taint
key. Update its taint check to match OpenShift’s node-role.kubernetes.io/arbiter
key while preserving the NoSchedule effect and existing nil-node behavior.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository: openshift/coderabbit/.coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 0f079ba3-fae7-48be-8b8c-0407a925d03f
📒 Files selected for processing (13)
docs/tnf/job-controllers.mdpkg/tnf/operator/nodehandler.gopkg/tnf/operator/starter.gopkg/tnf/pkg/jobs/jobcontroller.gopkg/tnf/pkg/jobs/jobcontroller_test.gopkg/tnf/pkg/jobs/lifecycle.gopkg/tnf/pkg/jobs/lifecycle_test.gopkg/tnf/pkg/jobs/tnf.gopkg/tnf/pkg/jobs/tnf_test.gopkg/tnf/pkg/jobs/utils.gopkg/tnf/pkg/tools/jobs.gopkg/tnf/pkg/tools/nodes.gopkg/tnf/update-setup/runner.go
💤 Files with no reviewable changes (2)
- pkg/tnf/pkg/jobs/tnf.go
- pkg/tnf/pkg/jobs/tnf_test.go
🚧 Files skipped from review as they are similar to previous changes (7)
- pkg/tnf/pkg/tools/jobs.go
- docs/tnf/job-controllers.md
- pkg/tnf/pkg/jobs/jobcontroller.go
- pkg/tnf/operator/starter.go
- pkg/tnf/pkg/jobs/utils.go
- pkg/tnf/pkg/jobs/jobcontroller_test.go
- pkg/tnf/operator/nodehandler.go
| func RestartNodeJobOrRunController( | ||
| ctx context.Context, | ||
| jobType tools.JobType, | ||
| node *corev1.Node, | ||
| retries int, | ||
| controllerContext *controllercmd.ControllerContext, | ||
| operatorClient v1helpers.StaticPodOperatorClient, | ||
| kubeClient kubernetes.Interface, | ||
| kubeInformersForNamespaces v1helpers.KubeInformersForNamespaces, | ||
| conditions []string, | ||
| existingJobCompletionTimeout time.Duration) error { | ||
|
|
||
| jobName := jobType.GetJobName(&node.Name) | ||
|
|
||
| // Acquire a lock for this specific job to prevent parallel execution | ||
| restartJobLocksMutex.Lock() | ||
| jobLock, exists := restartJobLocks[jobName] | ||
| if !exists { | ||
| jobLock = &sync.Mutex{} | ||
| restartJobLocks[jobName] = jobLock | ||
| } | ||
| restartJobLocksMutex.Unlock() | ||
|
|
||
| jobLock.Lock() | ||
| defer jobLock.Unlock() | ||
|
|
||
| // Check if job already exists | ||
| jobExists := true | ||
| _, err := kubeClient.BatchV1().Jobs(operatorclient.TargetNamespace).Get(ctx, jobName, v1.GetOptions{}) | ||
| if err != nil { | ||
| if !apierrors.IsNotFound(err) { | ||
| return fmt.Errorf("failed to check for existing job %s: %w", jobName, err) | ||
| } | ||
| jobExists = false | ||
| } | ||
|
|
||
| // Always run the controller (CEO might have been restarted) | ||
| RunNodeJobController(ctx, jobType, nodeTarget, retries, controllerContext, operatorClient, kubeClient, kubeInformersForNamespaces, conditions) | ||
|
|
||
| if !jobExists { | ||
| // No existing job - reset retry state to start fresh | ||
| resetJobRetryState(jobName) | ||
| return nil | ||
| } | ||
|
|
||
| // Job exists, wait for it to stop | ||
| klog.Infof("Job %s already exists, waiting for it to stop", jobName) | ||
| if err := WaitForStopped(ctx, kubeClient, jobName, operatorclient.TargetNamespace, existingJobCompletionTimeout); err != nil { | ||
| return fmt.Errorf("failed to wait for job %s to stop: %w", jobName, err) | ||
| } | ||
|
|
||
| // Delete the job so the controller can recreate it | ||
| klog.Infof("Deleting existing job %s", jobName) | ||
| if err := DeleteAndWait(ctx, kubeClient, jobName, operatorclient.TargetNamespace); err != nil { | ||
| return fmt.Errorf("failed to delete existing job %s: %w", jobName, err) | ||
| } | ||
|
|
||
| // Reset retry state when starting fresh after deleting old job | ||
| resetJobRetryState(jobName) | ||
|
|
||
| return nil | ||
| } |
There was a problem hiding this comment.
🎯 Functional Correctness | 🔴 Critical | ⚡ Quick win
Undefined identifier nodeTarget — this will not compile.
The function parameter is named node (line 647), but line 681 passes nodeTarget, which is never declared anywhere in this function. This is a build-breaking typo.
🐛 Proposed fix
// Always run the controller (CEO might have been restarted)
- RunNodeJobController(ctx, jobType, nodeTarget, retries, controllerContext, operatorClient, kubeClient, kubeInformersForNamespaces, conditions)
+ RunNodeJobController(ctx, jobType, node, retries, controllerContext, operatorClient, kubeClient, kubeInformersForNamespaces, conditions)📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| func RestartNodeJobOrRunController( | |
| ctx context.Context, | |
| jobType tools.JobType, | |
| node *corev1.Node, | |
| retries int, | |
| controllerContext *controllercmd.ControllerContext, | |
| operatorClient v1helpers.StaticPodOperatorClient, | |
| kubeClient kubernetes.Interface, | |
| kubeInformersForNamespaces v1helpers.KubeInformersForNamespaces, | |
| conditions []string, | |
| existingJobCompletionTimeout time.Duration) error { | |
| jobName := jobType.GetJobName(&node.Name) | |
| // Acquire a lock for this specific job to prevent parallel execution | |
| restartJobLocksMutex.Lock() | |
| jobLock, exists := restartJobLocks[jobName] | |
| if !exists { | |
| jobLock = &sync.Mutex{} | |
| restartJobLocks[jobName] = jobLock | |
| } | |
| restartJobLocksMutex.Unlock() | |
| jobLock.Lock() | |
| defer jobLock.Unlock() | |
| // Check if job already exists | |
| jobExists := true | |
| _, err := kubeClient.BatchV1().Jobs(operatorclient.TargetNamespace).Get(ctx, jobName, v1.GetOptions{}) | |
| if err != nil { | |
| if !apierrors.IsNotFound(err) { | |
| return fmt.Errorf("failed to check for existing job %s: %w", jobName, err) | |
| } | |
| jobExists = false | |
| } | |
| // Always run the controller (CEO might have been restarted) | |
| RunNodeJobController(ctx, jobType, nodeTarget, retries, controllerContext, operatorClient, kubeClient, kubeInformersForNamespaces, conditions) | |
| if !jobExists { | |
| // No existing job - reset retry state to start fresh | |
| resetJobRetryState(jobName) | |
| return nil | |
| } | |
| // Job exists, wait for it to stop | |
| klog.Infof("Job %s already exists, waiting for it to stop", jobName) | |
| if err := WaitForStopped(ctx, kubeClient, jobName, operatorclient.TargetNamespace, existingJobCompletionTimeout); err != nil { | |
| return fmt.Errorf("failed to wait for job %s to stop: %w", jobName, err) | |
| } | |
| // Delete the job so the controller can recreate it | |
| klog.Infof("Deleting existing job %s", jobName) | |
| if err := DeleteAndWait(ctx, kubeClient, jobName, operatorclient.TargetNamespace); err != nil { | |
| return fmt.Errorf("failed to delete existing job %s: %w", jobName, err) | |
| } | |
| // Reset retry state when starting fresh after deleting old job | |
| resetJobRetryState(jobName) | |
| return nil | |
| } | |
| func RestartNodeJobOrRunController( | |
| ctx context.Context, | |
| jobType tools.JobType, | |
| node *corev1.Node, | |
| retries int, | |
| controllerContext *controllercmd.ControllerContext, | |
| operatorClient v1helpers.StaticPodOperatorClient, | |
| kubeClient kubernetes.Interface, | |
| kubeInformersForNamespaces v1helpers.KubeInformersForNamespaces, | |
| conditions []string, | |
| existingJobCompletionTimeout time.Duration) error { | |
| jobName := jobType.GetJobName(&node.Name) | |
| // Acquire a lock for this specific job to prevent parallel execution | |
| restartJobLocksMutex.Lock() | |
| jobLock, exists := restartJobLocks[jobName] | |
| if !exists { | |
| jobLock = &sync.Mutex{} | |
| restartJobLocks[jobName] = jobLock | |
| } | |
| restartJobLocksMutex.Unlock() | |
| jobLock.Lock() | |
| defer jobLock.Unlock() | |
| // Check if job already exists | |
| jobExists := true | |
| _, err := kubeClient.BatchV1().Jobs(operatorclient.TargetNamespace).Get(ctx, jobName, v1.GetOptions{}) | |
| if err != nil { | |
| if !apierrors.IsNotFound(err) { | |
| return fmt.Errorf("failed to check for existing job %s: %w", jobName, err) | |
| } | |
| jobExists = false | |
| } | |
| // Always run the controller (CEO might have been restarted) | |
| RunNodeJobController(ctx, jobType, node, retries, controllerContext, operatorClient, kubeClient, kubeInformersForNamespaces, conditions) | |
| if !jobExists { | |
| // No existing job - reset retry state to start fresh | |
| resetJobRetryState(jobName) | |
| return nil | |
| } | |
| // Job exists, wait for it to stop | |
| klog.Infof("Job %s already exists, waiting for it to stop", jobName) | |
| if err := WaitForStopped(ctx, kubeClient, jobName, operatorclient.TargetNamespace, existingJobCompletionTimeout); err != nil { | |
| return fmt.Errorf("failed to wait for job %s to stop: %w", jobName, err) | |
| } | |
| // Delete the job so the controller can recreate it | |
| klog.Infof("Deleting existing job %s", jobName) | |
| if err := DeleteAndWait(ctx, kubeClient, jobName, operatorclient.TargetNamespace); err != nil { | |
| return fmt.Errorf("failed to delete existing job %s: %w", jobName, err) | |
| } | |
| // Reset retry state when starting fresh after deleting old job | |
| resetJobRetryState(jobName) | |
| return nil | |
| } |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@pkg/tnf/pkg/jobs/lifecycle.go` around lines 644 - 705, Fix the undefined
nodeTarget reference in RestartNodeJobOrRunController by passing the existing
node parameter to RunNodeJobController. Preserve the surrounding controller
invocation and restart handling unchanged.
| // HasArbiterNoScheduleTaint returns true if the node has the arbiter:NoSchedule taint. | ||
| // Arbiter nodes in SNO+1 configurations are not eligible for normal workload scheduling. | ||
| func HasArbiterNoScheduleTaint(node *corev1.Node) bool { | ||
| if node == nil { | ||
| return false | ||
| } | ||
| for _, taint := range node.Spec.Taints { | ||
| if taint.Key == "arbiter" && taint.Effect == corev1.TaintEffectNoSchedule { | ||
| return true | ||
| } | ||
| } | ||
| return false | ||
| } |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
# Look for the arbiter taint key elsewhere in the repo (installer/MCO/topology helpers)
rg -nP 'arbiter' --type=go -C2Repository: openshift/cluster-etcd-operator
Length of output: 169
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Map the target file and inspect nearby code/tests for the helper usage.
ast-grep outline pkg/tnf/pkg/tools/nodes.go --view expanded || true
printf '\n--- nodes.go ---\n'
cat -n pkg/tnf/pkg/tools/nodes.go | sed -n '1,220p'
printf '\n--- repo search: arbiter / taint key / schedulable nodes ---\n'
rg -n --hidden --glob '!**/*vendor*' --glob '!**/node_modules/**' 'node-role\.kubernetes\.io/arbiter|arbiter:NoSchedule|HasArbiterNoScheduleTaint|FilterSchedulableNodes|NoSchedule' .Repository: openshift/cluster-etcd-operator
Length of output: 4840
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Check public docs references in the repo, if any, that define the arbiter topology or taint key.
rg -n --hidden --glob '!**/*vendor*' --glob '!**/node_modules/**' 'Two-Node-with-Arbiter|arbiter topology|arbiter node|node-role\.kubernetes\.io/arbiter' .Repository: openshift/cluster-etcd-operator
Length of output: 1299
🌐 Web query:
OpenShift arbiter node taint key node-role.kubernetes.io/arbiter NoSchedule documentation
💡 Result:
The taint key node-role.kubernetes.io/arbiter with the effect NoSchedule is used in OpenShift to prevent standard application workloads and non-essential control plane components from being scheduled onto an arbiter node [1][2][3]. An arbiter node is a specialized, lightweight node role introduced to support topologies such as two-node OpenShift clusters [4][2][5]. Its primary function is to store etcd data to maintain quorum and prevent split-brain scenarios in compact, cost-effective deployments [4][5][6]. Because the arbiter node is intended solely for etcd consensus and does not run standard control plane components (like kube-apiserver or kube-controller-manager) or application workloads, this taint is applied to ensure it remains dedicated to its specific role [4][3][5]. If you need to schedule specific infrastructure components or workloads onto an arbiter node, you must add a corresponding toleration to the pod or deployment specification [3]: tolerations: - effect: NoSchedule key: node-role.kubernetes.io/arbiter While the arbiter node is a regular Red Hat CoreOS node running the Kubelet, the arbiter role prevents automatic scheduling of general pods [3]. Technical documentation and community resources emphasize that this node role is distinct and handled specifically by the OpenShift infrastructure [7][8][9].
Citations:
- 1: https://portworx.com/wp-content/uploads/2025/11/TNA-Design-Guide.pdf
- 2: https://medium.com/@tcij1013/architectural-paradigms-for-two-node-openshift-4-20-clusters-tna-vs-tnf-deep-dive-4498772453b4
- 3: https://www.redhat.com/en/blog/efficient-two-node-edge-infrastructure-red-hat-openshift-and-portworxpure-storage
- 4: https://docs.redhat.com/en/documentation/openshift_container_platform/4.21/observability/installing_a_two_node_openshift_cluster/about-two-node-arbiter-installation
- 5: https://docs.redhat.com/en/documentation/openshift_container_platform/4.22/html/installing_a_two_node_openshift_cluster/about-two-node-arbiter-installation
- 6: https://docs.redhat.com/en/documentation/openshift_container_platform/4.20/html/installing_a_two_node_openshift_cluster/about-two-node-arbiter-installation
- 7: OCPEDGE-1314: feat: add arbiter node role kubernetes#2109
- 8: openshift/kubernetes@cbbbafc
- 9: http://github.com/openshift/machine-config-operator/pull/5436
🌐 Web query:
site:docs.redhat.com OpenShift arbiter taint key effect NoSchedule "node-role.kubernetes.io/arbiter"
💡 Result:
In OpenShift Container Platform, the arbiter node is a specialized node type used in Two-Node OpenShift with Arbiter (TNA) cluster topologies [1][2]. Its primary purpose is to store etcd data to maintain quorum and prevent split-brain conditions in a configuration that only has two control plane nodes [1][3]. Arbiter nodes are explicitly restricted from running workloads and do not run additional control plane components like kube-apiserver or kube-controller-manager [1][2]. To enforce this, arbiter nodes are automatically tainted with the key node-role.kubernetes.io/arbiter and the effect NoSchedule [1]. Because of this NoSchedule taint, the Kubernetes scheduler will not place pods on an arbiter node unless those pods have a corresponding toleration that matches the taint [4][5]. This design ensures that the arbiter node remains dedicated to its etcd-quorum function and is not used for general-purpose application hosting [1][2].
Citations:
- 1: https://docs.redhat.com/en/documentation/openshift_container_platform/4.22/html/installing_a_two_node_openshift_cluster/about-two-node-arbiter-installation
- 2: https://docs.redhat.com/en/documentation/assisted_installer_for_openshift_container_platform/2026/html/installing_openshift_container_platform_with_the_assisted_installer/about-ai
- 3: https://docs.redhat.com/en/documentation/openshift_container_platform/4.21/observability/installing_a_two_node_openshift_cluster/two-node-with-fencing
- 4: https://docs.redhat.com/en/documentation/openshift_container_platform/4.20/html/nodes/controlling-pod-placement-onto-nodes-scheduling
- 5: https://docs.redhat.com/en/documentation/openshift_container_platform/4.15/html/nodes/controlling-pod-placement-onto-nodes-scheduling
Use the OpenShift arbiter taint key here
HasArbiterNoScheduleTaint checks arbiter, but OpenShift taints arbiter nodes with node-role.kubernetes.io/arbiter:NoSchedule. As written, arbiter nodes will never match and FilterSchedulableNodes will keep including them.
Suggested fix
- if taint.Key == "arbiter" && taint.Effect == corev1.TaintEffectNoSchedule {
+ if taint.Key == "node-role.kubernetes.io/arbiter" && taint.Effect == corev1.TaintEffectNoSchedule {📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| // HasArbiterNoScheduleTaint returns true if the node has the arbiter:NoSchedule taint. | |
| // Arbiter nodes in SNO+1 configurations are not eligible for normal workload scheduling. | |
| func HasArbiterNoScheduleTaint(node *corev1.Node) bool { | |
| if node == nil { | |
| return false | |
| } | |
| for _, taint := range node.Spec.Taints { | |
| if taint.Key == "arbiter" && taint.Effect == corev1.TaintEffectNoSchedule { | |
| return true | |
| } | |
| } | |
| return false | |
| } | |
| // HasArbiterNoScheduleTaint returns true if the node has the arbiter:NoSchedule taint. | |
| // Arbiter nodes in SNO+1 configurations are not eligible for normal workload scheduling. | |
| func HasArbiterNoScheduleTaint(node *corev1.Node) bool { | |
| if node == nil { | |
| return false | |
| } | |
| for _, taint := range node.Spec.Taints { | |
| if taint.Key == "node-role.kubernetes.io/arbiter" && taint.Effect == corev1.TaintEffectNoSchedule { | |
| return true | |
| } | |
| } | |
| return false | |
| } |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@pkg/tnf/pkg/tools/nodes.go` around lines 54 - 66, The
HasArbiterNoScheduleTaint function uses the wrong taint key. Update its taint
check to match OpenShift’s node-role.kubernetes.io/arbiter key while preserving
the NoSchedule effect and existing nil-node behavior.
59545f5 to
22df05d
Compare
Implements the foundational framework for multi-node job scheduling with round-robin retry logic and drift detection: - Multi-node job support with NodeTarget struct for node-specific jobs - Round-robin retry logic with state tracking (AttemptNumber, NodeIndex, LastJobConfig) - Job config drift detection (resets retry state when config changes) - Degradation when retry attempts exhausted - IsControllerRunning() helper to prevent duplicate controllers - Shared node utility functions (GetNodeNames, DetermineReconciliationActions, IsNodeReady) - Job type labels for filtering and cleanup See docs/tnf/job-controllers.md for detailed multi-node retry behavior and config drift examples. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> Refactoring updates: - Split API into RunNodeJobController/RunClusterJobController (eliminates conditionals) - Unified node readiness checking to avoid duplicate condition setting - Update ToPascalCase to capitalize 'tnf' as 'TNF' (acronym, not 'Tnf') - Extract duplicate blocked condition logic into manageBlockedCondition() helper - All exports verified as intentional (public API) OCPBUGS-84695: feat(tnf): bootstrap job controllers and startup logic Implements job controller startup with dual-mode behavior (bootstrap vs runtime) and individual job implementations: - StartJobControllers() with bootstrap vs runtime separation - Bootstrap mode: exponential backoff retry, requires exactly 2 ready nodes, sets degraded condition - Runtime mode: idempotent restart support, handles 1-2 nodes, fast path when setup job complete - startTnfJobcontrollers(): waits for etcd bootstrap, stable revision, creates per-node and cluster-wide controllers - Auth job implementation: sets hacluster password (required before Pacemaker operations) - After-setup job implementation: * Pre-transition: waits for setup job completion * Post-transition: waits for update-setup job containing this node * Disables kubelet service (Pacemaker owns it now) * Prevents kubelet disable race during Day 2 node addition - Update-setup controller check: on operator restart, starts controller for stopped update-setup job See docs/tnf/job-controllers.md for dual-mode operation and bootstrap sequence. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
22df05d to
6ffede6
Compare
|
@jaypoulz: The following tests failed, say
Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
2 participants
Implements the foundational framework for multi-node job scheduling with round-robin retry logic and drift detection:
See docs/tnf/job-controllers.md for detailed multi-node retry behavior and config drift examples.
Co-Authored-By: Claude Sonnet 4.5 noreply@anthropic.com
Summary by CodeRabbit
New Features
Bug Fixes
Documentation
Refactor