RFC: Two-level Failed Pod handling in RoleInstance Controller

[Syspretor]

Summary

This RFC proposes a clear separation of responsibilities for handling Failed/inactive pods in the RBG controller system. With the removal of RecreateRBGOnPodRestart policy (#340), the remaining restart policies need well-defined handling paths for pod failures.

Background

Pod failures in Kubernetes (Evicted, UnexpectedAdmissionError, OOM-killed, etc.) result in pods entering the Failed phase. Unlike container restarts (where the pod remains Running/Pending), Failed pods are terminal and require explicit cleanup before replacements can be created.

Two-Level Handling Design

Level 1: Delete Failed Pods (applies to ALL RestartPolicies)

Failed pods must be explicitly deleted because they block replacement creation — hasOrphanPod detects the same-name pod still exists and prevents new pod creation.

Handler: RoleInstance Controller (instance_scale.go → calculateDiffsWithExpectation)

// Delete inactive (Failed) pods so that replacements can be created on next reconcile.
for _, p := range inactivePods {
    if p.Status.Phase == v1.PodFailed && p.DeletionTimestamp == nil {
        toDeletePods = append(toDeletePods, p)
    }
}

Flow: Pod Failed → RoleInstance reconcile → delete Failed pod → next reconcile creates replacement

Level 2: Recreate Entire Instance (only for RecreateRoleInstanceOnPodRestart)

When restartPolicy=RecreateRoleInstanceOnPodRestart, a Pod failure should trigger recreation of ALL pods in the Instance (not just a replacement for the failed one). This ensures instance-level consistency.

Handler: RoleInstance Controller (instance_scale.go → shouldRecreateInstance)

func shouldRecreateInstance(instance, pods) bool {
    // Only for RecreateRoleInstanceOnPodRestart policy
    // Only when Instance was previously Ready (stable state)
    // Only when spec is not being changed (Generation == ObservedGeneration)
    // Trigger: any Pod in Failed phase
}

Flow: Pod Failed → RoleInstance reconcile → delete ALL pods → recreate entire Instance

Key Design Decisions

Aspect	Decision	Rationale
Failed pod detection	Check pod.Status.Phase == Failed	Direct, unambiguous signal
Only trigger on stable state	wasInstanceReady && Generation == ObservedGeneration	Avoid recreation storms during scaling/rollout
Container restart handling	Delegated to underlying workload controller (e.g. LWS)	RoleInstance Controller only handles terminal pod states
Succeeded pods	NOT handled (per Non-Goals)	Represents normal completion, not failure

Comparison: Container Restart vs Pod Failed

Scenario	Handler	Action
Container restart + None	Underlying workload controller	Let container restart
Container restart + RecreateRoleInstanceOnPodRestart	LWS Controller	Recreate RoleInstance
Pod Failed + None	RoleInstance Controller	Delete Failed pod → create replacement
Pod Failed + RecreateRoleInstanceOnPodRestart	RoleInstance Controller	Recreate entire Instance

Implementation

The implementation passes inactivePods (pods filtered out by IsPodActive) through the RoleInstance reconciler chain into calculateDiffsWithExpectation, making Failed pods visible to both Level 1 (deletion) and Level 2 (recreation) logic.

Related

• chore(rbg): deprecate and remove RecreateRBGOnPodRestart restart policy #340 (Remove RecreateRBGOnPodRestart)
• KEP: keps/inactive-pod-handling/README.md

[Syspretor]

Native Workload Comparison: How K8s handles Failed Pods

Deployment (via ReplicaSet Controller)

• Does NOT actively delete Failed Pods
• RS controller uses FilterActivePods() → IsPodActive() — Failed/Succeeded pods are excluded from active count
• Active count < desired → directly create a new pod (with a new random name)
• Failed pod remains in the cluster until Pod GC Controller cleans it up asynchronously (--terminated-pod-gc-threshold, default 12500)
• Key: Deployment pods have random names, so there's never a "same-name conflict" — replacement creation is never blocked by the Failed pod's existence

StatefulSet Controller

• Actively deletes Failed Pods
• StatefulSet pods have strict ordinal identity (pod-0, pod-1, ...)
• In UpdateStatefulSet(): if isFailed(replicas[i]) → explicitly delete the pod
• Next reconcile: ordinal slot is empty → create a new pod with the same ordinal name
• Key: Must delete before recreate because two pods with the same ordinal cannot coexist

RoleInstance (this PR)

RoleInstance is analogous to StatefulSet — pods have fixed names/identities. The hasOrphanPod check prevents creating a new pod when a same-name pod already exists (regardless of phase). Therefore:

1. If a Failed pod is NOT deleted: hasOrphanPod → skip creation → replica count deficit persists indefinitely until Pod GC runs (uncontrolled delay)
2. The correct approach (matching StatefulSet): explicitly delete Failed pod → next reconcile creates replacement

Summary

Workload	Pod Identity	Deletes Failed Pod?	Why
Deployment/RS	Random name	No	No name conflict; just create another
StatefulSet	Fixed ordinal	Yes	Same-ordinal pod blocks creation
RoleInstance	Fixed name	Yes	hasOrphanPod blocks same-name creation

This validates the two-level design in this PR: Level 1 deletes Failed pods (unblocking replacement), Level 2 optionally recreates the entire Instance for policy-driven consistency.