fix: prevent permanent reconcile hang when a runner pod dies mid-reconcile

[agentydragon]

Problem

When a node hosting tf-runner pods reboots (or the pods are evicted / OOM-killed) while the controller is mid-reconcile, the affected Terraform reconciles hang permanently. They freeze in Ready=Unknown / Initializing, stop logging, and never retry or self-heal. Only a controller restart clears it.

We hit this in a ~26-Terraform cluster after a worker node reboot: every Terraform whose runner was on that node wedged simultaneously and stayed wedged for 8h+ until we restarted the controller.

Root cause

Two things combine into an unbounded hang:

1. The runner gRPC client uses waitForReady: true — controllers/tf_controller_runner.go (getRunnerConnection, the retryPolicy service config). An RPC on a channel in TRANSIENT_FAILURE (backend pod gone) blocks waiting for READY instead of failing fast with UNAVAILABLE.

2. The runner RPCs carry no context deadline — e.g. runnerClient.Init(ctx, initRequest) at controllers/tf_controller_backend.go:361 uses the raw reconcile ctx. controller-runtime imposes no per-reconcile timeout by default, and the controller never wraps ctx. (RunnerCreationTimeout bounds only pod creation in reconcileRunnerPod, not the RPCs; the 30s dialCtx in LookupOrCreateRunner is cancelled before any RPC runs, and grpc.NewClient is lazy.)

waitForReady:true + a deadline-less context = an RPC to a vanished runner blocks the reconcile goroutine forever.

Because controller-runtime won't start a new Reconcile for a key while the previous one is still running, that object is never reconciled again, and the parked goroutine permanently consumes one of the --concurrent worker slots. Enough such events deadlock the whole controller.

Log signature: the wedged object's last line is "generated template" (tf_controller_backend.go), then silence — the next RPC (Init) never returns. A healthy reconcile logs "init reply:" or "error running Init:" next; a wedged one logs neither.

Secondary bug (dead runner pods pile up)

In reconcileRunnerPod, a pod in phase=Failed matches none of the state branches (label present, label matches, no DeletionTimestamp, phase ≠ Running) → podState = stateUnknown, and the switch has no case stateUnknown → silent no-op. The dead pod is never cleaned up, so Error runner pods accumulate.

Reproduction

1. Apply any Terraform with a runner pod.
2. While it is in setup/init, delete the runner pod (kubectl delete pod <name>-tf-runner) — or reboot/drain its node — so the pod vanishes mid-Init.
3. The Terraform freezes at Ready=Unknown/Initializing and never recovers; controller logs go silent for that object; a Failed runner pod is left behind.

Fix

1. Bound the runner setup RPCs with a configurable deadline. New --runner-rpc-timeout flag (default 30m — generous enough for provider downloads / long inits, finite enough that a dead runner surfaces as an error and requeues instead of hanging). Applied to the setupTerraform RPC batch, where the hang occurs. This also supersedes the long-commented-out ctx30s TODO in setupTerraform (introduced already-commented in b98b632, 2022, and never enabled).
2. Reap Failed runner pods. Add case stateUnknown in reconcileRunnerPod to force-delete a pod that exists but is neither Running nor Terminating, then recreate it, mirroring the stateMustBeDeleted path.

Split into two commits so they can be reviewed independently.

Alternative considered

Dropping waitForReady:true (so a broken channel returns UNAVAILABLE immediately) is smaller, but risks spurious failures during normal runner cold-start, when the pod is Running but its gRPC server isn't listening yet. A bounded context keeps the startup-smoothing benefit of waitForReady while removing the infinite-hang failure mode. Happy to go the other direction if maintainers prefer.

Scope note

The timeout is applied to setupTerraform (upload → backend config → init → workspace select), which is where the observed hang occurs. plan/apply are invoked separately and can legitimately run much longer, so they are intentionally left unbounded here; a similar (larger) budget could be added for them as a follow-up if desired.

Testing

• go build ./... and go vet ./controllers/ ./cmd/manager/ pass.
• The stateUnknown case falls through to the existing "wait for pod IP" path, consistent with the other create branches.
• The envtest integration suite (controllers/tc0002xx) exercises real runner pods and is intended to run in CI; I was not able to run it in my local sandbox (envtest apiserver instability, reproducible on unmodified main). Happy to add a focused tc000260-style case for the stateUnknown reap if you'd like it in this PR.

I'm running these patches on my own cluster to confirm the wedge no longer recurs on node reboots. Feedback on the approach (flag name/default, setupTerraform-only scope, or preferring the waitForReady removal) very welcome.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.