fix: disable waitForReady on gRPC runner client to prevent infinite block

[Eitan1112]

Problem

When a runner pod is killed mid-reconciliation (e.g. due to spot/preemptible node eviction),
the controller goroutine blocks forever on the gRPC call and never recovers.

This happens because the gRPC service config in getRunnerConnection sets waitForReady: true.
Per gRPC wait-for-ready semantics,
when the connection enters TRANSIENT_FAILURE (runner pod gone), the client queues RPCs and waits
for the connection to become READY again instead of failing. Since the runner pod is dead and the
reconciliation context carries no deadline, this wait is infinite.

The stuck goroutine blocks all future reconciliations of the same Terraform CR because
controller-runtime allows only one active reconciliation per object key. Annotating with
reconcile.fluxcd.io/requestedAt does not help — the event is enqueued but cannot be
processed until the stuck reconciliation completes, which it never does.

Observed behavior

Terraform CR permanently stuck in Reconciling / Initializing state. Controller logs show
the reconciliation reaching calling detectDrift ... → creating a plan and then silence.
No further logs for that CR. No runner pod present in the namespace.

Code path

1. Reconcile receives a context with no deadline (tf_controller.go:113)
2. getRunnerConnection creates a gRPC client with waitForReady: true (tf_controller_runner.go:143)
3. detectDrift calls runnerClient.Plan(ctx, planRequest) (tf_controller_drift_detect.go:89)
4. Runner pod is killed → connection goes to TRANSIENT_FAILURE
5. gRPC client buffers the Plan RPC, waits forever for connection to become READY
6. Goroutine is permanently blocked, no new reconciliation can start for this CR

Reproduction conditions

• Runner pods on preemptible/spot nodes
• Long-running tofu plan (large state, many resources)
• Runner pod killed during plan execution

Controller logs

17:09:44 [info] >> Started Generation: 17
17:09:44 [info] before lookup runner: checking ready condition
17:09:44 [info] getting source
17:09:44 [info] show runner pod state:
17:09:46 [info] runner is running
17:09:46 [info] setting up terraform
17:09:55 [info] write backend config: ok
17:09:55 [info] new terraform
17:09:55 [info] generated var files from spec
17:10:26 [info] init reply: ok
17:10:26 [info] workspace select reply: ok
17:10:26 [info] calling detectDrift ...
<--- silence, no further logs for this CR, 22+ hours later still stuck --->

Runner pod logs

17:09:45 Starting the runner... version  sha
17:09:55 preparing for Upload and Extraction
17:09:55 write backend config
17:09:55 creating new terraform
17:09:55 setting up the input variables
17:10:26 workspace select
17:10:26 creating a plan
<--- pod killed (spot preemption), no further logs --->

Terraform CR status (stuck)

status:
  conditions:
    - type: Reconciling
      status: "True"
      reason: Progressing
      message: Fulfilling prerequisites
    - type: Ready
      status: Unknown
      reason: Progressing
      message: Initializing

Fix

Set waitForReady to false in the gRPC service config. This makes RPCs fail immediately
with UNAVAILABLE when the connection is broken. The existing retry policy already handles
UNAVAILABLE (4 attempts with backoff). After retries are exhausted, the reconciliation
returns an error, the CR is re-queued by controller-runtime, and the next reconciliation
proceeds normally with a fresh runner pod.

• waitForReady: false is the gRPC default
  — this change restores default behavior
• The initial connection is already established and confirmed working before any RPCs are made
  (the dial has a 30s timeout at tf_controller_runner.go:109)
• The retry policy (UNAVAILABLE, 4 attempts) provides resilience against transient network blips
• If the runner pod is genuinely gone, failing fast and re-queuing is the correct behavior

[KarstenSiemer]

Hit this exact bug

Cluster had been silently stuck for days with Terraform CRs frozen on Reconciling=True (Fulfilling prerequisites) + Ready=Unknown (Initializing), no logs for the affected resources, dependents looping every 90s on dependency '...' is not ready.

Pulled metrics off the leader before doing anything destructive:

workqueue_longest_running_processor_seconds{controller="terraform"} 938959   # 10.86 days
workqueue_unfinished_work_seconds                                   1861919
controller_runtime_active_workers                                   3        # of 24 max
workqueue_depth                                                     1

Goroutine dump showed three workers parked in exactly the place this PR points at:

google.golang.org/grpc.(*pickerWrapper).pick
google.golang.org/grpc.(*clientStream).RecvMsg
runner.(*runnerClient).Plan
controllers.(*TerraformReconciler).detectDrift   tf_controller_drift_detect.go:89
controllers.(*TerraformReconciler).reconcile     tf_controller_reconcile.go:139

Three runner pods evicted at some point in the past, three worker slots wedged ever since, no way for the controller to recover on its own. Annotating with reconcile.fluxcd.io/requestedAt did nothing — confirmed your point about controller-runtime not enqueuing a second reconcile while one is in flight.

After deleting the leader pod, workqueue_longest_running_processor_seconds dropped to ~12 seconds and every stuck CR reconciled cleanly within ~15s using fresh runner pods. Same code, same workload, just no inherited stuck state.

Bumping in the hope it gets reviewer attention

[matikij]

Adding another data point in support of this fix — we hit exactly this pattern on EKS this week.

Versions: controller ghcr.io/flux-iac/tofu-controller:v0.16.2, runner built from the same upstream tag, gRPC v1.80.0, controller-runtime v0.23.3.

Symptom. Three Terraform CRs stuck at Ready=Unknown / Initializing for ~17h. No log lines mentioning these CRs across either controller pod's available log buffer, while many sibling Terraforms in the same namespaces reconciled normally throughout. reconciliationFailures was a frozen value, lastPlanAt was 17h stale, and Reconciling.lastTransitionTime had not advanced since the moment of failure. Annotating with reconcile.fluxcd.io/requestedAt had no effect — resourceVersion advanced but no Reconcile entered.

Trigger. A failed Apply (an upstream-module change had introduced a missing IAM permission); the next Reconcile cycle entered setupTerraform / detectDrift, made a gRPC call to the runner pod, and the call never returned.

Workqueue metrics from the leader pod, ~17h after the wedge:

controller_runtime_active_workers{controller="terraform"}            3
controller_runtime_max_concurrent_reconciles{controller="terraform"} 24
workqueue_longest_running_processor_seconds                          63847.93   (~17h44m)
workqueue_unfinished_work_seconds                                    191515.58  (~53h, ≈ 3 × 17h44m)
workqueue_retries_total                                              3001
workqueue_depth{priority="0"}                                        3

The three "active workers" each held one of the three stuck CRs.

Goroutine dump. Three Reconcile goroutines were parked in google.golang.org/grpc.(*pickerWrapper).pick at picker_wrapper.go:123 — the select{} that waits forever when waitForReady=true and the channel is in TRANSIENT_FAILURE. The bottom of the stack identifies the call site:

• one in runner.(*runnerClient).Plan ← controllers.detectDrift ← tf_controller_drift_detect.go:89
• two in runner.(*runnerClient).UploadAndExtract ← controllers.setupTerraform ← tf_controller_backend.go:54

Abbreviated stack from one of the UploadAndExtract cases (other two are structurally identical):

goroutine 361 [select]:
google.golang.org/grpc.(*pickerWrapper).pick(...)
        /go/pkg/mod/google.golang.org/grpc@v1.80.0/picker_wrapper.go:123 +0x17c
google.golang.org/grpc.(*csAttempt).getTransport(...)
        /go/pkg/mod/google.golang.org/grpc@v1.80.0/stream.go:503 +0x56
google.golang.org/grpc.newClientStreamWithParams.func2(...)
        /go/pkg/mod/google.golang.org/grpc@v1.80.0/stream.go:384 +0x25
...
google.golang.org/grpc.(*ClientConn).Invoke(...)
        /go/pkg/mod/google.golang.org/grpc@v1.80.0/call.go:37 +0x232
github.com/flux-iac/tofu-controller/runner.(*runnerClient).UploadAndExtract(...)
        /build/runner/runner_grpc.pb.go:139 +0x167
github.com/flux-iac/tofu-controller/controllers.(*TerraformReconciler).setupTerraform(...)
        /build/controllers/tf_controller_backend.go:54 +0x26d
github.com/flux-iac/tofu-controller/controllers.(*TerraformReconciler).reconcile(...)
        /build/controllers/tf_controller_reconcile.go:47 +0x18d
github.com/flux-iac/tofu-controller/controllers.(*TerraformReconciler).Reconcile(...)
        /build/controllers/tf_controller.go:511 +0x388e
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile(...)
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 +0x1ab
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler(...)
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 +0x39b
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem(...)
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 +0x1f8

Why annotation requeues didn't help (matches the PR description exactly): controller-runtime's workqueue dedup-coalesces an Add for a key that is currently in the processing set into a dirty flag, only re-emitting after Done(key). The wedged Reconcile never returns → never calls Done → every annotation-driven Add is silently absorbed.

Recovery. Deleting the leader pod (kubectl delete pod) unblocked all three within seconds — the workqueue rebuilt from a fresh List, the gRPC clients tore down, and the runners came back up against new pods.

This is on EKS (so not the GCP DNS class of cause referenced in #1181). Network/DNS between the controller and runners is healthy in steady state — sibling Terraforms in the same namespaces were reconciling concurrently throughout the wedge.

We can't say with certainty what made the specific runner pods go away (we don't have post-mortem logs from those pods); the most likely trigger is just node rotation — our cluster autoscaler turns nodes over routinely, and any runner pod that happened to be on a rotated node would have been killed mid-life along with everything else. Spot/preemptible eviction or the controller's own alwaysCleanupRunnerPod cleanup defer would produce the same end-state. Whatever the specific cause, the runner pod that handled the original failed Apply went away before its ClientConn was closed by the controller. With waitForReady: true and an unbounded reconcile context, the next Reconcile that re-uses that ClientConn blocks forever in pickerWrapper.pick.

The fix in this PR (waitForReady: false + the existing 4-attempt retry policy) would have surfaced the stale connection as UNAVAILABLE, reconcile would have errored, and the workqueue would have re-enqueued normally with a fresh dial. +1 from us.