From a1b0da0b6bd10176636ee85c3af4b656c6dc1bea Mon Sep 17 00:00:00 2001 From: Eitan Nargassi Date: Thu, 5 Mar 2026 17:37:31 +0200 Subject: [PATCH] fix: disable waitForReady on gRPC runner client to prevent infinite block When a runner pod is killed mid-reconciliation (e.g. spot/preemptible node eviction), the gRPC client blocks forever because waitForReady: true causes it to queue RPCs and wait for the dead connection to become READY again. Since the reconciliation context has no deadline, and the runner pod will never come back, the goroutine is permanently stuck. This blocks all future reconciliations of the same Terraform CR because controller-runtime only allows one active reconciliation per object key. Even annotating with reconcile.fluxcd.io/requestedAt cannot unblock it since the new event sits in the work queue behind the stuck one. Setting waitForReady to false makes RPCs fail immediately with UNAVAILABLE when the connection is broken, which is already handled by the retry policy (4 attempts). After retries are exhausted, the reconciliation returns an error, gets re-queued, and proceeds normally with a fresh runner pod. Co-Authored-By: Claude Opus 4.6 (1M context) --- controllers/tf_controller_runner.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/controllers/tf_controller_runner.go b/controllers/tf_controller_runner.go index d2a7eeca1..a38a04962 100644 --- a/controllers/tf_controller_runner.go +++ b/controllers/tf_controller_runner.go @@ -140,7 +140,7 @@ func (r *TerraformReconciler) getRunnerConnection(ctx context.Context, tlsSecret const retryPolicy = `{ "methodConfig": [{ "name": [{"service": "runner.Runner"}], - "waitForReady": true, + "waitForReady": false, "retryPolicy": { "MaxAttempts": 4, "InitialBackoff": ".01s",