docs(cluster): lesson — tofu-controller reconcile hang on runner death#2775
Merged
Conversation
RCA for the tofu-controller upstream bug where a reconcile goroutine parks forever in the runner Init RPC when a runner pod dies mid-reconcile (waitForReady + deadline-less context), which froze haku's email-account TF (agent-machine-access) after a wyrm2 reboot. Includes root cause with code refs, recovery, the compounding transient Cilium-policy/etcd-contention episode, and the upstream fix we filed (flux-iac/tofu-controller#1838).
…ooting runbook Add operational recognizers to troubleshooting.md so the failure modes are caught during an incident (not just after finding the dated lesson): - tofu-controller reconcile hang after a node reboot (recognize + restart). - transient 'Policy denied'/ClusterIP timeout during etcd contention (don't chase the datapath; a denied drop from an unrelated pod is correct). Also surface tofu-controller in the AGENTS.md troubleshooting pointer.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Documents the tofu-controller reconcile-hang incident that froze haku's email-account Terraform (
agent-machine-access) after a wyrm2 reboot, and propagates the recognizers into the operational runbook so the failure modes are caught during an incident.New lesson —
cluster/docs/lessons_learned/2026_07_03_tofu_controller_runner_rpc_hang.md:InitRPC —waitForReady:true+ deadline-less context; controller-runtime never re-runs the key).Failedrunner pods never reaped), aggravating factor (all runners co-locate on wyrm2), and the transientArtifactFailed/etcd-contention compounding factor.Runbook propagation —
cluster/docs/troubleshooting.md:Policy denied/ClusterIP Timeout During Control-Plane Instability' (don't chase the datapath; a denied drop from an unrelated pod is correct; check etcd first).cluster/AGENTS.md: surfacetofu-controllerin the troubleshooting pointer.Docs-only.