[libvirt_manager] Add retries to controller SSH key inject tasks#4057
[libvirt_manager] Add retries to controller SSH key inject tasks#4057skovili wants to merge 2 commits into
Conversation
Cloud-init may not finish writing authorized_keys before the inject tasks attempt SSH, causing "Permission denied (publickey)" on controller VMs. This is the same race the OCP SSH task already handles with retries. Related-Issue: #OSPRH-32065 Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
|
Build failed (check pipeline). Post ✔️ openstack-k8s-operators-content-provider SUCCESS in 2h 32m 27s |
Ansible's retries/until mechanism only retries FAILED tasks, not UNREACHABLE ones. When controller-0 SSH is not yet ready (cloud-init race), the host is marked unreachable and removed from the play without retrying. Add ignore_unreachable: true so the task result includes unreachable status instead of ejecting the host, and extend the until condition to retry on both failed and unreachable states.
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
1 participant
Summary
retries: 5, delay: 30to both "Inject private key" and "Inject public key" tasks inmanage_vms.ymlretries: 5, delay: 60)authorized_keysbefore the inject task attempts SSH, causingPermission denied (publickey)on controller VMsRoot cause
SSHD starts independently of cloud-init on RHEL 9. The SSH port opens ~16s after VM boot, but cloud-init may not finish writing SSH keys until 30-60s post-boot. Without retries, the inject task fails immediately on first attempt.
Observed in testproject build
5ac3f032on devin07 (reused hypervisor, routable IPs through physical bridge), but this is a latent race condition that could affect any environment where cloud-init is slower than usual.Test plan
Depends-On:pointing to this PRRelated-Issue: #OSPRH-32065
🤖 Generated with Claude Code