Add transport URL secret rotation with consumer finalizer#716
Conversation
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: lmiccini The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
b6d036f to
b27f224
Compare
|
/test keystone-operator-build-deploy-kuttl |
b27f224 to
37e535d
Compare
|
Build failed (check pipeline). Post ❌ openstack-k8s-operators-content-provider FAILURE in 7m 36s |
37e535d to
b956f42
Compare
When infra-operator rotates a RabbitMQ transport URL (creating a new secret and user), consumer operators must hold a consumer finalizer on the old secret until all their pods have rolled out with the new credentials. Without this, infra-operator cleans up the old RabbitMQ user while pods are still connected with old credentials, causing message bus outages. Design: 1. Add consumer finalizer to the current transport URL secret early in reconcile. Set instance.Status.TransportURLSecret for first-time setup only (when empty or unchanged); during rotation the status is updated solely by FinalizeSecretRotation at end of reconcile. 2. Pass transportURL.Status.SecretName directly to sub-CR creation functions and config generation as a parameter — never read from instance.Status.TransportURLSecret for sub-CR specs. 3. Use statefulset.IsReady() / deployment.IsReady() from lib-common in all sub-CR controllers for accurate rollout status. 4. Use object.ManageRotationGracePeriod() to enforce a 60-second grace period before evaluating the rotation guard. This gives sub-CRs time to detect config changes, update their workloads, and roll pods — without relying on informer cache freshness. 5. Guard: CredentialRotationGuardReady(true, conditions) — evaluates AllSubConditionIsTrue after the grace period expires. Only when all sub-CR conditions are True does FinalizeSecretRotation remove the consumer finalizer from the old secret. The same pattern applies to notification transport URL secrets and application credential secrets where applicable. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
b956f42 to
78e0ae1
Compare
|
PR needs rebase. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
|
@lmiccini: The following tests failed, say
Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
1 participant
When infra-operator rotates a RabbitMQ transport URL (creating a new secret and user), consumer operators must hold a consumer finalizer on the old secret until all their pods have rolled out with the new credentials. Without this, infra-operator cleans up the old RabbitMQ user while pods are still connected with old credentials, causing message bus outages.
Design:
Add consumer finalizer to the current transport URL secret early in reconcile. Set instance.Status.TransportURLSecret for first-time setup only (when empty or unchanged); during rotation the status is updated solely by FinalizeSecretRotation at end of reconcile.
Pass transportURL.Status.SecretName directly to sub-CR creation functions and config generation as a parameter — never read from instance.Status.TransportURLSecret for sub-CR specs.
Use statefulset.IsReady() / deployment.IsReady() from lib-common in all sub-CR controllers for accurate rollout status.
Use object.ManageRotationGracePeriod() to enforce a 60-second grace period before evaluating the rotation guard. This gives sub-CRs time to detect config changes, update their workloads, and roll pods — without relying on informer cache freshness.
Guard: CredentialRotationGuardReady(true, conditions) — evaluates AllSubConditionIsTrue after the grace period expires. Only when all sub-CR conditions are True does FinalizeSecretRotation remove the consumer finalizer from the old secret.
The same pattern applies to notification transport URL secrets and application credential secrets where applicable.