Skip to content

[BUG] etcd rollout has a probability of blocking #10873

Description

@JashBook

Describe the bug
A clear and concise description of what the bug is.
2026-09-09T10:01:54.639Z INFO build error: requeue after: 1s as: [pod etcd-nfauvf-etcd-3: action: memberJoin, executed on pod: etcd-nfauvf-etcd-3, error: exit code: 1: failed: action failed some replicas have not joined: [etcd-nfauvf-etcd-3]] {"controller": "component", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Component", "Component": {"name":"etcd-nfauvf-etcd","namespace":"default"}, "namespace": "default", "name": "etcd-nfauvf-etcd", "reconcileID": "683a8177-c771-461d-9c5b-97262b473463", "component": {"name":"etcd-nfauvf-etcd","namespace":"default"}}

To Reproduce
Steps to reproduce the behavior:

  1. create cluster
apiVersion: apps.kubeblocks.io/v1
kind: Cluster
metadata:
  name: etcd-nfauvf
  namespace: default
spec:
  terminationPolicy: DoNotTerminate
  componentSpecs:
    - name: etcd
      componentDef: etcd-3-1.1.0-alpha.0
      serviceVersion: 3.5.6
      tls: false
      replicas: 3
      resources:
        requests:
          cpu: 100m
          memory: 0.5Gi
        limits:
          cpu: 100m
          memory: 0.5Gi
      persistentVolumeClaimRetentionPolicy:
        whenDeleted: Delete
      volumeClaimTemplates:
        - name: data
          spec:
            storageClassName: 
            accessModes:
              - ReadWriteOnce
            resources:
              requests:
                storage: 20Gi
  services:
    - name: client
      serviceName: client
      spec:
        type: NodePort
        ports:
          - port: 2379
            targetPort: 2379
      componentSelector: etcd
      roleSelector: leader
      `kbcli cluster list-instances etcd-nfauvf --namespace default `
    
NAME                 NAMESPACE   CLUSTER       COMPONENT   STATUS    ROLE       ACCESSMODE   AZ               CPU(REQUEST/LIMIT)   MEMORY(REQUEST/LIMIT)   STORAGE     NODE                          CREATED-TIME                 
etcd-nfauvf-etcd-0   default     etcd-nfauvf   etcd        Running   leader                  cn-guangzhou-b   100m / 100m          512Mi / 512Mi           data:20Gi   192.168.0.165/192.168.0.165   Sep 09,2026 17:51 UTC+0800   
etcd-nfauvf-etcd-1   default     etcd-nfauvf   etcd        Running   follower                cn-guangzhou-b   100m / 100m          512Mi / 512Mi           data:20Gi   192.168.0.207/192.168.0.207   Sep 09,2026 17:51 UTC+0800   
etcd-nfauvf-etcd-2   default     etcd-nfauvf   etcd        Running   follower                cn-guangzhou-b   100m / 100m          512Mi / 512Mi           data:20Gi   192.168.0.125/192.168.0.125   Sep 09,2026 17:51 UTC+0800 
  1. rollout
apiVersion: apps.kubeblocks.io/v1alpha1
kind: Rollout
metadata:
  name: etcd-nfauvf-etcd-rollout-create-18618
  namespace: default
spec:
  clusterName: etcd-nfauvf
  components:
    - name: etcd
      compDef: etcd-3-1.1.0-alpha.0
      serviceVersion: "3.5.15"
      replicas: 1
      strategy:
        create:
          canary: true
          promotion:
            auto: true
            delaySeconds: 5
            scaleDownDelaySeconds: 30
      instanceMeta:
        canary:
          labels:
            traffic: canary
          annotations:
            rollout.apps.kubeblocks.io/version: "3.5.15"
kubectl get rollouts.apps.kubeblocks.io 
NAME                                        CLUSTER         REPLICAS   ROLLED-OUT   CANARY   STATE     AGE
etcd-nfauvf-etcd-rollout-create-18618       etcd-nfauvf     3          0            0        Rolling   11m
  1. See error
kubectl get cluster etcd-nfauvf
NAME          CLUSTER-DEFINITION   TERMINATION-POLICY   STATUS     AGE
etcd-nfauvf                        DoNotTerminate       Updating   9m40s
➜  ~ 
➜  ~ kubectl get pod -l app.kubernetes.io/instance=etcd-nfauvf    
NAME                 READY   STATUS    RESTARTS        AGE
etcd-nfauvf-etcd-0   2/2     Running   0               9m46s
etcd-nfauvf-etcd-1   2/2     Running   0               9m28s
etcd-nfauvf-etcd-2   2/2     Running   0               9m7s
etcd-nfauvf-etcd-3   2/2     Running   1 (8m11s ago)   8m30s
➜  ~ 
➜  ~ kbcli cluster list-instances etcd-nfauvf 
NAME                 NAMESPACE   CLUSTER       COMPONENT   STATUS    ROLE       ACCESSMODE   AZ               CPU(REQUEST/LIMIT)   MEMORY(REQUEST/LIMIT)   STORAGE     NODE                          CREATED-TIME                 
etcd-nfauvf-etcd-0   default     etcd-nfauvf   etcd        Running   leader                  cn-guangzhou-b   100m / 100m          512Mi / 512Mi           data:20Gi   192.168.0.165/192.168.0.165   Sep 09,2026 17:51 UTC+0800   
etcd-nfauvf-etcd-1   default     etcd-nfauvf   etcd        Running   follower                cn-guangzhou-b   100m / 100m          512Mi / 512Mi           data:20Gi   192.168.0.207/192.168.0.207   Sep 09,2026 17:51 UTC+0800   
etcd-nfauvf-etcd-2   default     etcd-nfauvf   etcd        Running   follower                cn-guangzhou-b   100m / 100m          512Mi / 512Mi           data:20Gi   192.168.0.125/192.168.0.125   Sep 09,2026 17:51 UTC+0800   
etcd-nfauvf-etcd-3   default     etcd-nfauvf   etcd        Running   follower                cn-guangzhou-b   100m / 100m          512Mi / 512Mi           data:20Gi   192.168.0.207/192.168.0.207   Sep 09,2026 17:52 UTC+0800   

logs etcd3 previous

kubectl logs etcd-nfauvf-etcd-3 --tail 10 --previous 
Defaulted container "etcd" out of: etcd, kbagent, inject-bash (init), init-kbagent (init), kbagent-worker (init)
{"level":"info","ts":"2026-09-09T09:52:55.722341Z","caller":"etcdmain/etcd.go:73","msg":"Running: ","args":["etcd","--config-file","/var/run/etcd/etcd.conf"]}
{"level":"warn","ts":"2026-09-09T09:52:55.722360Z","caller":"embed/config.go:687","msg":"Running http and grpc server on single port. This is not recommended for production."}
{"level":"info","ts":"2026-09-09T09:52:55.722369Z","caller":"embed/etcd.go:128","msg":"configuring peer listeners","listen-peer-urls":["http://0.0.0.0:2380"]}
{"level":"info","ts":"2026-09-09T09:52:55.722457Z","caller":"embed/etcd.go:136","msg":"configuring client listeners","listen-client-urls":["http://0.0.0.0:2379"]}
{"level":"info","ts":"2026-09-09T09:52:55.722499Z","caller":"embed/etcd.go:310","msg":"starting an etcd server","etcd-version":"3.5.15","git-sha":"9a5533382","go-version":"go1.21.12","go-os":"linux","go-arch":"amd64","max-cpu-set":8,"max-cpu-available":8,"member-initialized":false,"name":"etcd-nfauvf-etcd-3","data-dir":"/var/run/etcd/default.etcd","wal-dir":"","wal-dir-dedicated":"","member-dir":"/var/run/etcd/default.etcd/member","force-new-cluster":false,"heartbeat-interval":"100ms","election-timeout":"1s","initial-election-tick-advance":true,"snapshot-count":10000,"max-wals":5,"max-snapshots":5,"snapshot-catchup-entries":5000,"initial-advertise-peer-urls":["http://etcd-nfauvf-etcd-3.etcd-nfauvf-etcd-headless.default.svc.cluster.local:2380"],"listen-peer-urls":["http://0.0.0.0:2380"],"advertise-client-urls":["http://etcd-nfauvf-etcd-3.etcd-nfauvf-etcd-headless.default.svc.cluster.local:2379"],"listen-client-urls":["http://0.0.0.0:2379"],"listen-metrics-urls":[],"cors":["*"],"host-whitelist":["*"],"initial-cluster":"etcd-nfauvf-etcd-0=http://etcd-nfauvf-etcd-0.etcd-nfauvf-etcd-headless.default.svc.cluster.local:2380,etcd-nfauvf-etcd-1=http://etcd-nfauvf-etcd-1.etcd-nfauvf-etcd-headless.default.svc.cluster.local:2380,etcd-nfauvf-etcd-2=http://etcd-nfauvf-etcd-2.etcd-nfauvf-etcd-headless.default.svc.cluster.local:2380,etcd-nfauvf-etcd-3=http://etcd-nfauvf-etcd-3.etcd-nfauvf-etcd-headless.default.svc.cluster.local:2380","initial-cluster-state":"existing","initial-cluster-token":"etcd-nfauvf","quota-backend-bytes":2147483648,"max-request-bytes":1572864,"max-concurrent-streams":4294967295,"pre-vote":true,"initial-corrupt-check":false,"corrupt-check-time-interval":"0s","compact-check-time-enabled":false,"compact-check-time-interval":"1m0s","auto-compaction-mode":"periodic","auto-compaction-retention":"1h0m0s","auto-compaction-interval":"1h0m0s","discovery-url":"","discovery-proxy":"","downgrade-check-interval":"5s"}
{"level":"info","ts":"2026-09-09T09:52:55.724264Z","caller":"etcdserver/backend.go:81","msg":"opened backend db","path":"/var/run/etcd/default.etcd/member/snap/db","took":"1.274939ms"}
{"level":"info","ts":"2026-09-09T09:52:55.726946Z","caller":"embed/etcd.go:377","msg":"closing etcd server","name":"etcd-nfauvf-etcd-3","data-dir":"/var/run/etcd/default.etcd","advertise-peer-urls":["http://etcd-nfauvf-etcd-3.etcd-nfauvf-etcd-headless.default.svc.cluster.local:2380"],"advertise-client-urls":["http://etcd-nfauvf-etcd-3.etcd-nfauvf-etcd-headless.default.svc.cluster.local:2379"]}
{"level":"info","ts":"2026-09-09T09:52:55.726970Z","caller":"embed/etcd.go:379","msg":"closed etcd server","name":"etcd-nfauvf-etcd-3","data-dir":"/var/run/etcd/default.etcd","advertise-peer-urls":["http://etcd-nfauvf-etcd-3.etcd-nfauvf-etcd-headless.default.svc.cluster.local:2380"],"advertise-client-urls":["http://etcd-nfauvf-etcd-3.etcd-nfauvf-etcd-headless.default.svc.cluster.local:2379"]}
{"level":"warn","ts":"2026-09-09T09:52:55.726978Z","caller":"etcdmain/etcd.go:146","msg":"failed to start etcd","error":"error validating peerURLs {ClusterID:f4739fba6dd1323a Members:[&{ID:57fa33b6819ae24c RaftAttributes:{PeerURLs:[http://etcd-nfauvf-etcd-2.etcd-nfauvf-etcd-headless.default.svc.cluster.local:2380] IsLearner:false} Attributes:{Name:etcd-nfauvf-etcd-2 ClientURLs:[http://etcd-nfauvf-etcd-2.etcd-nfauvf-etcd-headless.default.svc.cluster.local:2379]}} &{ID:760f52a8a71061f8 RaftAttributes:{PeerURLs:[http://etcd-nfauvf-etcd-1.etcd-nfauvf-etcd-headless.default.svc.cluster.local:2380] IsLearner:false} Attributes:{Name:etcd-nfauvf-etcd-1 ClientURLs:[http://etcd-nfauvf-etcd-1.etcd-nfauvf-etcd-headless.default.svc.cluster.local:2379]}} &{ID:7c953dbdc36e588e RaftAttributes:{PeerURLs:[http://etcd-nfauvf-etcd-0.etcd-nfauvf-etcd-headless.default.svc.cluster.local:2380] IsLearner:false} Attributes:{Name:etcd-nfauvf-etcd-0 ClientURLs:[http://etcd-nfauvf-etcd-0.etcd-nfauvf-etcd-headless.default.svc.cluster.local:2379]}}] RemovedMemberIDs:[]}: member count is unequal"}
{"level":"fatal","ts":"2026-09-09T09:52:55.726991Z","caller":"etcdmain/etcd.go:204","msg":"discovery failed","error":"error validating peerURLs {ClusterID:f4739fba6dd1323a Members:[&{ID:57fa33b6819ae24c RaftAttributes:{PeerURLs:[http://etcd-nfauvf-etcd-2.etcd-nfauvf-etcd-headless.default.svc.cluster.local:2380] IsLearner:false} Attributes:{Name:etcd-nfauvf-etcd-2 ClientURLs:[http://etcd-nfauvf-etcd-2.etcd-nfauvf-etcd-headless.default.svc.cluster.local:2379]}} &{ID:760f52a8a71061f8 RaftAttributes:{PeerURLs:[http://etcd-nfauvf-etcd-1.etcd-nfauvf-etcd-headless.default.svc.cluster.local:2380] IsLearner:false} Attributes:{Name:etcd-nfauvf-etcd-1 ClientURLs:[http://etcd-nfauvf-etcd-1.etcd-nfauvf-etcd-headless.default.svc.cluster.local:2379]}} &{ID:7c953dbdc36e588e RaftAttributes:{PeerURLs:[http://etcd-nfauvf-etcd-0.etcd-nfauvf-etcd-headless.default.svc.cluster.local:2380] IsLearner:false} Attributes:{Name:etcd-nfauvf-etcd-0 ClientURLs:[http://etcd-nfauvf-etcd-0.etcd-nfauvf-etcd-headless.default.svc.cluster.local:2379]}}] RemovedMemberIDs:[]}: member count is unequal","stacktrace":"go.etcd.io/etcd/server/v3/etcdmain.startEtcdOrProxyV2\n\tgo.etcd.io/etcd/server/v3/etcdmain/etcd.go:204\ngo.etcd.io/etcd/server/v3/etcdmain.Main\n\tgo.etcd.io/etcd/server/v3/etcdmain/main.go:40\nmain.main\n\tgo.etcd.io/etcd/server/v3/main.go:31\nruntime.main\n\truntime/proc.go:267"}

logs etcd3

kubectl logs etcd-nfauvf-etcd-3 --tail 10
Defaulted container "etcd" out of: etcd, kbagent, inject-bash (init), init-kbagent (init), kbagent-worker (init)
{"level":"error","ts":"2026-09-09T10:05:10.660050Z","caller":"etcdserver/server.go:2356","msg":"Validation on configuration change failed","shouldApplyV3":true,"error":"membership: peerURL exists","stacktrace":"go.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).applyConfChange\n\tgo.etcd.io/etcd/server/v3/etcdserver/server.go:2356\ngo.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).apply\n\tgo.etcd.io/etcd/server/v3/etcdserver/server.go:2225\ngo.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).applyEntries\n\tgo.etcd.io/etcd/server/v3/etcdserver/server.go:1440\ngo.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).applyAll\n\tgo.etcd.io/etcd/server/v3/etcdserver/server.go:1255\ngo.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).run.func8\n\tgo.etcd.io/etcd/server/v3/etcdserver/server.go:1143\ngo.etcd.io/etcd/pkg/v3/schedule.(*fifo).run\n\tgo.etcd.io/etcd/pkg/v3@v3.5.15/schedule/schedule.go:157"}
{"level":"info","ts":"2026-09-09T10:05:10.660138Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"7696847d385604d switched to configuration voters=(534072688520814669 6339436284428870220 8507109105437008376 8977149317469853838)"}
{"level":"error","ts":"2026-09-09T10:05:11.718920Z","caller":"etcdserver/server.go:2356","msg":"Validation on configuration change failed","shouldApplyV3":true,"error":"membership: peerURL exists","stacktrace":"go.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).applyConfChange\n\tgo.etcd.io/etcd/server/v3/etcdserver/server.go:2356\ngo.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).apply\n\tgo.etcd.io/etcd/server/v3/etcdserver/server.go:2225\ngo.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).applyEntries\n\tgo.etcd.io/etcd/server/v3/etcdserver/server.go:1440\ngo.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).applyAll\n\tgo.etcd.io/etcd/server/v3/etcdserver/server.go:1255\ngo.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).run.func8\n\tgo.etcd.io/etcd/server/v3/etcdserver/server.go:1143\ngo.etcd.io/etcd/pkg/v3/schedule.(*fifo).run\n\tgo.etcd.io/etcd/pkg/v3@v3.5.15/schedule/schedule.go:157"}
{"level":"info","ts":"2026-09-09T10:05:11.718984Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"7696847d385604d switched to configuration voters=(534072688520814669 6339436284428870220 8507109105437008376 8977149317469853838)"}
{"level":"error","ts":"2026-09-09T10:05:12.774310Z","caller":"etcdserver/server.go:2356","msg":"Validation on configuration change failed","shouldApplyV3":true,"error":"membership: peerURL exists","stacktrace":"go.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).applyConfChange\n\tgo.etcd.io/etcd/server/v3/etcdserver/server.go:2356\ngo.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).apply\n\tgo.etcd.io/etcd/server/v3/etcdserver/server.go:2225\ngo.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).applyEntries\n\tgo.etcd.io/etcd/server/v3/etcdserver/server.go:1440\ngo.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).applyAll\n\tgo.etcd.io/etcd/server/v3/etcdserver/server.go:1255\ngo.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).run.func8\n\tgo.etcd.io/etcd/server/v3/etcdserver/server.go:1143\ngo.etcd.io/etcd/pkg/v3/schedule.(*fifo).run\n\tgo.etcd.io/etcd/pkg/v3@v3.5.15/schedule/schedule.go:157"}
{"level":"info","ts":"2026-09-09T10:05:12.774378Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"7696847d385604d switched to configuration voters=(534072688520814669 6339436284428870220 8507109105437008376 8977149317469853838)"}
{"level":"error","ts":"2026-09-09T10:05:13.831548Z","caller":"etcdserver/server.go:2356","msg":"Validation on configuration change failed","shouldApplyV3":true,"error":"membership: peerURL exists","stacktrace":"go.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).applyConfChange\n\tgo.etcd.io/etcd/server/v3/etcdserver/server.go:2356\ngo.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).apply\n\tgo.etcd.io/etcd/server/v3/etcdserver/server.go:2225\ngo.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).applyEntries\n\tgo.etcd.io/etcd/server/v3/etcdserver/server.go:1440\ngo.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).applyAll\n\tgo.etcd.io/etcd/server/v3/etcdserver/server.go:1255\ngo.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).run.func8\n\tgo.etcd.io/etcd/server/v3/etcdserver/server.go:1143\ngo.etcd.io/etcd/pkg/v3/schedule.(*fifo).run\n\tgo.etcd.io/etcd/pkg/v3@v3.5.15/schedule/schedule.go:157"}
{"level":"info","ts":"2026-09-09T10:05:13.831638Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"7696847d385604d switched to configuration voters=(534072688520814669 6339436284428870220 8507109105437008376 8977149317469853838)"}
{"level":"error","ts":"2026-09-09T10:05:14.892418Z","caller":"etcdserver/server.go:2356","msg":"Validation on configuration change failed","shouldApplyV3":true,"error":"membership: peerURL exists","stacktrace":"go.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).applyConfChange\n\tgo.etcd.io/etcd/server/v3/etcdserver/server.go:2356\ngo.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).apply\n\tgo.etcd.io/etcd/server/v3/etcdserver/server.go:2225\ngo.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).applyEntries\n\tgo.etcd.io/etcd/server/v3/etcdserver/server.go:1440\ngo.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).applyAll\n\tgo.etcd.io/etcd/server/v3/etcdserver/server.go:1255\ngo.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).run.func8\n\tgo.etcd.io/etcd/server/v3/etcdserver/server.go:1143\ngo.etcd.io/etcd/pkg/v3/schedule.(*fifo).run\n\tgo.etcd.io/etcd/pkg/v3@v3.5.15/schedule/schedule.go:157"}
{"level":"info","ts":"2026-09-09T10:05:14.892490Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"7696847d385604d switched to configuration voters=(534072688520814669 6339436284428870220 8507109105437008376 8977149317469853838)"}
kubectl get cluster etcd-nfauvf -oyaml
apiVersion: apps.kubeblocks.io/v1
kind: Cluster
metadata:
  annotations:
    kubeblocks.io/crd-api-version: apps.kubeblocks.io/v1
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"apps.kubeblocks.io/v1","kind":"Cluster","metadata":{"annotations":{},"name":"etcd-nfauvf","namespace":"default"},"spec":{"componentSpecs":[{"componentDef":"etcd-3-1.1.0-alpha.0","name":"etcd","persistentVolumeClaimRetentionPolicy":{"whenDeleted":"Delete"},"replicas":3,"resources":{"limits":{"cpu":"100m","memory":"0.5Gi"},"requests":{"cpu":"100m","memory":"0.5Gi"}},"serviceVersion":"3.5.6","tls":false,"volumeClaimTemplates":[{"name":"data","spec":{"accessModes":["ReadWriteOnce"],"resources":{"requests":{"storage":"20Gi"}},"storageClassName":null}}]}],"services":[{"componentSelector":"etcd","name":"client","roleSelector":"leader","serviceName":"client","spec":{"ports":[{"port":2379,"targetPort":2379}],"type":"NodePort"}}],"terminationPolicy":"DoNotTerminate"}}
  creationTimestamp: "2026-09-09T09:51:19Z"
  finalizers:
  - cluster.kubeblocks.io/finalizer
  generation: 4
  labels:
    apps.kubeblocks.io/rollout-name: etcd-nfauvf-etcd-rollout-create-18618
  name: etcd-nfauvf
  namespace: default
  resourceVersion: "269892081"
  uid: 98bb30c5-f68e-41d8-ab7b-427fafa35c52
spec:
  componentSpecs:
  - componentDef: etcd-3-1.1.0-alpha.0
    flatInstanceOrdinal: true
    instances:
    - annotations:
        rollout.apps.kubeblocks.io/version: 3.5.15
      canary: true
      compDef: etcd-3-1.1.0-alpha.0
      labels:
        traffic: canary
      name: 84c67a4b
      ordinals: {}
      replicas: 1
      serviceVersion: 3.5.15
    name: etcd
    ordinals: {}
    persistentVolumeClaimRetentionPolicy:
      whenDeleted: Delete
    podUpdatePolicy: PreferInPlace
    replicas: 4
    resources:
      limits:
        cpu: 100m
        memory: 512Mi
      requests:
        cpu: 100m
        memory: 512Mi
    serviceVersion: 3.5.6
    volumeClaimTemplates:
    - name: data
      spec:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 20Gi
  services:
  - componentSelector: etcd
    name: client
    roleSelector: leader
    serviceName: client
    spec:
      ports:
      - nodePort: 31035
        port: 2379
        protocol: TCP
        targetPort: 2379
      type: NodePort
  terminationPolicy: DoNotTerminate
status:
  components:
    etcd:
      observedGeneration: 4
      phase: Updating
  conditions:
  - lastTransitionTime: "2026-09-09T09:51:20Z"
    message: 'The operator has started the provisioning of Cluster: etcd-nfauvf'
    observedGeneration: 4
    reason: PreCheckSucceed
    status: "True"
    type: ProvisioningStarted
  - lastTransitionTime: "2026-09-09T09:52:36Z"
    message: 'component etcd-nfauvf-etcd is not available; '
    reason: Unavailable
    status: "False"
    type: Available
  - lastTransitionTime: "2026-09-09T09:51:20Z"
    message: Successfully applied for resources
    observedGeneration: 4
    reason: ApplyResourcesSucceed
    status: "True"
    type: ApplyResources
  - lastTransitionTime: "2026-09-09T09:52:23Z"
    message: cluster etcd-nfauvf is ready
    reason: ClusterReady
    status: "True"
    type: Ready
  observedGeneration: 4
  phase: Updating

logs kb pod

kubectl logs -n kb-system kubeblocks-76c4f9975b-mnt5f --tail 50 | grep etcd-nfauvf-etcd
Defaulted container "manager" out of: manager, tools (init)
2026-09-09T10:01:47.145Z	INFO	reconcile object *v1.Component with action STATUS OK	{"controller": "component", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Component", "Component": {"name":"etcd-nfauvf-etcd","namespace":"default"}, "namespace": "default", "name": "etcd-nfauvf-etcd", "reconcileID": "42736ba8-411f-4264-a550-ff0dbb6f2b07", "component": {"name":"etcd-nfauvf-etcd","namespace":"default"}}
2026-09-09T10:01:47.375Z	INFO	role probe event processed	{"controller": "event", "controllerGroup": "", "controllerKind": "Event", "Event": {"name":"etcd-nfauvf-etcd-1.ca91222a","namespace":"default"}, "namespace": "default", "name": "etcd-nfauvf-etcd-1.ca91222a", "reconcileID": "566891a8-08a2-4e37-9f39-76519c5435d9", "event": {"name":"etcd-nfauvf-etcd-1.ca91222a","namespace":"default"}, "eventName": "default/etcd-nfauvf-etcd-1.ca91222a", "eventUID": "35e120e2-e5fa-4e67-93a5-e38bd821565b", "eventPodUID": "005adad2-021a-4a7b-91b3-8a3b5b02fe7e", "podName": "default/etcd-nfauvf-etcd-1", "podUID": "005adad2-021a-4a7b-91b3-8a3b5b02fe7e", "role": "follower", "eventVersion": "1788948107350475", "branch": "instanceset", "result": "handled", "reason": "updated", "workload": "etcd-nfauvf-etcd", "previousRole": "follower", "roleDefined": true, "handled": true, "exclusiveClean": false}
2026-09-09T10:01:48.203Z	INFO	build error: requeue after: 1s as: [pod etcd-nfauvf-etcd-3: action: memberJoin, executed on pod: etcd-nfauvf-etcd-3, error: exit code: 1: failed: action failed some replicas have not joined: [etcd-nfauvf-etcd-3]]	{"controller": "component", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Component", "Component": {"name":"etcd-nfauvf-etcd","namespace":"default"}, "namespace": "default", "name": "etcd-nfauvf-etcd", "reconcileID": "f9d96185-24fa-4e1f-9223-834fbe38a41c", "component": {"name":"etcd-nfauvf-etcd","namespace":"default"}}
2026-09-09T10:01:48.209Z	INFO	reconcile object *v1.Component with action STATUS OK	{"controller": "component", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Component", "Component": {"name":"etcd-nfauvf-etcd","namespace":"default"}, "namespace": "default", "name": "etcd-nfauvf-etcd", "reconcileID": "f9d96185-24fa-4e1f-9223-834fbe38a41c", "component": {"name":"etcd-nfauvf-etcd","namespace":"default"}}
2026-09-09T10:01:49.268Z	INFO	build error: requeue after: 1s as: [pod etcd-nfauvf-etcd-3: action: memberJoin, executed on pod: etcd-nfauvf-etcd-3, error: exit code: 1: failed: action failed some replicas have not joined: [etcd-nfauvf-etcd-3]]	{"controller": "component", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Component", "Component": {"name":"etcd-nfauvf-etcd","namespace":"default"}, "namespace": "default", "name": "etcd-nfauvf-etcd", "reconcileID": "b7e1fa9a-725d-4473-8be4-b0dc8f36f33e", "component": {"name":"etcd-nfauvf-etcd","namespace":"default"}}
2026-09-09T10:01:49.275Z	INFO	reconcile object *v1.Component with action STATUS OK	{"controller": "component", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Component", "Component": {"name":"etcd-nfauvf-etcd","namespace":"default"}, "namespace": "default", "name": "etcd-nfauvf-etcd", "reconcileID": "b7e1fa9a-725d-4473-8be4-b0dc8f36f33e", "component": {"name":"etcd-nfauvf-etcd","namespace":"default"}}
2026-09-09T10:01:50.330Z	INFO	build error: requeue after: 1s as: [pod etcd-nfauvf-etcd-3: action: memberJoin, executed on pod: etcd-nfauvf-etcd-3, error: exit code: 1: failed: action failed some replicas have not joined: [etcd-nfauvf-etcd-3]]	{"controller": "component", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Component", "Component": {"name":"etcd-nfauvf-etcd","namespace":"default"}, "namespace": "default", "name": "etcd-nfauvf-etcd", "reconcileID": "354202d7-cd31-4ae7-81f0-94faeac1a2e7", "component": {"name":"etcd-nfauvf-etcd","namespace":"default"}}
2026-09-09T10:01:50.336Z	INFO	reconcile object *v1.Component with action STATUS OK	{"controller": "component", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Component", "Component": {"name":"etcd-nfauvf-etcd","namespace":"default"}, "namespace": "default", "name": "etcd-nfauvf-etcd", "reconcileID": "354202d7-cd31-4ae7-81f0-94faeac1a2e7", "component": {"name":"etcd-nfauvf-etcd","namespace":"default"}}
2026-09-09T10:01:50.553Z	INFO	role probe event processed	{"controller": "event", "controllerGroup": "", "controllerKind": "Event", "Event": {"name":"etcd-nfauvf-etcd-3.13470795","namespace":"default"}, "namespace": "default", "name": "etcd-nfauvf-etcd-3.13470795", "reconcileID": "cdef4365-39cf-4452-865a-ee91ef099b8f", "event": {"name":"etcd-nfauvf-etcd-3.13470795","namespace":"default"}, "eventName": "default/etcd-nfauvf-etcd-3.13470795", "eventUID": "7c49dc73-7ef0-433d-878c-aa2a67e236d8", "eventPodUID": "9afdee2f-6374-4b54-982d-b16c04afa3d4", "podName": "default/etcd-nfauvf-etcd-3", "podUID": "9afdee2f-6374-4b54-982d-b16c04afa3d4", "role": "follower", "eventVersion": "1788948110528207", "branch": "instanceset", "result": "handled", "reason": "updated", "workload": "etcd-nfauvf-etcd", "previousRole": "follower", "roleDefined": true, "handled": true, "exclusiveClean": false}
2026-09-09T10:01:51.393Z	INFO	build error: requeue after: 1s as: [pod etcd-nfauvf-etcd-3: action: memberJoin, executed on pod: etcd-nfauvf-etcd-3, error: exit code: 1: failed: action failed some replicas have not joined: [etcd-nfauvf-etcd-3]]	{"controller": "component", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Component", "Component": {"name":"etcd-nfauvf-etcd","namespace":"default"}, "namespace": "default", "name": "etcd-nfauvf-etcd", "reconcileID": "fc3383e9-b107-4b4f-9e35-321849db5f38", "component": {"name":"etcd-nfauvf-etcd","namespace":"default"}}
2026-09-09T10:01:51.399Z	INFO	reconcile object *v1.Component with action STATUS OK	{"controller": "component", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Component", "Component": {"name":"etcd-nfauvf-etcd","namespace":"default"}, "namespace": "default", "name": "etcd-nfauvf-etcd", "reconcileID": "fc3383e9-b107-4b4f-9e35-321849db5f38", "component": {"name":"etcd-nfauvf-etcd","namespace":"default"}}
2026-09-09T10:01:52.453Z	INFO	build error: requeue after: 1s as: [pod etcd-nfauvf-etcd-3: action: memberJoin, executed on pod: etcd-nfauvf-etcd-3, error: exit code: 1: failed: action failed some replicas have not joined: [etcd-nfauvf-etcd-3]]	{"controller": "component", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Component", "Component": {"name":"etcd-nfauvf-etcd","namespace":"default"}, "namespace": "default", "name": "etcd-nfauvf-etcd", "reconcileID": "a5e80380-7ffc-45d7-ba3a-cb51433e2b8d", "component": {"name":"etcd-nfauvf-etcd","namespace":"default"}}
2026-09-09T10:01:52.459Z	INFO	reconcile object *v1.Component with action STATUS OK	{"controller": "component", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Component", "Component": {"name":"etcd-nfauvf-etcd","namespace":"default"}, "namespace": "default", "name": "etcd-nfauvf-etcd", "reconcileID": "a5e80380-7ffc-45d7-ba3a-cb51433e2b8d", "component": {"name":"etcd-nfauvf-etcd","namespace":"default"}}
2026-09-09T10:01:52.988Z	INFO	role probe event processed	{"controller": "event", "controllerGroup": "", "controllerKind": "Event", "Event": {"name":"etcd-nfauvf-etcd-2.bd7355e2","namespace":"default"}, "namespace": "default", "name": "etcd-nfauvf-etcd-2.bd7355e2", "reconcileID": "0df32176-86bd-4681-bbe8-3368591435ab", "event": {"name":"etcd-nfauvf-etcd-2.bd7355e2","namespace":"default"}, "eventName": "default/etcd-nfauvf-etcd-2.bd7355e2", "eventUID": "bb73854d-cf67-4dfb-82f6-842d1b5ff18d", "eventPodUID": "0c5d5356-0b8f-4b9e-8b9e-c41d88ef486d", "podName": "default/etcd-nfauvf-etcd-2", "podUID": "0c5d5356-0b8f-4b9e-8b9e-c41d88ef486d", "role": "follower", "eventVersion": "1788948112966495", "branch": "instanceset", "result": "handled", "reason": "updated", "workload": "etcd-nfauvf-etcd", "previousRole": "follower", "roleDefined": true, "handled": true, "exclusiveClean": false}
2026-09-09T10:01:53.579Z	INFO	build error: requeue after: 1s as: [pod etcd-nfauvf-etcd-3: action: memberJoin, executed on pod: etcd-nfauvf-etcd-3, error: exit code: 1: failed: action failed some replicas have not joined: [etcd-nfauvf-etcd-3]]	{"controller": "component", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Component", "Component": {"name":"etcd-nfauvf-etcd","namespace":"default"}, "namespace": "default", "name": "etcd-nfauvf-etcd", "reconcileID": "6c25d112-e65d-4038-a075-280eb6f554b8", "component": {"name":"etcd-nfauvf-etcd","namespace":"default"}}
2026-09-09T10:01:53.585Z	INFO	reconcile object *v1.Component with action STATUS OK	{"controller": "component", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Component", "Component": {"name":"etcd-nfauvf-etcd","namespace":"default"}, "namespace": "default", "name": "etcd-nfauvf-etcd", "reconcileID": "6c25d112-e65d-4038-a075-280eb6f554b8", "component": {"name":"etcd-nfauvf-etcd","namespace":"default"}}
2026-09-09T10:01:54.639Z	INFO	build error: requeue after: 1s as: [pod etcd-nfauvf-etcd-3: action: memberJoin, executed on pod: etcd-nfauvf-etcd-3, error: exit code: 1: failed: action failed some replicas have not joined: [etcd-nfauvf-etcd-3]]	{"controller": "component", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Component", "Component": {"name":"etcd-nfauvf-etcd","namespace":"default"}, "namespace": "default", "name": "etcd-nfauvf-etcd", "reconcileID": "683a8177-c771-461d-9c5b-97262b473463", "component": {"name":"etcd-nfauvf-etcd","namespace":"default"}}
2026-09-09T10:01:54.646Z	INFO	reconcile object *v1.Component with action STATUS OK	{"controller": "component", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Component", "Component": {"name":"etcd-nfauvf-etcd","namespace":"default"}, "namespace": "default", "name": "etcd-nfauvf-etcd", "reconcileID": "683a8177-c771-461d-9c5b-97262b473463", "component": {"name":"etcd-nfauvf-etcd","namespace":"default"}}

Expected behavior
A clear and concise description of what you expected to happen.

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

kbcli version 
Kubernetes: v1.30.4-vke.4
KubeBlocks: 1.1.0-beta.11
kbcli: 1.0.2
WARNING: version difference between kbcli (1.0.2) and kubeblocks (1.1.0-beta.11) 
➜  ~ 
➜  ~ helm get notes -n kb-system kb-addon-etcd   
NOTES:
Release Information:
  Commit ID: "9456dc59e6f4f591b5e94642b6de47fa2492d863"
  Commit Time: "2026-09-08 16:00:39 +0800"
  Release Branch: "v1.1.0-beta.11"
  Release Time:  "2026-09-09 13:58:38 +0800"
  Enterprise: "false"

Additional context
Add any other context about the problem here.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

kind/bugSomething isn't working

Type

No type

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions