GR bootstrap startup probe races full-cluster-crash recovery: exit(1) on transient states kills clone donor, recovery loops on large datasets

[evilhamsterman]

Report

During full-cluster-crash recovery, the GR bootstrap startup probe (/opt/percona/bootstrap, failureThreshold: 1) races the recovery process itself. bootstrap deliberately os.Exit(1)s on transient recovery states (e.g. "Can't connect to any of the peers"), which makes the kubelet kill the mysql container — including a clone donor mid-addInstance. On clusters with large datasets (clone takes minutes) the recovery repeatedly kills its own members and the cluster loops through full-cluster-crash recoveries indefinitely.

The full-cluster-crash marker bypass added for the readiness/liveness probes (the #1392 / K8SPS fix, cmd/healthcheck/main.go) does not cover the startup probe, so bootstrap's exit-1 paths still terminate containers during recovery.

More about the problem

Observed on a 3-node GR cluster (~70 GB dataset, communication_stack=MYSQL) after a full cluster crash:

1. Operator crash recovery works as designed at first:

   Crash recovery  Attempting to reboot cluster from complete outage  pod=mysql-2
   Crash recovery  Cluster was successfully rebooted
   Crash recovery  Primary pod is mysql-0
   Crash recovery  Deleting secondary pod  pod=mysql-1
   Crash recovery  Deleting secondary pod  pod=mysql-2
   

2. Deleted secondaries return and start CLONE from the primary (addInstance ... waitRecovery: 2) — for a ~70 GB dataset this takes minutes. During this window a clone recipient also restarts its own mysqld (clone plugin behavior), and GR is still forming.
3. Meanwhile, whichever member's startup probe fires during a moment when no peer answers dba.getCluster() takes this path in cmd/bootstrap/gr/group_replication.go (v1.2.0):

   log.Printf("Can't connect to any of the peers, we need to reboot")
   ...
   os.Exit(1)   // force restart container
   

   With startupProbe.failureThreshold: 1, one non-zero exit = immediate container kill. mysqld logs show:

   [System] [MY-013172] [Server] Received SHUTDOWN from user <via user signal>. Shutting down mysqld
   

   ~27s after boot (15s initialDelaySeconds + one probe run).
4. When the killed member is the primary/clone donor, the in-flight clone aborts, the joiner fails, the group is dead again → operator declares FullClusterCrashDetected → reboot → GOTO 1. We observed this loop for ~90 minutes with zero client load until we intervened (paused the CR).

v1.2.0 cmd/bootstrap/gr/group_replication.go has five such os.Exit(1) paths (lines 503, 521, 546, 592, 600). Note the probe's timeoutSeconds: 43200 shows long-running bootstrap is expected and tolerated — it's the early deliberate exits interacting with failureThreshold: 1 that cause the kills.

Steps to reproduce

1. 3-node GR cluster with a dataset large enough that CLONE takes several minutes (tens of GB).
2. Induce a full cluster crash (e.g. make all members leave the group / kill mysqld on all pods).
3. Let the operator run rebootClusterFromCompleteOutage and delete the secondaries.
4. While the secondaries clone, the members' startup probes keep running bootstrap; observe container SIGTERMs shortly after boots ("Received SHUTDOWN from user "), aborted clones, and repeated Full cluster crash detected cycles.

Small datasets rarely hit this (the vulnerable window is seconds); large datasets make the window minutes wide and the loop quasi-deterministic.

Versions

1. Kubernetes: k3s v1.35.4 (bare metal)
2. Operator: 1.2.0
3. Database: Percona Server 8.0.46-37, group replication, communication_stack=MYSQL

Anything else?

Related: #1392 (state-monitor race — fixed in 1.2.0; this is a different race in the same recovery area; the 1.2.0 crash-marker bypass covers readiness/liveness probes but not the startup probe).

Possible directions:

• Make bootstrap marker-aware: while /var/lib/mysql/full-cluster-crash exists (or operator-driven recovery is in progress), return success / keep waiting instead of os.Exit(1) on transient "no peers" states.
• Or retry/tolerate transient states inside bootstrap (it already has a 12h probe timeout to work with) instead of forcing a container restart on the first failure.
• Or failureThreshold > 1 on the startup probe so a single transient failure isn't fatal.
• Never exit-1 on the member currently acting as a clone donor.

[mayankshah1607]

Hi @evilhamsterman , thanks for reporting. I want to try and reproduce this, do you have the YAML for the PerconaServerMySQL you used?

What I don't understand is why the primary/clone donor starts crashing (like you described in step 4), because the startup probe has already finished there. I'm not sure if what's described is the cause or the symptom

[evilhamsterman]

Nothing particularly unusual. The critical point was that it was a larger DB. This is the second issue I've encountered that appears to result from a race condition due to timing assumptions caused by the differences between a small DB and a larger DB. This occurred with a DB that was about 70GB. We'd just migrated it in and it was running fine till a network blip caused two of the pods to drop out (another point 5 sec GR timeout seems pretty aggressive). After that the cluster could not recover automatically it got stuck in the loop described.

apiVersion: ps.percona.com/v1
kind: PerconaServerMySQL
metadata:
  name: db
  namespace: db
spec:
  crVersion: "${psOperatorVersion}"
  updateStrategy: SmartUpdate
  upgradeOptions:
    apply: disabled
  enableVolumeExpansion: true
  pause: false
  mysql:
    autoRecovery: true
    clusterType: group-replication
    size: 3
    image: qcr.corp.qumulo.com/docker/percona/percona-server:${perconaServerVersion}
    volumeSpec:
      persistentVolumeClaim:
        resources:
          requests:
            storage: 150Gi
    resources:
      limits:
        memory: 8Gi
        cpu: 2
    configuration: |
      [mysqld]
      max_allowed_packet = 128M
      sql_mode = "STRICT_ALL_TABLES,NO_ENGINE_SUBSTITUTION"
      sql_generate_invisible_primary_key = ON
  proxy:
    haproxy:
      enabled: true
      size: 2
      image: qcr.corp.qumulo.com/docker/percona/haproxy:${perconaHaproxyVersion}
      expose:
        type: LoadBalancer
        externalTrafficPolicy: Local
        annotations:
          projectcalico.org/loadBalancerIPs: "[\"10.102.4.106\"]"
    router:
      enabled: false
      image: na
      size: 0
  backup:
    enabled: false
    image: qcr.corp.qumulo.com/docker/percona/percona-xtrabackup:${perconaXtrabackupVersion}
    storages:
      itdr:
        type: s3
        s3:
          bucket: sql-backups
          credentialsSecret: sql-backup
          endpointUrl: https://itdr.corp.qumulo.com:9000
          prefix: submit-prod-db
        containerOptions:
          args:
            xtrabackup: [ "--parallel=4", "--compress" ]
            xbstream: [ "--parallel=4" ]
    schedule:
      - name: "db-hourly"
        schedule: "50 * * * *"
        keep: 48
        storageName: s3-bucket
  orchestrator:
    enabled: false
    image: na
    size: 0

[mayankshah1607]

Thanks @evilhamsterman , it would also be great to have logs from the operator (as you were observing this loop)