Support for node-local NVMe across Kubernetes node upgrades

[brightsparc]

Summary

During an automatic GKE node-pool upgrade, an operator-managed Restate cluster using node-local NVMe for RESTATE_BASE_DIR became unrecoverable and lost its in-flight invocations.

Could you clarify whether node-local NVMe is a supported persistence option for RestateCluster, or whether the complete Restate base directory should always remain on the operator-managed PVC?

Configuration

• Restate Operator 3.0.1
• Restate Server 1.7.2
• Regional GKE cluster in us-east1
• 5 Restate nodes, replication factor 3
• Each node ran worker, log-server, and metadata-server
• Partition snapshots in object storage
• Operator-generated PDB with maxUnavailable: 1
• RESTATE_BASE_DIR pointed to an emptyDir backed by node-local NVMe
• The operator's durable PVCs still existed, but were not used as the base directory
• GKE node-pool management.autoUpgrade: true
• GKE upgrade strategy: BLUE_GREEN, batch_percentage: 1.0, no soak duration

Failure sequence

When GKE replaced the first node, its NVMe store was empty. The StatefulSet recreated the pod under the same stable name, and Restate exited with:

Node cannot start a log-server on N5, it has detected that it has lost its data.
storage-state is `data-loss`

The failed pod reduced PDB allowed disruptions to zero. GKE honored the PDB during its drain timeout, then force-evicted the rest of the blue-green batch. The cluster subsequently required manual recovery.

Our understanding is that a wiped Restate node needs a new node name, followed by catch-up and removal of the failed node. A StatefulSet normally recreates the ordinal with the same name.

Questions

1. Is the complete RESTATE_BASE_DIR supported on ephemeral node-local NVMe when using the Restate operator?
2. Does the operator support, or plan to support, replacing a wiped node with a fresh Restate node identity and waiting for it to catch up before the next disruption?
3. Can log-server and metadata-server identity remain on a durable PVC while only disposable partition/cache data uses NVMe?
4. If not, should we keep the entire base directory on a durable, provisioned-IOPS PVC?
5. For GKE, what node-pool upgrade configuration do you recommend? Would SURGE with maxSurge=1 and maxUnavailable=0 be preferred over our blue-green 100% batch, in addition to the operator PDB?

[tillrohrmann]

Hi @brightsparc,

1. The Restate operator currently does not support running with ephemeral disks. Restate itself brings all the tools to do it but operationally it is a tad bit more complex (nodes have to be properly removed from the cluster before they can join again instead of simply restarting them) and the operator hasn't implemented this control plane logic yet.

2. It's currently not part of the road map.

3. Yes, this should be possible from Restate's perspective.

4. You can split the partition store from the log-server and metadata server database as you've suggested.

5. @lukebond might be able to give you a better answer. From Restate's perspective the upgrade strategy should mainly affect availability. If you upgrade node pools such that you guarantee that no more than replication-property - 1 restate servers are down, then the cluster should stay available. This assumes that you don't lose disks as losing a disk is the same as a restate server being permanently down.