Summary
Backlog / discussion item to make the Higress gateway highly available on our on-prem bare-metal clusters by putting a proper LoadBalancer in front of it. No changes proposed yet — the goal is to explain the concept and agree on an approach.
The problem today
The Higress gateway runs with hostNetwork: true and is effectively tied to one physical node, so the DNS name our IT/VPN hands out points at that single node's IP. When that node goes down (as a15-31 did around July 4th), there is nothing to move traffic elsewhere — the gateway's Service endpoints go empty and everything through the gateway (including GitHub Actions job pickup) stops until someone manually re-pins the gateway and repoints the IP.
Pointing DNS at several node IPs (round-robin) doesn't fix this either: DNS is not health-aware, so it keeps handing out the dead node's IP until a human edits the record.
How a load balancer fixes it
The core idea is a virtual IP (VIP): one stable IP address that does not belong to any single node.
- IT/VPN points the domain name at that one VIP, permanently — it never has to change again.
- The load balancer controller keeps the VIP attached to a healthy node. If that node dies, it automatically moves the VIP to another healthy node (seconds), and traffic follows — no DNS edit, no manual re-pinning.
- Combined with running multiple gateway replicas across nodes, losing any one node takes out neither the entry IP nor the gateway.
In short: today "domain -> one node's IP (breaks if that node dies)"; with a load balancer it becomes "domain -> floating VIP -> whichever node is healthy." This also fixes why our higress-gateway LoadBalancer Service currently sits at EXTERNAL-IP: <pending> — there is no bare-metal LB controller installed to assign it a VIP.
The two options
Option A - MetalLB (CNI-agnostic) -- Also used by Silo.AI k8s stack
A bare-metal load balancer that assigns the VIP and, in L2 mode, answers for it on a healthy node (failing over automatically). Works with any CNI (flannel or Cilium) and Kubespray has a built-in addon for it. Because our team defaults to flannel and only sometimes uses Cilium, this "one approach for all clusters" portability is a strong point.
Option B - Cilium LB-IPAM (Cilium-only)
Cilium can provide the same VIP natively (via CiliumLoadBalancerIPPool + L2 announcements or BGP) with no extra component to install — but only on clusters where Cilium is the CNI. It can't cover our flannel clusters, so it can't be our single standard, but when we can use, it is likely to be less effort to set up and maintain, compared to MetalLB.
Notes
- Either option needs a reserved, unused IP on the node's subnet for the VIP (coordinate with the network team).
- To actually benefit, the gateway should also drop
hostNetwork, use the LoadBalancer VIP, and run 2-3 replicas spread across nodes (instead of being pinned to one).
Context: filed following the July 4th a15-31 gateway outage. Discussion only.
Summary
Backlog / discussion item to make the Higress gateway highly available on our on-prem bare-metal clusters by putting a proper LoadBalancer in front of it. No changes proposed yet — the goal is to explain the concept and agree on an approach.
The problem today
The Higress gateway runs with
hostNetwork: trueand is effectively tied to one physical node, so the DNS name our IT/VPN hands out points at that single node's IP. When that node goes down (asa15-31did around July 4th), there is nothing to move traffic elsewhere — the gateway's Service endpoints go empty and everything through the gateway (including GitHub Actions job pickup) stops until someone manually re-pins the gateway and repoints the IP.Pointing DNS at several node IPs (round-robin) doesn't fix this either: DNS is not health-aware, so it keeps handing out the dead node's IP until a human edits the record.
How a load balancer fixes it
The core idea is a virtual IP (VIP): one stable IP address that does not belong to any single node.
In short: today "domain -> one node's IP (breaks if that node dies)"; with a load balancer it becomes "domain -> floating VIP -> whichever node is healthy." This also fixes why our
higress-gatewayLoadBalancerService currently sits atEXTERNAL-IP: <pending>— there is no bare-metal LB controller installed to assign it a VIP.The two options
Option A - MetalLB (CNI-agnostic) -- Also used by Silo.AI k8s stack
A bare-metal load balancer that assigns the VIP and, in L2 mode, answers for it on a healthy node (failing over automatically). Works with any CNI (flannel or Cilium) and Kubespray has a built-in addon for it. Because our team defaults to flannel and only sometimes uses Cilium, this "one approach for all clusters" portability is a strong point.
Option B - Cilium LB-IPAM (Cilium-only)
Cilium can provide the same VIP natively (via
CiliumLoadBalancerIPPool+ L2 announcements or BGP) with no extra component to install — but only on clusters where Cilium is the CNI. It can't cover our flannel clusters, so it can't be our single standard, but when we can use, it is likely to be less effort to set up and maintain, compared to MetalLB.Notes
hostNetwork, use the LoadBalancer VIP, and run 2-3 replicas spread across nodes (instead of being pinned to one).Context: filed following the July 4th
a15-31gateway outage. Discussion only.