CdsLoadBalancer2 re-parsing LB config can fail if LoadBalancerRegistry is mutated

[lepistone]

What version of gRPC-Java are you using?

1.76.3 (regression from 1.70.0).

What is your environment?

Java 21, Linux (GKE), proxyless gRPC with xDS (SotW ADS) from a custom control plane.

One application has startup code that calls LoadBalancerRegistry.deregister() / register() to replace a ServiceLoader-discovered provider with a differently-configured instance.
We acknowledge that our application is likely at fault for creating gRPC channels during framework initialization, before all startup code has finished running.

What did you expect to see?

We'd expect CdsLoadBalancer2 to be resilient to registry mutations.

What did you see instead?

INTERNAL: CdsLb for xdstp://...: Unable to parse the LB config:
  Status{code=INTERNAL, description=Failed to parse child policy in wrr_locality LB policy:
    {childPolicy=[{els={}}]}}
Cause: None of [els] specified by Service Config are available.

The channel enters TRANSIENT_FAILURE and does not recover.

Steps to reproduce the bug

Setup: The xDS control plane sends a CDS load_balancing_policy with wrr_locality whose endpoint_picking_policy contains three policies in fallback order:

1. A custom LB policy (els) registered as a TypedStruct
2. LeastRequest
3. RoundRobin

The custom policy els has a LoadBalancerProvider registered via ServiceLoader. Application startup code deregisters it and conditionally re-registers a reconfigured instance.

We think that the failure sequence is this:

1. Phase 1 (CDS update) — XdsClusterResource parses a CDS response. LoadBalancingPolicyConverter finds the custom provider in the registry, selects it, and produces {"wrr_locality_experimental": {"childPolicy": [{"els": {}}]}}. Validation via WrrLocalityLoadBalancerProvider.parseLoadBalancingPolicyConfig succeeds. The raw JSON map is stored in CdsUpdate.lbPolicyConfig.

2. Registry mutation — Application startup code calls registry.deregister(oldProvider) then conditionally registry.register(newProvider). The custom provider is absent from the registry.

3. Phase 2 (any xDS update) — CdsLoadBalancer2.acceptResolvedAddresses re-parses the stored raw config. selectLbPolicyFromList calls getProvider("els") → returns null → error. The channel enters TRANSIENT_FAILURE and does not recover.

Critically, Phase 1 only runs on CDS updates (rare — when cluster config changes). But Phase 2 runs on every XdsConfig change, including EDS updates, which can be very frequent.

The re-parsing is guarded by the comment "Should be impossible, because XdsClusterResource validated this" (CdsLoadBalancer2.java:136), but the assumption that the registry is immutable after validation does not hold.

Why this didn't happen before #12140:

Probably because before #12140 CdsLoadBalancer2.acceptResolvedAddresses had an early return on subsequent calls.

// v1.70.0
if (this.resolvedAddresses != null) {
    return Status.OK;
}

Suggested action

Ideally, it would be great to find a way to not validate again the load balancer after the choice has been made somewhere else. If it's not fixed, it would still help to document that unregistering a load balancer at runtime can break the application.

Thanks!

[AgraVator]

Hey,
Mutating the global LoadBalancerRegistry after gRPC channels are created (or while xDS is processing updates) is fundamentally unsafe and unsupported. Providers should remain static for the application's lifetime once channels start resolving. CdsLoadBalancer2 intentionally re-parses during EDS updates to correctly propagate changes.

We agree with your suggestion to update the Javadoc to add an explicit warning that mutating the registry at runtime leads to undefined behavior.

[ejona86]

The channel enters TRANSIENT_FAILURE and does not recover.

This is because the config doesn't get re-parsed, right, because there are no updates? I think the problem with this issue is if we fix it, it still won't necessarily do what you want. The only way to fix this is by saving els and using it later. But you apparently don't want els to be used. Obviously you'd prefer "not failing" over "failing," but I don't think we'd want to guarantee that "the config from the xds server is the same but the parsing result would be different, thus we should apply a new config."

Application startup code calls registry.deregister(oldProvider) then conditionally registry.register(newProvider).

Do these providers share a config class? If we parse the config with oldProvider and pass the config to newProvider, that'd also fail?

What I'd like to do in general is have the parsing code save the actual provider and parsed config in the result; that way the config and the provider are guaranteed in-sync and there's no races with parsing. I've already been trying to do this from code health perspective, as there's just so many more error cases when you have to re-check the results and it makes the tests more annoying. To do that with LoadBalancer.parseLoadBalancingPolicyConfig(), I think I saw we'd really want to start passing LoadBalancerRegistry to the function in order for us to convert tests. And tests are the biggest pain with the change.

But I think cds is a place that is harder to do that. We're basically guaranteed to hit equals() using instance equality which will cause unnecessary config invalidation (which can be fixed, but is brittle and hard to detect before in-the-wild), and I think EDS results actually change the parsed config structure (or something like that).