Skip to content

Fix race conditions in ConnectionPool and SubProducer causing ThreadPool starvation - #292

Merged
blankensteiner merged 3 commits into
apache:masterfrom
eugene-polozhenkov:fix/connection-pool-race-conditions
Aug 11, 2026
Merged

Fix race conditions in ConnectionPool and SubProducer causing ThreadPool starvation#292
blankensteiner merged 3 commits into
apache:masterfrom
eugene-polozhenkov:fix/connection-pool-race-conditions

Conversation

@eugene-polozhenkov

@eugene-polozhenkov eugene-polozhenkov commented Aug 10, 2026

Copy link
Copy Markdown
Contributor

Fixes #291

Changes

SubProducer.EstablishNewChannel

The Task.Run lambda captured _dispatcherCts and _channel fields rather than local values. Under thread pool saturation a concurrent EstablishNewChannel call could dispose _dispatcherCts before the lambda read .Token, causing ObjectDisposedException -> fatal producer fault -> recreation death spiral.

Fix: materialize token = cts.Token and pin channel to locals before Task.Run.

ConnectionPool.GetConnection

No serialization on connection establishment: all N producers missing the cache simultaneously called EstablishNewConnection in parallel, spawning N(producers) TCP connections and background tasks.

Fix: SemaphoreSlim(1,1) gate per PulsarUrl with double-checked locking.

ConnectionPool.DisposeConnection

TryRemove(key) removed whichever connection was at the key, not specifically the dead one. Under load the ContinueWith delay could cause it to evict a freshly established replacement.

Fix: TryRemove(KeyValuePair<K,V>) / ICollection.Remove(kvp) for atomic
key+value comparison, with #if NETSTANDARD2_0 || NETSTANDARD2_1 following the existing library pattern.

Tests

  • SubProducerTests: unit test for the dispatcher field-capture race
  • ConnectionPoolTests: integration tests verifying all producers share a single
    connection after simultaneous reconnect (asserted via broker admin stats)
  • IntegrationFixture: CreateAdminClient() moved here from ConsumerTests

Same test as for the #291 with 3 sequential brokers ungraceful restarts, no thread spikes and N of connections stays within expected range.
fixedConnections

@eugene-polozhenkov
eugene-polozhenkov force-pushed the fix/connection-pool-race-conditions branch from 867b9e8 to 0a8dc86 Compare August 10, 2026 18:13
@blankensteiner
blankensteiner merged commit 4f4134d into apache:master Aug 11, 2026
3 checks passed
@blankensteiner

Copy link
Copy Markdown
Contributor

Very nice, thanks! :-)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Race conditions causing thread pool starvation and connection leaks on broker/proxy restarts

2 participants