feat(qwp): connect timeout, ingest callbacks, and lazy_connect (tolerant startup) on the QuestDB facade

[bluestreak01]

Tandem PRs:

• OSS core: feat(qwp): add a connect timeout and tolerant client startup questdb#7341
• Enterprise: https://github.com/questdb/questdb-enterprise/pull/1105

Summary

Related ergonomics/resilience improvements for the QWP (WebSocket) client:

1. Application-level TCP connect timeout (transport-wide) — bound the connect itself instead of riding the OS-level timeout.
2. Expose the ingest callbacks on the QuestDB facade — errorHandler / connectionListener, previously unreachable from the pooled facade.
3. Tolerant startup via lazy_connect=true — start the handle even when the server is down, buffer writes meanwhile, and read once it's up. Reads stay fully enabled.
4. Single cluster config on the facade — one ws/wss string (a single addr server list) configures the whole cluster, driving both the ingest and query pools.
5. Pooled-lease refactor + correctness fixes — reads and writes share one pooled-lease model (borrowQuery() mirrors borrowSender()), and a per-borrow generation closes a family of pool-corruption bugs (double-close, use-after-close, lost dispatch, stale cancel). This is internal hardening — see §5.
6. Invariant-B rework of the SF reconnect model — a store-and-forward sender never terminates on a connection error: async initial connect and mid-stream reconnects retry indefinitely; only genuine terminals (auth / non-421 upgrade / durable-ack gap) or SF exhaustion surface. Breaking: removes SenderConnectionEvent.Kind.RECONNECT_BUDGET_EXHAUSTED and narrows reconnect_max_duration_millis to the blocking SYNC initial connect — both shipped in 1.3.4. See §6.
7. NACK policy v2 — no drop, no lists, no dead senders — replaces the two-policy server-error model (DROP_AND_CONTINUE / HALT) with retriable/terminal classification that never silently loses data; a behavioral poison-frame detector (configurable via max_frame_rejections + a minimum escalation dwell poison_min_escalation_window_millis, and paced so a transient close never false-positives) replaces close-code lists. Breaking: SenderError.Policy renames; ackedFsn advances only on server OKs. See §7.
8. Review-milestone hardening (M8–M11) — drainer observability on the facade/builder, role-reject stream split out of durable-ack episodes, foreground-priority then lock-free connect walks, fast close() during outages, and a coverage-gap burn-down. See §8.

Items 1–3 are independently usable and off by default.

---

1. Configurable TCP connect timeout

A connect to a black-holed/firewalled host blocks on the OS-level TCP connect timeout (60–120s): the socket is created blocking, connect() runs, then it's switched to non-blocking. The code calls this out:

// SenderPool.java
// connect to a black-holed/firewalled host blocks on the OS connect timeout
// (the transport exposes no application-level connect timeout to clamp it).

Approach (native, cross-platform): non-blocking connect() (EINPROGRESS) → poll/select for writability bounded by the caller's budget → confirm via getsockopt(SO_ERROR). A sentinel (CONNECT_TIMEOUT = -3) lets Java raise a timeout-flagged exception. Generalises the existing handleEintrInConnect helper.

Sender.builder("https::addr=host:9000;connect_timeout=5000;")...   // or .connectTimeoutMillis(5000)
QwpQueryClient.fromConfig("ws::addr=host:9000;connect_timeout=5000;");

Touches: native share/net.c + windows/net.c + net.h; Net / NetworkFacade(Impl); HttpClientConfiguration.getConnectTimeout(); HttpClient.connect() / WebSocketClient.doConnect(); ConfigSchema COMMON key connect_timeout; Sender builder + both parsers; QwpWebSocketSender / QwpQueryClient (withConnectTimeout). Bounds the TCP connect and the TLS handshake (see below); the WebSocket upgrade stays under the request/auth timeout (auth_timeout_ms).

1b. Bound the TLS handshake (and stop it busy-spinning)

JavaTlsClientSocket.startTlsSession ran the TLS handshake with raw recv/send on the now-non-blocking socket and never waited on readiness. A peer that completed the TCP connect but stalled before its half of the handshake left the engine in NEED_UNWRAP with recv returning 0 (would-block), so the loop re-read in a tight cycle: a 100% CPU busy-spin with no deadline. connect_timeout bounded only the TCP connect, and the upgrade's request/auth timeout never covered the handshake, so a stalled wss:// host (e.g. QuestDB Cloud) could pin a core indefinitely and defeat the bounded connect. The busy-spin pre-dates this PR (it predates the connect_timeout work), but the same change closes it.

The handshake now runs through the client's existing deadline-aware ioWait (the same primitive recvOrDie/doSend use): when the socket would block it parks on epoll/kqueue/select for the remaining connect budget and throws a timeout-flagged exception once it is spent. doConnect registers the fd with the event loop before the handshake and bounds it by connect_timeout, falling back to the request timeout when connect_timeout is unset — so the handshake can no longer hang or spin even with the default config. Both HttpClient (https) and WebSocketClient (wss) share the fix, and the connect TLS block disconnects on any handshake error so the fd/native buffers do not leak.

Tradeoff: with connect_timeout unset, a wss:// handshake that stalls now fails after the request timeout instead of spinning/hanging forever. That is strictly better, but it is a behavior change for that edge.

Touches: JavaTlsClientSocket (handshake extracted into runHandshake), new SocketReadinessWaiter, Socket/PlainSocket, HttpClient.connect() / WebSocketClient.doConnect(). Tests: JavaTlsClientSocketTest.testHandshakeWaitsForReadabilityInsteadOfBusySpinning (a stalled peer must yield to the readiness waiter, not spin; a method-level timeout fails the test if the spin returns) and testHandshakeCompletesWithoutWaitingWhenEngineMakesProgress (happy path).

---

2. Ingest callbacks on the QuestDB facade

The facade built ingest senders from config strings only (SenderPool → Sender.fromConfig), so the programmatic SenderErrorHandler / SenderConnectionListener were unreachable — a facade user got the default loud-not-silent handlers with no way to observe async ingest errors or connection transitions.

QuestDB.builder()
    .fromConfig("ws::addr=host:9000;")
    .errorHandler(myErrorHandler)
    .connectionListener(myConnectionListener)
    .build();

QuestDBImpl / SenderPool each gain a full constructor carrying the callbacks; the white-box test-seam constructors are preserved as delegating shims. SenderPool.applyUserCallbacks() applies them to every pooled sender (non-SF and SF paths); internal recovery delegates are excluded. Defaults null.

---

3. Tolerant startup: lazy_connect=true

The facade prewarms a reader (QueryClientPool) synchronously and fail-fast (default query_pool_min=1; queries have no async connect), so a down server failed the whole build. The fix is a single connect-string flag that makes the handle tolerate a down server without giving up reads — "starts when the server is down" and "never reads" are different things, and you almost always want the first.

lazy_connect=true:

• a) starts even when the server is down — the ingest side goes async and the read pool defaults to query_pool_min=0, so neither side fail-fasts and build() returns promptly;
• b) buffers writes while down — the async sender accepts rows that flush once the wire is up;
• c) reads once the server is up — the read pool stays enabled (it is not disabled the way a write-only mode would). It defaults to query_pool_min=0, so nothing connects eagerly and borrowQuery() connects lazily on first use. While the server is still down a borrowQuery() throws (it has nowhere to connect — exactly like any read against a down server); once the server is up it connects and reads. The point is that lazy_connect defers the read connect rather than refusing reads.

// starts with no server present; writes buffer, reads work once it's up
try (QuestDB db = QuestDB.connect("ws::addr=host:9000;lazy_connect=true;")) {
    try (Sender s = db.borrowSender()) {
        s.table("t").longColumn("v", 1).atNow();    // buffers while the server is down
    }
    // ... later, once the server is up:
    try (Query q = db.borrowQuery()) {               // read pool connects lazily on first borrow
        q.sql("select 1").handler(handler).submit().await();
    }
}

Because both sides must start non-blocking, a knob that forces a blocking / fail-fast startup is a configuration conflict, rejected up front with a clear remedy rather than silently overridden:

• initial_connect_retry other than async (i.e. off/false/on/true/sync), and
• an explicit query_pool_min > 0 (connect string or builder call).

lazy_connect is a Side.POOL registry key — the two ws clients ignore it; the facade reads it, defaults query_pool_min to 0, and injects initial_connect_retry=async when the user set none.

---

4. Single cluster config on the facade

A QuestDB cluster is one logical target reached over QWP for both ingest and query, so the facade takes one cluster config: a single ws/wss string that lists every node in one addr server list and drives both the sender and query pools.

QuestDB.builder()
    .fromConfig("ws::addr=node1:9000,node2:9000,node3:9000;")  // whole cluster
    .errorHandler(myErrorHandler)
    .connectionListener(myConnectionListener)
    .build();

build() validates the one string with both the ingest (validateWsConfigString) and egress (QwpQueryClient.validateConfig) validators; each side applies the keys it owns and ignores the rest. Pool keys are read from a single ConfigView, and QuestDBImpl passes the one config to both pool slots (preserving the white-box reflection seam).

---

5. Pooled-handle lifecycle: lease/generation refactor + correctness fixes

This is the bulk of the diff and it is internal (no user-facing API beyond the borrow methods). It reshapes the facade so reads and writes share one pooled-lease model and fixes a family of pool-corruption bugs that the previous thread-affine / flag-guarded design allowed.

Refactor — symmetric pooled lease (1245c17). Reads and writes now use one model:

• Ingest: remove QuestDB.sender() / releaseSender() and the entire thread-pin subsystem behind them (pinToCurrentThread, releaseCurrentThread, clearPinIfCurrent, the threadAffine ThreadLocal, the PooledSender invalidated flag). borrowSender() is now the only way to lease a Sender.
• Egress: remove query(), newQuery(), and executeSql(); add QuestDB.borrowQuery(), a closeable, non-allocating Query lease that mirrors borrowSender(). Each pooled QueryWorker owns one pre-allocated QueryImpl, handed out reset on borrow; submit() dispatches on the held worker (single-flight) and close() returns it to the pool. Reads now connect at borrow time, so under lazy_connect the read pool defaults to query_pool_min=0 and build() does not fail-fast while the server is down. (This is why §3's read example uses borrowQuery() and not the removed executeSql.)

Correctness fixes carried on top of the refactor:

• f39e846 (M1) — stale pooled handle can corrupt the pool under double-close / use-after-close. The pooled handle was the reused per-slot object guarded by a non-volatile in-use flag, so a stale handle's close()/cancel()/write could leak into a different borrow (double-release → two concurrent borrowers on one non-thread-safe client). Fix: every borrow gets its own immutable generation, stamped under the pool lock and re-checked on every op — close()/cancel() no-op on a stale generation (idempotency preserved), submit()/writes throw, and release re-checks the generation under the lock so a slot can never be enqueued twice. Introduces QueryLease (wrapping QueryImpl) and SenderSlot (with PooledSender as the per-borrow wrapper).
• e30a59c — lost query-worker dispatch under single-flight reuse. runLoop() cleared current outside signalLock, so a fast submit() → await() → submit() loop could clobber a freshly dispatched job and discard its signal, hanging the caller forever. Fix: clear current under signalLock at the moment of consumption. (Surfaced as a 60s hang in testSustainedMixedConcurrency, mostly on aarch64 CI; 15/15 clean with the fix.)
• bcb1e7a — watchdog cancel must re-check the lease generation under the pool lock. cancel(gen) validated the generation with an unlocked volatile read then issued the wire cancel separately — a TOCTOU where a watchdog could abort a new borrower's query with a spurious STATUS_CANCELLED. Fix: route through QueryClientPool.cancelIfCurrent(worker, gen), which re-checks and cancels under the same lock the generation is bumped under; close()'s abort-on-unawaited-close uses the same path.
• f64567e — assert the lease wraps the same pooled QueryImpl. Pins the invariant that the freshly allocated per-borrow wrapper delegates back to the one pre-allocated, reused QueryImpl (so a regression that allocated per borrow would be caught).

---

6. Invariant B: store-and-forward senders never terminate on a connection error

This is the riskiest part of the diff and a behavior change for existing QWP senders. Previously connectLoop (async initial connect + every mid-stream reconnect on the I/O thread) enforced reconnect_max_duration_millis as a give-up deadline: on expiry it latched a terminal ...-budget-exhausted error that the next producer call surfaced. A long-but-recoverable outage (failover window, all-endpoints-replica, rolling upgrade) became a permanent producer-visible failure even though the rows were safe in SF.

Invariant B — once rows are accepted into store-and-forward, the background loop must never give up on a wall-clock budget (6be20d6, e9a1e7d):

• Async initial connect and mid-stream reconnect retry forever with capped exponential backoff + jitter. The only terminals are genuine ones — auth (401/403), non-421 upgrade reject, durable-ack capability gap — plus SF exhaustion, which surfaces to the producer as append backpressure (sf_append_deadline_millis, default 30s stall, then throws).
• SYNC initial connect (initial_connect_retry=on/sync) is unchanged: it still blocks up to reconnect_max_duration_millis on the user thread and fails loud.
• Outages stay loud: connectLoop WARNs on entry and per attempt (throttled to 5s), and the default connection listener still fires ENDPOINT_ATTEMPT_FAILED (INFO) per endpoint and ALL_ENDPOINTS_UNREACHABLE (WARN) per failed round — an outage is visible in logs, it just no longer kills the sender.

Breaking API change (e9a1e7d): SenderConnectionEvent.Kind.RECONNECT_BUDGET_EXHAUSTED is removed — under Invariant B it can never fire. It shipped in 1.3.4, so connection listeners that reference it (e.g. exhaustive switches) break at compile time. reconnect_max_duration_millis now bounds only the blocking SYNC initial connect; for async/mid-stream it is intentionally not consulted.

Drainer semantics aligned with the invariant:

• Orphan/background drainers no longer quarantine a down server (f9ece1a) or an all-replica window (d0a794a); both retry with capped backoff. An all-replica window is transient for durable-ack senders too (d920b9c).
• A mid-drain durable-ack capability gap re-enters the bounded settle-retry instead of quarantining on the first sweep (5b51236); transport windows between gap sweeps pause the settle wall clock instead of burning it (0f9857e, 4200697).
• Foreground role-reject retry backs off exponentially instead of storming fresh TLS handshakes at a fixed interval (70c706a).
• Background drainer connects no longer commit foreground sender state or fire foreground connection events (b990535); orphan drainers gate connects on their own lifecycle, not the foreground sender's (766832d).

Lifecycle/teardown hardening shaken out by the rework (retry-forever keeps the I/O thread alive in windows the old code never had):

• C5 (SEGV): never free the engine or client under a live I/O thread — failed-stop teardown protocol (4340371); the I/O thread disposes its own client on exit, and a connect racing close() discards the fresh client instead of installing it (2e0719d).
• C4 residuals: finite default connect deadline for background drainers; CAS-guarded WebSocketClient.close() (0e5125f).

Each behavior change landed red-first (a8fea31, 0d65c3a, 85dd6e8, ad25cdf, f6dfb32, c7419f4).

---

7. NACK policy v2 — no drop, no lists, no dead senders

Invariant B (§6) covers connection errors; NACK policy v2 (8302335) applies the same philosophy to server error ACKs. The old model had exactly two policies — DROP_AND_CONTINUE (silently lose the frame, advance the watermark) and HALT (kill the sender) — so a transient server-side condition either lost data or killed the producer. The new classification (design doc: design/qwp-nack-policy-v2.md):

• RETRIABLE (WRITE_ERROR, INTERNAL_ERROR, UNKNOWN fail-open): recycle the connection and replay from ackedFsn+1 through the existing wire-failure reconnect machinery. No watermark advance, nothing dropped.
• RETRIABLE_OTHER (NOT_WRITABLE, reserved wire byte 0x0C): same, biased to endpoint rotation. Read-only/demotion today arrives as a role-change NORMAL_CLOSURE close (see the OSS PR), so the byte is reserved for future servers.
• TERMINAL (SCHEMA_MISMATCH, PARSE_ERROR, SECURITY_ERROR on a writable node, PROTOCOL_VIOLATION): deterministic under byte-identical replay; latch loudly, bytes preserved in the SF log.

WS close codes carry no policy semantics anymore: every close is reconnect-eligible. A frame that deterministically kills the connection is caught behaviorally by the poison-frame detector — consecutive rejections of the same head-of-line FSN with no ack progress escalate to a typed PROTOCOL_VIOLATION terminal — instead of a close-code list.

The detector threshold is configurable (4278296): max_frame_rejections=N (>= 1, default 4) on the connect string (registered in ConfigSchema; the query client accepts it as a syntactic no-op — one vocabulary, side-owned application), LineSenderBuilder.maxFrameRejections(int), and through the QuestDB facade to every pooled sender. The threshold is forwarded to the cursor send loop and every BackgroundDrainer (drainers replay the owner's SF data, so they honor the same threshold).

Two refinements keep the detector true to its contract — never false-positive on an outage (a genuine outage fails at connect, not deterministically on one FSN):

• Below-threshold non-orderly-close recycles are paced, matching the NACK path. A middlebox/LB that completes the WS upgrade, accepts the head frame, then non-orderly-closes (e.g. 1006/1011) while its backend is briefly down succeeds at connect every cycle, so connectLoop's failed-connect backoff never engages. An unpaced recycle would burn max_frame_rejections strikes at connect+send+close RTT rate — well under a second — and latch a PROTOCOL_VIOLATION on a recoverable outage. The below-threshold close recycle now routes through the same strike-escalated failPaced the NACK path uses (previously it used the unpaced fail), so replay gaps widen and a transient gets a reset chance; genuine transport closes (orderly, or before any send) still reconnect immediately.
• Minimum escalation dwell (poison_min_escalation_window_millis, default 5000): even once the strike count is reached, escalation holds until the suspect has stayed poisoned for the window — decoupling "how many strikes prove determinism" from "how long a transient is allowed to look poisoned." With paced recycles strikes can accrue in well under a second, so the count alone could turn a brief outage into a producer-fatal terminal; the dwell guarantees an OK at/beyond the suspect a chance to reset the detector first. 0 = legacy immediate escalation at the threshold. Configurable on the connect string (registered in ConfigSchema; query-client no-op like max_frame_rejections), LineSenderBuilder.poisonMinEscalationWindowMillis(long), and forwarded to the cursor send loop and every BackgroundDrainer.

Breaking (pre-release surface): SenderError.Policy constants renamed; ackedFsn now advances only on server OKs (a NACKed frame no longer advances the watermark).

---

8. Review-milestone hardening (M8–M11)

Follow-ups from the milestone review of §6, mostly around observability and close/reconnect latency under outage:

• M10 — drainer observability (370e187, 514e789): role-reject notifications split out of onDurableAckUnavailable (a role reject is not a durable-ack capability episode), and BackgroundDrainerListener is now settable from the QuestDB facade and the Sender builder instead of only the internal pool.
• M11 — connect-walk concurrency (720d05f, then 9b33ae6): buildAndConnect no longer holds the sender monitor across network I/O. First pass: a dedicated foreground-priority lock (drainers tryLock and defer on contention). Final form: background walks are lock-free via walker-private round cursors on QwpHostHealthTracker (claim-at-pick, health-only ledger records), so concurrent drainer sweeps run in parallel with each other and the foreground, and can neither consume nor poison the foreground's round. close() during an outage also got a split stop policy: connect-phase drainers (never drained a row) stop within ~50ms; only actively-draining drainers get the graceful window — cutting outage-time close() from ~3s to one park chunk.
• Error containment (1582e3a): a half-built WebSocket client is closed (fd + native buffers) when an Error — not just an exception — escapes connect/upgrade.
• M8 — coverage-gap burn-down (2986827): mutant-verified pins plus a down-then-up outage-recovery e2e.
• M4–M7 residuals: stale Invariant-B docs fixed, assertMemoryLeak strict per-tag equality (single-tag leaks used to pass), native scratch-sink leak on failed engine construction fixed.

---

9. Deferred-commit ack integrity: never trim what the server can still roll back

Tandem with the server-side fix in questdb#7341: the server now withholds cumulative OK acks for FLAG_DEFER_COMMIT frames until their group-closing commit lands — an OK ack was letting the SF client trim slots whose rows the server rolls back on any error/demote/disconnect (silent data loss; the inverse of the deferred-commit feature's own replay contract). Client-side consequences land here:

• Close drains to the commit boundary. QwpWebSocketSender tracks lastCommitBoundaryFsn (updated at all four publish sites, residual seals included); drainOnClose targets min(publishedFsn, boundary) instead of publishedFsn — waiting for acks of an uncommitted deferred tail could only ever time out (was a 300s hang on abandoned transactions). Abandoning frames on close is WARN-logged with actionable guidance.
• Orphaned deferred tails abort instead of resurrecting. A producer killed mid-transaction leaves deferred frames with no covering commit frame at the top of its recovered SF log; replaying them would let the NEW session's next commit resurrect a partial transaction. Recovery locates the last commit-bearing frame (recoveredCommitBoundaryFsn); the send loop never transmits the tail and retires it with a cumulative self-acknowledge once everything below is server-acked (tryRetireOrphanTail). Fast path (everything below already acked — the common crash profile): retire at start(), zero wire cost. Slow path: replay below the tail first, then a single reconnect re-anchors the linear wireSeq↔FSN mapping past the gap. Crash-safe and idempotent; the BackgroundDrainer builds through the same constructor and inherits the containment.
• Positive identification only. The recovery walk classifies a frame as deferred only when it parses as a QWP message (min length + QWP1 magic before reading the flags byte). Anything unidentifiable is a retirement barrier, never trimmable — misclassifying an unknown frame as deferred would silently discard data that should replay.

Tests: CursorWebSocketSendLoopOrphanTailTest (recovery detection both ways, zero-send fast path, slow path end-to-end incl. the single recycle and ack attribution across the FSN gap).

Testing

• NetConnectTimeoutTest — loopback success, refused-vs-timeout disambiguation, black-hole timeout within budget.
• QuestDBFacadeCallbacksTest — facade-wired errorHandler receives the async auth-terminal SenderError from a 401-rejecting server (under Invariant B a plain connection error retries forever and never surfaces — only a genuine terminal like auth does); connectionListener observes connection events (no server needed).
• Invariant-B suite — InitialConnectAsyncTest (dead port: retries forever, no terminal; wasEverConnected() contract), ReconnectTest.testReconnectNeverGivesUpInvariantB, CloseSafetyNetTest / CloseOwnershipRaceTest (close() rethrows an unsurfaced genuine 401 terminal), orphan-drainer guards (down server, all-replica window, settle-clock pauses), and the C5 teardown races (interrupted close/stop under a live I/O thread).
• QuestDBLazyConnectTest — lazy_connect=true starts + buffers a write with the server down, keeps the read pool enabled (borrowQuery() deferred to first use, not disabled), and rejects both conflicts (blocking initial_connect_retry, explicit query_pool_min > 0) from the connect string and from builder calls; QuestDBServerRecoveryTest dogfoods lazy_connect=true for the full down → write → up → read lifecycle.
• QueryLeaseGenerationTest / SenderLeaseGenerationTest — the per-borrow generation guard: double-release, cross-borrow cancel/write, and the concurrent stale-cancel-after-reborrow TOCTOU (testConcurrentCancelDoesNotReachClientAfterReborrow).
• CursorWebSocketSendLoopPoisonFrameTest — poison-frame detector: OK-level strike keying under durable-ack replay re-OKs, terminal FSN attribution, NACK-recycle pacing, non-orderly-close-recycle pacing against an accepting-then-closing middlebox (unpaced code recorded ~200k recycles/1.2s; paced ≤10), and the minimum-escalation-dwell gate (strikes reaching the threshold must not escalate until the wall-clock window elapses). ServerErrorAckTerminalTest pins the end-to-end escalation with the dwell disabled (poison_min_escalation_window_millis=0) so the exact per-strike replay count stays assertable.
• QuestDBServerRecoveryTest — full lifecycle: server down → facade starts → client writes (buffered) → server starts → write side reconnects and the reader connects on the first borrowQuery().
• QuestDBBuilderTest — covers the single cluster-config surface; the shared-vocabulary and sender-pool-unwind tests run against one server with one config.
• TestWebSocketServer now serves both pools from one config like a real node: SERVER_INFO is emitted only on the egress /read path (the ingest /write ACK stream would choke on it), plus a setRejectReadUpgrade() toggle to fail only the query upgrade.

Full impl + network + facade suites pass locally on JDK 8 (source of truth) and the surface compiles on JDK 25 (java11+ front).

CI / native

• ci(native): the rebuild_native_libs.yml linux-x86-64 job moved from manylinux2014 (glibc 2.17) to manylinux_2_28 (glibc 2.28), mirroring linux-aarch64 — GitHub now forces actions onto Node 24 (glibc ≥ 2.27), which couldn't run in the 2.17 container (pre-existing breakage, unrelated to the C change).
• Native libraries are built on the test/release runners (no longer committed).

Compatibility

The connect_timeout knob and the ingest callbacks are additive and off by default. The direct QwpQueryClient API is unaffected.

The QuestDB facade shipped in 1.3.4 as unmarked public API, and §4/§5 make source-breaking changes to it (needs release-note coverage; note these removals ship without a deprecation cycle):

• Source-breaking (§5): sender(), releaseSender(), query(), newQuery(), and executeSql(CharSequence, QwpColumnBatchHandler) are removed. Migration: borrowSender() (already in 1.3.4) replaces the thread-pinned sender()/releaseSender(); the new borrowQuery() lease replaces the egress trio — db.executeSql(sql, handler) becomes try (Query q = db.borrowQuery()) { q.sql(sql).handler(handler).submit().await(); }.
• Source-breaking (§4): the two-arg connect(CharSequence ingest, CharSequence query) is removed. Migration: configure the whole cluster with a single ws/wss string (one addr server list) via connect(CharSequence) or builder().
• Behavior (§4): connect(CharSequence) now requires a ws/wss schema; 1.3.4 also accepted Sender-/QwpQueryClient-style http/https strings and mapped them.

The direct Sender API is affected by §6 (both surfaces shipped in 1.3.4 — needs release-note coverage):

• Source-breaking: SenderConnectionEvent.Kind.RECONNECT_BUDGET_EXHAUSTED is removed; connection listeners that reference it must drop that case (the event can no longer fire).
• Behavior: reconnect_max_duration_millis now bounds only the blocking SYNC initial connect. Async initial connect and mid-stream reconnects retry indefinitely instead of latching a producer-visible terminal on budget expiry; outages remain visible via the default connection listener and throttled WARN logs, and SF exhaustion still surfaces as append backpressure.

And by §7 (NACK policy v2):

• Source-breaking: SenderError.Policy constants renamed to the retriable/terminal taxonomy; error handlers that switch on them need the rename.
• Behavior: server error ACKs no longer drop frames (DROP_AND_CONTINUE is gone) — retriable NACKs replay from SF, deterministic rejections escalate to a typed terminal after max_frame_rejections deliveries; ackedFsn advances only on server OKs. A non-orderly close after a send now paces its recycle like a NACK, and poison escalation is gated on a minimum wall-clock dwell (poison_min_escalation_window_millis, default 5s) so a transient outage can't false-positive into a terminal — both are safer defaults (additive config; set the dwell to 0 for the previous escalate-at-threshold timing).

Note: the PR bundles several independent features. Happy to split any of them into standalone PRs off main if that's easier to review.

---

BREAKING CHANGE: removes QuestDB.sender(), QuestDB.releaseSender(), QuestDB.query(), QuestDB.newQuery(), QuestDB.executeSql(CharSequence, QwpColumnBatchHandler), and QuestDB.connect(CharSequence, CharSequence) (migrate to borrowSender() / borrowQuery() / single-config connect()); QuestDB.connect(CharSequence) now requires a ws/wss schema; removes SenderConnectionEvent.Kind.RECONNECT_BUDGET_EXHAUSTED; renames SenderError.Policy constants to the retriable/terminal taxonomy.

[mtopolnik]

[PR Coverage check]

😍 pass : 856 / 1108 (77.26%)

file detail

	path	covered line	new line	coverage
🔵	io/questdb/client/cutlass/qwp/client/WebSocketResponse.java	0	1	00.00%
🔵	io/questdb/client/std/MemoryTag.java	0	12	00.00%
🔵	io/questdb/client/cutlass/qwp/client/sf/cursor/BackgroundDrainerListener.java	0	1	00.00%
🔵	io/questdb/client/impl/PooledSender.java	14	54	25.93%
🔵	io/questdb/client/impl/QueryLease.java	6	23	26.09%
🔵	io/questdb/client/cutlass/http/client/HttpClient.java	5	14	35.71%
🔵	io/questdb/client/network/JavaTlsClientSocket.java	21	46	45.65%
🔵	io/questdb/client/impl/QueryImpl.java	37	77	48.05%
🔵	io/questdb/client/cutlass/qwp/client/sf/cursor/SegmentManager.java	1	2	50.00%
🔵	io/questdb/client/impl/QuestDBImpl.java	3	5	60.00%
🔵	io/questdb/client/Sender.java	43	60	71.67%
🔵	io/questdb/client/cutlass/http/client/WebSocketClient.java	22	28	78.57%
🔵	io/questdb/client/cutlass/qwp/client/QwpWebSocketSender.java	122	148	82.43%
🔵	io/questdb/client/cutlass/qwp/client/sf/cursor/CursorWebSocketSendLoop.java	160	188	85.11%
🔵	io/questdb/client/impl/QueryClientPool.java	31	35	88.57%
🔵	io/questdb/client/cutlass/qwp/client/sf/cursor/BackgroundDrainer.java	116	129	89.92%
🔵	io/questdb/client/cutlass/qwp/client/sf/cursor/CursorSendEngine.java	18	20	90.00%
🔵	io/questdb/client/impl/ConfigView.java	13	14	92.86%
🔵	io/questdb/client/cutlass/qwp/client/QwpQueryClient.java	12	13	92.31%
🔵	io/questdb/client/impl/SenderSlot.java	19	20	95.00%
🔵	io/questdb/client/impl/QueryWorker.java	19	20	95.00%
🔵	io/questdb/client/impl/SenderPool.java	79	82	96.34%
🔵	io/questdb/client/QuestDBBuilder.java	49	50	98.00%
🔵	io/questdb/client/cutlass/qwp/client/sf/cursor/BackgroundDrainerPool.java	7	7	100.00%
🔵	io/questdb/client/network/NetworkFacadeImpl.java	1	1	100.00%
🔵	io/questdb/client/cutlass/qwp/client/sf/cursor/MmapSegment.java	13	13	100.00%
🔵	io/questdb/client/cutlass/qwp/client/sf/cursor/SegmentRing.java	7	7	100.00%
🔵	io/questdb/client/cutlass/qwp/client/QwpHostHealthTracker.java	26	26	100.00%
🔵	io/questdb/client/HttpClientConfiguration.java	1	1	100.00%
🔵	io/questdb/client/cutlass/qwp/client/sf/cursor/DefaultSenderErrorHandler.java	1	1	100.00%
🔵	io/questdb/client/impl/ConfigSchema.java	5	5	100.00%
🔵	io/questdb/client/SenderConnectionEvent.java	1	1	100.00%
🔵	io/questdb/client/SenderError.java	4	4	100.00%