diff --git a/CLAUDE.md b/CLAUDE.md index 0cad286..897041c 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -330,6 +330,12 @@ behaviours: via `ValidateWebSocketKeys` (string-ctor path) or `ValidateWebSocketKeysAgainstDefaults` (programmatic-init path, default-comparison heuristic). +- `transaction=on` (WS-only): connection-scoped transactional mode — + auto-flush stages rows with `FLAG_DEFER_COMMIT`, `Send()`/`Commit()` + commits, and the server drops staged rows on connection close. + **Mutually exclusive with `sf_dir`** (`ConfigError`): SF replays + persisted frames across connections, which would re-stage and later + publish uncommitted (possibly abandoned) transactional rows. - `auto_flush=off` zeros `auto_flush_rows` / `auto_flush_bytes` / `auto_flush_interval` to `-1`. WS-specific defaults (`auto_flush_rows=1000`, `auto_flush_bytes=8 MiB`, `auto_flush_interval=100ms`) @@ -351,12 +357,162 @@ behaviours: ### Connection pooling -HTTP is thread-safe at the underlying `HttpClient` level; the Sender -itself is **not** thread-safe — one Sender per producer thread, or wrap -your own pool. There is no in-tree `LineSenderPool`; the HTTP transport -already shares `HttpClient`s under the hood via `IHttpClientFactory` -semantics in `HttpSender`. WS / SF manage their own concurrency model -(in-flight window, slot lock) and explicitly reject pooling. +A single `ISender` (or `IQwpQueryClient`) is **not** thread-safe — one per +producer thread. For multi-threaded producers there is now an in-tree pool, +`QuestDBClient` (mirrors the Java client's `QuestDB` handle), which pools +**both** ingest senders and (net7.0+) egress query clients. Files live in +`Pooling/` (public `QuestDBClient`/`IQuestDBClient`/`QuestDBClientBuilder` in +namespace `QuestDB`; internal `SenderPool`/`PooledSender`/`BorrowedSender`/`QueryClientPool`/ +`PooledQueryClient`/`PoolHousekeeper`/`QuestDBClientImpl` in `QuestDB.Pooling`). +The public `Query` builder (namespace `QuestDB`, `Query.cs`) is the query-side +surface. + +- **Entry points** mirror `Sender`/`ISender`: `QuestDBClient.Connect(confStr)` + or `QuestDBClient.Builder()…Build()` returns `IQuestDBClient`. Construct + once, share across threads. For distinct ingest/query endpoints use + `QuestDBClient.Connect(ingestConfStr, queryConfStr)` or + `Builder().IngestConfig(...).QueryConfig(...)` (Java `connect(ingest, query)` + parity); a single `ws`/`wss` `FromConfig` string serves both pools. +- **Borrow / return**: `BorrowSender()` (+ `BorrowSenderAsync`) returns an + `ISender` that is a fresh per-borrow `BorrowedSender` handle wrapping a + reusable `PooledSender` pool entry. **Dispose does not send** — it is pure + resource release: it discards any buffered-but-unsent rows (`Clear`), + **returns the entry to the pool** (it does NOT close the underlying sender), + and **never throws**. Call `Send()`/`SendAsync()` to hand rows to the + transport, and `Flush(timeout)`/`FlushAsync(timeout)` (drain = send + await + ACK) for delivery confirmation, **before** disposing; a sender whose buffer + can't be cleared (terminally failed) is discarded instead of re-pooled, as is + a transactional (`transaction=on`) ws sender still owing a commit for + auto-flushed rows staged server-side under `FLAG_DEFER_COMMIT` (the + `IPooledTransactionalSender` seam) — QWP has no rollback, so re-pooling the + live connection would let the next borrower's first commit publish the + abandoned rows; discarding closes the connection, which drops them. This is + airtight only because `transaction=on` + `sf_dir` is rejected at config + validation — SF replay would otherwise resurrect the discarded borrower's + deferred frames into the successor slot's connection. + **Pooled WS `Send()` is fast flush-to-ring** (the pool sets + `SenderOptions.SendAwaitsAck=false`): the pooled connection ships async after + return and delivery is confirmed via the pool-wide `Flush` below — unlike a + **standalone** WS `Send()`, which drains (`SendAwaitsAck=true` default) so + `Send()`-before-`Dispose` delivers on its own. The + handle is **use-after-return safe**: once disposed every ingest member throws + `ObjectDisposedException` (so a stale reference can't alias the entry a later + borrower now holds), and a second dispose is a tolerated no-op. There is no + context-affine / pinned-sender API — borrow per unit of work and dispose to + return (a single `ISender` is not thread-safe, so never share a borrowed one + across threads). +- **Pool-wide drain**: `IQuestDBClient.Flush(timeout)` / `FlushAsync(timeout)` + (→ `SenderPool.Flush`) fans `ISender.Flush` across every pooled sender — the + quiescence barrier for "confirm everything landed, then mark done". Call it + after all borrowed senders are returned; it does not synchronise against a + concurrent borrow. Returns `true` only if every sender drained within the + timeout. `close_flush_timeout_millis` is the default timeout for the no-arg + `Flush()` overloads (it no longer governs Dispose, which does not drain). +- **Sizing**: elastic between `sender_pool_min` and `sender_pool_max`, + bounded by a `SemaphoreSlim` capacity gate (counts in-use senders; + creation happens outside the lock). Idle senders sit in an + idle-time-sorted deque: borrowers pop/push the hot end (LIFO reuse, so + the excess goes genuinely cold and shrinks toward `min`), and the + housekeeper's `ReapIdle` sweeps from the cold end, stopping at the + first entry inside its timeout — O(reaped), not O(idle). `max_lifetime` + follows `CreatedAtUtc` (which the deque is NOT sorted by — an entry + can cross it while parked), so it runs as a separate walk gated on + `_idleOldestCreatedUtc`, a conservative lower bound over parked + entries' creation times that only fires when something actually + crossed the lifetime. + `BorrowSender` blocks up to `acquire_timeout_ms` then throws + `IngressError(ErrorCode.PoolExhausted)`. Reaping is **gated on full + drain**: an idle ws sender whose cursor-engine ring still holds + un-acked frames is skipped by both the idle and max-lifetime paths + (the idle sweep re-parks it at the hot end with a refreshed + `IdleSinceUtc`, so the idle clock effectively restarts at full-drain; + the age walk just leaves it for the next sweep), because tearing it down + would, in RAM mode, free the ring and silently drop those frames — the + pool-wide `Flush` can't cover an already-reaped sender. Surfaced via the net-agnostic `IPooledDrainAwareSender.IsFullyDrained` + seam (implemented by `QwpWebSocketSender` → `QwpCursorSendEngine.IsFullyDrained`; + HTTP/TCP deliver synchronously and are always drained). There is **no + bound**: a permanently wedged sender that never drains lives until the + pool is closed (data retained, not silently dropped) — consistent with + the "Flush before close" contract, since Dispose does not drain. +- **Config keys** (all-protocol, on `SenderOptions`, `[JsonIgnore]`d out of + `ToString` so a plain sender round-trips byte-identically): `sender_pool_min` + (1), `sender_pool_max` (4), `acquire_timeout_ms` (5000), `idle_timeout_ms` + (60000), `max_lifetime_ms` (1800000), `housekeeper_interval_ms` (5000), + `lazy_connect` (off). Validated by `SenderOptions.ValidatePoolOptions()`. +- **Tolerant startup (`lazy_connect=on`)**: a facade-only pool key (also + `QuestDBClientBuilder.LazyConnect()`) — in `QwpConnectStringKeys.Shared`, so a + plain `Sender` parses and ignores it on every scheme. It lets + `QuestDBClient` build while the server is down: the `ws`/`wss` ingest side + connects asynchronously (buffering writes) and the read pool defaults + `query_pool_min=0` so nothing connects eagerly — the read pool stays + **enabled** and a query connects lazily on first `NewQuery`. `QuestDBClientBuilder.Build` + reads it (`ResolveLazyConnect`), injects `initial_connect_retry=async` for the + pooled ws senders (via `SenderPool`'s `forceWsAsyncConnect`) when unset, and + rejects a conflicting blocking-startup knob up front (`IngressError`/`ConfigError`): + an explicit `initial_connect_retry` other than `async`, or an explicit + `query_pool_min > 0` (connect string or `QueryPoolMin(...)` builder call, tracked + across ingest + separate-query configs). Mirrors java-questdb-client `lazy_connect`. +- **Query pooling (net7.0+)**: `IQuestDBClient.NewQuery()` returns a fresh + `Query` builder (`Sql`/`Binds`/`Handler` then `ExecuteAsync`/`Execute`); + `ExecuteSqlAsync(sql, handler, ct)` is the convenience shortcut. There is + **no** public borrow/release for queries — the client lease is implicit per + `ExecuteAsync` and self-returns in a `finally` (Java `newQuery()`/ + `executeSql()` parity; Java's thread-local `query()` is **not** ported). The + `Task` returned by `ExecuteAsync` replaces Java's `Completion`. A clean return + re-pools the client (`GiveBack`); any throw — transport/protocol or a hard + `CancellationToken` cancel (which is permanently terminal) — discards it + (`MarkBroken` → `DiscardBroken`), since the egress client's terminal state is + sticky with no reset. `Query.Cancel()` is the **cooperative** path (forwards to + the in-flight client's `Cancel()`; the query ends normally and the client is + re-pooled). `PooledQueryClient` also checks an internal `IPooledQueryClientInner. + IsTerminalOrDisposed` seam (implemented by `QwpQueryWebSocketClient`) before + re-pooling. The query pool is the **stripped sibling** of `SenderPool` — no + SF/slot/leak-debt machinery (read side has no store-and-forward) and no + AsyncLocal pin. Query-pool config: `query_pool_min` (1), `query_pool_max` (4) + on `SenderOptions` (`[JsonIgnore]`, in `QwpConnectStringKeys.Shared` so both + `SenderOptions` and `QueryOptions` accept them), validated by the separate + `ValidateQueryPoolOptions()` (kept out of `EnsureValid` so a plain `Sender` + stays lenient); the acquire/idle/lifetime/housekeeper knobs are **shared** with + the sender pool, and one `PoolHousekeeper` sweeps both. The query pool is built + only when a `ws`/`wss` query config is present (single `ws` string, explicit + `QueryConfig`, or `Connect(ingest, query)`); an `http`/`tcp` ingest handle with + no query config throws `IngressError(ConfigError)` on `NewQuery`. All query + members are gated `#if NET7_0_OR_GREATER` (so `IQuestDBClient` differs by TFM); + net6.0 keeps the sender-only surface. +- **Store-and-forward**: when pooling `ws::`+`sf_dir`, each pooled sender + gets a distinct slot identity `sender_id = -` (via + `SenderPool.ApplySlotIdentity`) so siblings never collide on a slot + directory / flock. Free indices live in a LIFO stack (most recently freed + reused first, keeping the on-disk slot-directory working set compact). A + discarded/reaped sender's index is freed only after + `IPooledSlotSender.IsSlotLockReleased` confirms the lock dropped (a + net-agnostic seam implemented by `QwpWebSocketSender`, backed by + `QwpSlotLock.IsReleased`); otherwise it is **retired** (`_retired`) and one + capacity permit stays physically withheld on its behalf, shrinking effective + `max` (a discarded sender's own permit is simply not released; the reaper + competes through the same gate as borrowers — it takes a free permit up + front via `TryWithholdPermit` and **stops the sweep** when a mid-borrow + thread already drained the last one — that borrower reuses the idle + sender instead, so a legitimate borrow never sees a spurious `PoolExhausted`). + This shrink is **not permanent**: when a wedged/deferred engine teardown + (`QwpCursorSendEngine.Dispose` defers `ReleaseSharedResources` past its 5 s + pump-join budget) leaves the lock still held at dispose time, the housekeeper's + `ReclaimRetiredSlots` re-tests `IsInnerSlotLockReleased` each sweep — once it + flips true the index is freed and the withheld permit released + (`_capacity.Release()`). So `LeakedSlotCount` (= `_retired.Count`) is a live + gauge of currently-retired slots, not a monotonic counter. Pooled + senders pass their managed family to `QwpOrphanScanner.ClaimOrphans(..., + managedBase, managedCount)` so orphan adoption skips live/future siblings + but still drains true out-of-family orphans (when `drain_orphans=on`). + In-range stranded slots recover lazily — a pooled sender replays its own + slot's segments on open (`QwpSegmentRing.Open`) when the index is + (re)allocated. **Known limitation:** unlike the Java pool, there is no + proactive housekeeper-driven two-pass startup drain of in-range slots + that are not currently allocated (e.g. a pool that permanently shrank); + their data stays on disk until a future run reallocates the index. +- HTTP still shares `HttpClient`s under the hood per address inside + `HttpSender`; the pool is layered above the `ISender` seam and is + protocol-agnostic. ### Value types diff --git a/README.md b/README.md index d71e2a9..2fa3c75 100644 --- a/README.md +++ b/README.md @@ -78,8 +78,22 @@ This is equivalent to a config string of: using var sender = Sender.New("http:addr=localhost:9000;auto_flush=on;auto_flush_rows=75000;auto_flush_interval=1000;"); ``` -A final flush or send should always be used, as auto flush is not guaranteed to send all pending data before -the sender is disposed. +You must call `Send()` / `SendAsync()` explicitly before disposing — **`Dispose` no longer sends**. It is +pure resource release: any buffered-but-unsent rows are discarded and it never throws. Auto-flush is also not +guaranteed to have sent all pending rows by the time you dispose. + +Delivery semantics of `Send()`/`SendAsync()`: + +- **HTTP** — synchronous; returns once the server has committed the batch. +- **Standalone `ws`/`wss`** — **drains**: flushes and blocks until the server acknowledges (bounded by + `close_flush_timeout_millis`, throwing on timeout), so `Send()` before `Dispose` reliably delivers. +- **Pooled `ws`/`wss`** (from `IQuestDBClient.BorrowSender`) — hands the frame to the ring and returns + immediately; the pooled connection ships it asynchronously. Confirm delivery with the pool-wide + `IQuestDBClient.Flush(timeout)` once your producers have returned their senders. + +For an explicit bounded, bool-returning drain on any sender use `Flush(timeout)` / `FlushAsync(timeout)`; +for the whole pool use `IQuestDBClient.Flush(timeout)`. The identical `Send()`-before-dispose code works for +both standalone and pooled senders — only the fast bulk path differs (auto-flush + a final `Flush`). #### Flush every 1000 rows or every 1 second @@ -269,9 +283,9 @@ The config string format is: ``` | Name | Default | Description | -| ------------------------ | -------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `protocol` (schema) | `http` | The transport protocol to use. Options are http(s)/tcp(s)/ws(s). `ws::` / `wss::` requires .NET 7+. | -| `addr` | `localhost:9000` | The {host}:{port} pair denoting the QuestDB server. Default port 9000 for HTTP and ws/wss, 9009 for TCP. | +| ------------------------ | -------------------------- |-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| `protocol` (schema) | `http` | The transport protocol to use. Options are http(s)/tcp(s)/ws(s). `ws::` / `wss::` requires .NET 7+. | +| `addr` | `localhost:9000` | The {host}:{port} pair denoting the QuestDB server. Default port 9000 for HTTP and ws/wss, 9009 for TCP. | | `auto_flush` | `on` | Enables or disables auto-flushing functionality. By default, the buffer will be flushed every 75,000 rows, or every 1000ms, whichever comes first. | | `auto_flush_rows` | `75000 (HTTP)` `600 (TCP)` | The row count after which the buffer will be flushed. Effectively a batch size. | | `auto_flush_bytes` | `Int.MaxValue` | The byte buffer length which when exceeded, will trigger a flush. | @@ -281,10 +295,10 @@ The config string format is: | `username` | | The username for authentication. Used for Basic Authentication and TCP JWK Authentication. | | `password` | | The password for authentication. Used for Basic Authentication. | | `token` | | The token for authentication. Used for Token Authentication and TCP JWK Authentication. | -| `tls_verify` | `on` | Denotes whether TLS certificates should or should not be verified. Options are on/unsafe_off. | +| `tls_verify` | `on` | Denotes whether TLS certificates should or should not be verified. Options are on/unsafe_off. | | `tls_roots` | | Used to specify the filepath for a custom .pem certificate. | | `tls_roots_password` | | Used to specify the filepath for the private key/password corresponding to the `tls_roots` certificate. | -| `auth_timeout` | `15000` | The time period to wait for authenticating requests, in milliseconds. | +| `connect_timeout` | `15000` | Total time, in milliseconds, allowed to establish a connection across all layers — TCP socket connect, TLS handshake, WebSocket upgrade, and (on TCP) the ECDSA auth exchange. The attempt is aborted if exceeded. Applies to every transport. | | `request_timeout` | `30000` | Base timeout for HTTP requests before any additional time is added. | | `request_min_throughput` | `102400` | Expected minimum throughput of requests in bytes per second. Used to add additional time to `request_timeout` to prevent large requests timing out prematurely. | | `retry_timeout` | `10000` | The time period during which retries will be attempted, in milliseconds. | @@ -306,7 +320,7 @@ The config string format is: | `reconnect_max_backoff_millis` | `5000` | Cap on per-attempt backoff. | | `reconnect_max_duration_millis` | `300000` | Total per-outage budget; sender becomes terminal if exceeded. | | `initial_connect_retry` | `off` | `on` makes the first connect honour the same backoff loop. Default is "fail fast on first connect". | -| `close_flush_timeout_millis` | `60000` | Max wait at `Dispose` for the SF engine to drain (matches Java). `0` or `-1` for fast close. | +| `close_flush_timeout_millis` | `60000` | Drain timeout for a standalone `Send()` and the no-arg `Flush()` / `FlushAsync()`. **`Dispose` does not drain.** | | `drain_orphans` | `off` | `on` adopts unlocked sibling slots on startup and drains them in the background. | | `max_background_drainers` | `4` | Cap on concurrent orphan-drain workers. | diff --git a/examples.manifest.yaml b/examples.manifest.yaml index 27b1a3b..5f92f9c 100644 --- a/examples.manifest.yaml +++ b/examples.manifest.yaml @@ -43,3 +43,10 @@ header: |- [.NET client library](https://github.com/questdb/net-questdb-client) conf: ws::addr=localhost:9000;target=any; + +- name: sender-pool + lang: csharp + path: src/example-sender-pool/Program.cs + header: |- + [.NET client library](https://github.com/questdb/net-questdb-client) + conf: http::addr=localhost:9000; diff --git a/net-questdb-client.sln b/net-questdb-client.sln index 43fa637..cdca06c 100644 --- a/net-questdb-client.sln +++ b/net-questdb-client.sln @@ -29,6 +29,10 @@ Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "example-qwp-query", "src\ex EndProject Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "test-apps", "test-apps", "{827E0CD3-B72D-47B6-A68D-7590B98EB39B}" EndProject +Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "src", "src", "{827E0CD3-B72D-47B6-A68D-7590B98EB39B}" +EndProject +Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "example-sender-pool", "src\example-sender-pool\example-sender-pool.csproj", "{284F86C5-9B59-428C-B3E5-DAD7CA980BA0}" +EndProject Global GlobalSection(SolutionConfigurationPlatforms) = preSolution Debug|Any CPU = Debug|Any CPU @@ -183,6 +187,18 @@ Global {A247ACD9-F600-47D3-B8C6-33543B4FB95B}.Release|x64.Build.0 = Release|Any CPU {A247ACD9-F600-47D3-B8C6-33543B4FB95B}.Release|x86.ActiveCfg = Release|Any CPU {A247ACD9-F600-47D3-B8C6-33543B4FB95B}.Release|x86.Build.0 = Release|Any CPU + {284F86C5-9B59-428C-B3E5-DAD7CA980BA0}.Debug|Any CPU.ActiveCfg = Debug|Any CPU + {284F86C5-9B59-428C-B3E5-DAD7CA980BA0}.Debug|Any CPU.Build.0 = Debug|Any CPU + {284F86C5-9B59-428C-B3E5-DAD7CA980BA0}.Debug|x64.ActiveCfg = Debug|Any CPU + {284F86C5-9B59-428C-B3E5-DAD7CA980BA0}.Debug|x64.Build.0 = Debug|Any CPU + {284F86C5-9B59-428C-B3E5-DAD7CA980BA0}.Debug|x86.ActiveCfg = Debug|Any CPU + {284F86C5-9B59-428C-B3E5-DAD7CA980BA0}.Debug|x86.Build.0 = Debug|Any CPU + {284F86C5-9B59-428C-B3E5-DAD7CA980BA0}.Release|Any CPU.ActiveCfg = Release|Any CPU + {284F86C5-9B59-428C-B3E5-DAD7CA980BA0}.Release|Any CPU.Build.0 = Release|Any CPU + {284F86C5-9B59-428C-B3E5-DAD7CA980BA0}.Release|x64.ActiveCfg = Release|Any CPU + {284F86C5-9B59-428C-B3E5-DAD7CA980BA0}.Release|x64.Build.0 = Release|Any CPU + {284F86C5-9B59-428C-B3E5-DAD7CA980BA0}.Release|x86.ActiveCfg = Release|Any CPU + {284F86C5-9B59-428C-B3E5-DAD7CA980BA0}.Release|x86.Build.0 = Release|Any CPU EndGlobalSection GlobalSection(SolutionProperties) = preSolution HideSolutionNode = FALSE @@ -190,5 +206,6 @@ Global GlobalSection(NestedProjects) = preSolution {A1FE95A9-4761-4806-8891-A82F468624F8} = {827E0CD3-B72D-47B6-A68D-7590B98EB39B} {A247ACD9-F600-47D3-B8C6-33543B4FB95B} = {827E0CD3-B72D-47B6-A68D-7590B98EB39B} + {284F86C5-9B59-428C-B3E5-DAD7CA980BA0} = {827E0CD3-B72D-47B6-A68D-7590B98EB39B} EndGlobalSection EndGlobal diff --git a/sender-pool-plan.md b/sender-pool-plan.md new file mode 100644 index 0000000..00cab67 --- /dev/null +++ b/sender-pool-plan.md @@ -0,0 +1,277 @@ +# Plan: Sender connection-pool API for the C# QuestDB client (Java parity) + +Port the Java client's `QuestDB` sender-pool to C#, **including store-and-forward (SF) +slot management**. Egress/query pooling is explicitly **out of scope** for this effort +(see §8). + +Java sources analysed (`/Users/alpel/src/questdb-enterprise3/questdb/java-questdb-client`): +`QuestDB.java`, `QuestDBBuilder.java`, `impl/SenderPool.java` (1168 LOC), +`impl/PooledSender.java` (419), `impl/PoolHousekeeper.java` (137). + +--- + +## 1. Target public API (C#) + +Mirrors Java but adapts to repo idiom: the repo already pairs a static factory `Sender` +with interface `ISender`. We do the same. **The handle type cannot be named `QuestDB`** — +that's the root namespace. Use `QuestDBClient` (static factory) + `IQuestDBClient`. + +```csharp +namespace QuestDB; // reuse root ns, consistent with Sender/ISender + +public interface IQuestDBClient : IDisposable, IAsyncDisposable +{ + ISender BorrowSender(); // blocks ≤ acquire_timeout, throws on timeout/closed + ValueTask BorrowSenderAsync(CancellationToken ct); // C# value-add (see §6) + ISender Sender(); // thread-pinned borrow (ThreadLocal) + void ReleaseSender(); // unpin current thread + void Close(); // == Dispose(); idempotent +} + +public static class QuestDBClient // factory, mirrors Java QuestDB.connect/builder +{ + public static IQuestDBClient Connect(string confStr); + public static QuestDBClientBuilder Builder(); +} + +public sealed class QuestDBClientBuilder // mirrors QuestDBBuilder +{ + public QuestDBClientBuilder FromConfig(string confStr); // == IngestConfig for now + public QuestDBClientBuilder IngestConfig(string confStr); + public QuestDBClientBuilder SenderPoolMin(int n); + public QuestDBClientBuilder SenderPoolMax(int n); + public QuestDBClientBuilder SenderPoolSize(int n); // sets min==max + public QuestDBClientBuilder AcquireTimeout(TimeSpan t); + public QuestDBClientBuilder IdleTimeout(TimeSpan t); + public QuestDBClientBuilder MaxLifetime(TimeSpan t); + public QuestDBClientBuilder HousekeeperInterval(TimeSpan t); + public IQuestDBClient Build(); +} +``` + +Usage (Java parity — `close()` returns to pool, does NOT disconnect): +```csharp +using var client = QuestDBClient.Connect("http::addr=localhost:9000;sender_pool_max=8;"); +using (var s = client.BorrowSender()) // ISender; Dispose() flushes + returns to pool +{ + s.Table("trades").Column("px", 101.5).At(DateTime.UtcNow); +} // <- back in pool, real socket stays open +``` + +**Behavioral contract (matches Java `PooledSender`):** +- The borrowed `ISender` is a `PooledSender` decorator. Its `Dispose()`/`DisposeAsync()` + **flushes pending rows then returns the decorator to the pool** — it does *not* close the + underlying sender. The real sender closes only at `IQuestDBClient.Dispose()`. +- Idempotent: a second `Dispose()` after return is a no-op. +- If the return-flush throws, the sender is **discarded** (real delegate disposed, slot freed), + never returned. + +--- + +## 2. Config keys (added to `SenderOptions`) + +All-protocol keys (HTTP/TCP/WS), defaults from Java `QuestDBBuilder`: + +| Key | Type | Default | Notes | +|---|---|---|---| +| `sender_pool_min` | int | 1 | warm minimum, ≥ 0 | +| `sender_pool_max` | int | 4 | hard cap, ≥ 1 and ≥ min | +| `acquire_timeout_ms` | ms→TimeSpan | 5000 | borrow blocking budget | +| `idle_timeout_ms` | ms→TimeSpan | 60000 | reap idle (never below min) | +| `max_lifetime_ms` | ms→TimeSpan | 1800000 | reap over-age (30 min) | +| `housekeeper_interval_ms` | ms→TimeSpan | 5000 | reaper sweep, ≥ 100 | + +Wiring (verified seams in `Utils/SenderOptions.cs`): +- **Register** all six in `Qwp/QwpConnectStringKeys.cs` `Shared[]` (line ~42) — *not* in + `WebSocketOnlyKeys`; they are protocol-agnostic. `KnownConnectStringKeys` picks them up + automatically, so they pass `RejectUnknownConnectStringKeys` (SenderOptions.cs:341). +- **Parse** in the string ctor (after pool_timeout, ~line 230): `ParseIntWithDefault(...)` for + the two ints, `ParseMillisecondsWithDefault(...)` for the four ms keys. Add matching + `_*UserSet` flags (set from `ReadOptionFromBuilder(name) is not null`). +- **Fields/props**: backing field + get/set with `_*UserSet = true` in setter (repo idiom, + SenderOptions.cs:901). Field initializers carry the defaults. +- **Validate** — new `ValidatePoolOptions()` called from `EnsureValid()` (SenderOptions.cs:553): + `min >= 0`, `max >= 1`, `min <= max`, all timeouts `> 0`, `housekeeper >= 100ms`. Throw + `IngressError(ErrorCode.ConfigError, ...)` like the existing validators. +- **ToString round-trip**: reflection-based `ToString()` (SenderOptions.cs:1890) auto-emits + public props; TimeSpans already serialise as long-millis. Pool keys are not WS-only, so they + are not skipped → round-trips through `new SenderOptions(s.ToString())`. Add a + `SenderOptionsTests` round-trip case. + +Note: pool keys live in the **same** connect string as the sender keys (Java does the same — +`ConfigView.getInt` reads pool keys off the ingest view; there's no separate strip step). +`SenderOptions` already ignores pool keys for sender behavior; the pool reads them off the +parsed `SenderOptions`. + +--- + +## 3. New types (the pool itself) + +Under `src/net-questdb-client/Pooling/` (new folder), Apache banner on each file: + +| File | Role | Java analogue | +|---|---|---| +| `IQuestDBClient.cs` | public handle interface | `QuestDB` | +| `QuestDBClient.cs` | static `Connect`/`Builder` + internal `QuestDBClientImpl` | `QuestDB`/`QuestDBImpl` | +| `QuestDBClientBuilder.cs` | builder | `QuestDBBuilder` | +| `SenderPool.cs` | elastic pool core | `SenderPool` | +| `PooledSender.cs` | `ISender` decorator, return-on-Dispose | `PooledSender` | +| `PoolHousekeeper.cs` | background reaper task | `PoolHousekeeper` | + +`SenderPool` builds underlying senders via the existing `Sender.New(SenderOptions)` factory +(per-slot options cloned and `sender_id` overridden — see §5). + +--- + +## 4. Threading model (recommended C# adaptation) + +Java uses `ReentrantLock` + `Condition` + create-outside-lock. The idiomatic C# equivalent +that **also yields async acquire for free** is a `SemaphoreSlim` capacity gate + a short +`lock` for the free-list/counters: + +- `SemaphoreSlim _capacity = new(maxSize, maxSize)` — counts **in-use** senders. A permit is + taken on borrow, released on return. Because we only *create* a sender when `_available` is + empty *and* a permit is held, total alive senders never exceeds `max` (proof: when creating, + every other alive sender is in-use holding one of the remaining permits). +- `BorrowSender`: `_capacity.Wait(acquireTimeout)` (sync) / `await _capacity.WaitAsync(...)` + (async) → on success, under a short `lock(_gate)` pop an idle `PooledSender` or reserve a + creation; **create the sender outside the lock** (TLS/DNS won't block other borrowers); + add to `_all` under lock. On timeout → throw `IngressError` ("timed out waiting for a Sender + from the pool after Nms"). +- `giveBack`: under `lock(_gate)` push to `_available` (or discard if pool closed), then + `_capacity.Release()`. +- `_available` = `Stack` (LIFO keeps hot connections hot); `_all` = `List<>`. +- Counters/flags: `_closed` (`volatile bool`, checked before gate), in-flight creations folded + into the permit accounting. + +This diverges from the literal Java lock structure but is behaviorally equivalent and is the +clean way to support both `BorrowSender()` and `BorrowSenderAsync()`. (Alternative if strict +structural parity is wanted: `Monitor.Wait/Pulse` — but then async acquire needs a second +path. Recommend SemaphoreSlim.) + +**Decorator (`PooledSender`)** delegates the full `ISender` surface (~60 members: all `Column` +overloads, `Symbol`, `Table`, `At*`, `Send*`, `Transaction`, `Length`, etc.) to `_delegate`. +Only `Dispose`/`DisposeAsync` differ (flush+return, idempotent via `Interlocked` flag). This +is mechanical (~250 LOC, matches Java's 419-line explicit decorator). Returns `ISender` only — +QWP-only methods (`Ping`, ack getters) are not surfaced through the pool wrapper in v1, exactly +as Java exposes only `Sender`. + +--- + +## 5. Store-and-forward parity (the crux) — §4 chosen scope + +Each pooled WS+SF sender must get a **distinct, stable** slot identity so their mmap segment +dirs + flocks never collide. Java: `sender_id = -`, index ∈ `[0, maxSize)`, +lowest-free allocation. C# verification: +- `//` isolation is **automatic** (`QwpWebSocketSender` builds the slot path + via `Path.Combine(sf_dir, sender_id)`; `QwpSlotLock.Acquire` takes an exclusive `FileShare.None` + lock). So `sender_id = -` cleanly isolates segments + locks. ✅ reuse as-is. +- Reusable as-is: `QwpSlotLock.Acquire/TryAcquire`, `QwpOrphanScanner.ClaimOrphans`, + `QwpBackgroundDrainer` (already forces `InitialConnectMode.off`), `QwpBackgroundDrainerPool`, + `IQwpWebSocketSender.GetHighestAckedSeqTxn/Ping/AwaitAckedFsnAsync` for return-flush. ✅ + +### Two production-code gaps (MUST build before SF pool works) +These are in the **client library**, not the pool, and Java's pool depends on both: + +1. **Slot-lock-released probe — ABSENT in C#.** + Java reclaims a slot index only after `QwpWebSocketSender.isSlotLockReleased()` confirms the + OS flock actually dropped (the I/O thread may still hold it briefly after `close()`). + - Add `bool IsSlotLockReleased { get; }` to `QwpWebSocketSender` (and `IQwpWebSocketSender`), + backed by a released-flag on `QwpSlotLock` (set when the held `FileStream` is actually + released in `Dispose`). Non-WS senders report `true` (no flock). + - Without this, the pool cannot safely reuse a slot index after discard/reap → risk of + "slot already in use" on recreate. + +2. **Exclude-managed-slots from orphan draining — ABSENT in C#.** + `QwpOrphanScanner.ClaimOrphans(sfRoot, ourSenderId)` only skips a *single* id. A pooled + sender at `base-2` would otherwise see siblings `base-0/base-1/...` as "orphans" and fight + the pool for them. Java solves this with `orphanDrainExcludeManagedSlots(base, maxSize)`. + - Add an exclude-predicate/overload to `QwpOrphanScanner.ClaimOrphans` (skip any dir matching + `-<0..maxSize-1>`), plumbed via a new internal option (e.g. + `orphan_exclude_managed_base`/`...count`) set by the pool on each slot's `SenderOptions`. + +### Pool-side SF logic (port from `SenderPool.java`) +- `slotInUse[]` bitmap, `allocateSlotIndex()` (lowest-free), `freeSlotIndex(i)`. +- `closingSlots` / `leakedSlots` counters folded into capacity accounting (a slot whose flock + hasn't released yet must not be re-created → hold its permit until `IsSlotLockReleased`, else + retire the index permanently and log a warning, matching Java `reclaimSlot`). +- **Two-pass startup recovery** driven on the housekeeper thread (keeps `Connect()` non-blocking): + - Pass 1 — in-range indices `[0, maxSize)`: probe each `-` slot dir; if it has + stranded `sf-*.sfa` data, reserve it (counts against capacity), drain via a recovery sender + built with `initial_connect_mode=off` + `drain_orphans=off`, bounded per-drain (~1s). + - Pass 2 — out-of-range orphans `[maxSize, ...)` left by a previously larger pool: lazily + enumerate, drain, retire. + - Reuse `QwpOrphanScanner` + `QwpBackgroundDrainer` rather than re-implementing the drain. +- `senderFactory` test seam (`Func`) like Java's `IntFunction`. + +--- + +## 6. Async semantics (C# value-add) + +- `BorrowSenderAsync(ct)` over `_capacity.WaitAsync`. The decorator implements true + `DisposeAsync` doing **async** return-flush (WS flush is naturally async). Sync `Dispose` + blocks on it (matches existing `QwpWebSocketSender` dispose behavior). +- ⚠️ `Sender()`/`ReleaseSender()` thread-affinity uses `ThreadLocal`. Document the + Java-aligned caveat: **dedicated producer threads only**; do not hold a pinned sender across + `await` (continuations can resume on a different pool thread). v1 ports it verbatim with this + doc warning; an `AsyncLocal` variant can come later if needed. + +--- + +## 7. Phasing & deliverables + +**Phase 0 — Config plumbing.** Pool keys in `SenderOptions` + `QwpConnectStringKeys.Shared` + +`ValidatePoolOptions` + ToString round-trip. Tests: `SenderOptionsTests` (parse, defaults, +min>max rejected, round-trip). *No behavior yet.* Low risk. + +**Phase 1 — Core elastic pool (HTTP/TCP/WS-RAM, no SF).** `IQuestDBClient`, `QuestDBClient`, +`QuestDBClientBuilder`, `SenderPool`, `PooledSender`. Borrow/giveBack, min/max, acquire-timeout, +create-outside-lock, pool close (disposes all delegates), discard-broken, introspection +(`AvailableSize`/`TotalSize`). Sync + async borrow. Tests: port `SenderPoolTest` (recycle-same- +decorator, broken-not-returned, close-idempotent, close-rejects-borrow, exhaustion-timeout, +builds-min, grows-to-max). **This delivers ~80% of the value** (HTTP/TCP/WS-RAM all poolable). + +**Phase 2 — Housekeeper.** `PoolHousekeeper` as a background `Task` + `PeriodicTimer` +(net6.0+, available on all TFMs). Idle reap + max-lifetime reap, never below min; swallow +`Exception` (C# analogue of Java's `Throwable`) so a delegate-close fault can't kill the loop. +Tests: reap-shrinks-to-min, respects-min, survives-delegate-close-error. + +**Phase 3 — Thread-affine.** `Sender()`/`ReleaseSender()` via `ThreadLocal`, pin/unpin, +clear-pin-on-return, invalidate-on-close. Tests: per-thread distinct, pin-after-close rejected, +release-after-close safe. + +**Phase 4 — SF slot management (largest, riskiest).** +- Production: `IsSlotLockReleased` (+ `QwpSlotLock` released-flag); orphan exclude-managed-slots. +- Pool: per-slot `sender_id=-`, slot bitmap, closing/leaked accounting, flock-release + reclaim/retire, two-pass startup recovery on housekeeper. +- Tests: port `SenderPoolSfTest` (slot collision, recovery, concurrency), `SenderPoolSfTest`'s + retire-on-leak. Integration: a `QuestDbWebSocketIntegrationTests`-style pool case against a + live master (slots survive recycle; recovery replays stranded segments). + +**Phase 5 — Error-safety + polish.** Port `SenderPoolErrorSafetyTest`; close-during-borrow +release-with-error parity; XML docs; an `example-*` demo; CLAUDE.md update (replace the +"explicitly reject pooling" note with the new pool, keep the single-Sender-not-thread-safe +guidance for the un-pooled path). + +--- + +## 8. Out of scope (flag to user) + +- **Egress/query pool** (`query()`, `executeSql`, `newQuery`, Java `QueryClientPool`, + `query_pool_*` keys). Java's `QuestDB` owns it; the user asked specifically about *sender* + pooling. Can be a follow-up (`IQuestDBClient` could later grow `Query()`), and `connect` + could later require ws/wss like Java. v1: ingest-only handle, all ingest protocols. + +## 9. Open questions / risks + +- **net6.0 gating.** WS sender is net7.0+. The pool core is TFM-agnostic, but SF/WS-specific + paths (Phase 4) must sit behind `#if NET7_0_OR_GREATER` (HTTP/TCP pooling must still compile + on net6.0). The two production gaps in §5 touch `QwpWebSocketSender` → already net7.0+. +- **SF capacity accounting vs. SemaphoreSlim.** A closing-but-not-yet-released slot must not be + recreated; reconcile the permit model with `closingSlots`/`leakedSlots` (hold permit until + `IsSlotLockReleased`, retire index on leak). Detail to nail in Phase 4 design. +- **`InternalsVisibleTo`.** Pool tests need internals; `net-questdb-client-tests` is already a + friend assembly — keep new pool internals `internal`, expose only `IQuestDBClient` + + `QuestDBClient` + `QuestDBClientBuilder` publicly. +- **Decorator drift.** When `ISender` gains a method, `PooledSender` must too. Add an analyzer + test (reflection over `ISender` members ⊆ `PooledSender`) to catch drift. diff --git a/src/dummy-http-server/DummyQwpServer.cs b/src/dummy-http-server/DummyQwpServer.cs index 1bb549c..a2ed093 100644 --- a/src/dummy-http-server/DummyQwpServer.cs +++ b/src/dummy-http-server/DummyQwpServer.cs @@ -67,7 +67,7 @@ public DummyQwpServer(DummyQwpServerOptions? options = null) { webHost.UseKestrel(kestrel => { - kestrel.Listen(IPAddress.Loopback, 0, listen => + kestrel.Listen(IPAddress.Loopback, _options.Port, listen => { listen.Protocols = HttpProtocols.Http1; if (_options.TlsCertificate is not null) @@ -333,6 +333,13 @@ public sealed class DummyQwpServerOptions /// HTTP path to bind. Defaults to /write/v4. public string Path { get; init; } = "/write/v4"; + /// + /// Loopback TCP port to bind. Defaults to 0 (a random ephemeral port). Set a fixed port + /// when a test must know the address before the server starts (client-before-server) or must + /// restart the server on the same address (server-restart-mid-stream). + /// + public int Port { get; init; } + /// Value to return in the X-QWP-Version response header. Set to null to omit. public string? NegotiatedVersion { get; init; } = "1"; diff --git a/src/example-qwp-ingest-auth-tls/Program.cs b/src/example-qwp-ingest-auth-tls/Program.cs index 6a85405..9c1cd19 100644 --- a/src/example-qwp-ingest-auth-tls/Program.cs +++ b/src/example-qwp-ingest-auth-tls/Program.cs @@ -59,6 +59,9 @@ await sender.Table("trades") .Column("amount", 0.001) .AtAsync(DateTime.UtcNow); +// A standalone wss:: Send drains — it flushes the buffered rows AND blocks until the server acknowledges +// them — so "Send before dispose" reliably delivers even though Dispose does not wait. (Use Flush(timeout) +// for a bounded, bool-returning drain.) await sender.SendAsync(); // Per-table durable / committed seqTxn watermarks are exposed via IQwpWebSocketSender. They diff --git a/src/example-qwp-ingest/Program.cs b/src/example-qwp-ingest/Program.cs index 70a2c9e..2b9106b 100644 --- a/src/example-qwp-ingest/Program.cs +++ b/src/example-qwp-ingest/Program.cs @@ -33,7 +33,8 @@ // addr host:port (default port 9000, shared with HTTP) // auto_flush_rows rows before an automatic flush is triggered (default 1000 for ws) // auto_flush_interval milliseconds before an automatic flush (default 100 for ws) -// close_timeout ms to wait for in-flight ACKs on Dispose / Ping (default 5000) +// close_flush_timeout_millis drain timeout (default 60000) for a standalone Send() and the no-arg +// Flush(). NOTE: Dispose does NOT drain — a standalone Send() delivers on its own. // request_durable_ack on/off — opt in to per-table durable seqTxn watermarks // username/password Basic auth, or // token Bearer auth @@ -53,6 +54,9 @@ await sender.Table("trades") .Column("amount", 0.001) .AtAsync(DateTime.UtcNow); +// A standalone ws:: Send drains — it flushes the buffered rows AND blocks until the server acknowledges +// them — so "Send before dispose" reliably delivers even though Dispose does not wait. (Send throws on a +// drain timeout, bounded by close_flush_timeout_millis; use Flush(timeout) for a bounded, bool-returning drain.) await sender.SendAsync(); // When `request_durable_ack=on` is set, the WebSocket sender exposes per-table seqTxn watermarks diff --git a/src/example-sender-pool/Program.cs b/src/example-sender-pool/Program.cs new file mode 100644 index 0000000..925c507 --- /dev/null +++ b/src/example-sender-pool/Program.cs @@ -0,0 +1,62 @@ +/******************************************************************************* + * ___ _ ____ ____ + * / _ \ _ _ ___ ___| |_| _ \| __ ) + * | | | | | | |/ _ \/ __| __| | | | _ \ + * | |_| | |_| | __/\__ \ |_| |_| | |_) | + * \__\_\\__,_|\___||___/\__|____/|____/ + * + * Copyright (c) 2014-2019 Appsicle + * Copyright (c) 2019-2026 QuestDB + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + ******************************************************************************/ + +using System; +using System.Linq; +using System.Threading.Tasks; +using QuestDB; + +// A single ISender is not thread-safe. To ingest from many threads, construct one shared +// QuestDBClient pool and borrow a sender per unit of work. Call Send()/SendAsync() to send your rows; +// disposing the borrowed sender does NOT send — it discards anything un-sent and returns the sender to +// the pool (the underlying connection stays open and is reused). + +await using var client = QuestDBClient.Builder() + .FromConfig("http::addr=localhost:9000;") + .SenderPoolMin(2) + .SenderPoolMax(8) + .AcquireTimeout(TimeSpan.FromSeconds(5)) + .Build(); + +// Fan out work across threads; each borrows independently from the shared pool. +await Parallel.ForEachAsync(Enumerable.Range(0, 100), async (i, ct) => +{ + using var sender = client.BorrowSender(); + sender.Table("trades") + .Symbol("symbol", "ETH-USD") + .Column("price", 2615.54 + i) + .Column("amount", 0.00044); + await sender.AtAsync(DateTime.UtcNow, ct); + await sender.SendAsync(ct); +}); // each `using` returns its sender to the pool + +// Pool-wide drain barrier: flush every pooled sender and block until the server has acknowledged. +// Call it once all borrowed senders have been returned (as here). Over HTTP it returns immediately; +// over ws:: it waits for the ACK watermark, so `true` means every row has landed. +var delivered = await client.FlushAsync(TimeSpan.FromSeconds(30)); + +Console.WriteLine($"Done (delivered={delivered}). pool: {client.AvailableSenderCount} idle / {client.TotalSenderCount} total"); + +// Borrow one sender per unit of work and dispose it (a `using` block) to return it to the pool. +// A single sender is not thread-safe, so never share a borrowed sender across threads. diff --git a/src/example-sender-pool/example-sender-pool.csproj b/src/example-sender-pool/example-sender-pool.csproj new file mode 100644 index 0000000..19e4851 --- /dev/null +++ b/src/example-sender-pool/example-sender-pool.csproj @@ -0,0 +1,17 @@ + + + + QuestDBDemo + enable + 10 + QuestDB client - Sender Pool Example + Sharing a pooled QuestDBClient across producer threads + net6.0;net7.0;net8.0;net9.0;net10.0 + Exe + + + + + + + diff --git a/src/net-questdb-client-tests/HttpTests.cs b/src/net-questdb-client-tests/HttpTests.cs index d5da2d1..a5c7551 100644 --- a/src/net-questdb-client-tests/HttpTests.cs +++ b/src/net-questdb-client-tests/HttpTests.cs @@ -105,7 +105,6 @@ await sender.Table("metrics") .ColumnDecimal64("d64_rounded", 12.345m, 2) // half away from zero -> 12.35 .ColumnDecimal128("d128", 12.34m, 2) // raw-limb overloads reconstruct a System.Decimal for the ILP encoder. - .ColumnDecimal64("d64_lo", 100L, 2) // 1.00 .ColumnDecimal128("d128_lo", 255L, 0L, 0) // 255 .ColumnDecimal256("d256_lo", -1L, -1L, -1L, -1L, 0) // -1 .AtAsync(new DateTime(1970, 01, 01, 0, 0, 1)); @@ -116,7 +115,6 @@ await sender.Table("metrics") DecimalTestHelpers.AssertDecimalField(buffer, "d64", 2, new byte[] { 0x04, 0xD2 }); // 1234 DecimalTestHelpers.AssertDecimalField(buffer, "d64_rounded", 2, new byte[] { 0x04, 0xD3 }); // 1235 DecimalTestHelpers.AssertDecimalField(buffer, "d128", 2, new byte[] { 0x04, 0xD2 }); - DecimalTestHelpers.AssertDecimalField(buffer, "d64_lo", 2, new byte[] { 0x64 }); // 100 DecimalTestHelpers.AssertDecimalField(buffer, "d128_lo", 0, new byte[] { 0x00, 0xFF }); // 255 (positive, high bit set) DecimalTestHelpers.AssertDecimalField(buffer, "d256_lo", 0, new byte[] { 0xFF }); // -1 await server.StopAsync(); diff --git a/src/net-questdb-client-tests/PoolOptionsTests.cs b/src/net-questdb-client-tests/PoolOptionsTests.cs new file mode 100644 index 0000000..88f4bda --- /dev/null +++ b/src/net-questdb-client-tests/PoolOptionsTests.cs @@ -0,0 +1,109 @@ +/******************************************************************************* + * ___ _ ____ ____ + * / _ \ _ _ ___ ___| |_| _ \| __ ) + * | | | | | | |/ _ \/ __| __| | | | _ \ + * | |_| | |_| | __/\__ \ |_| |_| | |_) | + * \__\_\\__,_|\___||___/\__|____/|____/ + * + * Copyright (c) 2014-2019 Appsicle + * Copyright (c) 2019-2026 QuestDB + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + ******************************************************************************/ + +using NUnit.Framework; +using QuestDB.Utils; + +namespace net_questdb_client_tests; + +/// +/// Phase 0: connection-pool config keys on . +/// +public class PoolOptionsTests +{ + [Test] + public void Defaults() + { + var o = new SenderOptions("http::addr=localhost:9000;"); + Assert.Multiple(() => + { + Assert.That(o.sender_pool_min, Is.EqualTo(1)); + Assert.That(o.sender_pool_max, Is.EqualTo(4)); + Assert.That(o.acquire_timeout_ms, Is.EqualTo(TimeSpan.FromSeconds(5))); + Assert.That(o.idle_timeout_ms, Is.EqualTo(TimeSpan.FromSeconds(60))); + Assert.That(o.max_lifetime_ms, Is.EqualTo(TimeSpan.FromMinutes(30))); + Assert.That(o.housekeeper_interval_ms, Is.EqualTo(TimeSpan.FromSeconds(5))); + }); + } + + [Test] + public void ParseExplicitValues() + { + var o = new SenderOptions( + "http::addr=localhost:9000;sender_pool_min=2;sender_pool_max=8;acquire_timeout_ms=1000;" + + "idle_timeout_ms=30000;max_lifetime_ms=120000;housekeeper_interval_ms=250;"); + Assert.Multiple(() => + { + Assert.That(o.sender_pool_min, Is.EqualTo(2)); + Assert.That(o.sender_pool_max, Is.EqualTo(8)); + Assert.That(o.acquire_timeout_ms, Is.EqualTo(TimeSpan.FromSeconds(1))); + Assert.That(o.idle_timeout_ms, Is.EqualTo(TimeSpan.FromSeconds(30))); + Assert.That(o.max_lifetime_ms, Is.EqualTo(TimeSpan.FromSeconds(120))); + Assert.That(o.housekeeper_interval_ms, Is.EqualTo(TimeSpan.FromMilliseconds(250))); + }); + } + + [TestCase("http::addr=localhost:9000;")] + [TestCase("tcp::addr=localhost:9009;")] + [TestCase("ws::addr=localhost:9000;")] + public void AcceptedOnEveryScheme(string confStr) + { + // Protocol-agnostic: a plain Sender must accept (and ignore) the pool keys without tripping + // the unknown-key check or the WS-only / ILP-only guards. + var o = new SenderOptions(confStr + "sender_pool_max=8;acquire_timeout_ms=1000;"); + Assert.That(o.sender_pool_max, Is.EqualTo(8)); + } + + [Test] + public void ToStringExcludesPoolKeysAndRoundTrips() + { + var o = new SenderOptions("http::addr=localhost:9000;sender_pool_max=8;acquire_timeout_ms=1000;"); + var s = o.ToString(); + Assert.That(s, Does.Not.Contain("sender_pool_max")); + Assert.That(s, Does.Not.Contain("acquire_timeout_ms")); + // The serialized sender stays byte-identical to one that never set pool keys. + Assert.That(s, Is.EqualTo(new SenderOptions("http::addr=localhost:9000;").ToString())); + } + + [TestCase("sender_pool_min=5;sender_pool_max=2;", "must be ≤")] + [TestCase("sender_pool_max=0;", "sender_pool_max")] + [TestCase("sender_pool_min=-1;sender_pool_max=4;", "sender_pool_min")] + [TestCase("housekeeper_interval_ms=50;", "housekeeper_interval_ms")] + [TestCase("idle_timeout_ms=0;", "idle_timeout_ms")] + [TestCase("max_lifetime_ms=0;", "max_lifetime_ms")] + public void InvalidCombinationsThrow(string keys, string expectedFragment) + { + var ex = Assert.Throws(() => + _ = new SenderOptions("http::addr=localhost:9000;" + keys)); + Assert.That(ex!.Message, Does.Contain(expectedFragment)); + } + + [Test] + public void AcquireTimeoutZeroIsAllowed() + { + // Zero is the deliberate non-blocking-try opt-out. + var o = new SenderOptions("http::addr=localhost:9000;acquire_timeout_ms=0;"); + Assert.That(o.acquire_timeout_ms, Is.EqualTo(TimeSpan.Zero)); + } +} diff --git a/src/net-questdb-client-tests/Pooling/FakeQueryClient.cs b/src/net-questdb-client-tests/Pooling/FakeQueryClient.cs new file mode 100644 index 0000000..23c3211 --- /dev/null +++ b/src/net-questdb-client-tests/Pooling/FakeQueryClient.cs @@ -0,0 +1,162 @@ +/******************************************************************************* + * ___ _ ____ ____ + * / _ \ _ _ ___ ___| |_| _ \| __ ) + * | | | | | | |/ _ \/ __| __| | | | _ \ + * | |_| | |_| | __/\__ \ |_| |_| | |_) | + * \__\_\\__,_|\___||___/\__|____/|____/ + * + * Copyright (c) 2014-2019 Appsicle + * Copyright (c) 2019-2026 QuestDB + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + ******************************************************************************/ + +#if NET7_0_OR_GREATER +using QuestDB.Enums; +using QuestDB.Pooling; +using QuestDB.Qwp.Query; +using QuestDB.Senders; +using QuestDB.Utils; + +namespace net_questdb_client_tests.Pooling; + +/// +/// A no-op used to unit-test the query pool without a live server. +/// Tracks execute / dispose / cancel counts and can simulate failure, hard cancellation, a sticky +/// terminal state, or a blocking in-flight query (via ). +/// +internal sealed class FakeQueryClient : IQwpQueryClient, IPooledQueryClientInner +{ + public int ExecuteCount; + public int DisposeCount; + public int CancelCount; + public string? LastSql; + + public bool ThrowOnExecute; + public bool CancelOnExecute; + public bool TerminalOrDisposed; + + // When set, an in-flight ExecuteAsync parks on this until the test completes it. + public TaskCompletionSource? Gate; + + // When set, CancelRequest signals CancelRequestEntered (with the rid it was called with) and + // then parks on this before applying — lets a test hold a cancel dispatch across a + // return-and-re-borrow to reproduce the stale-cancel race deterministically. + public TaskCompletionSource? CancelGate; + public readonly TaskCompletionSource CancelRequestEntered = + new(TaskCreationOptions.RunContinuationsAsynchronously); + + // Mimics the real client's per-execution request ids: unique, monotonic, -1 while idle. + private long _nextRid; + private long _currentRid = -1; + + public bool Disposed => Volatile.Read(ref DisposeCount) > 0; + + public QwpServerInfo? ServerInfo => null; + public int NegotiatedVersion => 1; + public string? NegotiatedCompression => null; + public bool WasLastCloseTimedOut => false; + + public bool IsTerminalOrDisposed => TerminalOrDisposed || Disposed; + + public void Execute(string sql, QwpColumnBatchHandler handler) => RunSync(sql); + + public void Execute(string sql, QwpBindSetter binds, QwpColumnBatchHandler handler) => RunSync(sql); + + public Task ExecuteAsync(string sql, QwpColumnBatchHandler handler, CancellationToken cancellationToken = default) => + RunAsync(sql, cancellationToken); + + public Task ExecuteAsync(string sql, QwpBindSetter binds, QwpColumnBatchHandler handler, + CancellationToken cancellationToken = default) => + RunAsync(sql, cancellationToken); + + public void Cancel() => Interlocked.Increment(ref CancelCount); + + public long CurrentRequestId => Interlocked.Read(ref _currentRid); + + public void CancelRequest(long requestId) + { + CancelRequestEntered.TrySetResult(requestId); + CancelGate?.Task.GetAwaiter().GetResult(); + if (requestId >= 0 && requestId == Interlocked.Read(ref _currentRid)) + { + Interlocked.Increment(ref CancelCount); + } + } + + public void Dispose() => Interlocked.Increment(ref DisposeCount); + + public ValueTask DisposeAsync() + { + Dispose(); + return ValueTask.CompletedTask; + } + + private void RunSync(string sql) + { + Interlocked.Increment(ref ExecuteCount); + LastSql = sql; + Interlocked.Exchange(ref _currentRid, Interlocked.Increment(ref _nextRid)); + try + { + if (CancelOnExecute) + { + throw new OperationCanceledException(); + } + + if (ThrowOnExecute) + { + throw new IngressError(ErrorCode.SocketError, "fake execute failure"); + } + } + finally + { + Interlocked.Exchange(ref _currentRid, -1); + } + } + + private async Task RunAsync(string sql, CancellationToken ct) + { + Interlocked.Increment(ref ExecuteCount); + LastSql = sql; + Interlocked.Exchange(ref _currentRid, Interlocked.Increment(ref _nextRid)); + try + { + if (Gate is not null) + { + await Gate.Task.WaitAsync(ct).ConfigureAwait(false); + } + + if (CancelOnExecute) + { + throw new OperationCanceledException(); + } + + if (ThrowOnExecute) + { + throw new IngressError(ErrorCode.SocketError, "fake execute failure"); + } + } + finally + { + Interlocked.Exchange(ref _currentRid, -1); + } + } +} + +/// Minimal for tests; all callbacks are inherited no-ops. +internal sealed class NoopQueryHandler : QwpColumnBatchHandler +{ +} +#endif diff --git a/src/net-questdb-client-tests/Pooling/FakeQwpSender.cs b/src/net-questdb-client-tests/Pooling/FakeQwpSender.cs new file mode 100644 index 0000000..93adf6f --- /dev/null +++ b/src/net-questdb-client-tests/Pooling/FakeQwpSender.cs @@ -0,0 +1,75 @@ +/******************************************************************************* + * ___ _ ____ ____ + * / _ \ _ _ ___ ___| |_| _ \| __ ) + * | | | | | | |/ _ \/ __| __| | | | _ \ + * | |_| | |_| | __/\__ \ |_| |_| | |_) | + * \__\_\\__,_|\___||___/\__|____/|____/ + * + * Copyright (c) 2014-2019 Appsicle + * Copyright (c) 2019-2026 QuestDB + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + ******************************************************************************/ + +using QuestDB.Senders; + +namespace net_questdb_client_tests.Pooling; + +/// +/// A that also implements , standing in +/// for a real ws:: sender so pool tests can assert that a borrowed handle exposes (and +/// forwards) the QWP surface exactly when the pooled inner does. +/// +internal sealed class FakeQwpSender : FakeSender, IQwpWebSocketSender +{ + public int PingCount; + + public FakeQwpSender(int slotIndex) : base(slotIndex) + { + } + + public long GetHighestAckedSeqTxn(string tableName) => -1; + public long GetHighestDurableSeqTxn(string tableName) => -1; + public void Ping(CancellationToken ct = default) => Interlocked.Increment(ref PingCount); + + public ValueTask PingAsync(CancellationToken ct = default) + { + Ping(ct); + return ValueTask.CompletedTask; + } + + public long AckedFsn => -1; + public Task FlushAndGetSequenceAsync(CancellationToken ct = default) => Task.FromResult(-1L); + + public Task AwaitAckedFsnAsync(long targetFsn, TimeSpan timeout, CancellationToken ct = default) => + Task.FromResult(true); + + public IQwpWebSocketSender ColumnBinary(ReadOnlySpan name, ReadOnlySpan value) => this; + public IQwpWebSocketSender ColumnIPv4(ReadOnlySpan name, System.Net.IPAddress addr) => this; + public IQwpWebSocketSender ColumnByte(ReadOnlySpan name, sbyte value) => this; + public IQwpWebSocketSender ColumnShort(ReadOnlySpan name, short value) => this; + public IQwpWebSocketSender ColumnFloat(ReadOnlySpan name, float value) => this; + public IQwpWebSocketSender ColumnDate(ReadOnlySpan name, long millisSinceEpoch) => this; + public IQwpWebSocketSender ColumnGeohash(ReadOnlySpan name, ulong hash, int precisionBits) => this; + public IQwpWebSocketSender ColumnLong256(ReadOnlySpan name, System.Numerics.BigInteger value) => this; + + public long DroppedErrorNotifications => 0; + public long DroppedConnectionNotifications => 0; + public long TotalErrorNotificationsDelivered => 0; + public long TotalFramesSent => 0; + public long TotalAcks => 0; + public long TotalServerErrors => 0; + public long TotalReconnectAttempts => 0; + public long TotalReconnectsSucceeded => 0; +} diff --git a/src/net-questdb-client-tests/Pooling/FakeSender.cs b/src/net-questdb-client-tests/Pooling/FakeSender.cs new file mode 100644 index 0000000..2941336 --- /dev/null +++ b/src/net-questdb-client-tests/Pooling/FakeSender.cs @@ -0,0 +1,239 @@ +/******************************************************************************* + * ___ _ ____ ____ + * / _ \ _ _ ___ ___| |_| _ \| __ ) + * | | | | | | |/ _ \/ __| __| | | | _ \ + * | |_| | |_| | __/\__ \ |_| |_| | |_) | + * \__\_\\__,_|\___||___/\__|____/|____/ + * + * Copyright (c) 2014-2019 Appsicle + * Copyright (c) 2019-2026 QuestDB + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + ******************************************************************************/ + +using QuestDB.Enums; +using QuestDB.Pooling; +using QuestDB.Senders; +using QuestDB.Utils; + +namespace net_questdb_client_tests.Pooling; + +/// +/// A no-op used to unit-test the pool without a live server. Tracks +/// dispose / flush counts and can be told to throw on flush to exercise the broken-sender path. +/// Implements so SF slot reclaim / retire can be exercised, and +/// so the staged-uncommitted-rows discard path can be too. +/// +internal class FakeSender : ISender, IPooledSlotSender, IPooledTransactionalSender, IPooledDrainAwareSender +{ + private static readonly SenderOptions Opts = new("http::addr=localhost:9000;"); + + public int SlotIndex { get; } + public int DisposeCount; + public int SendCount; + public int ClearCount; + public int FlushCount; + public bool ThrowOnSend; + public bool ThrowOnClear; + public bool ThrowOnFlush; + public bool ThrowOnDispose; + + // When set, Send() parks here (simulating an in-flight flush) until the test releases it. Lets a test + // hold a borrowed sender mid-flush while another thread closes the pool. + public ManualResetEventSlim? SendGate; + + // When set, Clear() parks here — the return path Dispose() now runs (it discards un-sent rows via + // Clear instead of sending). Lets a test hold a borrowed sender mid-return while another thread closes. + public ManualResetEventSlim? ClearGate; + + // When set, Dispose() signals DisposeEntered then parks here until released — simulating a slow WS+SF + // teardown so a test can observe the pool's reap-dispose window (slot index held, sender already out of + // _all/_available) and race a concurrent Borrow against it. + public ManualResetEventSlim? DisposeGate; + public ManualResetEventSlim? DisposeEntered; + + // True while Send() is executing; flipped by Send(). Used to detect a Dispose racing an in-flight Send. + public volatile bool SendInProgress; + + // True while Clear() is executing; flipped by Clear(). Detects a Dispose racing an in-flight return. + public volatile bool ClearInProgress; + + // Set true if Dispose() was ever called while Send() was in progress — the exact non-thread-safe + // race a borrowed sender must never be subjected to by pool teardown. + public volatile bool DisposedDuringSend; + + // Set true if Dispose() was ever called while Clear() (the return path) was in progress. + public volatile bool DisposedDuringClear; + + // Pretend this sender holds a slot lock that does (true) or does not (false) release on dispose. + // Volatile: SF concurrency stress tests flip this from a housekeeper thread while the pool reads it + // under its gate on another thread. + public volatile bool SlotLockReleased = true; + + public FakeSender(int slotIndex) + { + SlotIndex = slotIndex; + } + + public bool IsSlotLockReleased => SlotLockReleased; + + // Pretend this sender's cursor-engine ring still holds un-acked frames (false) or has fully drained + // (true). Volatile: reaper tests flip it from the test thread while the pool reads it under its gate. + // Defaults to drained so existing reaper tests (which expect prompt reaping) are unaffected. + public volatile bool FullyDrained = true; + + public bool IsFullyDrained => FullyDrained; + + // Pretend this sender has transactional rows staged server-side awaiting commit (ws transaction=on + // after a deferred auto-flush). The return path must discard such a sender, not re-pool it. + public bool HasUncommittedDeferredRows { get; set; } + + public bool Disposed => Volatile.Read(ref DisposeCount) > 0; + + public int Length => 0; + public int RowCount => 0; + public bool WithinTransaction => false; + public DateTime LastFlush => DateTime.MinValue; + public SenderOptions Options => Opts; + + public void Dispose() + { + if (SendInProgress) + { + DisposedDuringSend = true; + } + + if (ClearInProgress) + { + DisposedDuringClear = true; + } + + DisposeEntered?.Set(); + DisposeGate?.Wait(); + + Interlocked.Increment(ref DisposeCount); + if (ThrowOnDispose) + { + throw new IngressError(ErrorCode.ServerFlushError, "fake dispose failure"); + } + } + + public ValueTask DisposeAsync() + { + Dispose(); + return ValueTask.CompletedTask; + } + + public void Send(CancellationToken ct = default) + { + SendInProgress = true; + try + { + SendGate?.Wait(ct); + Interlocked.Increment(ref SendCount); + } + finally + { + SendInProgress = false; + } + + if (ThrowOnSend) + { + throw new IngressError(ErrorCode.ServerFlushError, "fake flush failure"); + } + } + + public Task SendAsync(CancellationToken ct = default) + { + Send(ct); + return Task.CompletedTask; + } + + public bool Flush(TimeSpan timeout, CancellationToken ct = default) + { + Interlocked.Increment(ref FlushCount); + if (ThrowOnFlush) + { + throw new IngressError(ErrorCode.ServerFlushError, "fake flush failure"); + } + + return true; + } + + public Task FlushAsync(TimeSpan timeout, CancellationToken ct = default) => Task.FromResult(Flush(timeout, ct)); + + public ISender Transaction(ReadOnlySpan tableName) => this; + public void Rollback() { } + public Task CommitAsync(CancellationToken ct = default) => Task.CompletedTask; + public void Commit(CancellationToken ct = default) { } + public ISender Table(ReadOnlySpan name) => this; + public ISender Symbol(ReadOnlySpan name, ReadOnlySpan value) => this; + public ISender Column(ReadOnlySpan name, ReadOnlySpan value) => this; + public ISender Column(ReadOnlySpan name, long value) => this; + public ISender Column(ReadOnlySpan name, int value) => this; + public ISender Column(ReadOnlySpan name, bool value) => this; + public ISender Column(ReadOnlySpan name, double value) => this; + public ISender Column(ReadOnlySpan name, DateTime value) => this; + public ISender Column(ReadOnlySpan name, DateTimeOffset value) => this; + public ISender ColumnNanos(ReadOnlySpan name, long timestampNanos) => this; + public ISender Column(ReadOnlySpan name, decimal value) => this; + public ISender ColumnDecimal64(ReadOnlySpan name, decimal value, byte scale) => this; + public ISender ColumnDecimal128(ReadOnlySpan name, decimal value, byte scale) => this; + public ISender ColumnDecimal256(ReadOnlySpan name, decimal value, byte scale) => this; + public ISender ColumnDecimal128(ReadOnlySpan name, long lo, long hi, byte scale) => this; + public ISender ColumnDecimal256(ReadOnlySpan name, long l0, long l1, long l2, long l3, byte scale) => this; + public ISender Column(ReadOnlySpan name, Guid value) => this; + public ISender Column(ReadOnlySpan name, char value) => this; + public ISender Column(ReadOnlySpan name, IEnumerable value, IEnumerable shape) where T : struct => this; + public ISender Column(ReadOnlySpan name, Array value) => this; + public ISender Column(ReadOnlySpan name, ReadOnlySpan value) where T : struct => this; + + public ValueTask AtAsync(DateTime value, CancellationToken ct = default) => ValueTask.CompletedTask; + public ValueTask AtAsync(DateTimeOffset value, CancellationToken ct = default) => ValueTask.CompletedTask; + public ValueTask AtAsync(long value, CancellationToken ct = default) => ValueTask.CompletedTask; + + [Obsolete("Not compatible with deduplication.")] + public ValueTask AtNowAsync(CancellationToken ct = default) => ValueTask.CompletedTask; + + public void At(DateTime value, CancellationToken ct = default) { } + public void At(DateTimeOffset value, CancellationToken ct = default) { } + public void At(long value, CancellationToken ct = default) { } + public ValueTask AtNanosAsync(long timestampNanos, CancellationToken ct = default) => ValueTask.CompletedTask; + public void AtNanos(long timestampNanos, CancellationToken ct = default) { } + + [Obsolete("Not compatible with deduplication.")] + public void AtNow(CancellationToken ct = default) { } + + public void Truncate() { } + public void CancelRow() { } + + public void Clear() + { + ClearInProgress = true; + try + { + ClearGate?.Wait(); + Interlocked.Increment(ref ClearCount); + } + finally + { + ClearInProgress = false; + } + + if (ThrowOnClear) + { + throw new IngressError(ErrorCode.ServerFlushError, "fake clear failure"); + } + } +} diff --git a/src/net-questdb-client-tests/Pooling/LazyConnectRecoveryTests.cs b/src/net-questdb-client-tests/Pooling/LazyConnectRecoveryTests.cs new file mode 100644 index 0000000..229d9e6 --- /dev/null +++ b/src/net-questdb-client-tests/Pooling/LazyConnectRecoveryTests.cs @@ -0,0 +1,180 @@ +/******************************************************************************* + * ___ _ ____ ____ + * / _ \ _ _ ___ ___| |_| _ \| __ ) + * | | | | | | |/ _ \/ __| __| | | | _ \ + * | |_| | |_| | __/\__ \ |_| |_| | |_) | + * \__\_\\__,_|\___||___/\__|____/|____/ + * + * Copyright (c) 2014-2019 Appsicle + * Copyright (c) 2019-2026 QuestDB + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + ******************************************************************************/ + +#if NET7_0_OR_GREATER + +using System.Buffers.Binary; +using System.Net; +using System.Net.Sockets; +using NUnit.Framework; +using QuestDB; +using QuestDB.Enums; +using QuestDB.Qwp; +using dummy_http_server; + +namespace net_questdb_client_tests.Pooling; + +/// +/// End-to-end lazy_connect=true recovery lifecycle against an in-process QWP server whose +/// lifetime the test controls: start the client with the server down (writes buffer), bring the +/// server up (buffered writes drain), and restart the server mid-stream (the client reconnects to a +/// fresh instance on the same port and replays un-acked frames). Mirrors the Java client's +/// QuestDBServerRecoveryTest. +/// +/// The server ACKs each frame with a per-connection 0-based wire sequence (a fresh instance restarts +/// at 0), matching the cursor engine's fsnAtZero mapping so a restarted server replays cleanly. +/// +[TestFixture] +public class LazyConnectRecoveryTests +{ + // Single reused pooled sender so both batches flow through the same reconnecting engine/ring, and + // small reconnect backoffs so recovery is quick. auto_flush=off keeps flushing under test control. + private static string ConfFor(int port) => + $"ws::addr=127.0.0.1:{port};lazy_connect=true;auto_flush=off;" + + "sender_pool_min=1;sender_pool_max=1;" + + "reconnect_initial_backoff_millis=50;reconnect_max_backoff_millis=200;" + + "reconnect_max_duration_millis=60000;"; + + [Test] + public async Task StartClientBeforeServer_BufferedWriteDeliversOnceServerIsUp() + { + var port = FreeLoopbackPort(); + + // The server is DOWN. lazy_connect makes Build() return promptly (ingest connects async) and the + // default sender_pool_min=1 pre-warm does not block/throw on the missing server. + using var client = QuestDBClient.Connect(ConfFor(port)); + + // Buffer a write while the server is down: Send() flushes to the ring (non-blocking); nothing is + // on the wire yet. + using (var s = client.BorrowSender()) + { + s.Table("recovery").Column("v", 1L).AtNow(); + s.Send(); + } + + // The server appears. + await using var server = await StartAckingServerAsync(port); + + // Flush drains: the engine connects to the now-up server, replays the buffered frame and awaits + // its ACK. + Assert.That(client.Flush(TimeSpan.FromSeconds(30)), Is.True, + "the write buffered while the server was down must drain once it is up"); + await WaitFor(() => server.ReceivedFrames.Count >= 1, 5000); + Assert.That(server.ReceivedFrames.Count, Is.EqualTo(1), + "the server received exactly the one buffered frame (no duplicate delivery)"); + } + + [Test] + public async Task StartBeforeServerThenRestartMidStream_ClientBuffersReconnectsAndReplays() + { + var port = FreeLoopbackPort(); + + // Start the client with NO server present (lazy_connect). + using var client = QuestDBClient.Connect(ConfFor(port)); + + // Buffer batch 1 while the server is still down. + using (var s = client.BorrowSender()) + { + s.Table("recovery").Column("v", 1L).AtNow(); + s.Send(); + } + + // Bring the server up: the buffered batch 1 drains. + var server1 = await StartAckingServerAsync(port); + Assert.That(client.Flush(TimeSpan.FromSeconds(30)), Is.True, "batch 1 drains once the server is up"); + await WaitFor(() => server1.ReceivedFrames.Count >= 1, 5000); + Assert.That(server1.ReceivedFrames.Count, Is.EqualTo(1), "first server received exactly batch 1"); + + // Restart the server in the middle: stop it, buffer a write while it is down, then bring a fresh + // instance up on the SAME port. + await server1.DisposeAsync(); + + using (var s = client.BorrowSender()) + { + s.Table("recovery").Column("v", 2L).AtNow(); + s.Send(); + } + + await using var server2 = await StartAckingServerAsync(port); + + // The engine reconnects to the fresh server and replays only the un-acked batch 2 (batch 1 was + // already acked, so the cursor does not rewind past it). + Assert.That(client.Flush(TimeSpan.FromSeconds(30)), Is.True, + "batch 2 drains to the restarted server after reconnect"); + await WaitFor(() => server2.ReceivedFrames.Count >= 1, 5000); + Assert.That(server2.ReceivedFrames.Count, Is.EqualTo(1), + "the restarted server received exactly the replayed batch 2 (the acked batch 1 is not re-sent)"); + } + + // ---- helpers ---- + + private static async Task StartAckingServerAsync(int port) + { + // Per-instance 0-based wire sequence. Each phase uses a fresh instance serving one stable + // connection, so this equals the per-connection sequence the cursor engine's fsnAtZero expects. + long nextWireSeq = 0; + var server = new DummyQwpServer(new DummyQwpServerOptions + { + Port = port, + FrameHandler = _ => BuildOkAck(Interlocked.Increment(ref nextWireSeq) - 1), + }); + await server.StartAsync(); + return server; + } + + private static byte[] BuildOkAck(long sequence) + { + var bytes = new byte[QwpConstants.OffsetTableCountInOkAck + 2]; + bytes[0] = (byte)QwpStatusCode.Ok; + BinaryPrimitives.WriteInt64LittleEndian(bytes.AsSpan(1, 8), sequence); + BinaryPrimitives.WriteUInt16LittleEndian(bytes.AsSpan(QwpConstants.OffsetTableCountInOkAck, 2), 0); + return bytes; + } + + // Bind :0 to grab a free loopback port, then release it so the client can be configured with the + // address before the server exists and so a restarted server can re-bind it. + private static int FreeLoopbackPort() + { + var listener = new TcpListener(IPAddress.Loopback, 0); + listener.Start(); + try + { + return ((IPEndPoint)listener.LocalEndpoint).Port; + } + finally + { + listener.Stop(); + } + } + + private static async Task WaitFor(Func predicate, int timeoutMs) + { + var deadline = Environment.TickCount64 + timeoutMs; + while (!predicate() && Environment.TickCount64 < deadline) + { + await Task.Delay(20); + } + } +} +#endif diff --git a/src/net-questdb-client-tests/Pooling/LazyConnectTests.cs b/src/net-questdb-client-tests/Pooling/LazyConnectTests.cs new file mode 100644 index 0000000..68b93a8 --- /dev/null +++ b/src/net-questdb-client-tests/Pooling/LazyConnectTests.cs @@ -0,0 +1,347 @@ +/******************************************************************************* + * ___ _ ____ ____ + * / _ \ _ _ ___ ___| |_| _ \| __ ) + * | | | | | | |/ _ \/ __| __| | | | _ \ + * | |_| | |_| | __/\__ \ |_| |_| | |_) | + * \__\_\\__,_|\___||___/\__|____/|____/ + * + * Copyright (c) 2014-2019 Appsicle + * Copyright (c) 2019-2026 QuestDB + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + ******************************************************************************/ + +using NUnit.Framework; +using QuestDB; +using QuestDB.Enums; +using QuestDB.Senders; +using QuestDB.Utils; + +namespace net_questdb_client_tests.Pooling; + +/// +/// lazy_connect=true: tolerant startup for the handle. The +/// ingest side connects asynchronously and the read pool defaults to query_pool_min=0, so the +/// handle builds even while the server is down. Blocking-startup knobs are rejected up front. +/// +public class LazyConnectTests +{ + // ---- connect-string parsing ---- + + [Test] + public void FlagParsesAndDefaultsOff() + { + Assert.Multiple(() => + { + Assert.That(new SenderOptions("ws::addr=localhost:9000;").lazy_connect, Is.False); + Assert.That(new SenderOptions("ws::addr=localhost:9000;lazy_connect=true;").lazy_connect, Is.True); + Assert.That(new SenderOptions("ws::addr=localhost:9000;lazy_connect=on;").lazy_connect, Is.True); + Assert.That(new SenderOptions("ws::addr=localhost:9000;lazy_connect=off;").lazy_connect, Is.False); + }); + } + + [TestCase("http::addr=localhost:9000;")] + [TestCase("tcp::addr=localhost:9009;")] + [TestCase("ws::addr=localhost:9000;")] + public void AcceptedOnEveryScheme(string confStr) + { + // Pool key: a plain Sender parses and ignores it on every scheme. + Assert.DoesNotThrow(() => _ = new SenderOptions(confStr + "lazy_connect=true;")); + } + + [Test] + public void PlainSenderIgnoresFlag() + { + Assert.DoesNotThrow(() => + { + using var s = Sender.New("http::addr=localhost:9000;lazy_connect=true;"); + }); + } + + [Test] + public void ExcludedFromToStringAndRoundTrips() + { + var o = new SenderOptions("ws::addr=localhost:9000;lazy_connect=true;"); + var s = o.ToString(); + Assert.That(s, Does.Not.Contain("lazy_connect")); + Assert.DoesNotThrow(() => _ = new SenderOptions(s)); + } + + // ---- resolution logic (QuestDBClientBuilder.ResolveLazyConnect), net-agnostic ---- + + [Test] + public void Resolve_Ws_ForcesAsyncAndDefaultsQueryPoolMinZero() + { + var o = new SenderOptions("ws::addr=localhost:9000;"); + var forceAsync = QuestDBClientBuilder.ResolveLazyConnect( + lazy: true, o, ingestIsWebSocket: true, queryPoolMinExplicit: false); + Assert.Multiple(() => + { + Assert.That(forceAsync, Is.True); + Assert.That(o.query_pool_min, Is.EqualTo(0)); + }); + } + + [Test] + public void Resolve_Http_DoesNotForceAsyncButStillDefaultsQueryPoolMinZero() + { + var o = new SenderOptions("http::addr=localhost:9000;"); + var forceAsync = QuestDBClientBuilder.ResolveLazyConnect( + lazy: true, o, ingestIsWebSocket: false, queryPoolMinExplicit: false); + Assert.Multiple(() => + { + Assert.That(forceAsync, Is.False); + Assert.That(o.query_pool_min, Is.EqualTo(0)); + }); + } + + [Test] + public void Resolve_LazyFalse_IsNoOp() + { + var o = new SenderOptions("ws::addr=localhost:9000;"); // query_pool_min default 1 + var forceAsync = QuestDBClientBuilder.ResolveLazyConnect( + lazy: false, o, ingestIsWebSocket: true, queryPoolMinExplicit: false); + Assert.Multiple(() => + { + Assert.That(forceAsync, Is.False); + Assert.That(o.query_pool_min, Is.EqualTo(1)); + }); + } + + [Test] + public void Resolve_ExplicitQueryPoolMinZero_Allowed() + { + var o = new SenderOptions("ws::addr=localhost:9000;query_pool_min=0;"); + Assert.DoesNotThrow(() => QuestDBClientBuilder.ResolveLazyConnect( + lazy: true, o, ingestIsWebSocket: true, queryPoolMinExplicit: true)); + Assert.That(o.query_pool_min, Is.EqualTo(0)); + } + + [Test] + public void Resolve_ExplicitInitialConnectAsync_Allowed() + { + var o = new SenderOptions("ws::addr=localhost:9000;initial_connect_retry=async;"); + Assert.DoesNotThrow(() => QuestDBClientBuilder.ResolveLazyConnect( + lazy: true, o, ingestIsWebSocket: true, queryPoolMinExplicit: false)); + Assert.That(o.query_pool_min, Is.EqualTo(0)); + } + + [TestCase("off")] + [TestCase("on")] + [TestCase("sync")] + public void Resolve_ExplicitBlockingInitialConnect_Throws(string mode) + { + var o = new SenderOptions($"ws::addr=localhost:9000;initial_connect_retry={mode};"); + var ex = Assert.Throws(() => QuestDBClientBuilder.ResolveLazyConnect( + lazy: true, o, ingestIsWebSocket: true, queryPoolMinExplicit: false)); + Assert.Multiple(() => + { + Assert.That(ex!.code, Is.EqualTo(ErrorCode.ConfigError)); + Assert.That(ex.Message, Does.Contain("initial_connect_retry")); + }); + } + + [Test] + public void Resolve_ExplicitQueryPoolMinPositive_Throws() + { + var o = new SenderOptions("ws::addr=localhost:9000;query_pool_min=2;"); + var ex = Assert.Throws(() => QuestDBClientBuilder.ResolveLazyConnect( + lazy: true, o, ingestIsWebSocket: true, queryPoolMinExplicit: true)); + Assert.Multiple(() => + { + Assert.That(ex!.code, Is.EqualTo(ErrorCode.ConfigError)); + Assert.That(ex.Message, Does.Contain("query_pool_min")); + }); + } + + // ---- facade build behaviour: conflicts are rejected before any pool is built (no server needed) ---- + + [Test] + public void Conflict_ExplicitQueryPoolMinInConfig_Throws() + { + var ex = Assert.Throws(() => QuestDBClient.Connect( + "ws::addr=localhost:9000;lazy_connect=true;query_pool_min=2;sender_pool_min=0;")); + Assert.Multiple(() => + { + Assert.That(ex!.code, Is.EqualTo(ErrorCode.ConfigError)); + Assert.That(ex.Message, Does.Contain("query_pool_min")); + }); + } + + [Test] + public void Conflict_ExplicitBlockingInitialConnectInConfig_Throws() + { + var ex = Assert.Throws(() => QuestDBClient.Connect( + "ws::addr=localhost:9000;lazy_connect=true;initial_connect_retry=on;sender_pool_min=0;")); + Assert.Multiple(() => + { + Assert.That(ex!.code, Is.EqualTo(ErrorCode.ConfigError)); + Assert.That(ex.Message, Does.Contain("initial_connect_retry")); + }); + } + +#if NET7_0_OR_GREATER + [Test] + public void Conflict_QueryPoolMinViaBuilder_Throws() + { + var ex = Assert.Throws(() => QuestDBClient.Builder() + .FromConfig("ws::addr=localhost:9000;lazy_connect=true;sender_pool_min=0;") + .QueryPoolMin(2) + .Build()); + Assert.Multiple(() => + { + Assert.That(ex!.code, Is.EqualTo(ErrorCode.ConfigError)); + Assert.That(ex.Message, Does.Contain("query_pool_min")); + }); + } + + [Test] + public void Lazy_QueryPoolMinLoweredToZeroViaBuilder_DoesNotConflict() + { + // The config string asks for query_pool_min=2, but the higher-precedence builder call overrides it + // back to 0. The effective value is 0, so lazy_connect must not fire the eager-read-pool conflict. + using var h = QuestDBClient.Builder() + .FromConfig("ws::addr=localhost:9000;query_pool_min=2;sender_pool_min=0;") + .LazyConnect() + .QueryPoolMin(0) + .Build(); + Assert.That(h.TotalQueryClientCount, Is.EqualTo(0)); + } + + [Test] + public void Lazy_StartsWithReadPoolEnabledButNotPrewarmed() + { + // No server: without lazy_connect the query pool would prewarm one client and fail. lazy_connect + // defaults query_pool_min to 0 so the read pool stays enabled but connects only on first borrow. + using var h = QuestDBClient.Connect("ws::addr=localhost:9000;lazy_connect=true;sender_pool_min=0;"); + Assert.Multiple(() => + { + Assert.That(h.TotalQueryClientCount, Is.EqualTo(0)); + Assert.DoesNotThrow(() => _ = h.NewQuery()); + }); + } + + [Test] + public void Lazy_ViaBuilderMethod_StartsWithoutServer() + { + using var h = QuestDBClient.Builder() + .FromConfig("ws::addr=localhost:9000;sender_pool_min=0;") + .LazyConnect() + .Build(); + Assert.That(h.TotalQueryClientCount, Is.EqualTo(0)); + } + + [Test] + public void Lazy_WithExplicitAsyncInitialConnect_StartsWithoutServer() + { + using var h = QuestDBClient.Connect( + "ws::addr=localhost:9000;lazy_connect=true;initial_connect_retry=async;sender_pool_min=0;"); + Assert.That(h.TotalQueryClientCount, Is.EqualTo(0)); + } + + [Test] + public void Lazy_IngestBorrowBuffersWithoutServer() + { + // The pooled ws sender is created async, so borrowing and appending never blocks on the down + // server (a non-async sender would block on first-connect and throw here). + using var h = QuestDBClient.Connect("ws::addr=localhost:9000;lazy_connect=true;sender_pool_min=0;"); + Assert.DoesNotThrow(() => + { + using var s = h.BorrowSender(); + s.Table("lazy_connect_test").Column("v", 1L).AtNow(); + }); + } + + [Test] + public void Lazy_DefaultSenderPoolMin_PreWarmsAsyncWithoutServer() + { + // Headline behaviour (a): with the DEFAULT sender_pool_min=1, the sender pool pre-warms one ws + // sender at construction. lazy_connect forces it async, so Build() returns promptly against a down + // server; a non-async pre-warm would block on first-connect and throw. This exercises the + // PreWarm -> CreateDefaultInner forced-async path the sender_pool_min=0 tests skip. + using var h = QuestDBClient.Connect("ws::addr=localhost:9000;lazy_connect=true;"); + Assert.Multiple(() => + { + Assert.That(h.TotalSenderCount, Is.EqualTo(1)); + Assert.That(h.TotalQueryClientCount, Is.EqualTo(0)); + }); + } + + [Test] + public void Lazy_BuilderFalseOverridesConfigTrue() + { + // `LazyConnect(false)` wins over a connect-string `lazy_connect=true` (the `??` precedence). With + // lazy disabled the conflict guard is not evaluated, so the otherwise-conflicting + // `initial_connect_retry=on` is accepted and Build() succeeds (both pools sized 0 — no server). If + // the precedence regressed to OR, lazy would stay true and this would throw a ConfigError. + Assert.DoesNotThrow(() => QuestDBClient.Builder() + .FromConfig("ws::addr=localhost:9000;lazy_connect=true;initial_connect_retry=on;" + + "query_pool_min=0;sender_pool_min=0;") + .LazyConnect(false) + .Build() + .Dispose()); + } + + [Test] + public void Conflict_IngestQueryPoolMinWithSeparateQueryConfig_Throws() + { + // The ingest string's explicit query_pool_min>0 must still be rejected under lazy_connect even when + // a separate query config (which wins for sizing) omits the key. + var ex = Assert.Throws(() => QuestDBClient.Builder() + .FromConfig("ws::addr=localhost:9000;query_pool_min=5;sender_pool_min=0;") + .QueryConfig("ws::addr=localhost:9000;") + .LazyConnect() + .Build()); + Assert.Multiple(() => + { + Assert.That(ex!.code, Is.EqualTo(ErrorCode.ConfigError)); + Assert.That(ex.Message, Does.Contain("query_pool_min")); + }); + } + + [Test] + public void Conflict_QueryConfigQueryPoolMin_Throws() + { + // A query_pool_min>0 carried by the separate query config (the winning sizing source) must also + // trip the lazy_connect conflict — dropping it would eagerly pre-warm the read pool. + var ex = Assert.Throws(() => QuestDBClient.Builder() + .FromConfig("ws::addr=localhost:9000;sender_pool_min=0;") + .QueryConfig("wss::addr=localhost:9000;query_pool_min=2;") + .LazyConnect() + .BuildPoolConfig(out _, out _)); + Assert.Multiple(() => + { + Assert.That(ex!.code, Is.EqualTo(ErrorCode.ConfigError)); + Assert.That(ex.Message, Does.Contain("query_pool_min")); + }); + } + + [Test] + public void Lazy_BareSeparateQueryConfigStillDefaultsQueryPoolMinZero() + { + // A bare separate query config must not count as an explicit query_pool_min: lazy_connect still + // defaults the read pool to 0 (no eager pre-warm) and forces the ws ingest side async. + var cfg = QuestDBClient.Builder() + .IngestConfig("ws::addr=localhost:9000;sender_pool_min=0;") + .QueryConfig("wss::addr=localhost:9000;") + .LazyConnect() + .BuildPoolConfig(out _, out var forceWsAsyncConnect); + Assert.Multiple(() => + { + Assert.That(cfg.query_pool_min, Is.EqualTo(0)); + Assert.That(forceWsAsyncConnect, Is.True); + }); + } +#endif +} diff --git a/src/net-questdb-client-tests/Pooling/PoolErrorSafetyTests.cs b/src/net-questdb-client-tests/Pooling/PoolErrorSafetyTests.cs new file mode 100644 index 0000000..371cc3c --- /dev/null +++ b/src/net-questdb-client-tests/Pooling/PoolErrorSafetyTests.cs @@ -0,0 +1,288 @@ +/******************************************************************************* + * ___ _ ____ ____ + * / _ \ _ _ ___ ___| |_| _ \| __ ) + * | | | | | | |/ _ \/ __| __| | | | _ \ + * | |_| | |_| | __/\__ \ |_| |_| | |_) | + * \__\_\\__,_|\___||___/\__|____/|____/ + * + * Copyright (c) 2014-2019 Appsicle + * Copyright (c) 2019-2026 QuestDB + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + ******************************************************************************/ + +using System.Collections.Concurrent; +using System.Diagnostics; +using NUnit.Framework; +using QuestDB.Enums; +using QuestDB.Pooling; +using QuestDB.Senders; +using QuestDB.Utils; +#if NET7_0_OR_GREATER +using QuestDB.Qwp.Query; +#endif + +namespace net_questdb_client_tests.Pooling; + +public class PoolErrorSafetyTests +{ + private static SenderPool MakePool(string keys) + { + var options = new SenderOptions("http::addr=localhost:9000;" + keys); + return new SenderPool(options, null, slot => new FakeSender(slot)); + } + + [Test] + public void CloseWakesBlockedBorrowerPromptly() + { + var pool = MakePool("sender_pool_min=0;sender_pool_max=1;acquire_timeout_ms=10000;"); + var held = pool.Borrow(); + + Exception? caught = null; + var sw = new Stopwatch(); + var t = new Thread(() => + { + try + { + pool.Borrow(); + } + catch (Exception e) + { + caught = e; + } + }); + + t.Start(); + Thread.Sleep(200); // let the borrower block on the exhausted pool + sw.Start(); + pool.Close(); + t.Join(TimeSpan.FromSeconds(5)); + sw.Stop(); + + Assert.Multiple(() => + { + Assert.That(caught, Is.InstanceOf()); + Assert.That(caught!.Message, Does.Contain("closed")); + Assert.That(sw.ElapsedMilliseconds, Is.LessThan(4000), "woke on close, not on the 10s acquire timeout"); + }); + + Assert.DoesNotThrow(() => held.Dispose()); + } + + [Test] + public void DisposingReturnedSenderTwiceDoesNotDoubleReturn() + { + var pool = MakePool("sender_pool_min=0;sender_pool_max=2;"); + try + { + var s = pool.Borrow(); + s.Dispose(); + Assert.That(pool.AvailableSize, Is.EqualTo(1)); + s.Dispose(); // second dispose is a no-op + Assert.That(pool.AvailableSize, Is.EqualTo(1), "idempotent: not pooled twice"); + } + finally + { + pool.Close(); + } + } + + [Test] + public void UsingABorrowedSenderAfterDisposeThrowsObjectDisposed() + { + // The use-after-return guard: once a borrowed handle is disposed (returned to the pool), every + // ingest member is inert. A caller hanging on to the stale reference therefore cannot reach — and + // corrupt — the entry the pool may have since lent to a different borrower. + var pool = MakePool("sender_pool_min=0;sender_pool_max=2;"); + try + { + var s = pool.Borrow(); + s.Table("t").Column("x", 1L).At(DateTime.UtcNow); // fine while borrowed + s.Dispose(); + + Assert.Multiple(() => + { + Assert.Throws(() => s.Table("t")); + Assert.Throws(() => s.Column("x", 1L)); + Assert.Throws(() => s.At(DateTime.UtcNow)); + Assert.Throws(() => s.Send()); + Assert.Throws(() => _ = s.RowCount); + Assert.Throws(() => _ = s.Options); + }); + + // A second dispose is a tolerated no-op (no double-return, no throw). + Assert.DoesNotThrow(() => s.Dispose()); + Assert.That(pool.AvailableSize, Is.EqualTo(1), "idempotent: returned exactly once"); + } + finally + { + pool.Close(); + } + } + + [Test] + public void DisposingBorrowedSenderAfterHandleCloseIsSafe() + { + var pool = MakePool("sender_pool_min=0;sender_pool_max=2;"); + var s = pool.Borrow(); + pool.Close(); + Assert.DoesNotThrow(() => s.Dispose()); + Assert.That(pool.AvailableSize, Is.EqualTo(0), "not re-pooled into a closed pool"); + } + + [Test] + public void ConcurrentBorrowReturnAndCloseDoNotThrow() + { + var pool = MakePool("sender_pool_min=2;sender_pool_max=6;acquire_timeout_ms=2000;"); + var errors = new ConcurrentQueue(); + var stop = false; + + var workers = Enumerable.Range(0, 6).Select(_ => new Thread(() => + { + while (!Volatile.Read(ref stop)) + { + try + { + var s = pool.Borrow(); + Thread.SpinWait(50); + s.Dispose(); + } + catch (IngressError) + { + // expected once the pool closes + } + catch (Exception e) + { + errors.Enqueue(e); + } + } + })).ToArray(); + + foreach (var w in workers) + { + w.Start(); + } + + Thread.Sleep(300); + pool.Close(); + Volatile.Write(ref stop, true); + foreach (var w in workers) + { + w.Join(TimeSpan.FromSeconds(5)); + } + + Assert.That(errors, Is.Empty, "no unexpected exceptions under concurrent borrow/return/close"); + } + + [Test] + public void PooledSenderFromNonWsPoolDoesNotMatchQwpCapabilityProbe() + { + // The documented probe — `if (sender is IQwpWebSocketSender ws)` — must answer per transport, + // same as a standalone Sender.New: a borrowed handle from a non-WS pool must NOT match, so + // callers never get a "capable" handle whose QWP members would only throw at runtime. + var pool = MakePool("sender_pool_min=0;sender_pool_max=1;"); + try + { + ISender s = pool.Borrow(); + Assert.That(s, Is.Not.InstanceOf(), + "a non-WS pool must hand out a handle without the QWP surface"); + s.Dispose(); + } + finally + { + pool.Close(); + } + } + + [Test] + public void PooledSenderFromWsPoolExposesQwpSurfaceAndForwards() + { + var options = new SenderOptions("ws::addr=localhost:9000;sender_pool_min=0;sender_pool_max=1;"); + var pool = new SenderPool(options, null, slot => new FakeQwpSender(slot)); + try + { + ISender s = pool.Borrow(); + Assert.That(s, Is.InstanceOf(), + "a WS pool's handle matches the QWP capability probe"); + + var qwp = (IQwpWebSocketSender)s; + qwp.Ping(); + Assert.That(qwp.ColumnByte("b", 1), Is.SameAs(s), "fluent QWP calls chain on the handle"); + Assert.That(qwp.AckedFsn, Is.EqualTo(-1)); + + s.Dispose(); + // Same use-after-return gate as the ISender members. + Assert.Throws(() => qwp.Ping()); + Assert.Throws(() => _ = qwp.AckedFsn); + Assert.Throws(() => qwp.ColumnByte("b", 1)); + } + finally + { + pool.Close(); + } + } + +#if NET7_0_OR_GREATER + [Test] + public void CtorDisposesSenderPoolWhenQueryPoolPrewarmThrows() + { + // Distinct-endpoint shape: ingest is healthy (sender pool warms fine) but the query endpoint is + // down (query-pool prewarm throws). The half-built handle is never returned, so unless the ctor + // tears down on failure the warm ingest senders (and SF flocks) leak — one per Connect retry. + var options = new SenderOptions( + "ws::addr=localhost:9000;sender_pool_min=2;sender_pool_max=4;query_pool_min=1;query_pool_max=4;"); + var senders = new ConcurrentBag(); + + var ex = Assert.Throws(() => _ = new QuestDBClientImpl( + options, + slot => + { + var s = new FakeSender(slot); + senders.Add(s); + return s; + }, + () => throw new IngressError(ErrorCode.SocketError, "query endpoint down"))); + + Assert.Multiple(() => + { + Assert.That(ex!.Message, Does.Contain("query endpoint down")); + Assert.That(senders, Is.Not.Empty, "sender pool pre-warmed before the query pool threw"); + Assert.That(senders, Has.All.Matches(s => s.Disposed), + "warmed senders disposed when query-pool construction throws (no leak)"); + }); + } +#endif + + [Test] + public void FluentMethodsReturnTheWrapperNotTheDelegate() + { + var pool = MakePool("sender_pool_min=0;sender_pool_max=1;"); + try + { + ISender s = pool.Borrow(); + Assert.Multiple(() => + { + Assert.That(s.Table("t"), Is.SameAs(s)); + Assert.That(s.Symbol("k", "v"), Is.SameAs(s)); + Assert.That(s.Column("c", 1L), Is.SameAs(s)); + Assert.That(s.Column("d", 1.5), Is.SameAs(s)); + }); + s.Dispose(); + } + finally + { + pool.Close(); + } + } +} diff --git a/src/net-questdb-client-tests/Pooling/PoolReaperTests.cs b/src/net-questdb-client-tests/Pooling/PoolReaperTests.cs new file mode 100644 index 0000000..e6c307f --- /dev/null +++ b/src/net-questdb-client-tests/Pooling/PoolReaperTests.cs @@ -0,0 +1,387 @@ +/******************************************************************************* + * ___ _ ____ ____ + * / _ \ _ _ ___ ___| |_| _ \| __ ) + * | | | | | | |/ _ \/ __| __| | | | _ \ + * | |_| | |_| | __/\__ \ |_| |_| | |_) | + * \__\_\\__,_|\___||___/\__|____/|____/ + * + * Copyright (c) 2014-2019 Appsicle + * Copyright (c) 2019-2026 QuestDB + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + ******************************************************************************/ + +using System.Collections.Concurrent; +using System.Diagnostics; +using NUnit.Framework; +using QuestDB; +using QuestDB.Pooling; +using QuestDB.Senders; +using QuestDB.Utils; + +namespace net_questdb_client_tests.Pooling; + +public class PoolReaperTests +{ + private static SenderPool MakePool(string keys, out ConcurrentBag created) + { + var bag = new ConcurrentBag(); + created = bag; + var options = new SenderOptions("http::addr=localhost:9000;" + keys); + return new SenderPool(options, null, slot => + { + var s = new FakeSender(slot); + bag.Add(s); + return s; + }); + } + + // Borrow `count` senders then immediately return them, leaving them idle in the pool. + private static void Churn(SenderPool pool, int count) + { + var borrowed = new List(count); + for (var i = 0; i < count; i++) + { + borrowed.Add(pool.Borrow()); + } + + foreach (var s in borrowed) + { + s.Dispose(); + } + } + + [Test] + public void ReapIdleShrinksToMin() + { + var pool = MakePool("sender_pool_min=1;sender_pool_max=4;idle_timeout_ms=1;", out var created); + try + { + Churn(pool, 4); + Thread.Sleep(25); + pool.ReapIdle(); + + Assert.Multiple(() => + { + Assert.That(pool.TotalSize, Is.EqualTo(1)); + Assert.That(pool.AvailableSize, Is.EqualTo(1)); + Assert.That(created.Count(s => s.Disposed), Is.EqualTo(created.Count - 1)); + }); + } + finally + { + pool.Close(); + } + } + + [Test] + public void ReapIdleRespectsMin() + { + var pool = MakePool("sender_pool_min=2;sender_pool_max=4;idle_timeout_ms=1;", out _); + try + { + Churn(pool, 4); + Thread.Sleep(25); + pool.ReapIdle(); + Assert.That(pool.TotalSize, Is.EqualTo(2)); + } + finally + { + pool.Close(); + } + } + + [Test] + public void ReapIdleKeepsSendersWithinTimeout() + { + // idle_timeout defaults to 60s; freshly-returned senders are not reaped. + var pool = MakePool("sender_pool_min=0;sender_pool_max=4;", out _); + try + { + Churn(pool, 3); + pool.ReapIdle(); + Assert.That(pool.TotalSize, Is.EqualTo(3)); + } + finally + { + pool.Close(); + } + } + + [Test] + public void ReapIdleNeverTouchesInUseSenders() + { + var pool = MakePool("sender_pool_min=0;sender_pool_max=4;idle_timeout_ms=1;", out _); + try + { + var held = pool.Borrow(); + Thread.Sleep(25); + pool.ReapIdle(); + Assert.That(pool.TotalSize, Is.EqualTo(1), "the in-use sender is never reaped"); + held.Dispose(); + } + finally + { + pool.Close(); + } + } + + [Test] + public void ReapIdleByMaxLifetime() + { + // idle_timeout is long, but max_lifetime is tiny: an over-age but recently-returned sender is recycled. + var pool = MakePool("sender_pool_min=0;sender_pool_max=2;idle_timeout_ms=600000;max_lifetime_ms=1;", out _); + try + { + Churn(pool, 2); + Thread.Sleep(25); + pool.ReapIdle(); + Assert.That(pool.TotalSize, Is.EqualTo(0)); + } + finally + { + pool.Close(); + } + } + + [Test] + public void ReapIdleSurvivesDelegateDisposeError() + { + var pool = MakePool("sender_pool_min=0;sender_pool_max=3;idle_timeout_ms=1;", out var created); + try + { + Churn(pool, 3); + foreach (var s in created) + { + s.ThrowOnDispose = true; + } + + Thread.Sleep(25); + Assert.DoesNotThrow(() => pool.ReapIdle()); + Assert.That(pool.TotalSize, Is.EqualTo(0), "reap removed all idle senders despite dispose faults"); + } + finally + { + pool.Close(); + } + } + + [Test] + public void ReapIdleSkipsSendersWithUnAckedData() + { + var pool = MakePool("sender_pool_min=0;sender_pool_max=4;idle_timeout_ms=1;", out var created); + try + { + Churn(pool, 3); + foreach (var s in created) + { + s.FullyDrained = false; // ring still holds un-acked frames + } + + Thread.Sleep(25); + pool.ReapIdle(); + + Assert.Multiple(() => + { + Assert.That(pool.TotalSize, Is.EqualTo(3), "un-drained senders are never reaped, even over idle timeout"); + Assert.That(created.Count(s => s.Disposed), Is.EqualTo(0)); + }); + + // Once the rings drain, the next sweep reaps them (idle clock started at drain). + foreach (var s in created) + { + s.FullyDrained = true; + } + + Thread.Sleep(25); + pool.ReapIdle(); + Assert.That(pool.TotalSize, Is.EqualTo(0), "drained senders reap normally on the next sweep"); + } + finally + { + pool.Close(); + } + } + + [Test] + public void ReapIdleByMaxLifetimeSkipsUnAckedData() + { + // Even the max-lifetime path must not drop un-acked data: an over-age but un-drained sender survives. + var pool = MakePool("sender_pool_min=0;sender_pool_max=2;idle_timeout_ms=600000;max_lifetime_ms=1;", out var created); + try + { + Churn(pool, 2); + foreach (var s in created) + { + s.FullyDrained = false; + } + + Thread.Sleep(25); + pool.ReapIdle(); + Assert.That(pool.TotalSize, Is.EqualTo(2), "over-age senders with un-acked data are not aged out"); + + foreach (var s in created) + { + s.FullyDrained = true; + } + + pool.ReapIdle(); + Assert.That(pool.TotalSize, Is.EqualTo(0), "once drained, the over-age sender is recycled"); + } + finally + { + pool.Close(); + } + } + + [Test] + public void ReapStopsAtFirstEntryWithinIdleTimeout() + { + // The idle deque is sorted by idle time (borrowers pop/push the hot end), so the sweep scans from + // the cold end and stops at the first entry inside its timeout: the long-idle sender is reaped, + // the freshly-returned one behind it survives without being visited. Margins are ~1.5s so a + // loaded CI runner cannot tip the young entry over the timeout before the sweep runs. + var pool = MakePool("sender_pool_min=0;sender_pool_max=2;idle_timeout_ms=1500;", out var created); + try + { + var a = pool.Borrow(); + var b = pool.Borrow(); + var fakes = created.ToArray(); + a.Dispose(); // cold: idles past the timeout below + Thread.Sleep(2000); + b.Dispose(); // hot: freshly idle + + pool.ReapIdle(); + + Assert.Multiple(() => + { + Assert.That(pool.TotalSize, Is.EqualTo(1), "only the over-idle cold entry is reaped"); + Assert.That(fakes.Count(s => s.Disposed), Is.EqualTo(1)); + }); + } + finally + { + pool.Close(); + } + } + + [Test] + public void OverAgeSenderReapedDespiteYoungerColdEntryInFront() + { + // The deque is idle-sorted, not age-sorted, so an over-age sender at the hot end hides behind + // the younger-created (but colder) entry the idle sweep stops at. The max_lifetime walk (gated + // on the oldest parked CreatedAtUtc) must reap it anyway — and leave the young entry alone. + var pool = MakePool("sender_pool_min=0;sender_pool_max=2;idle_timeout_ms=600000;max_lifetime_ms=1500;", out var created); + try + { + var a = pool.Borrow(); // created now; held past max_lifetime + var fakeA = created.Single(); + Thread.Sleep(2000); + var b = pool.Borrow(); // created young + var fakeB = created.Single(s => !ReferenceEquals(s, fakeA)); + b.Dispose(); // young entry at the cold end + a.Dispose(); // over-age, at the hot end behind it + + pool.ReapIdle(); + + Assert.Multiple(() => + { + Assert.That(pool.TotalSize, Is.EqualTo(1), "over-age entry reaped despite returning last"); + Assert.That(fakeA.Disposed, Is.True, "the over-age sender was the one reaped"); + Assert.That(fakeB.Disposed, Is.False, "the young sender survives"); + }); + } + finally + { + pool.Close(); + } + } + + [Test] + public void SenderCrossingMaxLifetimeWhileParkedIsReaped() + { + // Regression: an entry that goes over-age only AFTER being returned sits at the hot end where + // the cold-end idle sweep never inspects it. The _idleOldestCreatedUtc-gated walk must catch it + // — reaping the aged sender, not the younger cold entry in front of it. + var pool = MakePool("sender_pool_min=0;sender_pool_max=2;idle_timeout_ms=600000;max_lifetime_ms=3000;", out var created); + try + { + var a = pool.Borrow(); // A created at t0 + var fakeA = created.Single(); + Thread.Sleep(1200); + var b = pool.Borrow(); // B created young, at t1200 + var fakeB = created.Single(s => !ReferenceEquals(s, fakeA)); + b.Dispose(); // B parks first: cold end + a.Dispose(); // A parks under-age (~1.2s < 3s): hot end, NOT over-age yet + + pool.ReapIdle(); + Assert.That(pool.TotalSize, Is.EqualTo(2), "nothing over-age or over-idle yet"); + + Thread.Sleep(2400); // t3600: A (3.6s) crossed max_lifetime while parked; B (2.4s) still young + + pool.ReapIdle(); + + Assert.Multiple(() => + { + Assert.That(pool.TotalSize, Is.EqualTo(1), "the entry that aged while parked is reaped"); + Assert.That(fakeA.Disposed, Is.True, "A crossed max_lifetime while parked"); + Assert.That(fakeB.Disposed, Is.False, "young B survives at the cold end"); + }); + } + finally + { + pool.Close(); + } + } + + [Test] + public void HousekeeperReapsInBackground() + { + var bag = new ConcurrentBag(); + var options = new SenderOptions( + "http::addr=localhost:9000;sender_pool_min=0;sender_pool_max=2;" + + "idle_timeout_ms=1;housekeeper_interval_ms=100;"); + using var client = new QuestDBClientImpl(options, slot => + { + var s = new FakeSender(slot); + bag.Add(s); + return s; + }); + + Churn2(client, 2); + Assert.That(client.TotalSenderCount, Is.EqualTo(2)); + + var sw = Stopwatch.StartNew(); + while (client.TotalSenderCount > 0 && sw.ElapsedMilliseconds < 3000) + { + Thread.Sleep(50); + } + + Assert.That(client.TotalSenderCount, Is.EqualTo(0), "background housekeeper reaped idle senders"); + } + + private static void Churn2(IQuestDBClient client, int count) + { + var borrowed = new List(count); + for (var i = 0; i < count; i++) + { + borrowed.Add(client.BorrowSender()); + } + + foreach (var s in borrowed) + { + s.Dispose(); + } + } +} diff --git a/src/net-questdb-client-tests/Pooling/PoolSlotTests.cs b/src/net-questdb-client-tests/Pooling/PoolSlotTests.cs new file mode 100644 index 0000000..5852fe3 --- /dev/null +++ b/src/net-questdb-client-tests/Pooling/PoolSlotTests.cs @@ -0,0 +1,395 @@ +/******************************************************************************* + * ___ _ ____ ____ + * / _ \ _ _ ___ ___| |_| _ \| __ ) + * | | | | | | |/ _ \/ __| __| | | | _ \ + * | |_| | |_| | __/\__ \ |_| |_| | |_) | + * \__\_\\__,_|\___||___/\__|____/|____/ + * + * Copyright (c) 2014-2019 Appsicle + * Copyright (c) 2019-2026 QuestDB + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + ******************************************************************************/ + +using System.Collections.Concurrent; +using NUnit.Framework; +using QuestDB.Enums; +using QuestDB.Pooling; +using QuestDB.Qwp.Sf; +using QuestDB.Utils; + +namespace net_questdb_client_tests.Pooling; + +public class PoolSlotTests +{ + private static SenderPool MakeSfPool(string keys, out ConcurrentBag created) + { + var bag = new ConcurrentBag(); + created = bag; + // ws + sf_dir => the pool runs in store-and-forward mode and hands out slot indices. The fake + // factory bypasses the real WS sender, so no sockets / files are touched. + var options = new SenderOptions("ws::addr=localhost:9000;sf_dir=/tmp/qdb-pool-test;" + keys); + return new SenderPool(options, null, slot => + { + var s = new FakeSender(slot); + bag.Add(s); + return s; + }); + } + + [Test] + public void ApplySlotIdentityStampsSenderIdAndManagedFamily() + { + var options = new SenderOptions("ws::addr=localhost:9000;sf_dir=/tmp/qdb-pool-test;"); + SenderPool.ApplySlotIdentity(options, "default", 2, 4); + + Assert.Multiple(() => + { + Assert.That(options.sender_id, Is.EqualTo("default-2")); + Assert.That(options.OrphanExcludeManagedBase, Is.EqualTo("default")); + Assert.That(options.OrphanExcludeManagedCount, Is.EqualTo(4)); + }); + } + + [Test] + public void SfModeAssignsDistinctSlotIndices() + { + var pool = MakeSfPool("sender_pool_min=3;sender_pool_max=4;", out var created); + try + { + var indices = created.Select(s => s.SlotIndex).OrderBy(i => i).ToArray(); + Assert.That(indices, Is.EqualTo(new[] { 0, 1, 2 })); + } + finally + { + pool.Close(); + } + } + + [Test] + public void NonSfPoolUsesNoSlotIndex() + { + var bag = new ConcurrentBag(); + var options = new SenderOptions("http::addr=localhost:9000;sender_pool_min=2;sender_pool_max=2;"); + var pool = new SenderPool(options, null, slot => + { + var s = new FakeSender(slot); + bag.Add(s); + return s; + }); + try + { + Assert.That(bag.All(s => s.SlotIndex == -1), Is.True, "non-SF senders carry no slot index"); + } + finally + { + pool.Close(); + } + } + + [Test] + public void FreedSlotIndexIsReusedMostRecentFirst() + { + var pool = MakeSfPool("sender_pool_min=0;sender_pool_max=4;", out var created); + try + { + var a = pool.Borrow(); // slot 0 + var b = pool.Borrow(); // slot 1 + var c = pool.Borrow(); // slot 2 + Assert.That(new[] { a.SlotIndex, b.SlotIndex, c.SlotIndex }, Is.EqualTo(new[] { 0, 1, 2 })); + + // Break the slot-1 sender so it is discarded (lock releases cleanly), freeing index 1. + created.Single(s => s.SlotIndex == 1).ThrowOnClear = true; + b.Dispose(); + + // LIFO free-index reuse: the just-freed index comes back before the never-used 3, keeping the + // on-disk working set of slot directories compact. + var d = pool.Borrow(); + Assert.That(d.SlotIndex, Is.EqualTo(1), "most recently freed index reused"); + Assert.That(pool.LeakedSlotCount, Is.EqualTo(0)); + + a.Dispose(); + c.Dispose(); + d.Dispose(); + } + finally + { + pool.Close(); + } + } + + [Test] + public void DiscardWithUnreleasedLockRetiresSlotAndShrinksCapacity() + { + var pool = MakeSfPool("sender_pool_min=0;sender_pool_max=2;acquire_timeout_ms=150;", out var created); + try + { + var a = pool.Borrow(); // slot 0 + var b = pool.Borrow(); // slot 1 + + // Slot-0 sender breaks AND refuses to release its lock -> the index must be retired. + var fakeA = created.Single(s => s.SlotIndex == 0); + fakeA.SlotLockReleased = false; + fakeA.ThrowOnClear = true; + a.Dispose(); + + Assert.That(pool.LeakedSlotCount, Is.EqualTo(1), "leaked slot retired"); + + // Effective capacity is now 1 (one permit permanently retained): with b still held, a new + // borrow must time out. + Assert.Throws(() => pool.Borrow()); + + // Returning b frees the one remaining permit; a single concurrent borrow now succeeds... + b.Dispose(); + var c = pool.Borrow(); + // ...but a second concurrent borrow still cannot (max shrank to 1). + Assert.Throws(() => pool.Borrow()); + c.Dispose(); + } + finally + { + pool.Close(); + } + } + + [Test] + public void ReapRetiringLeakedSlotShrinksEffectiveCapacity() + { + // Regression for the reap-path accounting bug: a reaped idle sender that fails to release its + // slot lock must retire its index AND shrink effective capacity, so the semaphore never + // advertises more in-use slots than the bitmap can allocate. + var pool = MakeSfPool("sender_pool_min=0;sender_pool_max=2;idle_timeout_ms=1;acquire_timeout_ms=150;", out var created); + try + { + var a = pool.Borrow(); // slot 0 + var b = pool.Borrow(); // slot 1 + a.Dispose(); + b.Dispose(); + + // Slot 0's sender will not release its lock when reaped -> its index is retired. + created.Single(s => s.SlotIndex == 0).SlotLockReleased = false; + Thread.Sleep(25); + pool.ReapIdle(); + + Assert.That(pool.LeakedSlotCount, Is.EqualTo(1)); + + // Effective max is now 1: one concurrent borrow works; the second fails with the DOCUMENTED + // PoolExhausted error, not an internal "no free slot index". + var c = pool.Borrow(); + var ex = Assert.Throws(() => pool.Borrow()); + Assert.That(ex!.code, Is.EqualTo(ErrorCode.PoolExhausted)); + c.Dispose(); + } + finally + { + pool.Close(); + } + } + + [Test] + public void ReclaimRestoresCapacityWhenRetiredSlotLockReleases() + { + // Fix for the deferred-teardown capacity-shrink bug: a slot retired because its lock had not yet + // released at dispose time must be reclaimed (index + permit restored) once the lock releases, so + // a transient / wedged teardown does not permanently shrink effective max toward PoolExhausted. + var pool = MakeSfPool("sender_pool_min=0;sender_pool_max=2;acquire_timeout_ms=150;", out var created); + try + { + var a = pool.Borrow(); // slot 0 + var b = pool.Borrow(); // slot 1 + + // Slot-0 sender breaks AND has not released its lock yet -> the index is retired and effective + // max shrinks to 1 (mirrors a deferred engine teardown still holding the slot lock). + var fakeA = created.Single(s => s.SlotIndex == 0); + fakeA.SlotLockReleased = false; + fakeA.ThrowOnClear = true; + a.Dispose(); + + Assert.That(pool.LeakedSlotCount, Is.EqualTo(1), "slot retired while lock held"); + + // Reclaim is a no-op while the lock is still held. + pool.ReclaimRetiredSlots(); + Assert.That(pool.LeakedSlotCount, Is.EqualTo(1), "not reclaimed while lock held"); + + // The deferred teardown finally releases the lock; the housekeeper sweep reclaims the slot. + fakeA.SlotLockReleased = true; + pool.ReclaimRetiredSlots(); + Assert.That(pool.LeakedSlotCount, Is.EqualTo(0), "retired slot reclaimed once lock released"); + + // Effective max is back to 2: with b still held, a fresh borrow now succeeds and reuses the + // reclaimed index 0 — no permanent shrink. + var c = pool.Borrow(); + Assert.That(c.SlotIndex, Is.EqualTo(0), "reclaimed index reused"); + c.Dispose(); + b.Dispose(); + } + finally + { + pool.Close(); + } + } + + [Test] + public void ReclaimRestoresCapacityForReapedRetiredSlot() + { + // Same reclaim, but via the reap path (idle sender whose lock was held when reaped). The reaper + // withheld the sender's permit from the free pool up front (TryWithholdPermit), so reclaim must + // Release a real permit back. + var pool = MakeSfPool("sender_pool_min=0;sender_pool_max=2;idle_timeout_ms=1;acquire_timeout_ms=150;", out var created); + try + { + var a = pool.Borrow(); // slot 0 + var b = pool.Borrow(); // slot 1 + a.Dispose(); + b.Dispose(); + + created.Single(s => s.SlotIndex == 0).SlotLockReleased = false; + Thread.Sleep(25); + pool.ReapIdle(); + Assert.That(pool.LeakedSlotCount, Is.EqualTo(1)); + + // Lock releases later; reclaim restores effective max to 2 (two concurrent borrows now work). + created.Single(s => s.SlotIndex == 0).SlotLockReleased = true; + pool.ReclaimRetiredSlots(); + Assert.That(pool.LeakedSlotCount, Is.EqualTo(0)); + + var c = pool.Borrow(); + var d = pool.Borrow(); + Assert.That(new[] { c.SlotIndex, d.SlotIndex }.OrderBy(i => i), Is.EqualTo(new[] { 0, 1 })); + c.Dispose(); + d.Dispose(); + } + finally + { + pool.Close(); + } + } + + [Test] + public void BorrowRacingReapNeverSpuriouslyExhausts() + { + // M2 regression. The old up-front retire in ReapIdle withheld its permit best-effort + // (SettleLeakDebtLocked's Wait(0)): a borrower that had already drained the last free permit — + // mid-Borrow, between _capacity.Wait and TakeOrCreate — left the reaper unable to withhold, yet + // the reap proceeded. The idle sender left the idle deque, its slot index stayed reserved for the + // dispose window, and the mid-flight borrower found neither an idle sender nor a free index -> + // instant spurious PoolExhausted, ignoring acquire_timeout_ms. Now the reaper only proceeds when + // it atomically takes a permit (TryWithholdPermit) and otherwise skips, leaving the idle sender + // for that borrower to reuse. Race the two continuously: a legitimate second borrower in a max-2 + // pool must never see PoolExhausted, whichever side wins. + var pool = MakeSfPool( + "sender_pool_min=1;sender_pool_max=2;idle_timeout_ms=1;acquire_timeout_ms=5000;", out _); + try + { + using var held = pool.Borrow(); // pin slot 0 so the reaper only ever sees one idle sender + for (var i = 0; i < 300; i++) + { + var warm = pool.Borrow(); + warm.Dispose(); + Thread.Sleep(2); // exceed idle_timeout so the parked sender is reap-eligible + + using var start = new ManualResetEventSlim(false); + var reap = Task.Run(() => + { + start.Wait(); + pool.ReapIdle(); + }); + var borrow = Task.Run(() => + { + start.Wait(); + return pool.Borrow(); + }); + start.Set(); + + Assert.That(reap.Wait(TimeSpan.FromSeconds(10)), Is.True, "reap completed"); + Assert.That(borrow.Wait(TimeSpan.FromSeconds(10)), Is.True, "borrow completed"); + borrow.Result.Dispose(); // faults here if the borrow spuriously threw (pre-fix behaviour) + } + + Assert.That(pool.LeakedSlotCount, Is.EqualTo(0)); + } + finally + { + pool.Close(); + } + } + + [Test] + public void ConcurrentBorrowDuringReapDisposeWindowDoesNotSpuriouslyThrow() + { + // Regression: a reaped idle SF sender is removed from _all/_idle under the pool lock, but its + // slot index is freed only AFTER the (slow WS+SF) DisposeInner runs outside the lock. A reaped + // sender holds no capacity permit, so during that dispose window the semaphore used to advertise a + // free permit whose slot index was still locked -> a concurrent Borrow acquired the permit, found no + // free index (AllocateSlotIndex -> -1), and threw a spurious PoolExhausted. The fix withholds the + // permit for the whole window, so the borrow waits for real capacity instead of failing. + var pool = MakeSfPool( + "sender_pool_min=1;sender_pool_max=2;idle_timeout_ms=1;acquire_timeout_ms=5000;", out var created); + BorrowedSender? held = null; + try + { + held = pool.Borrow(); // slot 0 stays borrowed (the "min" sender), so reap can't touch it + var warm = pool.Borrow(); // slot 1 + warm.Dispose(); // slot 1 now idle and reapable + + var slot1 = created.Single(s => s.SlotIndex == 1); + slot1.DisposeEntered = new ManualResetEventSlim(false); + slot1.DisposeGate = new ManualResetEventSlim(false); + + Thread.Sleep(25); // exceed idle_timeout so slot 1 is over-idle + + var reap = Task.Run(() => pool.ReapIdle()); + Assert.That(slot1.DisposeEntered.Wait(TimeSpan.FromSeconds(5)), Is.True, + "reap reached the (blocked) dispose window: slot 1 out of _all/_idle, index still held"); + + // The window is now open — slot 1's index is reserved while its dispose is parked. Race a borrow. + var borrow = Task.Run(() => pool.Borrow()); + + // Before the fix the borrow threw immediately (faulting the task); after it, the borrow must be + // parked waiting for real capacity to come back. + Thread.Sleep(300); + Assert.That(borrow.IsCompleted, Is.False, + "borrow waits for capacity instead of spuriously throwing PoolExhausted"); + + slot1.DisposeGate.Set(); // let the teardown finish -> slot index + withheld permit reclaimed + Assert.That(reap.Wait(TimeSpan.FromSeconds(5)), Is.True, "reap completed"); + Assert.That(borrow.Wait(TimeSpan.FromSeconds(5)), Is.True, "borrow completed once capacity freed"); + + var s2 = borrow.Result; // throws if the borrow faulted (the pre-fix behaviour) + Assert.That(s2.SlotIndex, Is.EqualTo(1), "reclaimed slot index reused"); + Assert.That(pool.LeakedSlotCount, Is.EqualTo(0)); + s2.Dispose(); + } + finally + { + held?.Dispose(); + pool.Close(); + } + } + + [TestCase("default-0", "default", 4, true)] + [TestCase("default-3", "default", 4, true)] + [TestCase("default-4", "default", 4, false)] // out of managed range -> a true orphan + [TestCase("default", "default", 4, false)] // no -index suffix + [TestCase("default-x", "default", 4, false)] // non-numeric suffix + [TestCase("default-03", "default", 4, false)] // parses to 3 but pool only ever allocates "default-3" + [TestCase("default-+3", "default", 4, false)] // signed lookalike, same reasoning + [TestCase("default- 3", "default", 4, false)] // whitespace lookalike, same reasoning + [TestCase("other-1", "default", 4, false)] // different base + [TestCase("default-1", null, 0, false)] // no managed family configured + public void OrphanScannerIdentifiesManagedSlots(string senderId, string? managedBase, int managedCount, bool expected) + { + Assert.That(QwpOrphanScanner.IsManagedSlot(senderId, managedBase, managedCount), Is.EqualTo(expected)); + } +} diff --git a/src/net-questdb-client-tests/Pooling/PoolTransactionalDiscardTests.cs b/src/net-questdb-client-tests/Pooling/PoolTransactionalDiscardTests.cs new file mode 100644 index 0000000..d6c9c0e --- /dev/null +++ b/src/net-questdb-client-tests/Pooling/PoolTransactionalDiscardTests.cs @@ -0,0 +1,148 @@ +/******************************************************************************* + * ___ _ ____ ____ + * / _ \ _ _ ___ ___| |_| _ \| __ ) + * | | | | | | |/ _ \/ __| __| | | | _ \ + * | |_| | |_| | __/\__ \ |_| |_| | |_) | + * \__\_\\__,_|\___||___/\__|____/|____/ + * + * Copyright (c) 2014-2019 Appsicle + * Copyright (c) 2019-2026 QuestDB + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + ******************************************************************************/ + +#if NET7_0_OR_GREATER + +using System.Buffers.Binary; +using NUnit.Framework; +using QuestDB; +using QuestDB.Enums; +using QuestDB.Qwp; +using dummy_http_server; + +namespace net_questdb_client_tests.Pooling; + +/// +/// End-to-end pin of the pooled transactional return path: a borrower whose auto-flush staged rows +/// server-side under FLAG_DEFER_COMMIT and who then disposes without committing must have its +/// entry discarded (connection closed → server drops the staged rows), never re-pooled. Re-pooling the +/// live connection would let the next borrower's first commit silently publish the abandoned rows. +/// +[TestFixture] +public class PoolTransactionalDiscardTests +{ + private static readonly DateTime Ts = new(2026, 1, 1, 0, 0, 0, DateTimeKind.Utc); + + private static bool IsDeferred(byte[] frame) + => (frame[QwpConstants.OffsetFlags] & QwpConstants.FlagDeferCommit) != 0; + + [Test] + public async Task ReturnWithStagedDeferredRows_DiscardsEntry_NextBorrowerGetsFreshConnection() + { + long nextWireSeq = 0; + await using var server = new DummyQwpServer(new DummyQwpServerOptions + { + FrameHandler = _ => BuildOkAck(Interlocked.Increment(ref nextWireSeq) - 1), + }); + await server.StartAsync(); + + using var client = QuestDBClient.Connect( + $"ws::addr=127.0.0.1:{server.Uri.Port};transaction=on;" + + "auto_flush_rows=1;auto_flush_interval=off;auto_flush_bytes=off;" + + "sender_pool_min=0;sender_pool_max=1;query_pool_min=0;"); + + // Borrower A: auto_flush_rows=1 ships the row immediately as a deferred (staged) frame; the + // borrow is then abandoned — disposed without Send()/Commit(), so a commit is still owed. + using (var a = client.BorrowSender()) + { + a.Table("t").Column("v", 1L).At(Ts); + await WaitFor(() => server.ReceivedFrames.Count >= 1); + } + + Assert.That(server.ReceivedFrames.Count, Is.EqualTo(1)); + Assert.That(IsDeferred(server.ReceivedFrames.First()), Is.True, "borrower A's auto-flush frame defers commit"); + Assert.That(server.UpgradeCount, Is.EqualTo(1)); + + // Borrower B: writes its own row and commits. Because A's entry was discarded, B runs on a + // fresh connection — its commit cannot reach (and publish) A's staged rows, which the server + // dropped when A's connection closed. + using (var b = client.BorrowSender()) + { + b.Table("t").Column("v", 2L).At(Ts); + b.Send(); + } + + await WaitFor(() => server.ReceivedFrames.Count >= 3); + Assert.Multiple(() => + { + Assert.That(server.UpgradeCount, Is.EqualTo(2), + "the abandoned transactional entry must be discarded, not re-pooled on the live connection"); + Assert.That(server.ReceivedFrames.Count, Is.EqualTo(3), "B ships a deferred frame then a commit frame"); + Assert.That(IsDeferred(server.ReceivedFrames.Last()), Is.False, "B's Send() commits"); + }); + } + + [Test] + public async Task ReturnAfterCommit_RepoolsEntry_ConnectionIsReused() + { + long nextWireSeq = 0; + await using var server = new DummyQwpServer(new DummyQwpServerOptions + { + FrameHandler = _ => BuildOkAck(Interlocked.Increment(ref nextWireSeq) - 1), + }); + await server.StartAsync(); + + using var client = QuestDBClient.Connect( + $"ws::addr=127.0.0.1:{server.Uri.Port};transaction=on;" + + "auto_flush_rows=1;auto_flush_interval=off;auto_flush_bytes=off;" + + "sender_pool_min=0;sender_pool_max=1;query_pool_min=0;"); + + // Borrower A commits before returning: no commit owed, the entry re-pools as usual. + using (var a = client.BorrowSender()) + { + a.Table("t").Column("v", 1L).At(Ts); + a.Send(); + } + + using (var b = client.BorrowSender()) + { + b.Table("t").Column("v", 2L).At(Ts); + b.Send(); + } + + await WaitFor(() => server.ReceivedFrames.Count >= 4); + Assert.That(server.UpgradeCount, Is.EqualTo(1), "a committed transactional entry is reused, not discarded"); + } + + // ---- helpers ---- + + private static byte[] BuildOkAck(long sequence) + { + var bytes = new byte[QwpConstants.OffsetTableCountInOkAck + 2]; + bytes[0] = (byte)QwpStatusCode.Ok; + BinaryPrimitives.WriteInt64LittleEndian(bytes.AsSpan(1, 8), sequence); + BinaryPrimitives.WriteUInt16LittleEndian(bytes.AsSpan(QwpConstants.OffsetTableCountInOkAck, 2), 0); + return bytes; + } + + private static async Task WaitFor(Func predicate, int timeoutMs = 5000) + { + var deadline = Environment.TickCount64 + timeoutMs; + while (!predicate() && Environment.TickCount64 < deadline) + { + await Task.Delay(20); + } + } +} +#endif diff --git a/src/net-questdb-client-tests/Pooling/PooledQueryClientTests.cs b/src/net-questdb-client-tests/Pooling/PooledQueryClientTests.cs new file mode 100644 index 0000000..273c9fe --- /dev/null +++ b/src/net-questdb-client-tests/Pooling/PooledQueryClientTests.cs @@ -0,0 +1,113 @@ +/******************************************************************************* + * ___ _ ____ ____ + * / _ \ _ _ ___ ___| |_| _ \| __ ) + * | | | | | | |/ _ \/ __| __| | | | _ \ + * | |_| | |_| | __/\__ \ |_| |_| | |_) | + * \__\_\\__,_|\___||___/\__|____/|____/ + * + * Copyright (c) 2014-2019 Appsicle + * Copyright (c) 2019-2026 QuestDB + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + ******************************************************************************/ + +#if NET7_0_OR_GREATER +using System.Collections.Concurrent; +using NUnit.Framework; +using QuestDB.Pooling; +using QuestDB.Senders; +using QuestDB.Utils; + +namespace net_questdb_client_tests.Pooling; + +public class PooledQueryClientTests +{ + private static QueryClientPool MakePool(string keys, out ConcurrentBag created) + { + var bag = new ConcurrentBag(); + created = bag; + var options = new SenderOptions("ws::addr=localhost:9000;" + keys); + return new QueryClientPool(options, null, () => + { + var c = new FakeQueryClient(); + bag.Add(c); + return new ValueTask(c); + }); + } + + [Test] + public void CleanDisposeReturnsWithoutDisposingInner() + { + var pool = MakePool("query_pool_min=1;query_pool_max=1;", out var created); + try + { + var c = pool.Borrow(); + c.Dispose(); + + Assert.Multiple(() => + { + Assert.That(created.Single().Disposed, Is.False, "clean return does not dispose the real client"); + Assert.That(pool.AvailableSize, Is.EqualTo(1)); + }); + } + finally + { + pool.Close(); + } + } + + [Test] + public void DoubleDisposeIsNoOp() + { + var pool = MakePool("query_pool_min=1;query_pool_max=1;", out _); + try + { + var c = pool.Borrow(); + c.Dispose(); + c.Dispose(); // second dispose must not re-return or double-count + + Assert.That(pool.AvailableSize, Is.EqualTo(1), "second dispose is a no-op"); + } + finally + { + pool.Close(); + } + } + + [Test] + public void DelegationForwardsToInner() + { + var pool = MakePool("query_pool_min=1;query_pool_max=1;", out var created); + try + { + var c = pool.Borrow(); + c.Execute("select 1", new NoopQueryHandler()); + c.Cancel(); + + var inner = created.Single(); + Assert.Multiple(() => + { + Assert.That(inner.ExecuteCount, Is.EqualTo(1)); + Assert.That(inner.LastSql, Is.EqualTo("select 1")); + Assert.That(inner.CancelCount, Is.EqualTo(1)); + }); + c.Dispose(); + } + finally + { + pool.Close(); + } + } +} +#endif diff --git a/src/net-questdb-client-tests/Pooling/QueryClientPoolTests.cs b/src/net-questdb-client-tests/Pooling/QueryClientPoolTests.cs new file mode 100644 index 0000000..fc32ff8 --- /dev/null +++ b/src/net-questdb-client-tests/Pooling/QueryClientPoolTests.cs @@ -0,0 +1,237 @@ +/******************************************************************************* + * ___ _ ____ ____ + * / _ \ _ _ ___ ___| |_| _ \| __ ) + * | | | | | | |/ _ \/ __| __| | | | _ \ + * | |_| | |_| | __/\__ \ |_| |_| | |_) | + * \__\_\\__,_|\___||___/\__|____/|____/ + * + * Copyright (c) 2014-2019 Appsicle + * Copyright (c) 2019-2026 QuestDB + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + ******************************************************************************/ + +#if NET7_0_OR_GREATER +using System.Collections.Concurrent; +using NUnit.Framework; +using QuestDB.Enums; +using QuestDB.Pooling; +using QuestDB.Senders; +using QuestDB.Utils; + +namespace net_questdb_client_tests.Pooling; + +public class QueryClientPoolTests +{ + private static QueryClientPool MakePool(string keys, out ConcurrentBag created) + { + var bag = new ConcurrentBag(); + created = bag; + var options = new SenderOptions("ws::addr=localhost:9000;" + keys); + return new QueryClientPool(options, null, () => + { + var c = new FakeQueryClient(); + bag.Add(c); + return new ValueTask(c); + }); + } + + [Test] + public void PreWarmsMinClients() + { + using var g = new PoolGuard(MakePool("query_pool_min=2;query_pool_max=4;", out var created)); + Assert.Multiple(() => + { + Assert.That(g.Pool.TotalSize, Is.EqualTo(2)); + Assert.That(g.Pool.AvailableSize, Is.EqualTo(2)); + Assert.That(created, Has.Count.EqualTo(2)); + }); + } + + [Test] + public void BorrowReturnRecyclesSameDecorator() + { + using var g = new PoolGuard(MakePool("query_pool_min=1;query_pool_max=1;", out var created)); + + var c1 = g.Pool.Borrow(); + c1.Dispose(); + var c2 = g.Pool.Borrow(); + + Assert.That(c2, Is.SameAs(c1)); + Assert.That(created, Has.Count.EqualTo(1), "no new underlying client created on reuse"); + c2.Dispose(); + } + + [Test] + public void BorrowAndReturnTrackSizes() + { + using var g = new PoolGuard(MakePool("query_pool_min=1;query_pool_max=2;", out _)); + + var c = g.Pool.Borrow(); + Assert.Multiple(() => + { + Assert.That(g.Pool.AvailableSize, Is.EqualTo(0)); + Assert.That(g.Pool.TotalSize, Is.EqualTo(1)); + }); + + c.Dispose(); + Assert.Multiple(() => + { + Assert.That(g.Pool.AvailableSize, Is.EqualTo(1)); + Assert.That(g.Pool.TotalSize, Is.EqualTo(1)); + }); + } + + [Test] + public void ExhaustedBorrowThrowsWithQueryPoolMaxInMessage() + { + using var g = new PoolGuard(MakePool("query_pool_min=0;query_pool_max=1;acquire_timeout_ms=150;", out _)); + + var held = g.Pool.Borrow(); + var ex = Assert.Throws(() => g.Pool.Borrow()); + + Assert.Multiple(() => + { + Assert.That(ex!.code, Is.EqualTo(ErrorCode.PoolExhausted)); + Assert.That(ex.Message, Does.Contain("query_pool_max=1")); + }); + held.Dispose(); + } + + [Test] + public void ClosedPoolBorrowThrows() + { + var pool = MakePool("query_pool_min=0;query_pool_max=2;", out _); + pool.Close(); + + var ex = Assert.Throws(() => pool.Borrow()); + Assert.That(ex!.code, Is.EqualTo(ErrorCode.InvalidApiCall)); + } + + [Test] + public void DiscardBrokenRemovesAndRestoresCapacity() + { + using var g = new PoolGuard(MakePool("query_pool_min=0;query_pool_max=1;acquire_timeout_ms=150;", out var created)); + + var c = (PooledQueryClient)g.Pool.Borrow(); + c.MarkBroken(); + c.Dispose(); // broken -> discard + + Assert.Multiple(() => + { + Assert.That(g.Pool.TotalSize, Is.EqualTo(0), "broken client removed from the pool"); + Assert.That(created.Single().Disposed, Is.True, "broken client's inner is disposed"); + }); + + // Capacity restored (no leak-debt analog): a fresh borrow succeeds and creates a new client. + var c2 = g.Pool.Borrow(); + Assert.That(created, Has.Count.EqualTo(2)); + c2.Dispose(); + } + + [Test] + public void TerminalClientIsDiscardedOnReturn() + { + using var g = new PoolGuard(MakePool("query_pool_min=0;query_pool_max=2;", out var created)); + + var c = g.Pool.Borrow(); + created.Single().TerminalOrDisposed = true; // becomes terminal without throwing + c.Dispose(); // belt-and-braces -> discard, not re-pool + + Assert.Multiple(() => + { + Assert.That(g.Pool.AvailableSize, Is.EqualTo(0)); + Assert.That(g.Pool.TotalSize, Is.EqualTo(0)); + Assert.That(created.Single().Disposed, Is.True); + }); + } + + [Test] + public void ReapIdleReapsDownToMin() + { + using var g = new PoolGuard(MakePool("query_pool_min=1;query_pool_max=3;idle_timeout_ms=1;", out _)); + + var a = g.Pool.Borrow(); + var b = g.Pool.Borrow(); + var c = g.Pool.Borrow(); + a.Dispose(); + b.Dispose(); + c.Dispose(); + Assert.That(g.Pool.AvailableSize, Is.EqualTo(3)); + + Thread.Sleep(30); // exceed idle_timeout_ms + g.Pool.ReapIdle(); + + Assert.That(g.Pool.TotalSize, Is.EqualTo(1), "reaped idle clients down to min"); + } + + [Test] + public void CloseDuringCreateDisposesFreshClientAndRestoresCapacity() + { + var bag = new ConcurrentBag(); + var options = new SenderOptions("ws::addr=localhost:9000;query_pool_min=0;query_pool_max=1;"); + QueryClientPool? poolRef = null; + var pool = new QueryClientPool(options, null, () => + { + // Simulate the pool closing while a client is being created. + poolRef!.Close(); + var c = new FakeQueryClient(); + bag.Add(c); + return new ValueTask(c); + }); + poolRef = pool; + + var ex = Assert.Throws(() => pool.Borrow()); + Assert.Multiple(() => + { + Assert.That(ex!.code, Is.EqualTo(ErrorCode.InvalidApiCall)); + Assert.That(bag.Single().Disposed, Is.True, "the client created during the close race is disposed"); + }); + } + + [Test] + public void CloseCancelsBorrowedClientAndLeavesTeardownToBorrower() + { + // Close disposes only idle clients; an in-flight one is merely cancelled (so a blocked + // ExecuteAsync unwinds) and torn down by its borrower on return — never disposed by Close, which + // would race the borrower's still-running query on the non-thread-safe client. + var pool = MakePool("query_pool_min=2;query_pool_max=2;", out var created); + + var borrowed = pool.Borrow(); // 1 in-use, 1 idle + + pool.Close(); + + var inUse = created.Single(c => !c.Disposed); // the borrowed client survived close (only cancelled) + Assert.Multiple(() => + { + Assert.That(created.Count(c => c.Disposed), Is.EqualTo(1), "idle client disposed by close"); + Assert.That(inUse.CancelCount, Is.EqualTo(1), "in-flight client cancelled so a blocked query unwinds"); + Assert.That(inUse.DisposeCount, Is.EqualTo(0), "close did not dispose the in-use client"); + }); + + borrowed.Dispose(); // borrower returns post-close: teardown happens here, on this thread + + Assert.That(inUse.DisposeCount, Is.EqualTo(1), "inner disposed exactly once, by the borrower"); + + Assert.DoesNotThrow(() => pool.Close(), "second close is idempotent"); + } + + private sealed class PoolGuard : IDisposable + { + public PoolGuard(QueryClientPool pool) => Pool = pool; + public QueryClientPool Pool { get; } + public void Dispose() => Pool.Close(); + } +} +#endif diff --git a/src/net-questdb-client-tests/Pooling/QueryHandleTests.cs b/src/net-questdb-client-tests/Pooling/QueryHandleTests.cs new file mode 100644 index 0000000..45cbc0e --- /dev/null +++ b/src/net-questdb-client-tests/Pooling/QueryHandleTests.cs @@ -0,0 +1,333 @@ +/******************************************************************************* + * ___ _ ____ ____ + * / _ \ _ _ ___ ___| |_| _ \| __ ) + * | | | | | | |/ _ \/ __| __| | | | _ \ + * | |_| | |_| | __/\__ \ |_| |_| | |_) | + * \__\_\\__,_|\___||___/\__|____/|____/ + * + * Copyright (c) 2014-2019 Appsicle + * Copyright (c) 2019-2026 QuestDB + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + ******************************************************************************/ + +#if NET7_0_OR_GREATER +using System.Collections.Concurrent; +using NUnit.Framework; +using QuestDB.Enums; +using QuestDB.Pooling; +using QuestDB.Senders; +using QuestDB.Utils; + +namespace net_questdb_client_tests.Pooling; + +public class QueryHandleTests +{ + private static QuestDBClientImpl MakeHandle(string keys, out ConcurrentBag created) + { + var bag = new ConcurrentBag(); + created = bag; + var options = new SenderOptions("ws::addr=localhost:9000;sender_pool_min=0;" + keys); + return new QuestDBClientImpl( + options, + slot => new FakeSender(slot), + () => + { + var c = new FakeQueryClient(); + bag.Add(c); + return new ValueTask(c); + }); + } + + [Test] + public async Task NewQueryExecutesAndReturnsClient() + { + using var h = MakeHandle("query_pool_min=1;query_pool_max=1;", out var created); + + await h.NewQuery().Sql("select 1").Handler(new NoopQueryHandler()).ExecuteAsync(); + + var c = created.Single(); + Assert.Multiple(() => + { + Assert.That(c.ExecuteCount, Is.EqualTo(1)); + Assert.That(c.LastSql, Is.EqualTo("select 1")); + Assert.That(c.Disposed, Is.False); + Assert.That(h.AvailableQueryClientCount, Is.EqualTo(1), "client returned to the pool"); + }); + } + + [Test] + public async Task ExecuteSqlAsyncIsEquivalentToNewQuery() + { + using var h = MakeHandle("query_pool_min=1;query_pool_max=1;", out var created); + + await h.ExecuteSqlAsync("select 2", new NoopQueryHandler()); + + Assert.That(created.Single().LastSql, Is.EqualTo("select 2")); + } + + [Test] + public void MissingSqlThrows() + { + using var h = MakeHandle("query_pool_min=0;query_pool_max=1;", out _); + + var ex = Assert.ThrowsAsync(async () => + await h.NewQuery().Handler(new NoopQueryHandler()).ExecuteAsync()); + Assert.Multiple(() => + { + Assert.That(ex!.code, Is.EqualTo(ErrorCode.InvalidApiCall)); + Assert.That(ex.Message, Does.Contain("sql is required")); + }); + } + + [Test] + public void MissingHandlerThrows() + { + using var h = MakeHandle("query_pool_min=0;query_pool_max=1;", out _); + + var ex = Assert.ThrowsAsync(async () => + await h.NewQuery().Sql("x").ExecuteAsync()); + Assert.Multiple(() => + { + Assert.That(ex!.code, Is.EqualTo(ErrorCode.InvalidApiCall)); + Assert.That(ex.Message, Does.Contain("handler is required")); + }); + } + + [Test] + public async Task SingleFlightOverlapThrows() + { + using var h = MakeHandle("query_pool_min=1;query_pool_max=1;", out var created); + var fake = created.Single(); + fake.Gate = new TaskCompletionSource(); + + var q = h.NewQuery().Sql("x").Handler(new NoopQueryHandler()); + var inFlight = q.ExecuteAsync(); + + var ex = Assert.ThrowsAsync(async () => await q.ExecuteAsync()); + Assert.That(ex!.code, Is.EqualTo(ErrorCode.InvalidApiCall)); + + fake.Gate.SetResult(true); + await inFlight; + } + + [Test] + public async Task ThrowingQueryDiscardsClientThenNextBorrowCreatesFresh() + { + using var h = MakeHandle("query_pool_min=1;query_pool_max=2;", out var created); + var first = created.Single(); + first.ThrowOnExecute = true; + + var ex = Assert.ThrowsAsync(async () => + await h.NewQuery().Sql("x").Handler(new NoopQueryHandler()).ExecuteAsync()); + Assert.That(ex!.code, Is.EqualTo(ErrorCode.SocketError)); + + Assert.Multiple(() => + { + Assert.That(first.Disposed, Is.True, "failed client is discarded"); + Assert.That(h.TotalQueryClientCount, Is.EqualTo(0)); + }); + + await h.NewQuery().Sql("y").Handler(new NoopQueryHandler()).ExecuteAsync(); + Assert.That(created, Has.Count.EqualTo(2), "next borrow creates a fresh client"); + } + + [Test] + public void CtCancelDiscardsClientAndSurfacesOce() + { + using var h = MakeHandle("query_pool_min=1;query_pool_max=2;", out var created); + var first = created.Single(); + first.CancelOnExecute = true; + + Assert.ThrowsAsync(async () => + await h.NewQuery().Sql("x").Handler(new NoopQueryHandler()).ExecuteAsync()); + + Assert.Multiple(() => + { + Assert.That(first.Disposed, Is.True, "hard-cancelled client is discarded"); + Assert.That(h.TotalQueryClientCount, Is.EqualTo(0)); + }); + } + + [Test] + public async Task CooperativeCancelEndsCleanlyAndRepools() + { + using var h = MakeHandle("query_pool_min=1;query_pool_max=1;", out var created); + var fake = created.Single(); + fake.Gate = new TaskCompletionSource(); + + var q = h.NewQuery().Sql("x").Handler(new NoopQueryHandler()); + var inFlight = q.ExecuteAsync(); + q.Cancel(); // cooperative: posts a CANCEL frame + fake.Gate.SetResult(true); // query then completes normally + await inFlight; + + Assert.Multiple(() => + { + Assert.That(fake.CancelCount, Is.EqualTo(1)); + Assert.That(fake.Disposed, Is.False, "cleanly-ending cooperative cancel re-pools the client"); + Assert.That(h.AvailableQueryClientCount, Is.EqualTo(1)); + }); + } + + [Test] + public async Task CancelAfterCompletionDoesNotCancelReborrowedClient() + { + // Pool size 1: q2 re-borrows the exact same inner client q1 just returned. A late Cancel() on the + // already-completed q1 must not forward to that re-borrowed client and abort q2's in-flight query. + // Guards the lease null-out in Query.ExecuteAsync's finally. + using var h = MakeHandle("query_pool_min=1;query_pool_max=1;", out var created); + var fake = created.Single(); + + var q1 = h.NewQuery().Sql("a").Handler(new NoopQueryHandler()); + await q1.ExecuteAsync(); // completes cleanly; client re-pooled, q1's lease nulled + + fake.Gate = new TaskCompletionSource(); + var q2 = h.NewQuery().Sql("b").Handler(new NoopQueryHandler()); + var t2 = q2.ExecuteAsync(); // re-borrows the same inner; parks in flight on the gate + + var sw = System.Diagnostics.Stopwatch.StartNew(); + while (Volatile.Read(ref fake.ExecuteCount) < 2 && sw.ElapsedMilliseconds < 2000) + { + await Task.Delay(5); + } + + Assert.That(fake.ExecuteCount, Is.EqualTo(2), "q2 re-borrowed the same client and is in flight"); + + q1.Cancel(); // late cancel on the finished query — must be a no-op, not reach q2's client + + Assert.That(fake.CancelCount, Is.EqualTo(0), + "a cancel on a completed query must not reach the re-borrowed client running q2"); + + fake.Gate.SetResult(true); + await t2; + + Assert.That(fake.CancelCount, Is.EqualTo(0), "q2 completed without a stray cancel"); + } + + [Test] + public async Task CancelRacingWithCompletionAndReborrowDoesNotCancelSuccessorQuery() + { + // The TOCTOU window: Cancel() resolves its target while q1 is still in flight, but the CANCEL + // dispatch lands only after q1 completed, its client was re-pooled, and q2 re-borrowed it. + // The cancel must be scoped to q1's request id so the late dispatch is dropped rather than + // cancelling q2. CancelGate parks the dispatch inside the client to pin that interleaving. + using var h = MakeHandle("query_pool_min=1;query_pool_max=1;", out var created); + var fake = created.Single(); + fake.Gate = new TaskCompletionSource(); + fake.CancelGate = new TaskCompletionSource(); + + var q1 = h.NewQuery().Sql("a").Handler(new NoopQueryHandler()); + var t1 = q1.ExecuteAsync(); // in flight, parked on the gate; fake rid 1 + + var cancelDispatch = Task.Run(q1.Cancel); // resolves rid 1, then parks on CancelGate + var resolvedRid = await fake.CancelRequestEntered.Task.WaitAsync(TimeSpan.FromSeconds(5)); + Assert.That(resolvedRid, Is.EqualTo(1), "cancel resolved q1's own request id"); + + fake.Gate.SetResult(true); // q1 completes and its client is returned to the pool + await t1; + + fake.Gate = new TaskCompletionSource(); + var q2 = h.NewQuery().Sql("b").Handler(new NoopQueryHandler()); + var t2 = q2.ExecuteAsync(); // re-borrows the same inner client; fake rid 2 + + var sw = System.Diagnostics.Stopwatch.StartNew(); + while (Volatile.Read(ref fake.ExecuteCount) < 2 && sw.ElapsedMilliseconds < 2000) + { + await Task.Delay(5); + } + + Assert.That(fake.ExecuteCount, Is.EqualTo(2), "q2 re-borrowed the same client and is in flight"); + + fake.CancelGate.SetResult(true); // stale cancel for rid 1 finally dispatches + await cancelDispatch; + + Assert.That(fake.CancelCount, Is.EqualTo(0), + "a cancel resolved against q1 must not cancel q2 on the re-borrowed client"); + + fake.Gate.SetResult(true); + await t2; + + Assert.That(fake.CancelCount, Is.EqualTo(0), "q2 completed without a stray cancel"); + } + + [Test] + public async Task ConcurrentNewQueriesGetDistinctClients() + { + // Shared gate keeps every borrowed client in flight so the pool must hand out distinct ones. + var gate = new TaskCompletionSource(); + var bag = new ConcurrentBag(); + var options = new SenderOptions( + "ws::addr=localhost:9000;sender_pool_min=0;query_pool_min=0;query_pool_max=3;acquire_timeout_ms=2000;"); + using var h = new QuestDBClientImpl( + options, + slot => new FakeSender(slot), + () => + { + var c = new FakeQueryClient { Gate = gate }; + bag.Add(c); + return new ValueTask(c); + }); + + var tasks = new List(); + for (var i = 0; i < 3; i++) + { + tasks.Add(h.NewQuery().Sql("q").Handler(new NoopQueryHandler()).ExecuteAsync()); + } + + var sw = System.Diagnostics.Stopwatch.StartNew(); + while (bag.Count < 3 && sw.ElapsedMilliseconds < 2000) + { + await Task.Delay(5); + } + + Assert.That(h.TotalQueryClientCount, Is.EqualTo(3), "three concurrent queries borrowed three distinct clients"); + gate.SetResult(true); + await Task.WhenAll(tasks); + } + + [Test] + public void DisposeClosesQueryPool() + { + var h = MakeHandle("query_pool_min=1;query_pool_max=1;", out _); + h.Dispose(); + + var ex = Assert.ThrowsAsync(async () => + await h.NewQuery().Sql("x").Handler(new NoopQueryHandler()).ExecuteAsync()); + Assert.That(ex!.code, Is.EqualTo(ErrorCode.InvalidApiCall)); + } + + [Test] + public async Task QueryReusableAfterPoolExhausted() + { + using var h = MakeHandle("query_pool_min=1;query_pool_max=1;acquire_timeout_ms=100;", out var created); + var fake = created.Single(); + fake.Gate = new TaskCompletionSource(); + + var q1 = h.NewQuery().Sql("a").Handler(new NoopQueryHandler()); + var t1 = q1.ExecuteAsync(); // holds the only client (gated) + + var q2 = h.NewQuery().Sql("b").Handler(new NoopQueryHandler()); + var ex = Assert.ThrowsAsync(async () => await q2.ExecuteAsync()); + Assert.That(ex!.code, Is.EqualTo(ErrorCode.PoolExhausted)); + + fake.Gate.SetResult(true); + await t1; + + // q2 is not poisoned by the earlier exhaustion: a retry succeeds. + await q2.ExecuteAsync(); + Assert.That(fake.LastSql, Is.EqualTo("b")); + } +} +#endif diff --git a/src/net-questdb-client-tests/Pooling/QueryPoolConfigTests.cs b/src/net-questdb-client-tests/Pooling/QueryPoolConfigTests.cs new file mode 100644 index 0000000..32b3e50 --- /dev/null +++ b/src/net-questdb-client-tests/Pooling/QueryPoolConfigTests.cs @@ -0,0 +1,221 @@ +/******************************************************************************* + * ___ _ ____ ____ + * / _ \ _ _ ___ ___| |_| _ \| __ ) + * | | | | | | |/ _ \/ __| __| | | | _ \ + * | |_| | |_| | __/\__ \ |_| |_| | |_) | + * \__\_\\__,_|\___||___/\__|____/|____/ + * + * Copyright (c) 2014-2019 Appsicle + * Copyright (c) 2019-2026 QuestDB + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + ******************************************************************************/ + +using NUnit.Framework; +using QuestDB; +using QuestDB.Enums; +using QuestDB.Qwp.Query; +using QuestDB.Utils; + +namespace net_questdb_client_tests.Pooling; + +public class QueryPoolConfigTests +{ + [Test] + public void DefaultsAreOneAndFour() + { + var o = new SenderOptions("ws::addr=localhost:9000;"); + Assert.Multiple(() => + { + Assert.That(o.query_pool_min, Is.EqualTo(1)); + Assert.That(o.query_pool_max, Is.EqualTo(4)); + }); + } + + [Test] + public void ParsesQueryPoolKeys() + { + var o = new SenderOptions("ws::addr=localhost:9000;query_pool_min=2;query_pool_max=8;"); + Assert.Multiple(() => + { + Assert.That(o.query_pool_min, Is.EqualTo(2)); + Assert.That(o.query_pool_max, Is.EqualTo(8)); + }); + } + + [Test] + public void ValidateQueryPoolOptionsRejectsBadValues() + { + var min = new SenderOptions("ws::addr=localhost:9000;") { query_pool_min = -1 }; + var max = new SenderOptions("ws::addr=localhost:9000;") { query_pool_max = 0 }; + var minGtMax = new SenderOptions("ws::addr=localhost:9000;") { query_pool_min = 5, query_pool_max = 2 }; + + Assert.Multiple(() => + { + Assert.That(Assert.Throws(() => min.ValidateQueryPoolOptions())!.code, + Is.EqualTo(ErrorCode.ConfigError)); + Assert.That(Assert.Throws(() => max.ValidateQueryPoolOptions())!.code, + Is.EqualTo(ErrorCode.ConfigError)); + Assert.That(Assert.Throws(() => minGtMax.ValidateQueryPoolOptions())!.code, + Is.EqualTo(ErrorCode.ConfigError)); + }); + } + + [Test] + public void QueryPoolKeysAreJsonIgnoredAndRoundTrip() + { + var o = new SenderOptions("ws::addr=localhost:9000;query_pool_max=8;"); + var s = o.ToString(); + + Assert.That(s, Does.Not.Contain("query_pool_max"), "pool knobs are excluded from ToString"); + Assert.DoesNotThrow(() => _ = new SenderOptions(s), "the serialized string round-trips"); + } + + [Test] + public void QueryOptionsAcceptsQueryPoolKeys() + { + Assert.DoesNotThrow(() => _ = new QueryOptions("ws::addr=localhost:9000;query_pool_max=8;")); + } + +#if NET7_0_OR_GREATER + [Test] + public void WsSingleStringHandleHasQueryPool() + { + using var h = QuestDBClient.Connect("ws::addr=localhost:9000;sender_pool_min=0;query_pool_min=0;"); + Assert.Multiple(() => + { + Assert.That(h.TotalQueryClientCount, Is.EqualTo(0)); + Assert.DoesNotThrow(() => _ = h.NewQuery()); + }); + } + + [Test] + public void HttpHandleWithoutQueryConfigRejectsQueries() + { + using var h = QuestDBClient.Connect("http::addr=localhost:9000;sender_pool_min=0;"); + var ex = Assert.Throws(() => h.NewQuery()); + Assert.That(ex!.code, Is.EqualTo(ErrorCode.ConfigError)); + } + + [Test] + public void DistinctIngestAndQueryConfigBuildsQueryPool() + { + using var h = QuestDBClient.Connect( + "http::addr=localhost:9000;sender_pool_min=0;", + "ws::addr=localhost:9000;query_pool_min=0;"); + Assert.Multiple(() => + { + Assert.That(h.TotalQueryClientCount, Is.EqualTo(0)); + Assert.DoesNotThrow(() => _ = h.NewQuery()); + }); + } + + [Test] + public void NonWsQueryConfigIsRejected() + { + var ex = Assert.Throws(() => + QuestDBClient.Builder().IngestConfig("http::addr=localhost:9000;").QueryConfig("http::addr=localhost:9000;")); + Assert.That(ex!.code, Is.EqualTo(ErrorCode.ConfigError)); + } + + // ---- sizing precedence with a separate query config (builder call > query config > ingest config + // > default), asserted on the assembled config so no pool is pre-warmed ---- + + [Test] + public void SeparateQueryConfigWithoutSizingKeepsIngestSizing() + { + var cfg = QuestDBClient.Builder() + .IngestConfig("ws::addr=ingest:9000;query_pool_min=3;query_pool_max=8;") + .QueryConfig("wss::addr=query:9000;") + .BuildPoolConfig(out var queryConfStr, out _); + Assert.Multiple(() => + { + Assert.That(queryConfStr, Is.EqualTo("wss::addr=query:9000;")); + Assert.That(cfg.query_pool_min, Is.EqualTo(3)); + Assert.That(cfg.query_pool_max, Is.EqualTo(8)); + }); + } + + [Test] + public void SeparateQueryConfigExplicitSizingWinsOverIngest() + { + var cfg = QuestDBClient.Builder() + .IngestConfig("ws::addr=ingest:9000;query_pool_min=3;query_pool_max=8;") + .QueryConfig("wss::addr=query:9000;query_pool_min=2;query_pool_max=6;") + .BuildPoolConfig(out _, out _); + Assert.Multiple(() => + { + Assert.That(cfg.query_pool_min, Is.EqualTo(2)); + Assert.That(cfg.query_pool_max, Is.EqualTo(6)); + }); + } + + [Test] + public void SeparateQueryConfigOverridesPerKeyNotWholesale() + { + var cfg = QuestDBClient.Builder() + .IngestConfig("ws::addr=ingest:9000;query_pool_min=3;query_pool_max=8;") + .QueryConfig("wss::addr=query:9000;query_pool_max=6;") + .BuildPoolConfig(out _, out _); + Assert.Multiple(() => + { + Assert.That(cfg.query_pool_min, Is.EqualTo(3), "min not carried by the query config survives from ingest"); + Assert.That(cfg.query_pool_max, Is.EqualTo(6), "max carried by the query config wins"); + }); + } + + [Test] + public void SeparateQueryConfigExplicitDefaultEqualValuesStillWin() + { + // Explicitness is key presence, not value: 0 and the default 4 carried by the query config + // must beat the ingest values, not be mistaken for "unset". + var cfg = QuestDBClient.Builder() + .IngestConfig("ws::addr=ingest:9000;query_pool_min=3;query_pool_max=8;") + .QueryConfig("wss::addr=query:9000;query_pool_min=0;query_pool_max=4;") + .BuildPoolConfig(out _, out _); + Assert.Multiple(() => + { + Assert.That(cfg.query_pool_min, Is.EqualTo(0)); + Assert.That(cfg.query_pool_max, Is.EqualTo(4)); + }); + } + + [Test] + public void BuilderSizingWinsOverBothConfigs() + { + var cfg = QuestDBClient.Builder() + .IngestConfig("ws::addr=ingest:9000;query_pool_min=3;query_pool_max=8;") + .QueryConfig("wss::addr=query:9000;query_pool_min=5;query_pool_max=6;") + .QueryPoolMin(2) + .QueryPoolMax(9) + .BuildPoolConfig(out _, out _); + Assert.Multiple(() => + { + Assert.That(cfg.query_pool_min, Is.EqualTo(2)); + Assert.That(cfg.query_pool_max, Is.EqualTo(9)); + }); + } + + [Test] + public void MergedCrossConfigSizingIsValidated() + { + // min from the ingest string, max from the query config — the merged pair must still validate. + var ex = Assert.Throws(() => QuestDBClient.Builder() + .IngestConfig("ws::addr=ingest:9000;query_pool_min=6;") + .QueryConfig("wss::addr=query:9000;query_pool_max=2;") + .BuildPoolConfig(out _, out _)); + Assert.That(ex!.code, Is.EqualTo(ErrorCode.ConfigError)); + } +#endif +} diff --git a/src/net-questdb-client-tests/Pooling/SenderPoolTests.cs b/src/net-questdb-client-tests/Pooling/SenderPoolTests.cs new file mode 100644 index 0000000..52d1f01 --- /dev/null +++ b/src/net-questdb-client-tests/Pooling/SenderPoolTests.cs @@ -0,0 +1,476 @@ +/******************************************************************************* + * ___ _ ____ ____ + * / _ \ _ _ ___ ___| |_| _ \| __ ) + * | | | | | | |/ _ \/ __| __| | | | _ \ + * | |_| | |_| | __/\__ \ |_| |_| | |_) | + * \__\_\\__,_|\___||___/\__|____/|____/ + * + * Copyright (c) 2014-2019 Appsicle + * Copyright (c) 2019-2026 QuestDB + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + ******************************************************************************/ + +using System.Collections.Concurrent; +using System.Diagnostics; +using NUnit.Framework; +using QuestDB; +using QuestDB.Enums; +using QuestDB.Pooling; +using QuestDB.Utils; + +namespace net_questdb_client_tests.Pooling; + +public class SenderPoolTests +{ + private static SenderPool MakePool(string keys, out ConcurrentBag created) + { + var bag = new ConcurrentBag(); + created = bag; + var options = new SenderOptions("http::addr=localhost:9000;" + keys); + return new SenderPool(options, null, slot => + { + var s = new FakeSender(slot); + bag.Add(s); + return s; + }); + } + + [Test] + public void PreWarmsMinSenders() + { + using var g = new PoolGuard(MakePool("sender_pool_min=2;sender_pool_max=4;", out var created)); + var pool = g.Pool; + Assert.Multiple(() => + { + Assert.That(pool.TotalSize, Is.EqualTo(2)); + Assert.That(pool.AvailableSize, Is.EqualTo(2)); + Assert.That(created, Has.Count.EqualTo(2)); + }); + } + + [Test] + public void BorrowReturnReusesUnderlyingSenderButHandsOutFreshHandle() + { + using var g = new PoolGuard(MakePool("sender_pool_min=1;sender_pool_max=1;", out var created)); + var pool = g.Pool; + + var s1 = pool.Borrow(); + s1.Dispose(); + var s2 = pool.Borrow(); + + Assert.That(s2, Is.Not.SameAs(s1), + "each borrow gets a fresh handle so a returned-then-reused reference can't alias the next borrower"); + Assert.That(created, Has.Count.EqualTo(1), "the underlying sender is reused: no new one created on re-borrow"); + s2.Dispose(); + } + + [Test] + public void BorrowAndReturnTrackSizes() + { + using var g = new PoolGuard(MakePool("sender_pool_min=1;sender_pool_max=2;", out _)); + var pool = g.Pool; + + var s = pool.Borrow(); + Assert.Multiple(() => + { + Assert.That(pool.AvailableSize, Is.EqualTo(0)); + Assert.That(pool.TotalSize, Is.EqualTo(1)); + }); + + s.Dispose(); + Assert.Multiple(() => + { + Assert.That(pool.AvailableSize, Is.EqualTo(1)); + Assert.That(pool.TotalSize, Is.EqualTo(1)); + }); + } + + [Test] + public void ReturnDiscardsUnsentAndRepools() + { + using var g = new PoolGuard(MakePool("sender_pool_min=1;sender_pool_max=1;", out var created)); + var pool = g.Pool; + + var s = pool.Borrow(); + s.Dispose(); + + Assert.Multiple(() => + { + Assert.That(created.Single().SendCount, Is.EqualTo(0), "return does NOT send — Dispose is pure release"); + Assert.That(created.Single().ClearCount, Is.EqualTo(1), "return discards un-sent rows so the entry is clean"); + Assert.That(created.Single().Disposed, Is.False, "return does not dispose the real sender"); + }); + } + + [Test] + public void BrokenSenderIsDiscardedNotReturned() + { + using var g = new PoolGuard(MakePool("sender_pool_min=0;sender_pool_max=2;", out var created)); + var pool = g.Pool; + + var s = pool.Borrow(); + created.Single().ThrowOnClear = true; + s.Dispose(); // the buffer can't be cleared (terminal) -> discard rather than re-pool + + Assert.Multiple(() => + { + Assert.That(pool.AvailableSize, Is.EqualTo(0), "broken sender not re-pooled"); + Assert.That(pool.TotalSize, Is.EqualTo(0), "broken sender removed from pool"); + Assert.That(created.Single().Disposed, Is.True, "broken sender disposed for real"); + }); + + // capacity restored: a fresh sender can still be borrowed + var s2 = pool.Borrow(); + Assert.That(created, Has.Count.EqualTo(2)); + s2.Dispose(); + } + + [Test] + public void TransactionalSenderWithStagedRowsIsDiscardedNotReturned() + { + using var g = new PoolGuard(MakePool("sender_pool_min=0;sender_pool_max=2;", out var created)); + var pool = g.Pool; + + var s = pool.Borrow(); + // Rows already shipped under FLAG_DEFER_COMMIT can't be rolled back: re-pooling would let the + // next borrower's first commit publish them. The return must discard the entry instead. + created.Single().HasUncommittedDeferredRows = true; + s.Dispose(); + + Assert.Multiple(() => + { + Assert.That(pool.AvailableSize, Is.EqualTo(0), "sender owing a commit not re-pooled"); + Assert.That(pool.TotalSize, Is.EqualTo(0), "sender owing a commit removed from pool"); + Assert.That(created.Single().ClearCount, Is.EqualTo(1), "local buffers still discarded first"); + Assert.That(created.Single().Disposed, Is.True, "disposed for real — closing the connection drops the staged rows"); + }); + + // capacity restored: a fresh sender can still be borrowed + var s2 = pool.Borrow(); + Assert.That(created, Has.Count.EqualTo(2)); + s2.Dispose(); + } + + [Test] + public void TransactionalSenderWithNoStagedRowsIsRepooled() + { + using var g = new PoolGuard(MakePool("sender_pool_min=0;sender_pool_max=1;", out var created)); + var pool = g.Pool; + + var s = pool.Borrow(); + created.Single().HasUncommittedDeferredRows = false; // commit not owed (committed, or never deferred) + s.Dispose(); + + Assert.Multiple(() => + { + Assert.That(pool.AvailableSize, Is.EqualTo(1), "clean transactional sender re-pools as usual"); + Assert.That(created.Single().Disposed, Is.False); + }); + } + + [Test] + public async Task AsyncTransactionalSenderWithStagedRowsIsDiscarded() + { + using var g = new PoolGuard(MakePool("sender_pool_min=0;sender_pool_max=1;", out var created)); + var pool = g.Pool; + + var s = await pool.BorrowAsync(); + created.Single().HasUncommittedDeferredRows = true; + await s.DisposeAsync(); + + Assert.That(pool.TotalSize, Is.EqualTo(0)); + Assert.That(created.Single().Disposed, Is.True); + } + + [Test] + public void DisposeNeverThrows() + { + using var g = new PoolGuard(MakePool("sender_pool_min=1;sender_pool_max=1;", out var created)); + var pool = g.Pool; + + var s = pool.Borrow(); + created.Single().ThrowOnClear = true; + // Even when the inner cannot be cleared AND its teardown throws, Dispose must swallow both. + created.Single().ThrowOnDispose = true; + + Assert.DoesNotThrow(() => s.Dispose()); + } + + [Test] + public void FlushDrainsEverySenderInThePool() + { + using var g = new PoolGuard(MakePool("sender_pool_min=3;sender_pool_max=3;", out var created)); + var pool = g.Pool; + + var drained = pool.Flush(TimeSpan.FromSeconds(1)); + + Assert.Multiple(() => + { + Assert.That(drained, Is.True, "all senders drained"); + Assert.That(created, Has.Count.EqualTo(3)); + Assert.That(created.All(s => s.FlushCount == 1), Is.True, "every pooled sender was drained once"); + Assert.That(created.All(s => s.SendCount == 0), Is.True, "Flush drains via Flush(), not Send()"); + }); + } + + [Test] + public void FlushReturnsFalseWhenASenderDoesNotDrain() + { + using var g = new PoolGuard(MakePool("sender_pool_min=2;sender_pool_max=2;", out var created)); + var pool = g.Pool; + + created.First().ThrowOnFlush = true; + + Assert.That(pool.Flush(TimeSpan.FromSeconds(1)), Is.False, + "one sender failing to drain makes the pool-wide Flush report failure"); + Assert.That(created.All(s => s.FlushCount == 1), Is.True, "a failing sender does not abort draining the rest"); + } + + [Test] + public void GrowsOnDemandUpToMax() + { + using var g = new PoolGuard(MakePool("sender_pool_min=0;sender_pool_max=3;acquire_timeout_ms=100;", out var created)); + var pool = g.Pool; + + var a = pool.Borrow(); + var b = pool.Borrow(); + var c = pool.Borrow(); + Assert.That(pool.TotalSize, Is.EqualTo(3)); + Assert.That(created, Has.Count.EqualTo(3)); + + Assert.Throws(() => pool.Borrow()); + + a.Dispose(); + b.Dispose(); + c.Dispose(); + } + + [Test] + public void ExhaustionThrowsAfterAcquireTimeout() + { + using var g = new PoolGuard(MakePool("sender_pool_min=0;sender_pool_max=1;acquire_timeout_ms=200;", out _)); + var pool = g.Pool; + + var held = pool.Borrow(); + var sw = Stopwatch.StartNew(); + var ex = Assert.Throws(() => pool.Borrow()); + sw.Stop(); + + Assert.That(ex!.code, Is.EqualTo(ErrorCode.PoolExhausted)); + Assert.That(sw.ElapsedMilliseconds, Is.GreaterThanOrEqualTo(150), "blocked for ~acquire_timeout"); + held.Dispose(); + } + + [Test] + public void CloseDisposesEverySender() + { + var pool = MakePool("sender_pool_min=2;sender_pool_max=4;", out var created); + pool.Close(); + + Assert.Multiple(() => + { + Assert.That(pool.AvailableSize, Is.EqualTo(0)); + Assert.That(pool.TotalSize, Is.EqualTo(0)); + Assert.That(created.All(s => s.Disposed), Is.True); + }); + } + + [Test] + public void CloseIsIdempotent() + { + var pool = MakePool("sender_pool_min=1;sender_pool_max=2;", out _); + pool.Close(); + Assert.DoesNotThrow(() => pool.Close()); + } + + [Test] + public void BorrowAfterCloseThrows() + { + var pool = MakePool("sender_pool_min=1;sender_pool_max=2;", out _); + pool.Close(); + var ex = Assert.Throws(() => pool.Borrow()); + Assert.That(ex!.Message, Does.Contain("closed")); + } + + [Test] + public void CloseLeavesBorrowedSenderForBorrowerToTearDown() + { + // Close must dispose only idle senders; an in-use one is torn down by its borrower on return, + // never by Close (which would race the borrower's flush on a non-thread-safe inner). + var pool = MakePool("sender_pool_min=2;sender_pool_max=2;", out var created); + + var borrowed = pool.Borrow(); // 1 in-use, 1 idle + + pool.Close(); + + var inUse = created.Single(s => !s.Disposed); // the borrowed sender survived close untouched + Assert.Multiple(() => + { + Assert.That(created.Count(s => s.Disposed), Is.EqualTo(1), "idle sender disposed by close"); + Assert.That(inUse.DisposeCount, Is.EqualTo(0), "not torn down yet — borrower hasn't returned"); + }); + + borrowed.Dispose(); // borrower returns post-close: discard un-sent rows, then teardown, same thread + + Assert.Multiple(() => + { + Assert.That(inUse.SendCount, Is.EqualTo(0), "return never sends"); + Assert.That(inUse.ClearCount, Is.EqualTo(1), "return discarded un-sent rows exactly once"); + Assert.That(inUse.DisposeCount, Is.EqualTo(1), "inner disposed exactly once, by the borrower"); + Assert.That(inUse.DisposedDuringClear, Is.False, "clear and dispose never overlapped"); + }); + + Assert.DoesNotThrow(() => pool.Close(), "second close is idempotent"); + } + + [Test] + public void CloseDoesNotDisposeASenderWhileItIsReturning() + { + // The M1 race, retargeted to the current return path: Close() running concurrently with a + // borrower's Dispose (which now discards un-sent rows via Clear, not a send) on the same + // non-thread-safe inner sender. The gate parks the borrowed sender mid-return so Close is forced + // to run during that exact window. + using var gate = new ManualResetEventSlim(false); + var pool = MakePool("sender_pool_min=2;sender_pool_max=2;", out var created); + foreach (var s in created) + { + s.ClearGate = gate; // only the borrowed sender actually calls Clear and parks here + } + + var borrowed = pool.Borrow(); // 1 in-use, 1 idle + + var ret = Task.Run(() => borrowed.Dispose()); // enters the return path, parks inside Clear + Assert.That(SpinWait.SpinUntil(() => created.Any(s => s.ClearInProgress), TimeSpan.FromSeconds(5)), + Is.True, "borrowed sender reached its return"); + var returning = created.Single(s => s.ClearInProgress); + + pool.Close(); // concurrent with the parked return + + Assert.Multiple(() => + { + Assert.That(returning.Disposed, Is.False, "close must not dispose a sender mid-return"); + Assert.That(returning.DisposedDuringClear, Is.False, "no Dispose raced the in-flight return"); + Assert.That(created.Count(s => s.Disposed), Is.EqualTo(1), "only the idle sender was disposed"); + }); + + gate.Set(); // let the return finish; borrower now tears its sender down (post-close return path) + Assert.That(ret.Wait(TimeSpan.FromSeconds(5)), Is.True, "return completed"); + + Assert.Multiple(() => + { + Assert.That(returning.SendCount, Is.EqualTo(0), "return never sends"); + Assert.That(returning.ClearCount, Is.EqualTo(1)); + Assert.That(returning.DisposeCount, Is.EqualTo(1), "inner disposed exactly once, by its borrower"); + Assert.That(returning.DisposedDuringClear, Is.False); + }); + } + + [Test] + public async Task AsyncBorrowAndReturn() + { + using var g = new PoolGuard(MakePool("sender_pool_min=0;sender_pool_max=1;", out var created)); + var pool = g.Pool; + + var s = await pool.BorrowAsync(); + Assert.That(pool.AvailableSize, Is.EqualTo(0)); + await s.DisposeAsync(); + Assert.That(pool.AvailableSize, Is.EqualTo(1)); + Assert.That(created.Single().SendCount, Is.EqualTo(0), "async return never sends"); + Assert.That(created.Single().ClearCount, Is.EqualTo(1), "async return discarded un-sent rows"); + } + + [Test] + public async Task AsyncBrokenSenderDiscarded() + { + using var g = new PoolGuard(MakePool("sender_pool_min=0;sender_pool_max=1;", out var created)); + var pool = g.Pool; + + var s = await pool.BorrowAsync(); + created.Single().ThrowOnClear = true; + await s.DisposeAsync(); + + Assert.That(pool.TotalSize, Is.EqualTo(0)); + Assert.That(created.Single().Disposed, Is.True); + } + + [Test] + public void ConcurrentBorrowsNeverExceedMax() + { + using var g = new PoolGuard(MakePool("sender_pool_min=0;sender_pool_max=4;acquire_timeout_ms=5000;", out var created)); + var pool = g.Pool; + + Parallel.For(0, 200, _ => + { + var s = pool.Borrow(); + Thread.SpinWait(200); + Assert.That(pool.TotalSize, Is.LessThanOrEqualTo(4)); + s.Dispose(); + }); + + Assert.That(created, Has.Count.LessThanOrEqualTo(4), "never created more than max underlying senders"); + Assert.That(pool.TotalSize, Is.LessThanOrEqualTo(4)); + } + + [Test] + public async Task HandleBorrowReturnsWorkingSender() + { + // End-to-end through the public QuestDBClient handle with an injected factory. + var bag = new ConcurrentBag(); + var options = new SenderOptions("http::addr=localhost:9000;sender_pool_min=1;sender_pool_max=2;"); + await using IQuestDBClient client = new QuestDBClientImpl(options, slot => + { + var s = new FakeSender(slot); + bag.Add(s); + return s; + }); + + using (var sender = client.BorrowSender()) + { + sender.Table("t").Column("x", 1).At(DateTime.UtcNow); + sender.Send(); // explicit send — Dispose no longer flushes + } + + Assert.That(client.AvailableSenderCount, Is.EqualTo(1)); + Assert.That(client.TotalSenderCount, Is.EqualTo(1)); + Assert.That(bag.Single().SendCount, Is.EqualTo(1), "rows were sent explicitly, not on dispose"); + } + + [Test] + public void ConnectBuildsRealHttpSenders() + { + // Exercises the production factory path (CreateDefaultInner -> Sender.New -> HttpSender) and + // real PooledSender delegation. No rows are written, so the return-clear is a no-op and no + // server is contacted. + using var client = QuestDBClient.Connect( + "http::addr=localhost:9000;sender_pool_min=1;sender_pool_max=2;"); + + Assert.That(client.TotalSenderCount, Is.EqualTo(1)); + + using (var s = client.BorrowSender()) + { + Assert.That(s.Options.protocol, Is.EqualTo(ProtocolType.http)); + } + + Assert.That(client.AvailableSenderCount, Is.EqualTo(1)); + } + + // Disposes the pool even when an assertion fails mid-test. + private sealed class PoolGuard : IDisposable + { + public PoolGuard(SenderPool pool) => Pool = pool; + public SenderPool Pool { get; } + public void Dispose() => Pool.Close(); + } +} diff --git a/src/net-questdb-client-tests/Pooling/SfPoolConcurrencyTests.cs b/src/net-questdb-client-tests/Pooling/SfPoolConcurrencyTests.cs new file mode 100644 index 0000000..caaf7c4 --- /dev/null +++ b/src/net-questdb-client-tests/Pooling/SfPoolConcurrencyTests.cs @@ -0,0 +1,379 @@ +/******************************************************************************* + * ___ _ ____ ____ + * / _ \ _ _ ___ ___| |_| _ \| __ ) + * | | | | | | |/ _ \/ __| __| | | | _ \ + * | |_| | |_| | __/\__ \ |_| |_| | |_) | + * \__\_\\__,_|\___||___/\__|____/|____/ + * + * Copyright (c) 2014-2019 Appsicle + * Copyright (c) 2019-2026 QuestDB + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + ******************************************************************************/ + +using System.Collections.Concurrent; +using NUnit.Framework; +using QuestDB.Enums; +using QuestDB.Pooling; +using QuestDB.Utils; + +namespace net_questdb_client_tests.Pooling; + +/// +/// Store-and-forward (ws + sf_dir) concurrency stress for the slot-index machinery — +/// AllocateSlotIndex / FreeSlotIndex / TryWithholdPermit / +/// SettleAfterDispose / ReclaimRetiredSlots and the permit-withholding contract they +/// share. That code is the most race-prone in the pool, yet the rest of the pool's +/// multi-threaded coverage runs only against HTTP pools, which carry no slot machinery at all +/// ( exercises these paths only +/// single-threaded). These tests drive the same paths under contention and assert the two +/// invariants the machinery must never break: (1) no two concurrently-borrowed senders ever share a +/// slot index, and (2) after a retire/reclaim storm fully quiesces, effective capacity is restored +/// to exactly max — no permanent shrink, no over-release. +/// +public class SfPoolConcurrencyTests +{ + private const int Max = 8; + + // SF pool over the fake-sender seam: ws + sf_dir flips the pool into store-and-forward mode (slot + // indices, retire/reclaim) without touching a socket or the filesystem. `ownerByIndex[i]` is the fake + // currently backing slot i — written by the factory at creation; since a slot index is held + // continuously by one PooledSender entry until that entry is discarded/reaped, a borrower can map its + // slot index back to its inner fake (to drive retire) with no race. + private static SenderPool MakeSfPool( + string keys, out ConcurrentBag created, out FakeSender?[] ownerByIndex) + { + var bag = new ConcurrentBag(); + var owners = new FakeSender?[Max]; + created = bag; + ownerByIndex = owners; + var options = new SenderOptions("ws::addr=localhost:9000;sf_dir=/tmp/qdb-sf-pool-stress;" + keys); + return new SenderPool(options, null, slot => + { + var s = new FakeSender(slot); + bag.Add(s); + if (slot >= 0 && slot < owners.Length) + { + Volatile.Write(ref owners[slot], s); + } + + return s; + }); + } + + // Asserts effective capacity is exactly `expected`: that many borrows succeed with distinct indices + // in [0, Max), and one more times out with the documented PoolExhausted error. Leaves the pool empty. + private static void AssertEffectiveCapacity(SenderPool pool, int expected) + { + var held = new List(expected); + for (var i = 0; i < expected; i++) + { + held.Add(pool.Borrow()); + } + + var indices = held.Select(h => h.SlotIndex).OrderBy(i => i).ToArray(); + Assert.That(indices, Is.EqualTo(Enumerable.Range(0, expected)), + "every live SF sender holds a distinct lowest-fill slot index"); + + var ex = Assert.Throws(() => pool.Borrow()); + Assert.That(ex!.code, Is.EqualTo(ErrorCode.PoolExhausted), "capacity is exactly the restored max"); + + foreach (var h in held) + { + h.Dispose(); + } + } + + [Test] + public void SfConcurrentCleanChurnNeverDuplicatesLiveSlotIndex() + { + // Pure borrow/return churn (no breaks) hammering AllocateSlotIndex during pool growth and + // FreeSlotIndex is never hit on a clean return, but the free-index stack under TakeOrCreate is. + // Two concurrently-borrowed senders sharing an index would be a correctness bug. + var pool = MakeSfPool("sender_pool_min=0;sender_pool_max=8;acquire_timeout_ms=5000;", out _, out _); + var liveSlot = new int[Max]; + var errors = new ConcurrentQueue(); + try + { + Parallel.For(0, 6000, new ParallelOptions { MaxDegreeOfParallelism = 16 }, _ => + { + var ps = pool.Borrow(); + var idx = ps.SlotIndex; + if (idx < 0 || idx >= Max) + { + errors.Enqueue($"slot index out of range: {idx}"); + } + else if (Interlocked.Exchange(ref liveSlot[idx], 1) != 0) + { + errors.Enqueue($"two live senders share slot index {idx}"); + } + + Thread.SpinWait(100); + + if (idx >= 0 && idx < Max) + { + Interlocked.Exchange(ref liveSlot[idx], 0); + } + + ps.Dispose(); + }); + + Assert.That(errors, Is.Empty); + Assert.That(pool.LeakedSlotCount, Is.EqualTo(0)); + Assert.That(pool.TotalSize, Is.LessThanOrEqualTo(Max)); + + // Clean churn must not have drifted capacity: exactly Max distinct slots remain borrowable. + AssertEffectiveCapacity(pool, Max); + } + finally + { + pool.Close(); + } + } + + [Test] + public void SfConcurrentRetireAndReclaimKeepsCapacityAccountingConsistent() + { + // The core race: workers retire slots (discard with the slot lock still held -> SettleAfterDispose + // parks the entry on _retired with its permit withheld, shrinking effective capacity) while a + // housekeeper thread releases those locks and reclaims the indices (ReclaimRetiredSlots, restoring + // capacity). Retire and reclaim mutate _retired and withhold/release the capacity semaphore from + // different threads continuously. The invariant: once everything quiesces and all retired slots + // are reclaimed, effective capacity is back to exactly Max — every withheld permit is released + // exactly once, none lost or double-released. + var pool = MakeSfPool( + "sender_pool_min=0;sender_pool_max=8;acquire_timeout_ms=200;idle_timeout_ms=1;", + out var created, out var ownerByIndex); + var liveSlot = new int[Max]; + var errors = new ConcurrentQueue(); + var stop = false; + + try + { + var keeper = new Thread(() => + { + while (!Volatile.Read(ref stop)) + { + foreach (var s in created.ToArray()) + { + s.SlotLockReleased = true; // a deferred teardown finally drops the OS lock + } + + try + { + pool.ReclaimRetiredSlots(); + pool.ReapIdle(); + } + catch (Exception e) + { + errors.Enqueue("keeper: " + e); + } + + Thread.Sleep(1); + } + }); + + var remaining = 6000; + var workers = Enumerable.Range(0, 12).Select(_ => new Thread(() => + { + while (Interlocked.Decrement(ref remaining) >= 0) + { + BorrowedSender ps; + try + { + ps = pool.Borrow(); + } + catch (IngressError) + { + continue; // transient capacity shrink -> PoolExhausted is expected, not a fault + } + + var idx = ps.SlotIndex; + if (idx < 0 || idx >= Max) + { + errors.Enqueue($"slot index out of range: {idx}"); + continue; + } + + if (Interlocked.Exchange(ref liveSlot[idx], 1) != 0) + { + errors.Enqueue($"two live senders share slot index {idx}"); + } + + // ~1/3 of borrows retire: break the inner and keep its slot lock held so the discard + // retires the index (capacity shrink) rather than freeing it. + if (idx % 3 == 0) + { + var inner = Volatile.Read(ref ownerByIndex[idx]); + if (inner != null) + { + inner.SlotLockReleased = false; + inner.ThrowOnClear = true; + } + } + + Thread.SpinWait(100); + Interlocked.Exchange(ref liveSlot[idx], 0); + ps.Dispose(); // clean return, or discard->retire when this borrow was marked above + } + })).ToArray(); + + keeper.Start(); + foreach (var w in workers) + { + w.Start(); + } + + foreach (var w in workers) + { + Assert.That(w.Join(TimeSpan.FromSeconds(30)), Is.True, "worker finished"); + } + + Volatile.Write(ref stop, true); + Assert.That(keeper.Join(TimeSpan.FromSeconds(30)), Is.True, "keeper finished"); + + // Quiesce: every lock is now releasable, so a final reclaim sweep must drain every retired + // slot back into circulation. + foreach (var s in created.ToArray()) + { + s.SlotLockReleased = true; + } + + for (var i = 0; i < 16 && pool.LeakedSlotCount > 0; i++) + { + pool.ReclaimRetiredSlots(); + } + + Assert.That(errors, Is.Empty); + Assert.That(pool.LeakedSlotCount, Is.EqualTo(0), "every retired slot reclaimed once locks released"); + + // The decisive end-to-end check: capacity is restored to exactly Max. Under-counting leak debt + // would leave it < Max (PoolExhausted before Max borrows); over-releasing would let a Max+1th + // borrow through. + AssertEffectiveCapacity(pool, Max); + } + finally + { + Volatile.Write(ref stop, true); + pool.Close(); + } + } + + [Test] + public void SfConcurrentBorrowDiscardReapAndCloseStayConsistent() + { + // Close racing the full SF slot storm: borrows, retire-discards, reaping and reclaiming all in + // flight when the handle closes. Close clears _retired while SettleAfterDispose appends to it, and + // disposes the capacity semaphore under returning permits. Must never corrupt the live-slot set, + // never surface an unexpected exception, and stay idempotent. + var pool = MakeSfPool( + "sender_pool_min=2;sender_pool_max=8;acquire_timeout_ms=2000;idle_timeout_ms=1;", + out var created, out var ownerByIndex); + var liveSlot = new int[Max]; + var errors = new ConcurrentQueue(); + var stop = false; + + var keeper = new Thread(() => + { + while (!Volatile.Read(ref stop)) + { + foreach (var s in created.ToArray()) + { + s.SlotLockReleased = true; + } + + try + { + pool.ReapIdle(); + pool.ReclaimRetiredSlots(); + } + catch (Exception e) + { + errors.Enqueue("keeper: " + e); + } + + Thread.Sleep(1); + } + }); + + var workers = Enumerable.Range(0, 10).Select(_ => new Thread(() => + { + var n = 0; + while (!Volatile.Read(ref stop)) + { + BorrowedSender ps; + try + { + ps = pool.Borrow(); + } + catch (IngressError) + { + continue; // exhausted or closed: both expected during the storm / after close + } + + var idx = ps.SlotIndex; + if (idx < 0 || idx >= Max) + { + errors.Enqueue($"slot index out of range: {idx}"); + continue; + } + + if (Interlocked.Exchange(ref liveSlot[idx], 1) != 0) + { + errors.Enqueue($"two live senders share slot index {idx}"); + } + + if ((n++ & 1) == 0) + { + var inner = Volatile.Read(ref ownerByIndex[idx]); + if (inner != null) + { + inner.SlotLockReleased = false; + inner.ThrowOnClear = true; + } + } + + Interlocked.Exchange(ref liveSlot[idx], 0); + try + { + ps.Dispose(); + } + catch (Exception e) + { + errors.Enqueue("dispose: " + e); + } + } + })).ToArray(); + + keeper.Start(); + foreach (var w in workers) + { + w.Start(); + } + + Thread.Sleep(400); // let the storm build up retired slots and in-flight borrows + pool.Close(); // race the close against the slot machinery + Volatile.Write(ref stop, true); + + foreach (var w in workers) + { + Assert.That(w.Join(TimeSpan.FromSeconds(15)), Is.True, "worker drained after close"); + } + + Assert.That(keeper.Join(TimeSpan.FromSeconds(15)), Is.True, "keeper drained after close"); + + Assert.That(errors, Is.Empty, "no slot corruption or unexpected exception under borrow/retire/reap/close"); + Assert.DoesNotThrow(() => pool.Close(), "close is idempotent"); + } +} diff --git a/src/net-questdb-client-tests/QuestDbQueryIntegrationTests.cs b/src/net-questdb-client-tests/QuestDbQueryIntegrationTests.cs index 713262b..0021259 100644 --- a/src/net-questdb-client-tests/QuestDbQueryIntegrationTests.cs +++ b/src/net-questdb-client-tests/QuestDbQueryIntegrationTests.cs @@ -221,6 +221,36 @@ private async Task SeedFixtureTableAsync() .At(DateTime.UtcNow); } await sender.SendAsync(); + + // ILP SendAsync returns once the rows are durably in the WAL, but the WAL applies to the + // table asynchronously. Block setup until all five rows are queryable so no test can race + // the apply — otherwise the first reader of this table (Bind_ParameterFiltersTable, which + // runs before SelectFromSeededTable alphabetically) intermittently sees zero rows. + await WaitForSeededRowsAsync("qwp_egress_int_test", expected: 5); + } + + private async Task WaitForSeededRowsAsync(string table, long expected) + { + using var http = new HttpClient { Timeout = TimeSpan.FromSeconds(10) }; + var endpoint = _questDb!.GetHttpEndpoint(); + for (var attempt = 0; attempt < 120; attempt++) + { + using var resp = await http.GetAsync( + $"http://{endpoint}/exec?query={Uri.EscapeDataString($"SELECT count(*) FROM {table}")}"); + if (resp.IsSuccessStatusCode) + { + var body = await resp.Content.ReadAsStringAsync(); + using var json = JsonDocument.Parse(body); + if (json.RootElement.TryGetProperty("dataset", out var ds) + && ds.GetArrayLength() > 0 && ds[0].GetArrayLength() > 0 + && ds[0][0].GetInt64() >= expected) + { + return; + } + } + await Task.Delay(250); + } + Assert.Fail($"Seed of {expected} rows into {table} did not become queryable"); } private sealed class RecordingHandler : QwpColumnBatchHandler diff --git a/src/net-questdb-client-tests/QuestDbWebSocketIntegrationTests.cs b/src/net-questdb-client-tests/QuestDbWebSocketIntegrationTests.cs index aff313d..e49de8c 100644 --- a/src/net-questdb-client-tests/QuestDbWebSocketIntegrationTests.cs +++ b/src/net-questdb-client-tests/QuestDbWebSocketIntegrationTests.cs @@ -253,6 +253,7 @@ public async Task Reconnect_DuringDbRestart_SfReplaysAllRows() await DropTableAsync("test_ws_restart"); var endpoint = _questDb!.GetWebSocketEndpoint(); var sfRoot = Path.Combine(Path.GetTempPath(), "qdb-int-restart-" + Guid.NewGuid().ToString("N")); + Task? restart = null; try { using (var sender = Sender.New( @@ -267,22 +268,36 @@ public async Task Reconnect_DuringDbRestart_SfReplaysAllRows() { sender.Table("test_ws_restart").Column("v", (long)i).At(DateTime.UtcNow); } - await sender.SendAsync(); - - await _questDb.StartAsync(); - var qwp = (IQwpWebSocketSender)sender; - for (var i = 0; i < 200; i++) + // SendAsync drains: it flushes the buffered rows into the store-and-forward ring and + // then blocks until the server ACKs them. With the DB down that ACK can only arrive + // once the SF engine reconnects, so the DB must come back *while* SendAsync is + // awaiting — bring it up concurrently, within the reconnect budget, and let the + // engine replay the buffered rows across the restart. (Restarting only *after* + // awaiting SendAsync would deadlock: the send can never complete against a dead DB.) + restart = Task.Run(async () => { - try { await qwp.PingAsync(); break; } - catch { await Task.Delay(100); } - } + await Task.Delay(1000); + await _questDb.StartAsync(); + }); + await sender.SendAsync(); + await restart; } await VerifyTableRowCountAsync("test_ws_restart", expected: 30, maxAttempts: 150); } finally { + // The shared fixture DB must be left running for the remaining tests in this fixture, + // regardless of how this test exited (StartAsync is a no-op when it is already up). + // Without this, a failure here strands the DB down and cascades into every subsequent + // WebSocket test as "connection refused". + if (restart is not null) + { + try { await restart; } catch { /* surfaced by the await/assert above */ } + } + await _questDb!.StartAsync(); + if (Directory.Exists(sfRoot)) { try { Directory.Delete(sfRoot, recursive: true); } catch { } diff --git a/src/net-questdb-client-tests/Qwp/Query/QueryOptionsTests.cs b/src/net-questdb-client-tests/Qwp/Query/QueryOptionsTests.cs index 0ccfe76..a715fcc 100644 --- a/src/net-questdb-client-tests/Qwp/Query/QueryOptionsTests.cs +++ b/src/net-questdb-client-tests/Qwp/Query/QueryOptionsTests.cs @@ -75,6 +75,37 @@ public void InitialCredit_Negative_Rejected() StringAssert.Contains("initial_credit", ex!.Message); } + [Test] + public void ConnectTimeout_Default_InheritsAuthTimeoutMs() + { + var o = new QueryOptions("ws::addr=h:9000;"); + Assert.That(o.connect_timeout, Is.Null); + Assert.That(o.EffectiveConnectTimeout, Is.EqualTo(o.auth_timeout_ms)); + } + + [Test] + public void Parse_ConnectTimeout_OverridesAuthTimeoutMs() + { + var o = new QueryOptions("ws::addr=h:9000;auth_timeout_ms=4000;connect_timeout=2000;"); + Assert.That(o.connect_timeout!.Value, Is.EqualTo(TimeSpan.FromMilliseconds(2000))); + Assert.That(o.EffectiveConnectTimeout, Is.EqualTo(TimeSpan.FromMilliseconds(2000))); + } + + [Test] + public void ConnectTimeout_Unset_InheritsExplicitAuthTimeoutMs() + { + var o = new QueryOptions("ws::addr=h:9000;auth_timeout_ms=4000;"); + Assert.That(o.EffectiveConnectTimeout, Is.EqualTo(TimeSpan.FromMilliseconds(4000))); + } + + [Test] + public void ConnectTimeout_NonPositive_Rejected() + { + var o = new QueryOptions { addr = "h:9000", connect_timeout = TimeSpan.Zero }; + var ex = Assert.Throws(() => o.EnsureValid()); + StringAssert.Contains("connect_timeout", ex!.Message); + } + [Test] public void Parse_MinimalWs_AssignsAddr() { @@ -111,6 +142,19 @@ public void Parse_Wss_SwitchesProtocol() Assert.That(o.protocol, Is.EqualTo(ProtocolType.wss)); } + // The ingest parser (SenderOptions) and QuestDBClientBuilder.IsWebSocketScheme both accept + // the scheme case-insensitively; the egress parser must match so an uppercased ws::/wss:: + // string that builds a query pool doesn't get rejected at query time. + [TestCase("WS::addr=h:9000;", ProtocolType.ws)] + [TestCase("Ws::addr=h:9000;", ProtocolType.ws)] + [TestCase("WSS::addr=h:443;", ProtocolType.wss)] + [TestCase("Wss::addr=h:443;", ProtocolType.wss)] + public void Parse_SchemeIsCaseInsensitive(string connStr, ProtocolType expected) + { + var o = new QueryOptions(connStr); + Assert.That(o.protocol, Is.EqualTo(expected)); + } + [Test] public void Parse_AllEgressKnobs_RoundTrip() { @@ -339,6 +383,8 @@ public void Parse_BadCompressionLevel_Rejected(int level) [TestCase(1)] [TestCase(5)] [TestCase(9)] + [TestCase(10)] + [TestCase(22)] public void Parse_CompressionLevelInRange_Accepted(int level) { Assert.DoesNotThrow(() => new QueryOptions( diff --git a/src/net-questdb-client-tests/Qwp/Query/QwpQueryClientEndToEndTests.cs b/src/net-questdb-client-tests/Qwp/Query/QwpQueryClientEndToEndTests.cs index 972a94d..2b65afe 100644 --- a/src/net-questdb-client-tests/Qwp/Query/QwpQueryClientEndToEndTests.cs +++ b/src/net-questdb-client-tests/Qwp/Query/QwpQueryClientEndToEndTests.cs @@ -665,6 +665,51 @@ public async Task UserCancellation_AbortsSocketAndMarksClientTerminal() Assert.That(ex.Message, Does.Contain("terminal")); } + [Test] + public async Task StaleCancelRequest_DoesNotRegressTheCancelMarker() + { + // A pooled cancel resolved against an earlier query can dispatch late, after newer queries + // ran on the same client. CancelCore's marker must be monotonic: the stale rid must neither + // send a CANCEL nor overwrite a newer pending cancel's marker (which would erase that + // query's failover abort, checked by rid equality in ExecuteCoreAsync). + var schema = new ResultSchema { Columns = { new SchemaColumn("c", QwpTypeCode.Long) } }; + await using var server = new DummyQwpServer(new DummyQwpServerOptions + { + Path = QwpConstants.ReadPath, + NegotiatedVersion = "1", + FrameHandlerMulti = frame => + { + if (frame[0] != QwpConstants.MsgKindQueryRequest) return null; + var rid = BinaryPrimitives.ReadInt64LittleEndian(frame.AsSpan(1, 8)); + return new[] + { + QwpEgressFrameBuilder.BuildResultBatch(rid, 0L, schema, + new ResultBatchData { RowCount = 1, Columns = { new FixedColumnData { DenseBytes = LongLe(1L) } } }), + QwpEgressFrameBuilder.BuildResultEnd(rid, 0L, 1L), + }; + }, + }); + await server.StartAsync(); + + using var client = QueryClient.New(BuildConnString(server)); + var inner = (QwpQueryWebSocketClient)client; + var seam = (QuestDB.Pooling.IPooledQueryClientInner)client; + + client.Execute("SELECT 1", new RecordingHandler()); // rid 1, completes + client.Execute("SELECT 2", new RecordingHandler()); // rid 2, completes + + seam.CancelRequest(2); + Assert.That(inner.CancelTargetRid, Is.EqualTo(2L)); + + seam.CancelRequest(1); // stale cancel dispatched late — must not regress the marker + Assert.That(inner.CancelTargetRid, Is.EqualTo(2L), "a stale rid must never regress the cancel marker"); + + // The stale rids matched no in-flight query, so nothing was cancelled and the client is intact. + var handler = new RecordingHandler(); + client.Execute("SELECT 3", handler); + Assert.That(handler.Ended, Is.True); + } + [Test] public async Task SqlExceedingMaxLength_Throws() { @@ -1206,9 +1251,55 @@ public void AuthTimeout_BoundsConnectAttemptToBlackholeHost() sw.Stop(); Assert.That(ex!.code, Is.EqualTo(ErrorCode.SocketError)); - StringAssert.Contains("auth_timeout", ex.Message); + // auth_timeout_ms is the legacy alias of connect_timeout on the egress path; the bound is + // surfaced under the canonical connect_timeout name. + StringAssert.Contains("connect_timeout", ex.Message); + Assert.That(sw.ElapsedMilliseconds, Is.LessThan(3000), + "auth_timeout_ms=300ms should bound connect well below OS-level TCP timeout"); + } + finally + { + acceptCts.Cancel(); + listener.Stop(); + foreach (var c in held) try { c.Close(); } catch { } + } + } + + [Test] + public void ConnectTimeout_BoundsUpgradeToBlackholeHost() + { + // The TCP connect succeeds (listener accepts) but the WebSocket upgrade never completes, so + // connect_timeout must abort the attempt at the upgrade layer. + var listener = new TcpListener(IPAddress.Loopback, 0); + listener.Start(); + var port = ((IPEndPoint)listener.LocalEndpoint).Port; + var held = new List(); + using var acceptCts = new CancellationTokenSource(); + var acceptTask = Task.Run(async () => + { + try + { + while (!acceptCts.IsCancellationRequested) + { + var c = await listener.AcceptTcpClientAsync(acceptCts.Token); + held.Add(c); + } + } + catch { } + }); + + try + { + var conn = $"ws::addr=127.0.0.1:{port};path={QwpConstants.ReadPath};" + + "connect_timeout=300;failover=off;"; + var sw = Stopwatch.StartNew(); + var ex = Assert.Throws(() => QueryClient.New(conn)); + sw.Stop(); + + Assert.That(ex!.code, Is.EqualTo(ErrorCode.SocketError)); + StringAssert.Contains("connect_timeout", ex.Message); Assert.That(sw.ElapsedMilliseconds, Is.LessThan(3000), - "auth_timeout=300ms should bound connect well below OS-level TCP timeout"); + "connect_timeout=300ms should bound the upgrade well below OS-level TCP timeout"); } finally { diff --git a/src/net-questdb-client-tests/Qwp/QwpColumnExtendedTypesTests.cs b/src/net-questdb-client-tests/Qwp/QwpColumnExtendedTypesTests.cs index 8e1f3de..6539bba 100644 --- a/src/net-questdb-client-tests/Qwp/QwpColumnExtendedTypesTests.cs +++ b/src/net-questdb-client-tests/Qwp/QwpColumnExtendedTypesTests.cs @@ -461,6 +461,36 @@ public void AppendDecimal128_NegativeRescale_PreservesValue() Assert.That(second, Is.EqualTo(new BigInteger(-150))); } + // Each case: the first append locks the column scale; the second carries a *different* source + // scale but is representable at the locked scale, so QwpColumn.AppendDecimalRescaled takes the + // System.Decimal success branch — `targetScale <= MaxBclDecimalScale && TryRescaleDecimal(...)` — + // and returns there without ever touching the BigInteger fallback. Breakpoint that return. + private static IEnumerable DecimalRescaleSuccessCases() + { + yield return new TestCaseData(1.50m, 2.5m, (byte)2, new BigInteger(250)).SetName("UpScale_Positive"); + yield return new TestCaseData(-1.50m, -2.5m, (byte)2, new BigInteger(-250)).SetName("UpScale_Negative"); + yield return new TestCaseData(0.001m, 12.5m, (byte)3, new BigInteger(12500)).SetName("UpScale_WiderGap"); + yield return new TestCaseData(10.00m, 7m, (byte)2, new BigInteger(700)).SetName("UpScale_FromInteger"); + yield return new TestCaseData(100m, 5.00m, (byte)0, new BigInteger(5)).SetName("DownScale_ExactTrailingZeros"); + } + + [TestCaseSource(nameof(DecimalRescaleSuccessCases))] + public void AppendDecimal128_DecimalRescaleSuccessPath_WritesExpectedUnscaled( + decimal firstValue, decimal secondValue, byte expectedScale, BigInteger expectedSecondUnscaled) + { + var col = new QwpColumn("c", 0); + col.AppendDecimal128(firstValue); // locks the column scale + col.AppendDecimal128(secondValue); // scale mismatch → rescaled to the locked scale via System.Decimal + + Assert.That(col.DecimalScale, Is.EqualTo(expectedScale)); + Assert.That(col.NonNullCount, Is.EqualTo(2)); + + // The locked scale is unchanged, and the rescaled second value is byte-for-byte what the decimal + // success branch produced (the unscaled mantissa at the locked scale). + var second = ReadInt128(col.FixedData!.AsSpan(16, 16)); + Assert.That(second, Is.EqualTo(expectedSecondUnscaled)); + } + [Test] public void AppendVarchar_LoneSurrogate_ThrowsStrictUtf8() { @@ -580,20 +610,6 @@ public void AppendIPv4_TypeMismatch_Throws() Assert.Throws(() => col.AppendIPv4(1)); } - [Test] - public void AppendDecimal64_Limbs_WritesEightBytesLittleEndian() - { - var col = new QwpColumn("p", 0); - col.AppendDecimal64(unchecked((long)0x123456789ABCDEF0UL), scale: 5); - - Assert.That(col.TypeCode, Is.EqualTo(QwpTypeCode.Decimal64)); - Assert.That(col.DecimalScale, Is.EqualTo((byte)5)); - Assert.That(col.DecimalScaleSet, Is.True); - Assert.That(col.FixedLen, Is.EqualTo(8)); - Assert.That(BinaryPrimitives.ReadInt64LittleEndian(col.FixedData!.AsSpan(0, 8)), - Is.EqualTo(unchecked((long)0x123456789ABCDEF0UL))); - } - [Test] public void AppendDecimal128_Limbs_WritesTwoLimbsLsbFirst() { @@ -634,8 +650,8 @@ public void AppendDecimal256_Limbs_WritesFourLimbsLsbFirst() public void AppendDecimal64_Limbs_LosslessRescale_Allowed() { var col = new QwpColumn("p", 0); - col.AppendDecimal64(100L, scale: 2); - Assert.DoesNotThrow(() => col.AppendDecimal64(2000L, scale: 3)); + col.AppendDecimal64(100m, scale: 2); + Assert.DoesNotThrow(() => col.AppendDecimal64(2.0001m, scale: 2)); Assert.That(col.DecimalScale, Is.EqualTo((byte)2)); } @@ -645,7 +661,7 @@ public void AppendDecimal64_Limbs_LossyRescale_Throws() var col = new QwpColumn("p", 0); col.AppendDecimal64(100L, scale: 2); var ex = Assert.Throws(() => col.AppendDecimal64(201L, scale: 3)); - Assert.That(ex!.Message, Does.Contain("precision loss")); + Assert.That(ex!.Message, Does.Contain("scale is locked at 2")); } [Test] @@ -664,7 +680,7 @@ public void AppendDecimal64_LimbAndDecimalForms_ShareScaleLock() Assert.That(col.DecimalScale, Is.EqualTo((byte)2)); Assert.That(col.NonNullCount, Is.EqualTo(2)); var span = col.FixedData!.AsSpan(0, 16); - Assert.That(BinaryPrimitives.ReadInt64LittleEndian(span.Slice(0, 8)), Is.EqualTo(100L)); + Assert.That(BinaryPrimitives.ReadInt64LittleEndian(span.Slice(0, 8)), Is.EqualTo(10000L)); Assert.That(BinaryPrimitives.ReadInt64LittleEndian(span.Slice(8, 8)), Is.EqualTo(55L)); } diff --git a/src/net-questdb-client-tests/Qwp/QwpEncoderTests.cs b/src/net-questdb-client-tests/Qwp/QwpEncoderTests.cs index d48aa88..e78e425 100644 --- a/src/net-questdb-client-tests/Qwp/QwpEncoderTests.cs +++ b/src/net-questdb-client-tests/Qwp/QwpEncoderTests.cs @@ -361,22 +361,6 @@ public void Encode_Decimal256Column_WritesScalePrefixAnd32BytesPerValue() } } - [Test] - public void Encode_Decimal64Column_LimbForm_WritesScaleAndUnscaledLimb() - { - var t = new QwpTableBuffer("t"); - t.AppendDecimal64("p", 1234567890123L, scale: 4); - t.At(0); - - var bytes = QwpEncoder.Encode(new[] { t }, new QwpSymbolDictionary()); - var pos = FindFirstColumnDataOffset(bytes, tableNameLen: 1, userColCount: 1, userColDefSize: 1 + 1 + 1); - - Assert.That(bytes[pos++], Is.EqualTo(0x00), "null flag"); - Assert.That(bytes[pos++], Is.EqualTo((byte)4), "scale = 4"); - Assert.That(BinaryPrimitives.ReadInt64LittleEndian(bytes.AsSpan(pos, 8)), - Is.EqualTo(1234567890123L)); - } - [Test] public void Encode_Decimal128Column_LimbForm_WritesLoThenHiLittleEndian() { diff --git a/src/net-questdb-client-tests/Qwp/QwpWebSocketSenderTests.cs b/src/net-questdb-client-tests/Qwp/QwpWebSocketSenderTests.cs index 85c4246..13c4708 100644 --- a/src/net-questdb-client-tests/Qwp/QwpWebSocketSenderTests.cs +++ b/src/net-questdb-client-tests/Qwp/QwpWebSocketSenderTests.cs @@ -175,7 +175,8 @@ public async Task EndToEnd_ServerErrorAck_TurnsSenderTerminal() var qwp = (IQwpWebSocketSender)sender; sender.Table("t").Column("v", 1L).At(DateTime.UtcNow); - sender.Send(); + // Standalone Send drains, so the server's error-ack surfaces right here and turns it terminal. + Assert.Catch(() => sender.Send()); Assert.CatchAsync(async () => await qwp.PingAsync()); Assert.Catch(() => sender.Table("t").Column("v", 2L).At(DateTime.UtcNow)); @@ -396,12 +397,14 @@ public async Task Tls_SelfSignedCert_VerifyOff_ConnectsAndSends() } [Test] - public async Task DisposeAsync_FlushesAndCleansUp() + public async Task DisposeAsync_CleansUpAndBlocksReuse() { await using var server = StartServerWithOkAcks(); var sender = NewSender(server, "auto_flush=off;"); sender.Table("t").Column("v", 1L).At(DateTime.UtcNow); - sender.Send(); + // Flush (drain) — not just Send — so the frame is guaranteed delivered before teardown; Dispose + // does not drain. + Assert.That(await sender.FlushAsync(TimeSpan.FromSeconds(5)), Is.True); await ((IAsyncDisposable)sender).DisposeAsync(); @@ -411,7 +414,7 @@ public async Task DisposeAsync_FlushesAndCleansUp() } [Test] - public async Task DisposeAsync_OnTerminalSender_RethrowsLatchedError() + public async Task DisposeAsync_OnTerminalSender_DoesNotThrow() { await using var server = new DummyQwpServer(new DummyQwpServerOptions { @@ -420,17 +423,16 @@ public async Task DisposeAsync_OnTerminalSender_RethrowsLatchedError() await server.StartAsync(); var sender = NewSender(server, "auto_flush=off;on_server_error=halt;"); - var qwp = (IQwpWebSocketSender)sender; sender.Table("t").Column("v", 1L).At(DateTime.UtcNow); - sender.Send(); - Assert.CatchAsync(async () => await qwp.PingAsync()); + // Standalone Send drains → the server's error-ack surfaces here and turns the sender terminal. + Assert.Catch(() => sender.Send()); - Assert.ThrowsAsync( - async () => await ((IAsyncDisposable)sender).DisposeAsync()); + // Dispose is pure resource release and must never throw, even on a terminal sender. + Assert.DoesNotThrowAsync(async () => await ((IAsyncDisposable)sender).DisposeAsync()); } [Test] - public async Task SendAsync_CompletesFastWhileServerStalls() + public async Task SendAsync_Standalone_DrainsUntilAcked() { using var ackGate = new SemaphoreSlim(0, int.MaxValue); await using var server = new DummyQwpServer(new DummyQwpServerOptions @@ -446,10 +448,14 @@ public async Task SendAsync_CompletesFastWhileServerStalls() using var sender = NewSender(server, "auto_flush=off;"); sender.Table("t").Column("v", 1L).At(DateTime.UtcNow); - // Cursor engine path: SendAsync returns once the frame is in the ring, regardless of ACK. - await sender.SendAsync().WaitAsync(TimeSpan.FromSeconds(5)); + // A standalone Send drains: it must NOT complete until the server ACKs (server is parked on ackGate). + var send = sender.SendAsync(); + await Task.Delay(150); + Assert.That(send.IsCompleted, Is.False, "standalone Send drains — pending until the server ACKs"); ackGate.Release(8); + await send.WaitAsync(TimeSpan.FromSeconds(5)); + Assert.That(send.IsCompletedSuccessfully, Is.True); } [Test] @@ -549,27 +555,40 @@ public async Task ServerClosesWithProtocolViolation_TerminatesWithoutReconnect( try { sender.Table("t").Column("v", 1L).At(DateTime.UtcNow); - sender.Send(); + + // Standalone Send drains, so the protocol-violation close may surface at this first Send + // (it races the OK-ack), or on a later Send once the close reaches the engine. Handle both. + IngressError? caught = null; + try + { + sender.Send(); + } + catch (IngressError ex) + { + caught = ex; + } await WaitFor(() => server.ReceivedFrames.Count >= 1); - // Once the protocol-violation close hits the engine, the next API call must surface a - // terminal IngressError carrying ProtocolViolation — without sitting in reconnect. - IngressError? caught = null; - await WaitFor(() => + // Once the protocol-violation close hits the engine, an API call must surface a terminal + // IngressError carrying ProtocolViolation — without sitting in reconnect. + if (caught is null) { - try - { - sender.Table("t").Column("v", 2L).At(DateTime.UtcNow); - sender.Send(); - return false; - } - catch (IngressError ex) + await WaitFor(() => { - caught = ex; - return true; - } - }, timeoutMs: 5000); + try + { + sender.Table("t").Column("v", 2L).At(DateTime.UtcNow); + sender.Send(); + return false; + } + catch (IngressError ex) + { + caught = ex; + return true; + } + }, timeoutMs: 5000); + } Assert.That(caught, Is.Not.Null); var rootCode = caught!.code is ErrorCode.ProtocolViolation @@ -724,21 +743,45 @@ public async Task Transaction_CommitAsync_FlushesBufferedRowsNonDeferred() } [Test] - public async Task Transaction_Dispose_CommitsDeferredRows() + public async Task Transaction_Send_CommitsDeferredRows() { await using var server = StartServerWithOkAcks(); - await using (var sender = NewSender(server, - "transaction=on;auto_flush_rows=1;auto_flush_interval=off;auto_flush_bytes=off;close_flush_timeout_millis=2000;")) - { - sender.Table("t").Column("v", 1L).At(TxnTs); // auto-flush row=1 → deferred frame - await WaitFor(() => server.ReceivedFrames.Count >= 1); - } + using var sender = NewSender(server, + "transaction=on;auto_flush_rows=1;auto_flush_interval=off;auto_flush_bytes=off;"); + sender.Table("t").Column("v", 1L).At(TxnTs); // auto-flush row=1 → deferred frame + await WaitFor(() => server.ReceivedFrames.Count >= 1); + + // explicit Send commits the deferred rows (Dispose no longer flushes) + sender.Send(); - // dispose must emit a committing frame for the deferred rows await WaitFor(() => server.ReceivedFrames.Count >= 2); Assert.That(IsDeferred(server.ReceivedFrames.Last()), Is.False); } + [Test] + public async Task Transaction_HasUncommittedDeferredRows_TracksOwedCommit() + { + await using var server = StartServerWithOkAcks(); + using var sender = NewSender(server, + "transaction=on;auto_flush_rows=1;auto_flush_interval=off;auto_flush_bytes=off;"); + // The pool reads this seam on return to decide re-pool vs discard; pin its transitions here. + QuestDB.Pooling.IPooledTransactionalSender seam = sender; + + Assert.That(seam.HasUncommittedDeferredRows, Is.False, "no commit owed before any flush"); + + sender.Table("t").Column("v", 1L).At(TxnTs); // auto-flush row=1 → deferred frame + await WaitFor(() => server.ReceivedFrames.Count >= 1); + Assert.That(seam.HasUncommittedDeferredRows, Is.True, "deferred auto-flush leaves a commit owed"); + + // Clear() resets local buffers only — the staged rows remain on the server, commit still owed. + sender.Clear(); + Assert.That(seam.HasUncommittedDeferredRows, Is.True, "Clear() cannot un-stage shipped rows"); + + sender.Commit(); + await WaitFor(() => server.ReceivedFrames.Count >= 2); + Assert.That(seam.HasUncommittedDeferredRows, Is.False, "explicit commit settles the owed commit"); + } + [Test] public async Task NonTransactional_FramesNeverDefer() { @@ -824,14 +867,16 @@ public async Task DurableAck_ServerSendsPerTableSeqTxns_TrackedSeparately() }, }); await server.StartAsync(); - // close_flush_timeout_millis=0: durable-ack deliberately lags OK-ack here, so a close drain would time out. - using var sender = NewSender(server, "auto_flush=off;request_durable_ack=on;close_flush_timeout_millis=0;"); + // Auto-flush (not a draining Send): this test's durable watermarks (100/101) are deliberately + // below the committed ones (200/201) to prove separate tracking, so a standalone Send would drain + // forever waiting for a durability that never covers the frame. Auto-flush ships to the ring + // without waiting; the watermarks are then observed via WaitFor as the async acks arrive. + using var sender = NewSender(server, + "auto_flush=on;auto_flush_rows=1;auto_flush_interval=off;auto_flush_bytes=off;request_durable_ack=on;"); var ws = (IQwpWebSocketSender)sender; - sender.Table("trades").Column("v", 1L).At(DateTime.UtcNow); - sender.Send(); - sender.Table("trades").Column("v", 2L).At(DateTime.UtcNow); - sender.Send(); + sender.Table("trades").Column("v", 1L).At(DateTime.UtcNow); // auto-flush → batch seq 0 + sender.Table("trades").Column("v", 2L).At(DateTime.UtcNow); // auto-flush → batch seq 1 await WaitFor(() => ws.GetHighestAckedSeqTxn("trades") == 201L && ws.GetHighestDurableSeqTxn("trades") == 101L); @@ -931,7 +976,8 @@ public async Task AsyncMode_ServerErrorOnBatch_TurnsTerminal() sender.Send(); // first batch — OK sender.Table("t").Column("v", 2L).At(DateTime.UtcNow); - sender.Send(); + // Standalone Send drains → the second batch's error-ack surfaces here and turns it terminal. + Assert.Catch(() => sender.Send()); Assert.CatchAsync(async () => await qwp.PingAsync()); } finally @@ -1088,34 +1134,53 @@ public async Task EndToEnd_Sf_TwoSendersSameSlot_SecondFailsLockCollision() } [Test] - public async Task Dispose_Sf_ZeroCloseFlushTimeout_StillPersistsBufferedRowsToDisk() + public async Task Sf_Flush_PersistsRowsToDisk_DisposeDiscardsUnsent() { - // close_flush_timeout_millis=0 means "don't wait for acks", not "discard the buffer". In SF - // mode the close-time encode-to-ring is a local disk append (durability), not a network wait, - // so buffered-but-unsent rows must still be persisted to the segment ring on dispose. - // FrameHandler => null: the server never acks, so the persisted frame is never trimmed off disk. + // Dispose no longer encodes buffered rows to the ring: a flush (here auto-flush) is the path to + // the on-disk SF ring, and un-sent rows are discarded on dispose. FrameHandler => null: the server + // never acks, so a persisted frame is never trimmed off disk (and a draining Send would block, so + // part (a) uses auto-flush, which writes to the ring without waiting for an ACK). await using var server = new DummyQwpServer(new DummyQwpServerOptions { FrameHandler = _ => null }); await server.StartAsync(); - var sfRoot = Path.Combine(Path.GetTempPath(), "qwp-sf-closeflush0-" + Guid.NewGuid().ToString("N")); + + // (a) A flushed row is persisted to the segment ring. + var sentRoot = Path.Combine(Path.GetTempPath(), "qwp-sf-sent-" + Guid.NewGuid().ToString("N")); try { using (var sender = NewSender(server, - $"auto_flush=off;sf_dir={sfRoot};sender_id=svc-a;sf_max_bytes=4096;close_flush_timeout_millis=0;")) + $"auto_flush=on;auto_flush_rows=1;auto_flush_interval=off;auto_flush_bytes=off;" + + $"sf_dir={sentRoot};sender_id=svc-a;sf_max_bytes=4096;")) { - // Buffered only — auto_flush is off and Send() is never called, so the close-time encode - // is the row's only path to the on-disk ring. - sender.Table("trades") - .Symbol("ticker", "ETH-USD") - .Column("price", 2615.54) + sender.Table("trades").Symbol("ticker", "ETH-USD").Column("price", 2615.54) + .At(new DateTime(2026, 4, 28, 12, 0, 0, DateTimeKind.Utc)); // auto-flush → on-disk ring + } + + Assert.That(SegmentBytesOnDisk(sentRoot), Is.GreaterThan(0), + "a flushed row is persisted to the SF segment ring"); + } + finally + { + TryDeleteDirectory(sentRoot); + } + + // (b) A buffered-only row (never Sent) is discarded on dispose — nothing reaches disk. + var unsentRoot = Path.Combine(Path.GetTempPath(), "qwp-sf-unsent-" + Guid.NewGuid().ToString("N")); + try + { + using (var sender = NewSender(server, + $"auto_flush=off;sf_dir={unsentRoot};sender_id=svc-b;sf_max_bytes=4096;")) + { + sender.Table("trades").Symbol("ticker", "ETH-USD").Column("price", 2615.54) .At(new DateTime(2026, 4, 28, 12, 0, 0, DateTimeKind.Utc)); + // no Send() — Dispose discards the buffered row } - Assert.That(SegmentBytesOnDisk(sfRoot), Is.GreaterThan(0), - "SF segment ring must retain the buffered row after a zero close-flush-timeout dispose"); + Assert.That(SegmentBytesOnDisk(unsentRoot), Is.EqualTo(0), + "Dispose discards un-sent rows — nothing is persisted to the ring"); } finally { - TryDeleteDirectory(sfRoot); + TryDeleteDirectory(unsentRoot); } } @@ -1301,8 +1366,13 @@ public async Task ColumnDecimal64_LimbOverload_RoundTripsToServer() var ws = (IQwpWebSocketSender)sender; sender.Table("t"); - ws.ColumnDecimal64("p", unscaledValue: 1234567890123L, scale: 4); + ws.ColumnDecimal64("p", 123456789.0123m, scale: 4); + sender.At(DateTime.UtcNow); + + sender.Table("t"); + ws.ColumnDecimal64("p", 123456789.01m, scale: 4); sender.At(DateTime.UtcNow); + await sender.SendAsync(); await ws.PingAsync(); @@ -1312,7 +1382,7 @@ public async Task ColumnDecimal64_LimbOverload_RoundTripsToServer() // After type code: TS col def (nameLen=0, type 0x10), then user col data: null flag + scale + 8-byte LE. var scaleOffset = typeCodeOffset + 1 + 2 + 1; Assert.That(payload[scaleOffset], Is.EqualTo((byte)4), "scale prefix"); - Assert.That(System.Buffers.Binary.BinaryPrimitives.ReadInt64LittleEndian(payload.Slice(scaleOffset + 1, 8)), + Assert.That(BinaryPrimitives.ReadInt64LittleEndian(payload.Slice(scaleOffset + 1, 8)), Is.EqualTo(1234567890123L)); } @@ -1827,11 +1897,9 @@ public async Task PostTerminal_MutatorsThrow() var sender = NewSender(server, "auto_flush=off;on_server_error=halt;"); try { - var qwp = (IQwpWebSocketSender)sender; - sender.Table("t").Column("v", 1L).At(DateTime.UtcNow); - sender.Send(); - try { await qwp.PingAsync(); } catch { /* expected terminal */ } + // Standalone Send drains → the error-ack surfaces here and turns the sender terminal. + try { sender.Send(); } catch { /* expected terminal */ } Assert.Throws(() => sender.Truncate()); Assert.Throws(() => sender.CancelRow()); diff --git a/src/net-questdb-client-tests/Qwp/Sf/QwpTrackedCursorTransportTests.cs b/src/net-questdb-client-tests/Qwp/Sf/QwpTrackedCursorTransportTests.cs index 80dfcd6..6d87d1e 100644 --- a/src/net-questdb-client-tests/Qwp/Sf/QwpTrackedCursorTransportTests.cs +++ b/src/net-questdb-client-tests/Qwp/Sf/QwpTrackedCursorTransportTests.cs @@ -24,9 +24,12 @@ #if NET7_0_OR_GREATER +using System.Diagnostics; using NUnit.Framework; +using QuestDB.Enums; using QuestDB.Qwp; using QuestDB.Qwp.Sf; +using QuestDB.Utils; namespace net_questdb_client_tests.Qwp.Sf; @@ -135,6 +138,46 @@ public void Success_WithoutZoneHeader_DoesNotTouchZoneTier() Assert.That(tracker.GetState(0), Is.EqualTo(QwpHostState.Healthy)); } + [Test] + public void ConnectTimeout_ExpiresOnStalledUpgrade_ThrowsConnectTimeoutAndRecordsTransportError() + { + // The inner transport never completes the upgrade; the connect_timeout deadline must abort it + // (not the caller's token) and surface the canonical connect_timeout message. + var tracker = new QwpHostHealthTracker(new[] { "h1:9000" }, clientZone: null, targetIsPrimary: true); + var tracked = new QwpTrackedCursorTransport(new StallingStub(), tracker, hostIndex: 0, + TimeSpan.FromMilliseconds(150)); + + var sw = Stopwatch.StartNew(); + var ex = Assert.ThrowsAsync( + async () => await tracked.ConnectAsync(CancellationToken.None)); + sw.Stop(); + + Assert.That(ex!.code, Is.EqualTo(ErrorCode.SocketError)); + Assert.That(ex.Message, Does.Contain("connect_timeout")); + Assert.That(sw.ElapsedMilliseconds, Is.LessThan(3000), + "connect_timeout=150ms must abort the stalled upgrade promptly"); + Assert.That(tracker.GetState(0), Is.Not.EqualTo(QwpHostState.Healthy)); + } + + private sealed class StallingStub : IQwpCursorTransport + { + public (string Host, int Port)? Endpoint => ("h", 9000); + + // Blocks until the (connect_timeout-linked) token cancels, then throws OperationCanceledException. + public Task ConnectAsync(CancellationToken cancellationToken) => + Task.Delay(System.Threading.Timeout.Infinite, cancellationToken); + + public Task SendBinaryAsync(ReadOnlyMemory data, CancellationToken cancellationToken) => + Task.CompletedTask; + + public Task ReceiveFrameAsync(Memory destination, CancellationToken cancellationToken) => + Task.FromResult(0); + + public Task CloseAsync(CancellationToken cancellationToken) => Task.CompletedTask; + + public void Dispose() { } + } + private sealed class RejectingStub : IQwpCursorTransport { private readonly string _role; diff --git a/src/net-questdb-client-tests/SenderOptionsTests.cs b/src/net-questdb-client-tests/SenderOptionsTests.cs index 6d01446..e9a16e2 100644 --- a/src/net-questdb-client-tests/SenderOptionsTests.cs +++ b/src/net-questdb-client-tests/SenderOptionsTests.cs @@ -341,6 +341,29 @@ public void Sf_DurabilityNonMemory_Throws() Throws.TypeOf().With.Message.Contains("sf_durability")); } + [Test] + public void Transaction_WithSfDir_Throws() + { + // Connection-scoped transactional staging cannot coexist with cross-connection SF replay: + // a replayed deferred frame would re-stage abandoned rows for the next commit to publish. + Assert.That( + () => new SenderOptions("ws::addr=localhost:9000;transaction=on;sf_dir=/tmp;"), + Throws.TypeOf().With.Message.Contains("transaction=on").And.Message.Contains("sf_dir")); + + // Programmatic-init path is caught by the same EnsureValid pass. + var opts = new SenderOptions + { + protocol = ProtocolType.ws, addr = "h:9000", + transaction = true, sf_dir = "/tmp", + }; + Assert.That(() => opts.EnsureValid(), + Throws.TypeOf().With.Message.Contains("transaction=on")); + + // Each key remains valid on its own. + Assert.That(() => new SenderOptions("ws::addr=localhost:9000;transaction=on;"), Throws.Nothing); + Assert.That(() => new SenderOptions("ws::addr=localhost:9000;sf_dir=/tmp;"), Throws.Nothing); + } + [Test] public void Sf_KeysOnHttpScheme_Throws() { @@ -627,6 +650,99 @@ public void Http_ToString_KeepsAuthTimeoutLegacyName() Assert.That(opts.ToString(), Does.Contain("auth_timeout=4000")); } + [Test] + public void ConnectTimeout_Default_IsUnsetAndInheritsAuthTimeout() + { + var opts = new SenderOptions("http::addr=localhost:9000;"); + Assert.That(opts.connect_timeout, Is.Null); + // Unset connect_timeout inherits auth_timeout (default 15s). + Assert.That(opts.EffectiveConnectTimeout, Is.EqualTo(opts.auth_timeout)); + } + + [TestCase("http")] + [TestCase("https")] + [TestCase("tcp")] + [TestCase("tcps")] + [TestCase("ws")] + [TestCase("wss")] + public void ConnectTimeout_ParsesOnEveryScheme(string scheme) + { + var addr = scheme.StartsWith("tcp") ? "addr=localhost:9009" : "addr=localhost:9000"; + var opts = new SenderOptions($"{scheme}::{addr};connect_timeout=2500;"); + Assert.That(opts.connect_timeout!.Value, Is.EqualTo(TimeSpan.FromMilliseconds(2500))); + Assert.That(opts.EffectiveConnectTimeout, Is.EqualTo(TimeSpan.FromMilliseconds(2500))); + } + + [Test] + public void ConnectTimeout_Unset_InheritsExplicitAuthTimeout() + { + // auth_timeout is the alias of connect_timeout where there is no separate auth step (HTTP). + var opts = new SenderOptions("http::addr=localhost:9000;auth_timeout=4000;"); + Assert.That(opts.EffectiveConnectTimeout, Is.EqualTo(TimeSpan.FromMilliseconds(4000))); + } + + [Test] + public void ConnectTimeout_Explicit_OverridesAuthTimeout() + { + var opts = new SenderOptions("http::addr=localhost:9000;auth_timeout=4000;connect_timeout=2000;"); + Assert.That(opts.EffectiveConnectTimeout, Is.EqualTo(TimeSpan.FromMilliseconds(2000))); + // auth_timeout retains its own value for the (TCP) auth step. + Assert.That(opts.auth_timeout, Is.EqualTo(TimeSpan.FromMilliseconds(4000))); + } + + [TestCase("connect_timeout=0")] + [TestCase("connect_timeout=-5")] + public void ConnectTimeout_NonPositive_Rejected(string kv) + { + var ex = Assert.Throws( + () => new SenderOptions($"http::addr=localhost:9000;{kv};")); + Assert.That(ex!.Message, Does.Contain("connect_timeout")); + } + + [Test] + public void ConnectTimeout_ToString_OmittedWhenUnset_PreservesInheritance() + { + var s = new SenderOptions("http::addr=localhost:9000;auth_timeout=4000;").ToString(); + Assert.That(s, Does.Not.Contain("connect_timeout")); + // Re-parse keeps connect_timeout unset, so it still inherits the (round-tripped) auth_timeout. + var rt = new SenderOptions(s); + Assert.That(rt.EffectiveConnectTimeout, Is.EqualTo(TimeSpan.FromMilliseconds(4000))); + } + + [TestCase("http::addr=localhost:9000;connect_timeout=7000;")] + [TestCase("ws::addr=localhost:9000;connect_timeout=7000;")] + public void ConnectTimeout_ToString_RoundTripsWhenSet(string confStr) + { + var opts = new SenderOptions(confStr); + var s = opts.ToString(); + Assert.That(s, Does.Contain("connect_timeout=7000")); + var rt = new SenderOptions(s); + Assert.That(rt.connect_timeout!.Value, Is.EqualTo(TimeSpan.FromMilliseconds(7000))); + Assert.That(rt.EffectiveConnectTimeout, Is.EqualTo(TimeSpan.FromMilliseconds(7000))); + } + + [Test] + public void ConnectTimeout_ProgrammaticSet_OverridesAuthTimeout() + { + var opts = new SenderOptions { auth_timeout = TimeSpan.FromSeconds(9) }; + // Unset programmatically → inherits auth_timeout. + Assert.That(opts.connect_timeout, Is.Null); + Assert.That(opts.EffectiveConnectTimeout, Is.EqualTo(TimeSpan.FromSeconds(9))); + + opts.connect_timeout = TimeSpan.FromSeconds(2); + Assert.That(opts.EffectiveConnectTimeout, Is.EqualTo(TimeSpan.FromSeconds(2))); + } + + [Test] + public void ConnectTimeout_ProgrammaticOverflow_RejectedByEnsureValid() + { + // The connect-string parser is int-ms so it can't express this; the TimeSpan setter can, + // and EnsureValid (run by Build) must reject it before it reaches CancellationTokenSource. + var opts = new SenderOptions { connect_timeout = TimeSpan.FromDays(30) }; + var ex = Assert.Throws(() => opts.EnsureValid()); + Assert.That(ex!.Message, Does.Contain("connect_timeout")); + } + [TestCase("off", InitialConnectMode.off)] [TestCase("false", InitialConnectMode.off)] [TestCase("OFF", InitialConnectMode.off)] diff --git a/src/net-questdb-client-tests/TcpTests.cs b/src/net-questdb-client-tests/TcpTests.cs index 5a5c5d6..928768d 100644 --- a/src/net-questdb-client-tests/TcpTests.cs +++ b/src/net-questdb-client-tests/TcpTests.cs @@ -23,7 +23,9 @@ ******************************************************************************/ +using System.Diagnostics; using System.Net; +using System.Net.Sockets; using System.Text; using NUnit.Framework; using Org.BouncyCastle.Asn1.Sec; @@ -31,6 +33,7 @@ using Org.BouncyCastle.Math; using Org.BouncyCastle.Security; using QuestDB; +using QuestDB.Enums; using QuestDB.Utils; #pragma warning disable CS0618 // Type or member is obsolete @@ -653,6 +656,48 @@ public Task CannotConnect() return Task.CompletedTask; } + [Test] + public void ConnectTimeout_BoundsTlsHandshakeToStalledHost() + { + // The raw listener accepts the TCP connection but never answers the TLS ClientHello, so + // connect_timeout must abort the attempt at the handshake layer instead of hanging. + var listener = new TcpListener(IPAddress.Loopback, 0); + listener.Start(); + var port = ((IPEndPoint)listener.LocalEndpoint).Port; + var held = new List(); + using var acceptCts = new CancellationTokenSource(); + var acceptTask = Task.Run(async () => + { + try + { + while (!acceptCts.IsCancellationRequested) + { + held.Add(await listener.AcceptTcpClientAsync(acceptCts.Token)); + } + } + catch { } + }); + + try + { + var conn = $"tcps::addr=127.0.0.1:{port};tls_verify=unsafe_off;connect_timeout=300;auto_flush=off;"; + var sw = Stopwatch.StartNew(); + var ex = Assert.Throws(() => Sender.New(conn)); + sw.Stop(); + + Assert.That(ex!.code, Is.EqualTo(ErrorCode.TlsError)); + StringAssert.Contains("connect_timeout", ex.Message); + Assert.That(sw.ElapsedMilliseconds, Is.LessThan(3000), + "connect_timeout=300ms should bound the TLS handshake well below OS-level timeouts"); + } + finally + { + acceptCts.Cancel(); + listener.Stop(); + foreach (var c in held) try { c.Close(); } catch { } + } + } + [Test] public Task SendNegativeLongMin() { diff --git a/src/net-questdb-client/Enums/ErrorCode.cs b/src/net-questdb-client/Enums/ErrorCode.cs index a788b0f..cf8c099 100644 --- a/src/net-questdb-client/Enums/ErrorCode.cs +++ b/src/net-questdb-client/Enums/ErrorCode.cs @@ -113,4 +113,10 @@ public enum ErrorCode /// grow unbounded waiting for ack frames that will never arrive. /// DurableAckNotSupported, + + /// + /// A QuestDBClient sender pool was exhausted: every sender up to + /// sender_pool_max was in use and none freed within acquire_timeout_ms. + /// + PoolExhausted, } \ No newline at end of file diff --git a/src/net-questdb-client/IQuestDBClient.cs b/src/net-questdb-client/IQuestDBClient.cs new file mode 100644 index 0000000..8203e60 --- /dev/null +++ b/src/net-questdb-client/IQuestDBClient.cs @@ -0,0 +1,140 @@ +/******************************************************************************* + * ___ _ ____ ____ + * / _ \ _ _ ___ ___| |_| _ \| __ ) + * | | | | | | |/ _ \/ __| __| | | | _ \ + * | |_| | |_| | __/\__ \ |_| |_| | |_) | + * \__\_\\__,_|\___||___/\__|____/|____/ + * + * Copyright (c) 2014-2019 Appsicle + * Copyright (c) 2019-2026 QuestDB + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + ******************************************************************************/ + +using QuestDB.Senders; +#if NET7_0_OR_GREATER +using QwpColumnBatchHandler = QuestDB.Qwp.Query.QwpColumnBatchHandler; +#endif + +namespace QuestDB; + +/// +/// High-level handle that owns a pool of instances. Construct once with +/// or , then share across +/// threads — and may be called +/// concurrently from any thread. +/// +public interface IQuestDBClient : IDisposable, IAsyncDisposable +{ + /// Number of idle senders currently parked in the pool. + int AvailableSenderCount { get; } + + /// Total senders alive in the pool (idle + in-use). + int TotalSenderCount { get; } + + /// + /// Borrows a sender from the pool. The caller MUST dispose the returned instance (a + /// using block is the idiom) to release it back to the pool. + /// + /// Disposing does NOT send. It discards any buffered-but-unsent rows and returns the sender + /// to the pool; the real connection is only closed when this handle is disposed. Call + /// (hand rows to the transport) — and, for delivery + /// confirmation, per sender or + /// across the whole pool — before disposing. + /// + /// Blocks up to acquire_timeout_ms when every sender up to sender_pool_max is in + /// use, then throws. + /// + /// Do not use the sender after disposing it. Dispose returns the instance to the pool, + /// where another thread may immediately re-borrow it. The returned handle is inert, not aliased: + /// every ingest member (Table / Column / At / Send, …) throws + /// once disposed, so a stale reference can never reach — or + /// corrupt — the entry the pool has since lent to a different borrower (a second dispose is a + /// tolerated no-op). That guard makes accidental misuse fail loudly; it is not a licence to share a + /// live borrowed sender across threads — a single is not + /// thread-safe. Borrow per unit of work, drop the reference when the using scope ends, and + /// never cache or share a borrowed sender across threads. There is no thread-affine / pinned-sender + /// API. + /// + /// A sender leased from the pool; release it with . + /// If the pool is exhausted past the acquire timeout, or the handle is closed. + ISender BorrowSender(); + + /// + /// Cancels the wait for a free sender. + ValueTask BorrowSenderAsync(CancellationToken ct = default); + + /// + /// Drains every sender in the pool: flushes each one's buffered rows and blocks until the server has + /// acknowledged them, or elapses (applied per sender). The pool-wide + /// delivery barrier — the analogue of Java's drain() at pool scope. + /// + /// Call this during quiescence — after every borrowed sender has been returned. It does not + /// synchronise against a concurrent borrow (draining a sender another thread is writing to is a data + /// race). Typical use: producers finish and return their senders, then the coordinator calls + /// Flush and only marks the batch done when it returns true. + /// + /// Upper bound on the ACK wait, per sender. + /// Cancels the flush and the wait. + /// true if every sender drained fully; false if any timed out or latched an error. + bool Flush(TimeSpan timeout, CancellationToken ct = default); + + /// Convenience overload using close_flush_timeout_millis as the per-sender drain timeout. + bool Flush(CancellationToken ct = default); + + /// + ValueTask FlushAsync(TimeSpan timeout, CancellationToken ct = default); + + /// + ValueTask FlushAsync(CancellationToken ct = default); + +#if NET7_0_OR_GREATER + /// Number of idle query clients currently parked in the pool. + /// If the handle has no query configuration. + int AvailableQueryClientCount { get; } + + /// Total query clients alive in the pool (idle + in-use). + /// If the handle has no query configuration. + int TotalQueryClientCount { get; } + + /// + /// Allocates a fresh bound to this handle's query-client pool. Configure it + /// with Sql / Binds / Handler, then ExecuteAsync. Each execution + /// borrows a pooled query client for the duration of one query and returns it automatically — + /// there is no explicit borrow/release for queries (unlike senders). + /// + /// Allocate a fresh NewQuery() per query when running queries concurrently; a single + /// allows only one in-flight execution. + /// + /// Requires a ws/wss query configuration (a single ws connect string, or an + /// explicit QueryConfig / ); throws + /// otherwise. net7.0+ only. + /// + Query NewQuery(); + + /// + /// Convenience for a bind-less query: equivalent to + /// NewQuery().Sql(sql).Handler(handler).ExecuteAsync(ct). Borrows a pooled query client, + /// runs the query, and returns the client (or discards it on a hard cancel / transport failure). + /// + Task ExecuteSqlAsync(string sql, QwpColumnBatchHandler handler, CancellationToken ct = default); +#endif + + /// + /// Shuts the pool down, closing every underlying sender. Idempotent. Threads blocked in + /// are released with an error. Equivalent to + /// . + /// + void Close(); +} diff --git a/src/net-questdb-client/Pooling/BorrowedQwpSender.cs b/src/net-questdb-client/Pooling/BorrowedQwpSender.cs new file mode 100644 index 0000000..d984bbe --- /dev/null +++ b/src/net-questdb-client/Pooling/BorrowedQwpSender.cs @@ -0,0 +1,126 @@ +/******************************************************************************* + * ___ _ ____ ____ + * / _ \ _ _ ___ ___| |_| _ \| __ ) + * | | | | | | |/ _ \/ __| __| | | | _ \ + * | |_| | |_| | __/\__ \ |_| |_| | |_) | + * \__\_\\__,_|\___||___/\__|____/|____/ + * + * Copyright (c) 2014-2019 Appsicle + * Copyright (c) 2019-2026 QuestDB + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + ******************************************************************************/ + +using QuestDB.Senders; + +namespace QuestDB.Pooling; + +/// +/// The borrow handle for a ws:: / wss:: pool. Extends +/// with the full QWP surface (QWP-only column types, Ping, seqTxn / FSN watermarks) so a +/// borrowed sender can be probed and cast exactly like a standalone one: +/// sender is IQwpWebSocketSender ws. allocates this +/// subtype only when the pooled entry's real sender implements , +/// so a handle from an HTTP/TCP pool never matches the probe. Same lifecycle as the base: every +/// member (QWP ones included) throws once the handle is +/// returned to the pool. +/// +internal sealed class BorrowedQwpSender : BorrowedSender, IQwpWebSocketSender +{ + private readonly IQwpWebSocketSender _qwpInner; + + internal BorrowedQwpSender(PooledSender entry, SenderPool pool) + : base(entry, pool) + { + _qwpInner = (IQwpWebSocketSender)entry.Inner; + } + + // Same use-after-return gate as the base's ISender members: Active() throws once the handle is + // returned; _qwpInner is the same object it returns, pre-cast at construction. + private IQwpWebSocketSender Qwp + { + get + { + Active(); + return _qwpInner; + } + } + + public long GetHighestAckedSeqTxn(string tableName) => Qwp.GetHighestAckedSeqTxn(tableName); + public long GetHighestDurableSeqTxn(string tableName) => Qwp.GetHighestDurableSeqTxn(tableName); + public void Ping(CancellationToken ct = default) => Qwp.Ping(ct); + public ValueTask PingAsync(CancellationToken ct = default) => Qwp.PingAsync(ct); + public long AckedFsn => Qwp.AckedFsn; + public Task FlushAndGetSequenceAsync(CancellationToken ct = default) => Qwp.FlushAndGetSequenceAsync(ct); + + public Task AwaitAckedFsnAsync(long targetFsn, TimeSpan timeout, CancellationToken ct = default) => + Qwp.AwaitAckedFsnAsync(targetFsn, timeout, ct); + + public IQwpWebSocketSender ColumnBinary(ReadOnlySpan name, ReadOnlySpan value) + { + Qwp.ColumnBinary(name, value); + return this; + } + + public IQwpWebSocketSender ColumnIPv4(ReadOnlySpan name, System.Net.IPAddress addr) + { + Qwp.ColumnIPv4(name, addr); + return this; + } + + public IQwpWebSocketSender ColumnByte(ReadOnlySpan name, sbyte value) + { + Qwp.ColumnByte(name, value); + return this; + } + + public IQwpWebSocketSender ColumnShort(ReadOnlySpan name, short value) + { + Qwp.ColumnShort(name, value); + return this; + } + + public IQwpWebSocketSender ColumnFloat(ReadOnlySpan name, float value) + { + Qwp.ColumnFloat(name, value); + return this; + } + + public IQwpWebSocketSender ColumnDate(ReadOnlySpan name, long millisSinceEpoch) + { + Qwp.ColumnDate(name, millisSinceEpoch); + return this; + } + + public IQwpWebSocketSender ColumnGeohash(ReadOnlySpan name, ulong hash, int precisionBits) + { + Qwp.ColumnGeohash(name, hash, precisionBits); + return this; + } + + public IQwpWebSocketSender ColumnLong256(ReadOnlySpan name, System.Numerics.BigInteger value) + { + Qwp.ColumnLong256(name, value); + return this; + } + + public long DroppedErrorNotifications => Qwp.DroppedErrorNotifications; + public long DroppedConnectionNotifications => Qwp.DroppedConnectionNotifications; + public long TotalErrorNotificationsDelivered => Qwp.TotalErrorNotificationsDelivered; + public long TotalFramesSent => Qwp.TotalFramesSent; + public long TotalAcks => Qwp.TotalAcks; + public long TotalServerErrors => Qwp.TotalServerErrors; + public long TotalReconnectAttempts => Qwp.TotalReconnectAttempts; + public long TotalReconnectsSucceeded => Qwp.TotalReconnectsSucceeded; +} diff --git a/src/net-questdb-client/Pooling/BorrowedSender.cs b/src/net-questdb-client/Pooling/BorrowedSender.cs new file mode 100644 index 0000000..dc0b846 --- /dev/null +++ b/src/net-questdb-client/Pooling/BorrowedSender.cs @@ -0,0 +1,338 @@ +/******************************************************************************* + * ___ _ ____ ____ + * / _ \ _ _ ___ ___| |_| _ \| __ ) + * | | | | | | |/ _ \/ __| __| | | | _ \ + * | |_| | |_| | __/\__ \ |_| |_| | |_) | + * \__\_\\__,_|\___||___/\__|____/|____/ + * + * Copyright (c) 2014-2019 Appsicle + * Copyright (c) 2019-2026 QuestDB + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + ******************************************************************************/ + +using QuestDB.Senders; +using QuestDB.Utils; + +namespace QuestDB.Pooling; + +/// +/// The single-borrow handle handed out by / +/// . A fresh handle is allocated per borrow and wraps a +/// reusable pool entry; it is the only object the caller ever touches. +/// +/// (and ) are pure resource release: they discard any +/// buffered-but-unsent rows and return the underlying entry to the pool — they do NOT send, and do NOT +/// close the real sender (that happens only when the owning handle is +/// closed). Call / (and, on ws, +/// to await ACK) before disposing if the data must +/// land. Two invariants make a borrowed handle misuse-proof: +/// +/// No use after return. Once disposed, every ingest member throws +/// . Because each borrow gets its own handle, a caller +/// that hangs on to a returned handle can never reach (and corrupt) the entry the pool has +/// since lent to a different borrower — the stale handle is inert, not aliased. +/// Double dispose is safe. The first returns the entry; +/// any later one is a no-op, so the entry is never returned (or flushed) twice. +/// +/// A single is still not thread-safe: a borrowed handle must not be shared +/// across threads. Borrow one per unit of work and dispose it to return. +/// +/// When the pool's transport is ws:: / wss::, hands out the +/// subtype instead, so the standard capability probe +/// (sender is IQwpWebSocketSender) answers truthfully per transport: it matches on a WS +/// pool and fails on HTTP/TCP, exactly as it does for a standalone . +/// +internal class BorrowedSender : ISender +{ + private readonly ISender _inner; + private readonly PooledSender _entry; + private readonly SenderPool _pool; + + // 0 while live, 1 once returned. Interlocked.Exchange in Dispose makes the return-once / no-op-after + // semantics atomic; every ingest member reads it first and throws if it has flipped. + private int _disposed; + + private protected BorrowedSender(PooledSender entry, SenderPool pool) + { + _entry = entry; + _pool = pool; + _inner = entry.Inner; + } + + /// Allocates the per-borrow handle, picking the QWP-capable subtype when (and only when) + /// the entry's real sender exposes the QWP surface — the inner's type is fixed at entry creation, + /// so the capability is known at borrow time. + internal static BorrowedSender For(PooledSender entry, SenderPool pool) + { + return entry.Inner is IQwpWebSocketSender + ? new BorrowedQwpSender(entry, pool) + : new BorrowedSender(entry, pool); + } + + /// SF slot index the backing entry owns, or -1 when store-and-forward is disabled. Pool-internal + /// metadata (stable for the entry's lifetime); exposed for tests, not gated on dispose. + internal int SlotIndex => _entry.SlotIndex; + + /// + public void Dispose() + { + // First dispose returns the entry; any later call is an idempotent no-op. Dispose is pure resource + // release: it does NOT send — buffered-but-unsent rows are discarded so the reused entry starts + // clean — and it never throws. Call Send()/Flush() before disposing for delivery. The pool — not + // Dispose — tears the real sender down. + if (Interlocked.Exchange(ref _disposed, 1) != 0) + { + return; + } + + if (TryDiscardUnsent()) + { + _pool.GiveBack(_entry); + } + else + { + _pool.DiscardBroken(_entry); + } + } + + /// + public ValueTask DisposeAsync() + { + if (Interlocked.Exchange(ref _disposed, 1) != 0) + { + return default; + } + + return TryDiscardUnsent() + ? _pool.GiveBackAsync(_entry) + : _pool.DiscardBrokenAsync(_entry); + } + + // Discard this borrow's un-sent rows so the reused entry starts with a clean buffer. Returns false + // (→ discard, don't re-pool) when the inner sender is terminally failed and cannot be cleared — so a + // dead sender never aliases a later borrower — or when it still owes a transactional commit for rows + // staged server-side. Never throws — Dispose must not throw. + private bool TryDiscardUnsent() + { + try + { + _inner.Clear(); + + // A transactional (`transaction=on`) ws sender whose auto-flush staged rows server-side under + // FLAG_DEFER_COMMIT still owes a commit; Clear() only resets local buffers. QWP has no rollback, + // so re-pooling the live connection would let the next borrower's first commit silently publish + // this borrow's abandoned rows. Discarding closes the connection, which drops the staged rows. + return _inner is not IPooledTransactionalSender t || !t.HasUncommittedDeferredRows; + } + catch + { + return false; + } + } + + // Gate + reach the delegate in one step: throws if this handle was already returned, otherwise hands + // back the inner sender to forward the call to. Centralises the use-after-return check so every member + // below is a one-liner that cannot forget it. + private protected ISender Active() + { + if (_disposed != 0) + { + throw new ObjectDisposedException(nameof(ISender), + "this pooled sender has been returned to the pool; borrow a fresh one per unit of work"); + } + + return _inner; + } + + // ---- Pure delegation below. Fluent methods return `this` so chaining stays on the handle. ---- + + public int Length => Active().Length; + public int RowCount => Active().RowCount; + public bool WithinTransaction => Active().WithinTransaction; + public DateTime LastFlush => Active().LastFlush; + public SenderOptions Options => Active().Options; + + public ISender Transaction(ReadOnlySpan tableName) + { + Active().Transaction(tableName); + return this; + } + + public void Rollback() => Active().Rollback(); + public Task CommitAsync(CancellationToken ct = default) => Active().CommitAsync(ct); + public void Commit(CancellationToken ct = default) => Active().Commit(ct); + public Task SendAsync(CancellationToken ct = default) => Active().SendAsync(ct); + public void Send(CancellationToken ct = default) => Active().Send(ct); + public bool Flush(TimeSpan timeout, CancellationToken ct = default) => Active().Flush(timeout, ct); + public Task FlushAsync(TimeSpan timeout, CancellationToken ct = default) => Active().FlushAsync(timeout, ct); + public bool Flush(CancellationToken ct = default) => Active().Flush(ct); + public Task FlushAsync(CancellationToken ct = default) => Active().FlushAsync(ct); + + public ISender Table(ReadOnlySpan name) + { + Active().Table(name); + return this; + } + + public ISender Symbol(ReadOnlySpan name, ReadOnlySpan value) + { + Active().Symbol(name, value); + return this; + } + + public ISender Column(ReadOnlySpan name, ReadOnlySpan value) + { + Active().Column(name, value); + return this; + } + + public ISender Column(ReadOnlySpan name, long value) + { + Active().Column(name, value); + return this; + } + + public ISender Column(ReadOnlySpan name, int value) + { + Active().Column(name, value); + return this; + } + + public ISender Column(ReadOnlySpan name, bool value) + { + Active().Column(name, value); + return this; + } + + public ISender Column(ReadOnlySpan name, double value) + { + Active().Column(name, value); + return this; + } + + public ISender Column(ReadOnlySpan name, DateTime value) + { + Active().Column(name, value); + return this; + } + + public ISender Column(ReadOnlySpan name, DateTimeOffset value) + { + Active().Column(name, value); + return this; + } + + public ISender ColumnNanos(ReadOnlySpan name, long timestampNanos) + { + Active().ColumnNanos(name, timestampNanos); + return this; + } + + public ISender Column(ReadOnlySpan name, decimal value) + { + Active().Column(name, value); + return this; + } + + public ISender ColumnDecimal64(ReadOnlySpan name, decimal value, byte scale) + { + Active().ColumnDecimal64(name, value, scale); + return this; + } + + public ISender ColumnDecimal128(ReadOnlySpan name, decimal value, byte scale) + { + Active().ColumnDecimal128(name, value, scale); + return this; + } + + public ISender ColumnDecimal256(ReadOnlySpan name, decimal value, byte scale) + { + Active().ColumnDecimal256(name, value, scale); + return this; + } + + public ISender ColumnDecimal128(ReadOnlySpan name, long lo, long hi, byte scale) + { + Active().ColumnDecimal128(name, lo, hi, scale); + return this; + } + + public ISender ColumnDecimal256(ReadOnlySpan name, long l0, long l1, long l2, long l3, byte scale) + { + Active().ColumnDecimal256(name, l0, l1, l2, l3, scale); + return this; + } + + public ISender Column(ReadOnlySpan name, Guid value) + { + Active().Column(name, value); + return this; + } + + public ISender Column(ReadOnlySpan name, char value) + { + Active().Column(name, value); + return this; + } + + public ISender Column(ReadOnlySpan name, IEnumerable value, IEnumerable shape) where T : struct + { + Active().Column(name, value, shape); + return this; + } + + public ISender Column(ReadOnlySpan name, Array value) + { + Active().Column(name, value); + return this; + } + + public ISender Column(ReadOnlySpan name, ReadOnlySpan value) where T : struct + { + Active().Column(name, value); + return this; + } + + public ValueTask AtAsync(DateTime value, CancellationToken ct = default) => Active().AtAsync(value, ct); + public ValueTask AtAsync(DateTimeOffset value, CancellationToken ct = default) => Active().AtAsync(value, ct); + public ValueTask AtAsync(long value, CancellationToken ct = default) => Active().AtAsync(value, ct); + + [Obsolete("Not compatible with deduplication. Please use `AtAsync(DateTime.UtcNow)` instead.")] + public ValueTask AtNowAsync(CancellationToken ct = default) + { +#pragma warning disable CS0618 + return Active().AtNowAsync(ct); +#pragma warning restore CS0618 + } + + public void At(DateTime value, CancellationToken ct = default) => Active().At(value, ct); + public void At(DateTimeOffset value, CancellationToken ct = default) => Active().At(value, ct); + public void At(long value, CancellationToken ct = default) => Active().At(value, ct); + public ValueTask AtNanosAsync(long timestampNanos, CancellationToken ct = default) => Active().AtNanosAsync(timestampNanos, ct); + public void AtNanos(long timestampNanos, CancellationToken ct = default) => Active().AtNanos(timestampNanos, ct); + + [Obsolete("Not compatible with deduplication. Please use `At(DateTime.UtcNow)` instead.")] + public void AtNow(CancellationToken ct = default) + { +#pragma warning disable CS0618 + Active().AtNow(ct); +#pragma warning restore CS0618 + } + + public void Truncate() => Active().Truncate(); + public void CancelRow() => Active().CancelRow(); + public void Clear() => Active().Clear(); +} diff --git a/src/net-questdb-client/Pooling/IPooledDrainAwareSender.cs b/src/net-questdb-client/Pooling/IPooledDrainAwareSender.cs new file mode 100644 index 0000000..0d1d8a0 --- /dev/null +++ b/src/net-questdb-client/Pooling/IPooledDrainAwareSender.cs @@ -0,0 +1,43 @@ +/******************************************************************************* + * ___ _ ____ ____ + * / _ \ _ _ ___ ___| |_| _ \| __ ) + * | | | | | | |/ _ \/ __| __| | | | _ \ + * | |_| | |_| | __/\__ \ |_| |_| | |_) | + * \__\_\\__,_|\___||___/\__|____/|____/ + * + * Copyright (c) 2014-2019 Appsicle + * Copyright (c) 2019-2026 QuestDB + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + ******************************************************************************/ + +namespace QuestDB.Pooling; + +/// +/// Implemented by senders that ship asynchronously and can hold un-acked in-flight data after a +/// borrow returns (the WS / QWP sender, whose cursor-engine ring keeps frames until the server +/// acks them). The pool reads during idle reaping and keeps the idle +/// clock from starting until it is true, so a pooled sender is never torn down — which in RAM mode +/// frees the ring, silently dropping those frames — while it still owes un-acked rows. Once drained, +/// the sender reaps normally; a permanently wedged one simply lives until the pool is closed. +/// +/// Kept net-agnostic (no dependency on the net7-only WS type) so the pool compiles on net6.0, and +/// so it can be faked in tests. Senders that deliver synchronously (HTTP / TCP) don't implement it +/// and are treated as always drained. +/// +internal interface IPooledDrainAwareSender +{ + /// True when every appended frame has been acknowledged — the sender owes no un-acked data. + bool IsFullyDrained { get; } +} diff --git a/src/net-questdb-client/Pooling/IPooledQueryClientInner.cs b/src/net-questdb-client/Pooling/IPooledQueryClientInner.cs new file mode 100644 index 0000000..bece8dc --- /dev/null +++ b/src/net-questdb-client/Pooling/IPooledQueryClientInner.cs @@ -0,0 +1,49 @@ +/******************************************************************************* + * ___ _ ____ ____ + * / _ \ _ _ ___ ___| |_| _ \| __ ) + * | | | | | | |/ _ \/ __| __| | | | _ \ + * | |_| | |_| | __/\__ \ |_| |_| | |_) | + * \__\_\\__,_|\___||___/\__|____/|____/ + * + * Copyright (c) 2014-2019 Appsicle + * Copyright (c) 2019-2026 QuestDB + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + ******************************************************************************/ + +namespace QuestDB.Pooling; + +/// +/// Implemented by the egress query client so the pool can tell whether a returned client is still +/// reusable. The client's terminal state is sticky and has no public reset; a clean +/// ExecuteAsync leaves it reusable, but the pool checks this seam as belt-and-braces before +/// re-pooling, so a future non-throwing terminal path can never re-pool a dead client. +/// +/// Kept as a small seam (no net7-only types in the signature) so it can be faked in tests. +/// +internal interface IPooledQueryClientInner +{ + /// True when the client has transitioned to a non-recoverable (terminal) or disposed state. + bool IsTerminalOrDisposed { get; } + + /// The request id of the in-flight query, or -1 when none. Ids are unique per client. + long CurrentRequestId { get; } + + /// + /// Cooperatively cancels the query with exactly this request id; a no-op if that query is no + /// longer the in-flight one. Lets a caller that resolved the id while it provably owned the + /// client dispatch the cancel late without ever hitting a successor borrower's query. + /// + void CancelRequest(long requestId); +} diff --git a/src/net-questdb-client/Pooling/IPooledSlotSender.cs b/src/net-questdb-client/Pooling/IPooledSlotSender.cs new file mode 100644 index 0000000..2fe9a6d --- /dev/null +++ b/src/net-questdb-client/Pooling/IPooledSlotSender.cs @@ -0,0 +1,40 @@ +/******************************************************************************* + * ___ _ ____ ____ + * / _ \ _ _ ___ ___| |_| _ \| __ ) + * | | | | | | |/ _ \/ __| __| | | | _ \ + * | |_| | |_| | __/\__ \ |_| |_| | |_) | + * \__\_\\__,_|\___||___/\__|____/|____/ + * + * Copyright (c) 2014-2019 Appsicle + * Copyright (c) 2019-2026 QuestDB + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + ******************************************************************************/ + +namespace QuestDB.Pooling; + +/// +/// Implemented by senders that hold a store-and-forward slot lock (the WS / QWP sender). After a +/// pooled sender is disposed, the pool reads to decide whether +/// the slot index is safe to reuse: a wedged I/O path could leave the OS lock held, in which case +/// the index must be retired rather than handed to a new sender. +/// +/// Kept net-agnostic (no dependency on the net7-only WS type) so the pool compiles on net6.0, and +/// so it can be faked in tests. +/// +internal interface IPooledSlotSender +{ + /// True when this sender holds no slot lock, or its slot lock has been released. + bool IsSlotLockReleased { get; } +} diff --git a/src/net-questdb-client/Pooling/IPooledTransactionalSender.cs b/src/net-questdb-client/Pooling/IPooledTransactionalSender.cs new file mode 100644 index 0000000..87e2e48 --- /dev/null +++ b/src/net-questdb-client/Pooling/IPooledTransactionalSender.cs @@ -0,0 +1,42 @@ +/******************************************************************************* + * ___ _ ____ ____ + * / _ \ _ _ ___ ___| |_| _ \| __ ) + * | | | | | | |/ _ \/ __| __| | | | _ \ + * | |_| | |_| | __/\__ \ |_| |_| | |_) | + * \__\_\\__,_|\___||___/\__|____/|____/ + * + * Copyright (c) 2014-2019 Appsicle + * Copyright (c) 2019-2026 QuestDB + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + ******************************************************************************/ + +namespace QuestDB.Pooling; + +/// +/// Implemented by senders whose transactional mode can stage rows server-side ahead of an explicit +/// commit (the WS / QWP sender under transaction=on). On return, the pool reads +/// to decide whether the entry is safe to re-pool: rows +/// already shipped under FLAG_DEFER_COMMIT cannot be rolled back client-side, so an entry +/// still owing a commit must be discarded (closing the connection drops the staged rows) — re-pooling +/// it would let the next borrower's first commit silently publish the previous borrow's abandoned rows. +/// +/// Kept net-agnostic (no dependency on the net7-only WS type) so the pool compiles on net6.0, and +/// so it can be faked in tests. +/// +internal interface IPooledTransactionalSender +{ + /// True while rows shipped with FLAG_DEFER_COMMIT are still awaiting a commit frame. + bool HasUncommittedDeferredRows { get; } +} diff --git a/src/net-questdb-client/Pooling/PoolHousekeeper.cs b/src/net-questdb-client/Pooling/PoolHousekeeper.cs new file mode 100644 index 0000000..0746e2e --- /dev/null +++ b/src/net-questdb-client/Pooling/PoolHousekeeper.cs @@ -0,0 +1,168 @@ +/******************************************************************************* + * ___ _ ____ ____ + * / _ \ _ _ ___ ___| |_| _ \| __ ) + * | | | | | | |/ _ \/ __| __| | | | _ \ + * | |_| | |_| | __/\__ \ |_| |_| | |_) | + * \__\_\\__,_|\___||___/\__|____/|____/ + * + * Copyright (c) 2014-2019 Appsicle + * Copyright (c) 2019-2026 QuestDB + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + ******************************************************************************/ + +namespace QuestDB.Pooling; + +/// +/// Background sweeper that periodically reaps idle / over-age senders from a +/// (and, on net7.0+, query clients from a +/// ). Runs on a pooled driven by a +/// ; the loop swallows every fault (the C# analogue of Java's +/// catch (Throwable)) so a single bad delegate teardown can never kill all future reaping. +/// +internal sealed class PoolHousekeeper : IDisposable +{ + private static readonly TimeSpan JoinBudget = TimeSpan.FromSeconds(2); + + private readonly CancellationTokenSource _cts = new(); + private readonly Task _loop; + private readonly SenderPool _pool; + private readonly PeriodicTimer _timer; + private int _stopped; +#if NET7_0_OR_GREATER + private readonly QueryClientPool? _queryPool; +#endif + + internal PoolHousekeeper(SenderPool pool, TimeSpan interval) + { + _pool = pool; + _timer = new PeriodicTimer(interval); + _loop = Task.Run(RunAsync); + } + +#if NET7_0_OR_GREATER + /// Sweeps both the sender pool and the (optional) query pool. + internal PoolHousekeeper(SenderPool pool, QueryClientPool? queryPool, TimeSpan interval) + { + _pool = pool; + _queryPool = queryPool; + _timer = new PeriodicTimer(interval); + _loop = Task.Run(RunAsync); + } +#endif + + public void Dispose() + { + Stop(); + _cts.Dispose(); + } + + /// Signals the loop to stop and joins it (bounded), like Java's housekeeper.stop(). + internal void Stop() + { + if (Interlocked.Exchange(ref _stopped, 1) == 0) + { + SignalStop(); + } + + try + { + // Bounded join: a reap in flight finishes well within this; we never block close for long. + _loop.Wait(JoinBudget); + } + catch + { + // the loop swallows its own faults; nothing actionable here + } + } + + /// Async variant of so the async client-dispose path does not block on the join. + internal async ValueTask StopAsync() + { + if (Interlocked.Exchange(ref _stopped, 1) == 0) + { + SignalStop(); + } + + try + { + await _loop.WaitAsync(JoinBudget).ConfigureAwait(false); + } + catch + { + // bounded; the loop swallows its own faults + } + } + + private void SignalStop() + { + try + { + _cts.Cancel(); + } + catch (ObjectDisposedException) + { + // already stopped + } + + _timer.Dispose(); + } + + private async Task RunAsync() + { + try + { + while (await _timer.WaitForNextTickAsync(_cts.Token).ConfigureAwait(false)) + { + try + { + _pool.ReapIdle(); + } + catch + { + // Best-effort housekeeping; a delegate teardown fault must not stop the sweeper. + } + + try + { + // Recover capacity from slots retired by a deferred / wedged teardown whose lock has + // since released, so a transient stall does not permanently shrink effective max. + _pool.ReclaimRetiredSlots(); + } + catch + { + // Best-effort; a faulty reclaim sweep must not stop future housekeeping. + } +#if NET7_0_OR_GREATER + try + { + _queryPool?.ReapIdle(); + } + catch + { + // Independent try/catch so a query-pool fault can't stop sender reaping (or vice versa). + } +#endif + } + } + catch (OperationCanceledException) + { + // Stop() cancelled the loop. + } + catch (ObjectDisposedException) + { + // timer disposed by Stop() + } + } +} diff --git a/src/net-questdb-client/Pooling/PooledQueryClient.cs b/src/net-questdb-client/Pooling/PooledQueryClient.cs new file mode 100644 index 0000000..1554680 --- /dev/null +++ b/src/net-questdb-client/Pooling/PooledQueryClient.cs @@ -0,0 +1,181 @@ +/******************************************************************************* + * ___ _ ____ ____ + * / _ \ _ _ ___ ___| |_| _ \| __ ) + * | | | | | | |/ _ \/ __| __| | | | _ \ + * | |_| | |_| | __/\__ \ |_| |_| | |_) | + * \__\_\\__,_|\___||___/\__|____/|____/ + * + * Copyright (c) 2014-2019 Appsicle + * Copyright (c) 2019-2026 QuestDB + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + ******************************************************************************/ + +#if NET7_0_OR_GREATER +using QuestDB.Qwp.Query; +using QuestDB.Senders; + +namespace QuestDB.Pooling; + +/// +/// Decorator that lends a real from a . +/// One decorator is allocated per pooled client and reused across borrows. +/// +/// Unlike a borrowed ingest sender () there is no flush on return +/// (queries are request/response), and — because the query client is never handed to the user (the +/// runner borrows and returns it internally per execute) — there is no +/// use-after-return hazard to guard, so the entry is reused directly rather than behind a per-borrow +/// handle. / route the client back to the pool: a clean +/// borrow re-pools it (); a borrow that failed +/// () or left the client terminal/disposed is discarded +/// (), because the egress client's terminal state is +/// sticky with no public reset. A second dispose after a return is a no-op. +/// +internal sealed class PooledQueryClient : IQwpQueryClient +{ + private readonly IQwpQueryClient _delegate; + private readonly QueryClientPool _pool; + + // 1 while borrowed, 0 while idle. Interlocked so a double dispose is a safe no-op and only the + // first returns the client to the pool. + private int _inUse; + + // Guards the underlying client's teardown so Close / DiscardBroken / ReapIdle racing can never + // dispose the same delegate twice. + private int _innerDisposed; + + // Set by the query runner when an Execute throws / cancels, so the dispose path discards instead + // of re-pooling. Read on the return path. + private volatile bool _broken; + + internal PooledQueryClient(IQwpQueryClient inner, QueryClientPool pool) + { + _delegate = inner; + _pool = pool; + CreatedAtUtc = DateTime.UtcNow; + IdleSinceUtc = CreatedAtUtc; + } + + /// Wall-clock creation time; drives max-lifetime reaping. + internal DateTime CreatedAtUtc { get; } + + /// Last time this client was returned to the pool; drives idle reaping. Guarded by the pool lock. + internal DateTime IdleSinceUtc { get; set; } + + /// The wrapped client, exposed to the pool for teardown. + internal IQwpQueryClient Inner => _delegate; + + // Belt-and-braces: re-pool only when the inner client is still reusable. A clean Execute leaves it + // reusable, but a future non-throwing terminal path must not silently re-pool a dead client. + private bool IsInnerTerminalOrDisposed => + _delegate is IPooledQueryClientInner s && s.IsTerminalOrDisposed; + + /// The in-flight request id (-1 when none), so can scope a cancel to it. + internal long CurrentRequestId => + _delegate is IPooledQueryClientInner s ? s.CurrentRequestId : -1; + + /// Cancels only the query with this request id; a no-op once that query is no longer in flight. + internal void CancelRequest(long requestId) + { + if (_delegate is IPooledQueryClientInner s) + { + s.CancelRequest(requestId); + } + } + + /// Marks this wrapper as handed out and clears the broken flag. Called by the pool under its lock. + internal void MarkBorrowed() + { + _broken = false; + Volatile.Write(ref _inUse, 1); + } + + /// Flags the client as unusable so the return path discards rather than re-pools it. + internal void MarkBroken() => _broken = true; + + /// Disposes the underlying client for real. Called only by the pool (close / discard / reap). Idempotent. + internal void DisposeInner() + { + if (Interlocked.Exchange(ref _innerDisposed, 1) == 0) + { + _delegate.Dispose(); + } + } + + /// Asynchronously disposes the underlying client for real. Called only by the pool. Idempotent. + internal ValueTask DisposeInnerAsync() + { + return Interlocked.Exchange(ref _innerDisposed, 1) == 0 + ? _delegate.DisposeAsync() + : ValueTask.CompletedTask; + } + + /// + public void Dispose() + { + if (Interlocked.Exchange(ref _inUse, 0) == 0) + { + return; + } + + if (_broken || IsInnerTerminalOrDisposed) + { + _pool.DiscardBroken(this); + } + else + { + _pool.GiveBack(this); + } + } + + /// + public async ValueTask DisposeAsync() + { + if (Interlocked.Exchange(ref _inUse, 0) == 0) + { + return; + } + + if (_broken || IsInnerTerminalOrDisposed) + { + await _pool.DiscardBrokenAsync(this).ConfigureAwait(false); + } + else + { + await _pool.GiveBackAsync(this).ConfigureAwait(false); + } + } + + // ---- Pure delegation below. ---- + + public QwpServerInfo? ServerInfo => _delegate.ServerInfo; + public int NegotiatedVersion => _delegate.NegotiatedVersion; + public string? NegotiatedCompression => _delegate.NegotiatedCompression; + public bool WasLastCloseTimedOut => _delegate.WasLastCloseTimedOut; + + public void Execute(string sql, QwpColumnBatchHandler handler) => _delegate.Execute(sql, handler); + + public void Execute(string sql, QwpBindSetter binds, QwpColumnBatchHandler handler) => + _delegate.Execute(sql, binds, handler); + + public Task ExecuteAsync(string sql, QwpColumnBatchHandler handler, CancellationToken cancellationToken = default) => + _delegate.ExecuteAsync(sql, handler, cancellationToken); + + public Task ExecuteAsync(string sql, QwpBindSetter binds, QwpColumnBatchHandler handler, + CancellationToken cancellationToken = default) => + _delegate.ExecuteAsync(sql, binds, handler, cancellationToken); + + public void Cancel() => _delegate.Cancel(); +} +#endif diff --git a/src/net-questdb-client/Pooling/PooledSender.cs b/src/net-questdb-client/Pooling/PooledSender.cs new file mode 100644 index 0000000..e18b133 --- /dev/null +++ b/src/net-questdb-client/Pooling/PooledSender.cs @@ -0,0 +1,99 @@ +/******************************************************************************* + * ___ _ ____ ____ + * / _ \ _ _ ___ ___| |_| _ \| __ ) + * | | | | | | |/ _ \/ __| __| | | | _ \ + * | |_| | |_| | __/\__ \ |_| |_| | |_) | + * \__\_\\__,_|\___||___/\__|____/|____/ + * + * Copyright (c) 2014-2019 Appsicle + * Copyright (c) 2019-2026 QuestDB + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + ******************************************************************************/ + +using QuestDB.Senders; + +namespace QuestDB.Pooling; + +/// +/// A pool entry: the long-lived box around one real plus the bookkeeping the +/// needs (slot index, creation / idle timestamps, lock-release probe). One +/// entry is allocated per pool slot and reused for every borrow, so steady-state reuse allocates +/// nothing on the inner sender. +/// +/// The entry is never handed to callers directly — each borrow wraps it in a fresh +/// handle, which owns the flush-and-return lifecycle and the +/// use-after-return guard. The entry only knows how to be torn down for real +/// (), which exactly one party (close / discard / reap) ever does. +/// +internal sealed class PooledSender +{ + private readonly ISender _inner; + + // Guards the underlying sender's teardown so Close / DiscardBroken / ReapIdle racing each other can + // never dispose the same delegate twice. + private int _innerDisposed; + + internal PooledSender(ISender inner, int slotIndex) + { + _inner = inner; + SlotIndex = slotIndex; + CreatedAtUtc = DateTime.UtcNow; + IdleSinceUtc = CreatedAtUtc; + } + + /// The wrapped real sender, lent to a for the duration of a borrow. + internal ISender Inner => _inner; + + /// SF slot index this entry owns, or -1 when store-and-forward is disabled. + internal int SlotIndex { get; } + + /// Wall-clock creation time; drives max-lifetime reaping. + internal DateTime CreatedAtUtc { get; } + + /// Last time this entry was returned to the pool; drives idle reaping. Guarded by the pool lock. + internal DateTime IdleSinceUtc { get; set; } + + /// + /// True if the inner sender holds no SF slot lock or has released it — i.e. the slot index is + /// safe to reuse. Always true for non-SF senders (HTTP/TCP/WS-RAM). Call only after the inner + /// sender has been disposed. + /// + internal bool IsInnerSlotLockReleased => _inner is not IPooledSlotSender s || s.IsSlotLockReleased; + + /// + /// True when the inner sender owes no un-acked in-flight data, or isn't drain-aware (HTTP / TCP, + /// which deliver synchronously). The pool holds the idle-reap clock while this is false so a WS + /// sender whose ring still carries un-acked frames is never reaped — in RAM mode that would free + /// the ring and drop the frames with no error. See . + /// + internal bool IsInnerFullyDrained => _inner is not IPooledDrainAwareSender d || d.IsFullyDrained; + + /// Disposes the underlying sender for real. Called only by the pool (close / discard / reap). Idempotent. + internal void DisposeInner() + { + if (Interlocked.Exchange(ref _innerDisposed, 1) == 0) + { + _inner.Dispose(); + } + } + + /// Asynchronously disposes the underlying sender for real. Called only by the pool. Idempotent. + internal ValueTask DisposeInnerAsync() + { + return Interlocked.Exchange(ref _innerDisposed, 1) == 0 + ? _inner.DisposeAsync() + : ValueTask.CompletedTask; + } +} diff --git a/src/net-questdb-client/Pooling/QueryClientPool.cs b/src/net-questdb-client/Pooling/QueryClientPool.cs new file mode 100644 index 0000000..a51e14a --- /dev/null +++ b/src/net-questdb-client/Pooling/QueryClientPool.cs @@ -0,0 +1,660 @@ +/******************************************************************************* + * ___ _ ____ ____ + * / _ \ _ _ ___ ___| |_| _ \| __ ) + * | | | | | | |/ _ \/ __| __| | | | _ \ + * | |_| | |_| | __/\__ \ |_| |_| | |_) | + * \__\_\\__,_|\___||___/\__|____/|____/ + * + * Copyright (c) 2014-2019 Appsicle + * Copyright (c) 2019-2026 QuestDB + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + ******************************************************************************/ + +#if NET7_0_OR_GREATER +using QuestDB.Enums; +using QuestDB.Senders; +using QuestDB.Utils; + +namespace QuestDB.Pooling; + +/// +/// Elastic pool of instances, each wrapped in a reusable +/// decorator. The read-side analog of , +/// deliberately stripped of all store-and-forward slot machinery (queries have no SF) and the +/// leaked-slot / leak-debt accounting. +/// +/// Capacity is bounded by a counting in-use clients: a permit is +/// taken on borrow and released on return. A client is only created when no idle one exists and a +/// permit is held, so the total alive count ≤ max. The WebSocket upgrade happens OUTSIDE +/// the lock so a slow connect cannot block other borrowers. +/// +internal sealed class QueryClientPool +{ + private readonly SemaphoreSlim _capacity; + private readonly CancellationTokenSource _closeCts = new(); + private readonly object _gate = new(); + private readonly TimeSpan _idleTimeout; + private readonly TimeSpan _maxLifetime; + private readonly int _max; + private readonly int _min; + private readonly int _acquireTimeoutMs; + private readonly string? _queryConfStr; + private readonly Func>? _clientFactory; + + private readonly Stack _available = new(); + private readonly List _all = new(); + private bool _closed; + + /// Production constructor: pool sizes from , clients built from + /// (a ws/wss connect string). + internal QueryClientPool(SenderOptions poolConfig, string queryConfStr) + : this(poolConfig, queryConfStr, null) + { + } + + /// Test seam: inject a client factory so unit tests need no live server. + /// may be null when a factory is supplied. + internal QueryClientPool(SenderOptions poolConfig, string? queryConfStr, Func>? clientFactory) + { + // Re-validate: builder methods may have mutated min/max after the connect-string parse. + poolConfig.ValidateQueryPoolOptions(); + + _min = poolConfig.query_pool_min; + _max = poolConfig.query_pool_max; + _acquireTimeoutMs = checked((int)poolConfig.acquire_timeout_ms.TotalMilliseconds); + _idleTimeout = poolConfig.idle_timeout_ms; + _maxLifetime = poolConfig.max_lifetime_ms; + _queryConfStr = queryConfStr; + _clientFactory = clientFactory; + _capacity = new SemaphoreSlim(_max, _max); + + try + { + PreWarm(); + } + catch + { + _closeCts.Dispose(); + _capacity.Dispose(); + throw; + } + } + + /// Number of idle query clients currently parked in the pool. + internal int AvailableSize + { + get + { + lock (_gate) + { + return _available.Count; + } + } + } + + /// Total query clients alive (idle + in-use). + internal int TotalSize + { + get + { + lock (_gate) + { + return _all.Count; + } + } + } + + internal bool IsClosed + { + get + { + lock (_gate) + { + return _closed; + } + } + } + + private void PreWarm() + { + var created = new List(_min); + try + { + for (var i = 0; i < _min; i++) + { + // Client creation is async (WS upgrade); prewarm eagerly, off any captured + // SyncContext, matching QueryClient.New's sync-over-async idiom. + created.Add(Task.Run(() => CreateClientAsync().AsTask()).GetAwaiter().GetResult()); + } + } + catch + { + foreach (var pc in created) + { + try + { + pc.DisposeInner(); + } + catch + { + // best-effort teardown of the partially-warmed pool + } + } + + throw; + } + + lock (_gate) + { + foreach (var pc in created) + { + _all.Add(pc); + _available.Push(pc); + } + } + } + + /// Borrows a query client, blocking up to acquire_timeout_ms. + internal PooledQueryClient Borrow() + { + ThrowIfClosed(); + bool acquired; + try + { + acquired = _capacity.Wait(_acquireTimeoutMs, _closeCts.Token); + } + catch (OperationCanceledException) + { + throw Closed(); + } + catch (ObjectDisposedException) + { + throw Closed(); + } + + if (!acquired) + { + throw Exhausted(); + } + + return TakeOrCreate(); + } + + /// + internal async ValueTask BorrowAsync(CancellationToken ct = default) + { + ThrowIfClosed(); + bool acquired; + CancellationTokenSource? linked = null; + try + { + // Skip the linked-source allocation on the common ct=default path: WaitAsync takes a single + // token, so pass _closeCts.Token straight through and only link when the caller can also cancel. + // Reading _closeCts.Token must be inside the try: a concurrent Close that disposes the CTS + // makes the getter throw, which we want surfaced as the friendly closed error. + CancellationToken waitToken; + if (ct.CanBeCanceled) + { + linked = CancellationTokenSource.CreateLinkedTokenSource(ct, _closeCts.Token); + waitToken = linked.Token; + } + else + { + waitToken = _closeCts.Token; + } + + acquired = await _capacity.WaitAsync(_acquireTimeoutMs, waitToken).ConfigureAwait(false); + } + catch (OperationCanceledException) when (_closeCts.IsCancellationRequested) + { + throw Closed(); + } + catch (ObjectDisposedException) + { + throw Closed(); + } + finally + { + linked?.Dispose(); + } + + if (!acquired) + { + throw Exhausted(); + } + + return await TakeOrCreateAsync().ConfigureAwait(false); + } + + // Permit already held. Reuse an idle client or create a fresh one outside the lock. + private async ValueTask TakeOrCreateAsync() + { + lock (_gate) + { + if (_closed) + { + ReleaseCapacity(); + throw Closed(); + } + + if (_available.Count > 0) + { + var reused = _available.Pop(); + reused.MarkBorrowed(); + return reused; + } + } + + return await CreateOutsideLockAsync().ConfigureAwait(false); + } + + private PooledQueryClient TakeOrCreate() + { + lock (_gate) + { + if (_closed) + { + ReleaseCapacity(); + throw Closed(); + } + + if (_available.Count > 0) + { + var reused = _available.Pop(); + reused.MarkBorrowed(); + return reused; + } + } + + // Off the captured SyncContext so sync-over-async creation can't deadlock. + return Task.Run(() => CreateOutsideLockAsync().AsTask()).GetAwaiter().GetResult(); + } + + private async ValueTask CreateOutsideLockAsync() + { + PooledQueryClient created; + try + { + created = await CreateClientAsync().ConfigureAwait(false); + } + catch + { + ReleaseCapacity(); + throw; + } + + lock (_gate) + { + if (_closed) + { + try + { + created.DisposeInner(); + } + catch + { + // best effort + } + + ReleaseCapacity(); + throw Closed(); + } + + _all.Add(created); + } + + created.MarkBorrowed(); + return created; + } + + private async ValueTask CreateClientAsync() + { + var factory = _clientFactory ?? DefaultFactoryAsync; + var inner = await factory().ConfigureAwait(false); + return new PooledQueryClient(inner, this); + } + + private ValueTask DefaultFactoryAsync() + { + if (_queryConfStr is null) + { + throw new IngressError(ErrorCode.InvalidApiCall, + "QueryClientPool has no query connect string and no client factory"); + } + + return new ValueTask(QueryClient.NewAsync(_queryConfStr)); + } + + /// Returns a borrowed query client to the pool. + internal void GiveBack(PooledQueryClient pc) + { + if (GiveBackOrTakeOwnership(pc)) + { + try + { + pc.DisposeInner(); + } + catch + { + // best effort + } + } + + ReleaseCapacity(); + } + + /// + internal async ValueTask GiveBackAsync(PooledQueryClient pc) + { + if (GiveBackOrTakeOwnership(pc)) + { + try + { + await pc.DisposeInnerAsync().ConfigureAwait(false); + } + catch + { + // best effort + } + } + + ReleaseCapacity(); + } + + // Re-pool an idle-again client, or — if the pool closed while it was on loan — hand teardown back to + // the caller. Close() only disposes idle clients and merely *cancels* in-flight ones; it never + // disposes an in-use client (that would race the borrower's still-running query on the + // non-thread-safe client). So a client returning post-close is disposed here by its borrower, on the + // borrower's own stack after its query already unwound. Returns true when the caller must dispose. + private bool GiveBackOrTakeOwnership(PooledQueryClient pc) + { + lock (_gate) + { + if (_closed) + { + return true; + } + + pc.IdleSinceUtc = DateTime.UtcNow; + _available.Push(pc); + return false; + } + } + + /// Evicts a broken / terminal client: dispose it for real and release its permit. Unlike the + /// sender pool there is no slot to reclaim or retire, so effective max never shrinks. + internal void DiscardBroken(PooledQueryClient pc) + { + lock (_gate) + { + _all.Remove(pc); + } + + try + { + pc.DisposeInner(); + } + catch + { + // best effort + } + + ReleaseCapacity(); + } + + /// + internal async ValueTask DiscardBrokenAsync(PooledQueryClient pc) + { + lock (_gate) + { + _all.Remove(pc); + } + + try + { + await pc.DisposeInnerAsync().ConfigureAwait(false); + } + catch + { + // best effort + } + + ReleaseCapacity(); + } + + /// Reaps idle / over-age clients down to min. Driven by the housekeeper. + internal void ReapIdle() + { + if (IsClosed) + { + return; + } + + var now = DateTime.UtcNow; + List? toDispose = null; + + lock (_gate) + { + if (_closed) + { + return; + } + + var idle = _available.ToArray(); + _available.Clear(); + foreach (var pc in idle) + { + var overIdle = now - pc.IdleSinceUtc >= _idleTimeout; + var overAge = now - pc.CreatedAtUtc >= _maxLifetime; + if ((overIdle || overAge) && _all.Count > _min) + { + _all.Remove(pc); + (toDispose ??= new List()).Add(pc); + } + else + { + _available.Push(pc); + } + } + } + + if (toDispose is null) + { + return; + } + + foreach (var pc in toDispose) + { + try + { + pc.DisposeInner(); + } + catch + { + // best effort; reaping must never throw + } + } + } + + /// Shuts the pool down, closing every idle underlying client. Idempotent. Clients currently + /// borrowed are only *cancelled* here (so a blocked ExecuteAsync unwinds) and torn down by their + /// borrower on return — never disposed here, to avoid racing a non-thread-safe in-flight query. + internal void Close() + { + var snapshot = BeginClose(); + if (snapshot is null) + { + return; + } + + // Cancel in-flight queries first so blocked ExecuteAsync calls unwind and return their clients + // (GiveBack/DiscardBroken dispose them). Never dispose an in-use client here. + foreach (var pc in snapshot.Value.InUse) + { + CancelInner(pc); + } + + foreach (var pc in snapshot.Value.Idle) + { + try + { + pc.DisposeInner(); + } + catch + { + // best effort + } + } + + DisposePrimitives(); + } + + /// + internal async ValueTask CloseAsync() + { + var snapshot = BeginClose(); + if (snapshot is null) + { + return; + } + + foreach (var pc in snapshot.Value.InUse) + { + CancelInner(pc); + } + + foreach (var pc in snapshot.Value.Idle) + { + try + { + await pc.DisposeInnerAsync().ConfigureAwait(false); + } + catch + { + // best effort + } + } + + DisposePrimitives(); + } + + // ---- internals ---- + + // Cooperatively cancel an in-flight query so a blocked ExecuteAsync returns before we dispose the + // client (Java cancel-then-join). No-op on an idle client. + private static void CancelInner(PooledQueryClient pc) + { + try + { + pc.Inner.Cancel(); + } + catch + { + // best effort + } + } + + // Idle clients are disposed by Close(); in-use clients are only cancelled there and disposed by their + // borrower on return. Splitting them keeps Close from disposing a client whose query is still running. + private readonly struct CloseSet + { + internal CloseSet(List idle, List inUse) + { + Idle = idle; + InUse = inUse; + } + + internal List Idle { get; } + internal List InUse { get; } + } + + private CloseSet? BeginClose() + { + List idle; + List inUse; + lock (_gate) + { + if (_closed) + { + return null; + } + + _closed = true; + idle = new List(_available); + var idleSet = new HashSet(_available, ReferenceEqualityComparer.Instance); + inUse = new List(); + foreach (var pc in _all) + { + if (!idleSet.Contains(pc)) + { + inUse.Add(pc); + } + } + + _all.Clear(); + _available.Clear(); + } + + try + { + _closeCts.Cancel(); + } + catch (ObjectDisposedException) + { + // already torn down + } + + return new CloseSet(idle, inUse); + } + + private void DisposePrimitives() + { + _closeCts.Dispose(); + _capacity.Dispose(); + } + + private void ReleaseCapacity() + { + try + { + _capacity.Release(); + } + catch (ObjectDisposedException) + { + // pool closed concurrently + } + catch (SemaphoreFullException) + { + // defensive: never expected, the borrow/return accounting is 1:1 + } + } + + private void ThrowIfClosed() + { + if (IsClosed) + { + throw Closed(); + } + } + + private static IngressError Closed() => + new(ErrorCode.InvalidApiCall, "QuestDBClient handle is closed"); + + private IngressError Exhausted() => + new(ErrorCode.PoolExhausted, + $"timed out waiting for a query client from the pool after {_acquireTimeoutMs}ms " + + $"(query_pool_max={_max})"); +} +#endif diff --git a/src/net-questdb-client/Pooling/QuestDBClientImpl.cs b/src/net-questdb-client/Pooling/QuestDBClientImpl.cs new file mode 100644 index 0000000..03cb2b2 --- /dev/null +++ b/src/net-questdb-client/Pooling/QuestDBClientImpl.cs @@ -0,0 +1,248 @@ +/******************************************************************************* + * ___ _ ____ ____ + * / _ \ _ _ ___ ___| |_| _ \| __ ) + * | | | | | | |/ _ \/ __| __| | | | _ \ + * | |_| | |_| | __/\__ \ |_| |_| | |_) | + * \__\_\\__,_|\___||___/\__|____/|____/ + * + * Copyright (c) 2014-2019 Appsicle + * Copyright (c) 2019-2026 QuestDB + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + ******************************************************************************/ + +using QuestDB.Enums; +using QuestDB.Senders; +using QuestDB.Utils; +#if NET7_0_OR_GREATER +using QwpColumnBatchHandler = QuestDB.Qwp.Query.QwpColumnBatchHandler; +#endif + +namespace QuestDB.Pooling; + +/// +/// Default : owns a and, on net7.0+ when a +/// query config is supplied, a . A single +/// reaps idle / over-age instances from both pools. +/// +internal sealed class QuestDBClientImpl : IQuestDBClient +{ + private readonly PoolHousekeeper _housekeeper; + private readonly SenderPool _pool; +#if NET7_0_OR_GREATER + private readonly QueryClientPool? _queryPool; +#endif + + internal QuestDBClientImpl(SenderOptions poolConfig, string ingestConfStr, string? queryConfStr, + bool forceWsAsyncConnect = false) + { + SenderPool? pool = null; + PoolHousekeeper? housekeeper = null; +#if NET7_0_OR_GREATER + QueryClientPool? queryPool = null; +#endif + // The sender pool warms `sender_pool_min` live connections (and, in ws::+sf_dir mode, takes + // slot flocks / mmaps) the moment it is constructed. If a later step throws — most realistically + // the query pool's live /read/v1 prewarm against a down endpoint — the half-built handle is + // never returned, so Close() never runs and everything already built leaks. Tear it down here. + try + { + pool = new SenderPool(poolConfig, ingestConfStr, forceWsAsyncConnect); +#if NET7_0_OR_GREATER + queryPool = queryConfStr is null ? null : new QueryClientPool(poolConfig, queryConfStr); + housekeeper = new PoolHousekeeper(pool, queryPool, poolConfig.housekeeper_interval_ms); +#else + housekeeper = new PoolHousekeeper(pool, poolConfig.housekeeper_interval_ms); +#endif + } + catch + { + // Same order as Close(): stop the sweeper, then query pool, then sender pool (owns SF/IO). + SafeTeardownOnConstructionFailure(housekeeper, +#if NET7_0_OR_GREATER + queryPool, +#endif + pool); + throw; + } + + _pool = pool; + _housekeeper = housekeeper; +#if NET7_0_OR_GREATER + _queryPool = queryPool; +#endif + } + + // Test seam: inject a sender factory (sender-only handle, no query pool). + internal QuestDBClientImpl(SenderOptions poolConfig, Func senderFactory) + { + SenderPool? pool = null; + PoolHousekeeper? housekeeper = null; + try + { + pool = new SenderPool(poolConfig, null, senderFactory); + housekeeper = new PoolHousekeeper(pool, poolConfig.housekeeper_interval_ms); + } + catch + { + SafeTeardownOnConstructionFailure(housekeeper, +#if NET7_0_OR_GREATER + null, +#endif + pool); + throw; + } + + _pool = pool; + _housekeeper = housekeeper; + } + +#if NET7_0_OR_GREATER + // Test seam: inject both a sender factory and a query-client factory. + internal QuestDBClientImpl(SenderOptions poolConfig, Func senderFactory, + Func> queryFactory) + { + SenderPool? pool = null; + QueryClientPool? queryPool = null; + PoolHousekeeper? housekeeper = null; + try + { + pool = new SenderPool(poolConfig, null, senderFactory); + queryPool = new QueryClientPool(poolConfig, null, queryFactory); + housekeeper = new PoolHousekeeper(pool, queryPool, poolConfig.housekeeper_interval_ms); + } + catch + { + SafeTeardownOnConstructionFailure(housekeeper, queryPool, pool); + throw; + } + + _pool = pool; + _queryPool = queryPool; + _housekeeper = housekeeper; + } +#endif + + // Best-effort teardown of whatever was built before a constructor threw. Each step is independently + // guarded so one failing teardown can't strand the resources owned by the others. + private static void SafeTeardownOnConstructionFailure( + PoolHousekeeper? housekeeper, +#if NET7_0_OR_GREATER + QueryClientPool? queryPool, +#endif + SenderPool? pool) + { + try + { + housekeeper?.Dispose(); + } + catch + { + // best effort + } + +#if NET7_0_OR_GREATER + try + { + queryPool?.Close(); + } + catch + { + // best effort + } +#endif + + try + { + pool?.Close(); + } + catch + { + // best effort + } + } + + public int AvailableSenderCount => _pool.AvailableSize; + public int TotalSenderCount => _pool.TotalSize; + + public ISender BorrowSender() + { + return _pool.Borrow(); + } + + public async ValueTask BorrowSenderAsync(CancellationToken ct = default) + { + return await _pool.BorrowAsync(ct).ConfigureAwait(false); + } + + public bool Flush(TimeSpan timeout, CancellationToken ct = default) => _pool.Flush(timeout, ct); + + public bool Flush(CancellationToken ct = default) => _pool.Flush(ct); + + public ValueTask FlushAsync(TimeSpan timeout, CancellationToken ct = default) => _pool.FlushAsync(timeout, ct); + + public ValueTask FlushAsync(CancellationToken ct = default) => _pool.FlushAsync(ct); + +#if NET7_0_OR_GREATER + public int AvailableQueryClientCount => RequireQueryPool().AvailableSize; + public int TotalQueryClientCount => RequireQueryPool().TotalSize; + + public Query NewQuery() + { + return new Query(RequireQueryPool()); + } + + public Task ExecuteSqlAsync(string sql, QwpColumnBatchHandler handler, CancellationToken ct = default) + { + return NewQuery().Sql(sql).Handler(handler).ExecuteAsync(ct); + } + + private QueryClientPool RequireQueryPool() + { + return _queryPool ?? throw new IngressError(ErrorCode.ConfigError, + "no query configuration; pass a ws:: / wss:: query config via QueryConfig() or " + + "QuestDBClient.Connect(ingest, query)"); + } +#endif + + public void Close() + { + // Stop the sweeper before tearing the pools down so it cannot reap mid-close. + _housekeeper.Dispose(); +#if NET7_0_OR_GREATER + // Query pool first (no flock/mmap); sender pool last (owns SF/flock/IO resources). + _queryPool?.Close(); +#endif + _pool.Close(); + } + + public void Dispose() + { + Close(); + } + + public async ValueTask DisposeAsync() + { + // Async join so we don't block the caller for up to the housekeeper's 2s budget. + await _housekeeper.StopAsync().ConfigureAwait(false); + _housekeeper.Dispose(); +#if NET7_0_OR_GREATER + if (_queryPool is not null) + { + await _queryPool.CloseAsync().ConfigureAwait(false); + } +#endif + await _pool.CloseAsync().ConfigureAwait(false); + } +} diff --git a/src/net-questdb-client/Pooling/SenderPool.cs b/src/net-questdb-client/Pooling/SenderPool.cs new file mode 100644 index 0000000..6e05fcc --- /dev/null +++ b/src/net-questdb-client/Pooling/SenderPool.cs @@ -0,0 +1,1083 @@ +/******************************************************************************* + * ___ _ ____ ____ + * / _ \ _ _ ___ ___| |_| _ \| __ ) + * | | | | | | |/ _ \/ __| __| | | | _ \ + * | |_| | |_| | __/\__ \ |_| |_| | |_) | + * \__\_\\__,_|\___||___/\__|____/|____/ + * + * Copyright (c) 2014-2019 Appsicle + * Copyright (c) 2019-2026 QuestDB + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + ******************************************************************************/ + +using QuestDB.Enums; +using QuestDB.Senders; +using QuestDB.Utils; + +namespace QuestDB.Pooling; + +/// +/// Elastic pool of instances, each boxed in a reusable +/// entry and lent out per borrow as a fresh +/// handle. Keeps at least min senders warm, grows on +/// demand to max, and (via , driven by the housekeeper) reaps +/// idle / over-age senders down to min. +/// +/// Capacity is bounded by a that counts in-use senders: a permit is +/// taken on borrow and released on return. A sender is only created when no idle one exists and a +/// permit is held, which keeps the total alive count ≤ max. Sender construction (TLS / DNS +/// / connect) happens OUTSIDE the lock so a slow connect cannot block other borrowers. The reaper +/// competes through the same gate (): it only removes an idle +/// sender after taking a permit for it, and a reaped or retired SF sender keeps that permit +/// physically withheld until its slot index is reusable, so the semaphore never advertises +/// capacity whose slot is still locked. +/// +internal sealed class SenderPool +{ + private readonly SemaphoreSlim _capacity; + private readonly CancellationTokenSource _closeCts = new(); + private readonly string? _confStr; + private readonly object _gate = new(); + private readonly TimeSpan _idleTimeout; + private readonly TimeSpan _maxLifetime; + private readonly int _max; + private readonly int _min; + private readonly Func _senderFactory; + private readonly int _acquireTimeoutMs; + private readonly TimeSpan _defaultFlushTimeout; + + // lazy_connect: pooled ws senders connect asynchronously so a down server doesn't fail-fast the + // pre-warm; writes buffer until the wire is up. No effect on non-ws senders. + private readonly bool _forceWsAsyncConnect; + + // Idle senders as a deque sorted by idle time: index 0 is the coldest (longest idle), the tail the + // hottest. Borrowers pop/push the hot end — LIFO reuse concentrates traffic on few senders so the + // excess goes genuinely cold and shrinks toward min — which keeps the list IdleSinceUtc-ordered, so + // the reaper only ever inspects the cold end and stops at the first entry inside its timeout: + // O(reaped), not O(idle). max_lifetime is NOT covered by that order (age follows CreatedAtUtc, not + // IdleSinceUtc), so it is handled by a separate walk gated on _idleOldestCreatedUtc: a conservative + // lower bound over the parked entries' CreatedAtUtc, only lowered when an entry is parked and + // retightened by the walk itself — the walk therefore runs only when some parked entry has actually + // crossed max_lifetime, keeping the common sweep O(reaped). + private readonly List _idle = new(); + private DateTime _idleOldestCreatedUtc = DateTime.MaxValue; + private readonly List _all = new(); + private bool _closed; + + // Store-and-forward slot management. Each pooled WS+SF sender owns a distinct slot identity + // (`-`) so siblings never collide on a slot directory / flock. `_freeSlots` holds the + // indices owned by no live (or not-yet-fully-torn-down) sender; LIFO reuse keeps the on-disk + // working set of slot directories compact. A retired index (lock not yet released after dispose) is + // simply absent from the stack and counts against effective capacity, matching the Java pool's + // `leakedSlots`. Unlike a permanent leak, the housekeeper re-tests retired slots + // (`ReclaimRetiredSlots`) and frees the index + permit once the engine's deferred teardown finally + // releases the lock, so retirement is a live shrink, not a monotonic counter. + private readonly bool _storeAndForward; + private readonly string _slotBaseId; + private readonly Stack _freeSlots = new(); + + // Senders retired with their slot lock still held, awaiting reclaim. Every entry has exactly one + // capacity permit physically withheld on its behalf: a discarded sender's own permit is simply not + // released, and the reap path takes a free permit up front — skipping the reap entirely when none is + // free (see ReapIdle). Reclaiming therefore always frees the index AND releases one permit; there is + // no deferred "leak debt" to settle. + private readonly List _retired = new(); + + /// Production constructor: pool sizes come from , senders are + /// built from . is set by the + /// lazy_connect facade path so pooled ws senders connect asynchronously. + internal SenderPool(SenderOptions poolConfig, string confStr, bool forceWsAsyncConnect = false) + : this(poolConfig, confStr, null, forceWsAsyncConnect) + { + } + + /// Test seam: inject a sender factory so unit tests need no live server. + /// may be null when a factory is supplied. + internal SenderPool(SenderOptions poolConfig, string? confStr, Func? senderFactory, + bool forceWsAsyncConnect = false) + { + // Re-validate: builder methods may have mutated min/max after the connect-string parse. + poolConfig.ValidatePoolOptions(); + + _min = poolConfig.sender_pool_min; + _max = poolConfig.sender_pool_max; + _acquireTimeoutMs = checked((int)poolConfig.acquire_timeout_ms.TotalMilliseconds); + _defaultFlushTimeout = poolConfig.close_flush_timeout_millis; + _idleTimeout = poolConfig.idle_timeout_ms; + _maxLifetime = poolConfig.max_lifetime_ms; + _confStr = confStr; + _senderFactory = senderFactory ?? CreateDefaultInner; + _forceWsAsyncConnect = forceWsAsyncConnect; + _capacity = new SemaphoreSlim(_max, _max); + + _storeAndForward = poolConfig.IsWebSocket() && !string.IsNullOrEmpty(poolConfig.sf_dir); + _slotBaseId = poolConfig.sender_id; + if (_storeAndForward) + { + // Push high-to-low so the first pops hand out 0, 1, 2... — fresh pools fill from index 0. + for (var i = _max - 1; i >= 0; i--) + { + _freeSlots.Push(i); + } + } + + try + { + PreWarm(); + } + catch + { + // A failed pre-warm (e.g. a slow/refused warm connect) must not leak the primitives. + _closeCts.Dispose(); + _capacity.Dispose(); + throw; + } + } + + /// Number of idle senders currently parked in the pool. + internal int AvailableSize + { + get + { + lock (_gate) + { + return _idle.Count; + } + } + } + + /// Total senders alive (idle + in-use). + internal int TotalSize + { + get + { + lock (_gate) + { + return _all.Count; + } + } + } + + internal bool IsClosed + { + get + { + lock (_gate) + { + return _closed; + } + } + } + + /// Count of SF slot indices currently retired because a sender has not yet released its lock. + /// Reclaimed back toward zero by the housekeeper once the lock releases (see ). + internal int LeakedSlotCount + { + get + { + lock (_gate) + { + return _retired.Count; + } + } + } + + private void PreWarm() + { + var created = new List(_min); + try + { + for (var i = 0; i < _min; i++) + { + created.Add(CreateSender(AllocateSlotIndex())); + } + } + catch + { + foreach (var ps in created) + { + try + { + ps.DisposeInner(); + } + catch + { + // best-effort teardown of the partially-warmed pool + } + + FreeSlotIndex(ps.SlotIndex); + } + + throw; + } + + lock (_gate) + { + foreach (var ps in created) + { + _all.Add(ps); + _idle.Add(ps); + if (ps.CreatedAtUtc < _idleOldestCreatedUtc) + { + _idleOldestCreatedUtc = ps.CreatedAtUtc; + } + } + } + } + + /// Borrows a sender, blocking up to acquire_timeout_ms. The returned + /// is a single-use handle: dispose it to return it to the pool + /// (dispose does not send — call Send()/Flush() first for delivery). + internal BorrowedSender Borrow() + { + ThrowIfClosed(); + bool acquired; + try + { + acquired = _capacity.Wait(_acquireTimeoutMs, _closeCts.Token); + } + catch (OperationCanceledException) + { + throw Closed(); + } + catch (ObjectDisposedException) + { + throw Closed(); + } + + if (!acquired) + { + throw Exhausted(); + } + + return BorrowedSender.For(TakeOrCreate(), this); + } + + /// + internal async ValueTask BorrowAsync(CancellationToken ct = default) + { + ThrowIfClosed(); + bool acquired; + CancellationTokenSource? linked = null; + try + { + // Reading _closeCts.Token must be inside the try: a concurrent Close that disposes the CTS + // makes the getter throw, which we want surfaced as the friendly closed error. + // Skip the linked-source allocation on the common ct=default path: WaitAsync takes a single + // token, so pass _closeCts.Token straight through and only link when the caller can also cancel. + CancellationToken waitToken; + if (ct.CanBeCanceled) + { + linked = CancellationTokenSource.CreateLinkedTokenSource(ct, _closeCts.Token); + waitToken = linked.Token; + } + else + { + waitToken = _closeCts.Token; + } + + acquired = await _capacity.WaitAsync(_acquireTimeoutMs, waitToken).ConfigureAwait(false); + } + catch (OperationCanceledException) when (_closeCts.IsCancellationRequested) + { + throw Closed(); + } + catch (ObjectDisposedException) + { + throw Closed(); + } + finally + { + linked?.Dispose(); + } + + if (!acquired) + { + throw Exhausted(); + } + + return BorrowedSender.For(TakeOrCreate(), this); + } + + /// + /// Drains every sender currently in the pool: flushes each one's buffered rows and blocks until the + /// server has acknowledged them, or elapses (per sender). This is the + /// pool-wide delivery barrier — the equivalent of calling Flush on each borrowed sender. + /// + /// Quiescence barrier: intended to be called when no senders are borrowed (all returned). It + /// does not synchronise against a concurrent borrow — draining a sender another thread is actively + /// writing to is a data race. Returns true only if every sender drained fully within the + /// timeout; false if any timed out or latched an error. + /// + internal bool Flush(TimeSpan timeout, CancellationToken ct = default) + { + ThrowIfClosed(); + var snapshot = SnapshotAll(); + if (snapshot.Count == 0) + { + return true; + } + + // Drain the senders concurrently on the thread pool — they are independent connections, so this + // turns the barrier's wall-clock cost into the slowest single drain rather than their sum, mirroring + // FlushAsync. DrainOneSync swallows a sender's terminal error (reports not-drained) and re-throws + // OperationCanceledException, so the only fault WaitAll can surface is cancellation. + var tasks = new Task[snapshot.Count]; + for (var i = 0; i < snapshot.Count; i++) + { + var ps = snapshot[i]; + tasks[i] = Task.Run(() => DrainOneSync(ps, timeout, ct)); + } + + try + { + Task.WaitAll(tasks); + } + catch (AggregateException ex) + { + foreach (var inner in ex.InnerExceptions) + { + if (inner is OperationCanceledException oce) + { + throw oce; + } + } + + throw; + } + + var ok = true; + foreach (var t in tasks) + { + ok &= t.Result; + } + + return ok; + } + + /// + internal bool Flush(CancellationToken ct = default) => Flush(_defaultFlushTimeout, ct); + + /// + internal async ValueTask FlushAsync(TimeSpan timeout, CancellationToken ct = default) + { + ThrowIfClosed(); + var snapshot = SnapshotAll(); + if (snapshot.Count == 0) + { + return true; + } + + // Drain the senders concurrently — they are independent connections, so this is safe and turns the + // barrier's wall-clock cost into the slowest single drain rather than their sum. + var tasks = new List>(snapshot.Count); + foreach (var ps in snapshot) + { + tasks.Add(DrainOneAsync(ps, timeout, ct)); + } + + var results = await Task.WhenAll(tasks).ConfigureAwait(false); + var ok = true; + foreach (var r in results) + { + ok &= r; + } + + return ok; + } + + /// + internal ValueTask FlushAsync(CancellationToken ct = default) => FlushAsync(_defaultFlushTimeout, ct); + + private static async Task DrainOneAsync(PooledSender ps, TimeSpan timeout, CancellationToken ct) + { + try + { + return await ps.Inner.FlushAsync(timeout, ct).ConfigureAwait(false); + } + catch (OperationCanceledException) + { + throw; + } + catch + { + return false; + } + } + + private static bool DrainOneSync(PooledSender ps, TimeSpan timeout, CancellationToken ct) + { + try + { + return ps.Inner.Flush(timeout, ct); + } + catch (OperationCanceledException) + { + throw; + } + catch + { + // One sender's terminal error must not abort draining the rest; report it as not-drained. + return false; + } + } + + private List SnapshotAll() + { + lock (_gate) + { + return new List(_all); + } + } + + // Permit already held. Reuse an idle entry or create a fresh one outside the lock. + private PooledSender TakeOrCreate() + { + int slotIndex; + lock (_gate) + { + if (_closed) + { + ReleaseCapacity(); + throw Closed(); + } + + if (_idle.Count > 0) + { + var ps = _idle[^1]; + _idle.RemoveAt(_idle.Count - 1); + return ps; + } + + slotIndex = AllocateSlotIndex(); + if (_storeAndForward && slotIndex < 0) + { + // Defence in depth: every retire/reap withholds its permit physically (reap skips when it + // can't — see TryWithholdPermit), so a held permit always implies an idle sender or a free + // index. Surface the documented exhaustion error rather than a factory failure. + ReleaseCapacity(); + throw Exhausted(); + } + } + + PooledSender created; + try + { + created = CreateSender(slotIndex); + } + catch + { + lock (_gate) + { + FreeSlotIndex(slotIndex); + } + + ReleaseCapacity(); + throw; + } + + lock (_gate) + { + if (_closed) + { + FreeSlotIndex(slotIndex); + try + { + created.DisposeInner(); + } + catch + { + // best effort + } + + ReleaseCapacity(); + throw Closed(); + } + + _all.Add(created); + } + + return created; + } + + /// Returns a borrowed sender to the pool (after Dispose discarded its un-sent rows). + internal void GiveBack(PooledSender ps) + { + if (GiveBackOrTakeOwnership(ps)) + { + try + { + ps.DisposeInner(); + } + catch + { + // best effort + } + } + + ReleaseCapacity(); + } + + /// + internal async ValueTask GiveBackAsync(PooledSender ps) + { + if (GiveBackOrTakeOwnership(ps)) + { + try + { + await ps.DisposeInnerAsync().ConfigureAwait(false); + } + catch + { + // best effort + } + } + + ReleaseCapacity(); + } + + // Re-pool an idle-again sender, or — if the pool closed while it was on loan — hand teardown back to + // the caller. Close() only disposes idle senders, never an in-use one (its borrower may be mid-clear in + // Dispose on a non-thread-safe inner), so a sender returning post-close is disposed here by its + // borrower, on the borrower's own stack after Dispose finished. Exactly one party ever disposes a given + // inner. Returns true when the caller must dispose the inner. + private bool GiveBackOrTakeOwnership(PooledSender ps) + { + lock (_gate) + { + if (_closed) + { + return true; + } + + ps.IdleSinceUtc = DateTime.UtcNow; + _idle.Add(ps); + if (ps.CreatedAtUtc < _idleOldestCreatedUtc) + { + _idleOldestCreatedUtc = ps.CreatedAtUtc; + } + + return false; + } + } + + /// Evicts a sender that can't be re-pooled (terminally failed, or holding un-rollback-able + /// transactional state): dispose it for real, then reclaim or retire its slot. + internal void DiscardBroken(PooledSender ps) + { + lock (_gate) + { + _all.Remove(ps); + } + + try + { + ps.DisposeInner(); + } + catch + { + // best effort + } + + FinishDiscard(ps); + } + + /// + internal async ValueTask DiscardBrokenAsync(PooledSender ps) + { + lock (_gate) + { + _all.Remove(ps); + } + + try + { + await ps.DisposeInnerAsync().ConfigureAwait(false); + } + catch + { + // best effort + } + + FinishDiscard(ps); + } + + // A discarded sender held a capacity permit. If its slot lock dropped, free the index and release + // the permit. Otherwise retire the index and keep the permit withheld — simply by not releasing it — + // so effective max shrinks by one. Mirrors the Java pool's leaked-slot accounting. The shrink is + // reversible: the housekeeper re-tests retired slots and reclaims the index + permit once the + // (possibly deferred) lock release lands. + private void FinishDiscard(PooledSender ps) + { + if (SettleAfterDispose(ps)) + { + ReleaseCapacity(); + } + } + + /// Reaps idle / over-age senders down to min. Driven by the housekeeper. The idle + /// deque is sorted by idle time, so the idle sweep only inspects the cold end and stops at the + /// first entry inside its timeout — O(reaped), not O(idle). max_lifetime is age-ordered, not + /// idle-ordered, so it runs as a separate walk gated on _idleOldestCreatedUtc, which + /// fires only when some parked entry has actually crossed the lifetime. + internal void ReapIdle() + { + var now = DateTime.UtcNow; + ReapOverAge(now); + + // Bound the sweep to the entries present at its start: an un-drained cold entry is re-parked at + // the hot end below, and without the bound a deque full of un-drained entries would be revisited + // forever. Entries returned mid-sweep are fresh and land at the hot end — next sweep's problem. + int budget; + lock (_gate) + { + budget = _closed ? 0 : _idle.Count; + } + + while (budget-- > 0) + { + PooledSender victim; + lock (_gate) + { + if (_closed || _idle.Count == 0) + { + return; + } + + var ps = _idle[0]; + // The cold end is the longest-idle entry: if it is inside its timeout, everything + // behind it is too. The min floor is global, so it ends the sweep as well. + if (now - ps.IdleSinceUtc < _idleTimeout || _all.Count <= _min) + { + return; + } + + // Don't reap until every in-flight frame is acked. Reaping a WS sender whose ring still + // holds un-acked data would, in RAM mode, free that data with no delivery and no error + // (the pool-wide Flush can't cover an already-reaped sender). Re-park it at the hot end + // with a fresh IdleSinceUtc so the idle clock effectively restarts (and the reap timer + // starts at full-drain); a wedged sender that never drains simply lives until the pool + // is closed (its data is retained, not silently dropped). No-op for HTTP/TCP, which + // deliver synchronously. + if (!ps.IsInnerFullyDrained) + { + _idle.RemoveAt(0); + ps.IdleSinceUtc = now; + _idle.Add(ps); + continue; + } + + // Withhold one capacity permit for the whole dispose window — atomically, BEFORE the + // sender leaves the pool. An idle sender holds no permit (its unit of capacity sits + // free in the semaphore), so taking it here guarantees the semaphore never advertises + // capacity whose slot index is still locked mid-dispose — the spurious-PoolExhausted + // race. If a mid-borrow thread already drained the last free permit, stop the sweep and + // leave the deque untouched: that borrower is about to reuse this very sender, and with + // zero free permits every other idle entry is equally spoken for. IdleSinceUtc is not + // refreshed, so a skipped sender stays reap-eligible next sweep. + if (!TryWithholdPermit()) + { + return; + } + + _idle.RemoveAt(0); + _all.Remove(ps); + victim = ps; + } + + DisposeAndSettle(victim); + } + } + + // max_lifetime pass. The deque is IdleSinceUtc-ordered, so an entry that crosses max_lifetime while + // parked can hide behind younger-created but colder entries where the cold-end sweep never sees it. + // Gated on the conservative _idleOldestCreatedUtc bound and retightens it while walking, so the + // walk only runs when some parked entry has actually crossed — or while an over-age entry survives + // it (un-drained, min floor, withhold loss) and must be re-checked next sweep. + private void ReapOverAge(DateTime now) + { + List? victims = null; + lock (_gate) + { + if (_closed || now - _idleOldestCreatedUtc < _maxLifetime) + { + return; + } + + var oldestKept = DateTime.MaxValue; + var reapable = true; + for (var i = 0; i < _idle.Count;) + { + var ps = _idle[i]; + if (reapable && now - ps.CreatedAtUtc >= _maxLifetime && ps.IsInnerFullyDrained) + { + if (_all.Count <= _min || !TryWithholdPermit()) + { + // Floor reached / a mid-borrow thread owns the free permits: stop reaping but + // keep walking to retighten the bound (survivors keep it low, so we re-check). + reapable = false; + } + else + { + _idle.RemoveAt(i); + _all.Remove(ps); + (victims ??= new List()).Add(ps); + continue; + } + } + + if (ps.CreatedAtUtc < oldestKept) + { + oldestKept = ps.CreatedAtUtc; + } + + i++; + } + + _idleOldestCreatedUtc = oldestKept; + } + + if (victims is null) + { + return; + } + + foreach (var ps in victims) + { + DisposeAndSettle(ps); + } + } + + // Disposes a reaped sender outside the pool lock, then frees the slot and returns the withheld + // permit — but only if the lock actually released; a new sender must never open a slot directory + // whose flock is still held. If a deferred teardown is still holding it, SettleAfterDispose parks + // the entry on _retired (permit still withheld) for ReclaimRetiredSlots to settle when the release + // finally lands. + private void DisposeAndSettle(PooledSender ps) + { + try + { + ps.DisposeInner(); + } + catch + { + // best effort; reaping must never throw + } + + if (SettleAfterDispose(ps)) + { + ReleaseCapacity(); + } + } + + /// + /// Re-tests slots retired by a deferred / wedged teardown and reclaims any whose slot lock has + /// since released — freeing the index and returning the withheld capacity permit. Driven by the + /// housekeeper. Turns the slot retirement of a transient teardown stall into a temporary capacity + /// dip rather than a permanent shrink toward PoolExhausted. + /// + internal void ReclaimRetiredSlots() + { + if (!_storeAndForward) + { + return; + } + + var releaseCount = 0; + lock (_gate) + { + if (_closed || _retired.Count == 0) + { + return; + } + + for (var i = _retired.Count - 1; i >= 0; i--) + { + var ps = _retired[i]; + if (ps.IsInnerSlotLockReleased) + { + _retired.RemoveAt(i); + FreeSlotIndex(ps.SlotIndex); + releaseCount++; + } + } + } + + ReleaseReclaimedPermits(releaseCount); + } + + /// Shuts the pool down, closing every idle underlying sender. Idempotent. Senders currently + /// borrowed are torn down by their borrower on return (never disposed here, to avoid racing a + /// non-thread-safe in-use sender) — return all borrowed senders before closing the handle. + internal void Close() + { + var snapshot = BeginClose(); + if (snapshot is null) + { + return; + } + + foreach (var ps in snapshot) + { + try + { + ps.DisposeInner(); + } + catch + { + // best effort + } + } + + DisposePrimitives(); + } + + /// + internal async ValueTask CloseAsync() + { + var snapshot = BeginClose(); + if (snapshot is null) + { + return; + } + + foreach (var ps in snapshot) + { + try + { + await ps.DisposeInnerAsync().ConfigureAwait(false); + } + catch + { + // best effort + } + } + + DisposePrimitives(); + } + + // ---- internals ---- + + private List? BeginClose() + { + List idle; + lock (_gate) + { + if (_closed) + { + return null; + } + + _closed = true; + + // Tear down only the idle senders here. A borrowed (in-use) sender is non-thread-safe and may + // be mid-clear inside its borrower's Dispose(); disposing it concurrently would corrupt its + // buffer / transport. Its owning borrower disposes it instead when it returns post-close (see + // GiveBackOrTakeOwnership / DiscardBroken), so each inner is torn down by exactly one party. + // Callers wanting a hard "all I/O has stopped" guarantee must return every borrowed sender + // before closing the handle. + idle = new List(_idle); + _all.Clear(); + _idle.Clear(); + // Drop references to retired-but-unreclaimed senders; their OS locks release independently + // (deferred continuation / process exit) and the pool no longer needs to track them. + _retired.Clear(); + } + + try + { + _closeCts.Cancel(); + } + catch (ObjectDisposedException) + { + // already torn down + } + + return idle; + } + + private void DisposePrimitives() + { + _closeCts.Dispose(); + _capacity.Dispose(); + } + + private PooledSender CreateSender(int slotIndex) + { + var inner = _senderFactory(slotIndex); + return new PooledSender(inner, slotIndex); + } + + private ISender CreateDefaultInner(int slotIndex) + { + if (_confStr is null) + { + throw new IngressError(ErrorCode.InvalidApiCall, + "SenderPool has no connect string and no sender factory"); + } + + // Each sender gets independent options parsed from the original connect string. + var options = new SenderOptions(_confStr); + // Pooled senders don't drain on Send() — the pooled connection survives the borrow's return and + // ships asynchronously; delivery is confirmed via the pool-wide IQuestDBClient.Flush(). (Standalone + // senders keep SendAwaitsAck=true so "Send() before Dispose" delivers.) + options.SendAwaitsAck = false; + // lazy_connect: the ingest side must not block the pre-warm on a down server. The builder has + // already rejected an explicit non-async initial_connect_retry, so forcing async here is either a + // no-op (already async) or the intended injection (unset / reconnect-promoted). ws only — the flag + // is off for http/tcp handles. + if (_forceWsAsyncConnect && options.IsWebSocket()) + { + options.initial_connect_mode = InitialConnectMode.async; + } + if (_storeAndForward) + { + if (slotIndex < 0) + { + throw new IngressError(ErrorCode.InvalidApiCall, + "no free store-and-forward slot index (all slots leaked?)"); + } + + ApplySlotIdentity(options, _slotBaseId, slotIndex, _max); + } + + return Sender.New(options); + } + + /// + /// Stamps a per-slot identity onto so pooled WS+SF senders never + /// collide: a unique sender_id and the managed-slot family for the orphan scanner. + /// + internal static void ApplySlotIdentity(SenderOptions options, string baseId, int slotIndex, int managedCount) + { + options.sender_id = $"{baseId}-{slotIndex}"; + options.OrphanExcludeManagedBase = baseId; + options.OrphanExcludeManagedCount = managedCount; + } + + // Next free index in [0, max), LIFO: the most recently freed index is reused first, keeping the + // on-disk working set of slot directories compact. Caller holds _gate (or is the single-threaded + // pre-warm path). + private int AllocateSlotIndex() + { + if (!_storeAndForward) + { + return -1; + } + + return _freeSlots.Count > 0 ? _freeSlots.Pop() : -1; + } + + private void FreeSlotIndex(int idx) + { + if (!_storeAndForward || idx < 0) + { + return; + } + + _freeSlots.Push(idx); + } + + // Non-blocking permit withhold used by the reap path: a reaped idle sender holds no permit, so the + // reaper takes one out of the free pool before removing the sender, keeping the semaphore in + // lock-step with allocatable slots for the whole dispose window. Returns false when no free permit + // exists (a mid-borrow thread drained it) — the caller must then skip the reap. Caller holds _gate; + // Wait(0) never blocks so holding the lock is safe. + private bool TryWithholdPermit() + { + try + { + return _capacity.Wait(0); + } + catch (ObjectDisposedException) + { + return false; + } + } + + // Post-dispose slot settlement shared by the discard and reap paths — both arrive holding exactly + // one withheld capacity permit for ps. Returns true (caller releases the permit) when the slot lock + // actually dropped and the index was freed; returns false after parking the entry on _retired with + // the permit still withheld, for ReclaimRetiredSlots to settle once the deferred teardown finally + // releases the lock. A new sender must never open a slot directory whose flock is still held. + private bool SettleAfterDispose(PooledSender ps) + { + lock (_gate) + { + if (!_closed && ps.SlotIndex >= 0 && !ps.IsInnerSlotLockReleased) + { + _retired.Add(ps); + // The index stays off _freeSlots: the directory / flock may still be held. + return false; + } + + FreeSlotIndex(ps.SlotIndex); + return true; + } + } + + // Releases reclaimed capacity permits back to borrowers, outside the pool lock. Tolerates a concurrent + // close (disposed / full semaphore) — the accounting is best-effort once the pool is tearing down. + private void ReleaseReclaimedPermits(int count) + { + for (var i = 0; i < count; i++) + { + try + { + _capacity.Release(); + } + catch (ObjectDisposedException) + { + break; + } + catch (SemaphoreFullException) + { + break; + } + } + } + + private void ReleaseCapacity() + { + try + { + _capacity.Release(); + } + catch (ObjectDisposedException) + { + // pool closed concurrently + } + catch (SemaphoreFullException) + { + // defensive: never expected, the borrow/return accounting is 1:1 + } + } + + private void ThrowIfClosed() + { + if (IsClosed) + { + throw Closed(); + } + } + + private static IngressError Closed() => + new(ErrorCode.InvalidApiCall, "QuestDBClient handle is closed"); + + private IngressError Exhausted() => + new(ErrorCode.PoolExhausted, + $"timed out waiting for a sender from the pool after {_acquireTimeoutMs}ms " + + $"(sender_pool_max={_max})"); +} diff --git a/src/net-questdb-client/Query.cs b/src/net-questdb-client/Query.cs new file mode 100644 index 0000000..6928163 --- /dev/null +++ b/src/net-questdb-client/Query.cs @@ -0,0 +1,195 @@ +/******************************************************************************* + * ___ _ ____ ____ + * / _ \ _ _ ___ ___| |_| _ \| __ ) + * | | | | | | |/ _ \/ __| __| | | | _ \ + * | |_| | |_| | __/\__ \ |_| |_| | |_) | + * \__\_\\__,_|\___||___/\__|____/|____/ + * + * Copyright (c) 2014-2019 Appsicle + * Copyright (c) 2019-2026 QuestDB + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + ******************************************************************************/ + +#if NET7_0_OR_GREATER +using QuestDB.Enums; +using QuestDB.Pooling; +using QuestDB.Utils; +// Alias the QWP types so this file does not `using QuestDB.Qwp.Query;` — that namespace shares its +// last segment with this `QuestDB.Query` type and the import would create a type/namespace ambiguity. +using QwpBindSetter = QuestDB.Qwp.Query.QwpBindSetter; +using QwpColumnBatchHandler = QuestDB.Qwp.Query.QwpColumnBatchHandler; + +namespace QuestDB; + +/// +/// Reusable builder for one egress query, obtained from . +/// Configure with , optional , and , +/// then call (or ). Each execution borrows a +/// query client from the handle's pool, runs the query, and returns the client. +/// +/// One in-flight execution per Query instance (single-flight). For concurrent queries, +/// allocate a fresh per query. +/// +/// Cancellation: is the cooperative path (posts a CANCEL frame, the +/// query ends normally and the client is re-pooled). Passing a to +/// and cancelling it is a hard cancel that tears the connection down — +/// the client is then discarded, not re-pooled. +/// +public sealed class Query +{ + private readonly QueryClientPool _pool; + private string? _sql; + private QwpBindSetter? _binds; + private QwpColumnBatchHandler? _handler; + + // 0 idle, 1 while an ExecuteAsync is in flight (single-flight guard). + private int _inFlight; + + // The client leased for the duration of the in-flight ExecuteAsync, so Cancel() can reach it. + // Cleared before the client is returned to the pool so a late Cancel() can't hit a client now + // serving another query. + private volatile PooledQueryClient? _inFlightLease; + + // Excludes Cancel()'s lease-read + request-id resolution against the lease-clear in + // ExecuteAsync's finally: a lease seen non-null under this gate cannot have been returned to + // the pool yet, so the request id resolved is this execution's own (or -1), never a successor + // borrower's. Held only for those field reads — never across the cancel send. + private readonly object _cancelGate = new(); + + internal Query(QueryClientPool pool) + { + _pool = pool; + } + + /// Sets the SQL text. Returns this for chaining. + public Query Sql(string sql) + { + _sql = sql; + return this; + } + + /// Sets the bind-value setter (optional). Returns this for chaining. + public Query Binds(QwpBindSetter binds) + { + _binds = binds; + return this; + } + + /// Sets the result-batch handler. Returns this for chaining. + public Query Handler(QwpColumnBatchHandler handler) + { + _handler = handler; + return this; + } + + /// + /// Borrows a pooled query client, runs the configured query, and returns the client to the + /// pool on clean completion (or discards it on any failure / hard cancellation). + /// + /// + /// for missing sql/handler, an overlapping execution, + /// or a closed handle; on acquire timeout. + /// + public async Task ExecuteAsync(CancellationToken ct = default) + { + var sql = _sql; + var handler = _handler; + var binds = _binds; + + if (string.IsNullOrEmpty(sql)) + { + throw new IngressError(ErrorCode.InvalidApiCall, "sql is required; call Sql(...) before Execute"); + } + + if (handler is null) + { + throw new IngressError(ErrorCode.InvalidApiCall, "handler is required; call Handler(...) before Execute"); + } + + if (Interlocked.Exchange(ref _inFlight, 1) != 0) + { + throw new IngressError(ErrorCode.InvalidApiCall, + "a previous Execute() is still in flight on this Query; use NewQuery() for concurrent queries"); + } + + try + { + var pooled = await _pool.BorrowAsync(ct).ConfigureAwait(false); + _inFlightLease = pooled; + try + { + if (binds is null) + { + await pooled.ExecuteAsync(sql, handler, ct).ConfigureAwait(false); + } + else + { + await pooled.ExecuteAsync(sql, binds, handler, ct).ConfigureAwait(false); + } + } + catch + { + // Any throw (transport/protocol, or a hard CT cancel) leaves the client terminal. + pooled.MarkBroken(); + throw; + } + finally + { + // Clear the lease BEFORE returning the client so a late Cancel() can't reach it. + lock (_cancelGate) + { + _inFlightLease = null; + } + + await pooled.DisposeAsync().ConfigureAwait(false); + } + } + finally + { + Interlocked.Exchange(ref _inFlight, 0); + } + } + + /// Synchronous wrapper over . + public void Execute() + { + // Threadpool hop drops any captured SyncContext so sync-over-async can't deadlock. + Task.Run(() => ExecuteAsync(CancellationToken.None)).GetAwaiter().GetResult(); + } + + /// + /// Cooperatively cancels the in-flight query (posts a CANCEL frame). The query ends with + /// a cancelled or normal terminator and the client is re-pooled. No-op when no query is in + /// flight. + /// + public void Cancel() + { + PooledQueryClient? lease; + long requestId; + lock (_cancelGate) + { + lease = _inFlightLease; + if (lease is null) return; + requestId = lease.CurrentRequestId; + } + + if (requestId < 0) return; + // Dispatched outside the gate: the CANCEL is addressed to this exact request id (ids are + // unique per client), so even if the client is returned and re-borrowed before the frame is + // sent, the client drops it rather than cancelling the new borrower's query. + lease.CancelRequest(requestId); + } +} +#endif diff --git a/src/net-questdb-client/QuestDBClient.cs b/src/net-questdb-client/QuestDBClient.cs new file mode 100644 index 0000000..0c6f9ad --- /dev/null +++ b/src/net-questdb-client/QuestDBClient.cs @@ -0,0 +1,63 @@ +/******************************************************************************* + * ___ _ ____ ____ + * / _ \ _ _ ___ ___| |_| _ \| __ ) + * | | | | | | |/ _ \/ __| __| | | | _ \ + * | |_| | |_| | __/\__ \ |_| |_| | |_) | + * \__\_\\__,_|\___||___/\__|____/|____/ + * + * Copyright (c) 2014-2019 Appsicle + * Copyright (c) 2019-2026 QuestDB + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + ******************************************************************************/ + +namespace QuestDB; + +/// +/// Factory for , a sender pool you construct once and share across +/// threads. Mirrors the / factory pairing. +/// +/// Use when the connect string carries everything, or +/// to set pool sizes / timeouts programmatically. +/// +public static class QuestDBClient +{ + /// + /// Connects with a single connect string (any ingest scheme: http, https, + /// tcp, tcps, ws, wss). Pool knobs may be embedded in the string + /// (sender_pool_max, acquire_timeout_ms, …). + /// + public static IQuestDBClient Connect(string confStr) + { + return Builder().FromConfig(confStr).Build(); + } + +#if NET7_0_OR_GREATER + /// + /// Connects with distinct ingest and query (egress) connect strings, e.g. an http/tcp + /// ingest endpoint plus a ws/wss query endpoint. The query string must be + /// ws/wss. net7.0+ only. + /// + public static IQuestDBClient Connect(string ingestConfStr, string queryConfStr) + { + return Builder().IngestConfig(ingestConfStr).QueryConfig(queryConfStr).Build(); + } +#endif + + /// Opens a builder for programmatic pool configuration. + public static QuestDBClientBuilder Builder() + { + return new QuestDBClientBuilder(); + } +} diff --git a/src/net-questdb-client/QuestDBClientBuilder.cs b/src/net-questdb-client/QuestDBClientBuilder.cs new file mode 100644 index 0000000..bb03b1f --- /dev/null +++ b/src/net-questdb-client/QuestDBClientBuilder.cs @@ -0,0 +1,346 @@ +/******************************************************************************* + * ___ _ ____ ____ + * / _ \ _ _ ___ ___| |_| _ \| __ ) + * | | | | | | |/ _ \/ __| __| | | | _ \ + * | |_| | |_| | __/\__ \ |_| |_| | |_) | + * \__\_\\__,_|\___||___/\__|____/|____/ + * + * Copyright (c) 2014-2019 Appsicle + * Copyright (c) 2019-2026 QuestDB + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + ******************************************************************************/ + +using QuestDB.Enums; +using QuestDB.Pooling; +using QuestDB.Utils; + +namespace QuestDB; + +/// +/// Fluent builder for . A pool knob set here always wins over the +/// same key embedded in the connect string. +/// +public sealed class QuestDBClientBuilder +{ + private TimeSpan? _acquireTimeout; + private string? _confStr; + private TimeSpan? _housekeeperInterval; + private TimeSpan? _idleTimeout; + private bool? _lazyConnect; + private TimeSpan? _maxLifetime; + private int? _poolMax; + private int? _poolMin; +#if NET7_0_OR_GREATER + private string? _queryConfStr; + private int? _queryPoolMax; + private int? _queryPoolMin; +#endif + + /// + /// Sets the ingest connect string. Any ingest scheme; if it is ws/wss and no + /// separate is given, the same string also configures the query pool. + /// + public QuestDBClientBuilder FromConfig(string confStr) + { + _confStr = confStr; + return this; + } + + /// Alias of . + public QuestDBClientBuilder IngestConfig(string confStr) + { + _confStr = confStr; + return this; + } + +#if NET7_0_OR_GREATER + /// + /// Sets the query (egress) connect string for the query-client pool. Must be ws/wss. + /// Use this when ingest and query endpoints differ (e.g. an http/tcp ingest handle + /// that also needs to query). net7.0+ only. + /// + public QuestDBClientBuilder QueryConfig(string confStr) + { + ArgumentNullException.ThrowIfNull(confStr); + if (!IsWebSocketScheme(confStr)) + { + throw new IngressError(ErrorCode.ConfigError, + "query configuration must use the ws or wss scheme"); + } + + _queryConfStr = confStr; + return this; + } + + /// Minimum warm query clients. + public QuestDBClientBuilder QueryPoolMin(int min) + { + _queryPoolMin = min; + return this; + } + + /// Maximum query clients. + public QuestDBClientBuilder QueryPoolMax(int max) + { + _queryPoolMax = max; + return this; + } + + /// Fixes the query pool to exactly clients (min == max). + public QuestDBClientBuilder QueryPoolSize(int size) + { + _queryPoolMin = size; + _queryPoolMax = size; + return this; + } +#endif + + /// Minimum warm senders. + public QuestDBClientBuilder SenderPoolMin(int min) + { + _poolMin = min; + return this; + } + + /// Maximum senders. + public QuestDBClientBuilder SenderPoolMax(int max) + { + _poolMax = max; + return this; + } + + /// Fixes the pool to exactly senders (min == max). + public QuestDBClientBuilder SenderPoolSize(int size) + { + _poolMin = size; + _poolMax = size; + return this; + } + + /// How long a borrow blocks before throwing. + public QuestDBClientBuilder AcquireTimeout(TimeSpan timeout) + { + _acquireTimeout = timeout; + return this; + } + + /// Idle duration before the housekeeper reaps a sender. + public QuestDBClientBuilder IdleTimeout(TimeSpan timeout) + { + _idleTimeout = timeout; + return this; + } + + /// Maximum age before the housekeeper recycles a sender. + public QuestDBClientBuilder MaxLifetime(TimeSpan lifetime) + { + _maxLifetime = lifetime; + return this; + } + + /// Housekeeper sweep interval. + public QuestDBClientBuilder HousekeeperInterval(TimeSpan interval) + { + _housekeeperInterval = interval; + return this; + } + + /// + /// Tolerant startup: build the handle even while the server is down. The ingest side connects + /// asynchronously (buffering writes meanwhile) and the read pool defaults to + /// 0 so nothing connects eagerly — a query connects lazily on first + /// once the server is up. Set explicitly here wins over the + /// connect-string lazy_connect key. Rejects a conflicting blocking-startup knob (an explicit + /// initial_connect_retry other than async, or an explicit query_pool_min > 0). + /// + public QuestDBClientBuilder LazyConnect(bool enabled = true) + { + _lazyConnect = enabled; + return this; + } + + /// Builds and pre-warms the pool. + public IQuestDBClient Build() + { + var poolConfig = BuildPoolConfig(out var queryConfStr, out var forceWsAsyncConnect); + return new QuestDBClientImpl(poolConfig, _confStr!, queryConfStr, forceWsAsyncConnect); + } + + // Assembles the effective pool config without constructing any pool — split out so tests can + // assert precedence without pre-warming live connections. + internal SenderOptions BuildPoolConfig(out string? queryConfStr, out bool forceWsAsyncConnect) + { + if (_confStr is null) + { + throw new IngressError(ErrorCode.ConfigError, + "ingest configuration is required; call FromConfig() or IngestConfig()"); + } + + var poolConfig = new SenderOptions(_confStr); + if (_poolMin.HasValue) + { + poolConfig.sender_pool_min = _poolMin.Value; + } + + if (_poolMax.HasValue) + { + poolConfig.sender_pool_max = _poolMax.Value; + } + + if (_acquireTimeout.HasValue) + { + poolConfig.acquire_timeout_ms = _acquireTimeout.Value; + } + + if (_idleTimeout.HasValue) + { + poolConfig.idle_timeout_ms = _idleTimeout.Value; + } + + if (_maxLifetime.HasValue) + { + poolConfig.max_lifetime_ms = _maxLifetime.Value; + } + + if (_housekeeperInterval.HasValue) + { + poolConfig.housekeeper_interval_ms = _housekeeperInterval.Value; + } + + poolConfig.ValidatePoolOptions(); + queryConfStr = null; + + // lazy_connect may be carried by the ingest config or the (separate) query config, and an + // explicit builder call wins over either. `queryPoolMinExplicit` tracks whether any source set + // query_pool_min at all (drives the default-to-0 decision); the conflict-rejection signal is + // derived from the effective post-precedence value inside ResolveLazyConnect, not accumulated here, + // so a higher-precedence source lowering the effective value back to 0 clears the conflict. + var configLazy = poolConfig.lazy_connect; + var queryPoolMinExplicit = poolConfig.IsQueryPoolMinExplicit; + var ingestIsWebSocket = IsWebSocketScheme(_confStr); + +#if NET7_0_OR_GREATER + // Resolve the query config: an explicit QueryConfig wins; otherwise a ws/wss ingest string + // serves both pools (Java parity). An http/tcp ingest handle with no QueryConfig has no query + // pool (NewQuery then throws a clear ConfigError). + queryConfStr = _queryConfStr; + if (queryConfStr is null && ingestIsWebSocket) + { + queryConfStr = _confStr; + } + + // Query-pool sizing precedence: explicit builder call > query-config value > ingest-config + // value > default. poolConfig (parsed from the ingest string) already carries the ingest value, + // so a query-config key overwrites it only when carried explicitly — a bare query config must + // not stomp explicit ingest-string sizing with defaults. + if (queryConfStr is not null && !ReferenceEquals(queryConfStr, _confStr)) + { + var queryOpts = new SenderOptions(queryConfStr); + if (queryOpts.IsQueryPoolMinExplicit) + { + poolConfig.query_pool_min = queryOpts.query_pool_min; + queryPoolMinExplicit = true; + } + + if (queryOpts.IsQueryPoolMaxExplicit) + { + poolConfig.query_pool_max = queryOpts.query_pool_max; + } + + configLazy = configLazy || queryOpts.lazy_connect; + } + + if (_queryPoolMin.HasValue) + { + poolConfig.query_pool_min = _queryPoolMin.Value; + queryPoolMinExplicit = true; + } + + if (_queryPoolMax.HasValue) + { + poolConfig.query_pool_max = _queryPoolMax.Value; + } +#endif + + var lazy = _lazyConnect ?? configLazy; + forceWsAsyncConnect = ResolveLazyConnect( + lazy, poolConfig, ingestIsWebSocket, queryPoolMinExplicit); + +#if NET7_0_OR_GREATER + if (queryConfStr is not null) + { + poolConfig.ValidateQueryPoolOptions(); + } +#endif + + return poolConfig; + } + + /// + /// Applies lazy_connect tolerant-startup semantics to the assembled pool config. + /// Returns whether pooled ws/wss senders must connect asynchronously so the + /// pre-warm does not fail-fast on a down server. When enabled it defaults + /// to 0 (lazy first-borrow reads) unless the user set + /// it, and throws () on a knob that + /// forces a blocking / fail-fast startup: an explicit initial_connect_retry other than + /// async, or an explicit query_pool_min > 0. + /// + /// is true when any source set query_pool_min at all + /// (drives the default-to-0 decision); the conflict rejection is driven by the effective + /// post-precedence value, so a higher-precedence source that overrides it back to 0 clears the + /// conflict. + /// + internal static bool ResolveLazyConnect( + bool lazy, SenderOptions poolConfig, bool ingestIsWebSocket, + bool queryPoolMinExplicit) + { + if (!lazy) + { + return false; + } + + if (poolConfig.IsInitialConnectModeExplicit + && poolConfig.initial_connect_mode != InitialConnectMode.async) + { + throw new IngressError(ErrorCode.ConfigError, + "`lazy_connect` requires a non-blocking startup, but `initial_connect_retry` is set to a " + + "blocking mode; drop `initial_connect_retry` or set it to `async` to use `lazy_connect`"); + } + + if (queryPoolMinExplicit && poolConfig.query_pool_min > 0) + { + throw new IngressError(ErrorCode.ConfigError, + "`lazy_connect` defers the read connect, but `query_pool_min` is set > 0 which pre-warms " + + "the read pool eagerly; drop `query_pool_min` or set it to 0 to use `lazy_connect`"); + } + + // Default the read pool to lazy first-borrow creation when the user left it unset. + if (!queryPoolMinExplicit) + { + poolConfig.query_pool_min = 0; + } + + // The ingest side goes async so Build() does not block on a down server; ws/wss only. + return ingestIsWebSocket; + } + + private static bool IsWebSocketScheme(string conf) + { + var idx = conf.IndexOf("::", StringComparison.Ordinal); + var scheme = idx < 0 ? conf : conf.Substring(0, idx); + return scheme.Equals("ws", StringComparison.OrdinalIgnoreCase) + || scheme.Equals("wss", StringComparison.OrdinalIgnoreCase); + } +} diff --git a/src/net-questdb-client/Qwp/Query/QueryOptions.cs b/src/net-questdb-client/Qwp/Query/QueryOptions.cs index e364b0f..ca7d33a 100644 --- a/src/net-questdb-client/Qwp/Query/QueryOptions.cs +++ b/src/net-questdb-client/Qwp/Query/QueryOptions.cs @@ -22,6 +22,7 @@ * ******************************************************************************/ +using System.ComponentModel; using System.Data.Common; using System.Globalization; using QuestDB.Enums; @@ -187,9 +188,20 @@ public IReadOnlyList addresses public TimeSpan failover_backoff_max_ms { get; set; } = TimeSpan.FromMilliseconds(1000); /// Total wall-clock budget for the failover loop across all attempts; = unbounded. Whichever of or this fires first ends the loop. public TimeSpan failover_max_duration_ms { get; set; } = TimeSpan.FromSeconds(30); - /// Per-endpoint timeout applied to the WebSocket upgrade (TCP+TLS+HTTP+SERVER_INFO). Without this, an unreachable address can block on OS-level TCP timeouts (~21s Linux, ~75s macOS). + /// Supported for backward compatibility but not advertised; use instead. Per-endpoint timeout applied to the WebSocket upgrade (TCP+TLS+HTTP+SERVER_INFO) when is unset. Defaults to 15 seconds. + [EditorBrowsable(EditorBrowsableState.Never)] public TimeSpan auth_timeout_ms { get; set; } = TimeSpan.FromSeconds(15); + /// + /// Total wall-clock budget for bringing up the egress connection (TCP socket connect + TLS + + /// WebSocket upgrade). Aborts the attempt when exceeded. When unset, the effective budget is + /// 15 seconds. Read the value actually applied via . + /// + public TimeSpan? connect_timeout { get; set; } + + /// The connect budget actually applied: when set, otherwise the compatibility fallback. + internal TimeSpan EffectiveConnectTimeout => connect_timeout ?? auth_timeout_ms; + /// /// Client zone identifier (opaque, case-insensitive; e.g. eu-west-1a). When set with /// target=any or target=replica, prefers endpoints whose server-advertised @@ -246,7 +258,7 @@ private void Parse(string connStr) } var schemeText = connStr.Substring(0, sep); - if (!Enum.TryParse(schemeText, out ProtocolType scheme) + if (!Enum.TryParse(schemeText, ignoreCase: true, out ProtocolType scheme) || (scheme != ProtocolType.ws && scheme != ProtocolType.wss)) { throw new IngressError(ErrorCode.ConfigError, @@ -338,6 +350,10 @@ private void Parse(string connStr) ReadInt(builder, "failover_max_duration_ms", 30000)); auth_timeout_ms = TimeSpan.FromMilliseconds( ReadInt(builder, "auth_timeout_ms", 15000)); + if (builder.ContainsKey("connect_timeout")) + { + connect_timeout = TimeSpan.FromMilliseconds(ReadInt(builder, "connect_timeout", 15000)); + } zone = ReadString(builder, "zone"); if (builder.ContainsKey("max_batch_rows")) @@ -487,6 +503,21 @@ private void ValidateNumericRanges() "`auth_timeout_ms` must be positive"); } + if (connect_timeout is { } ct) + { + if (ct <= TimeSpan.Zero) + { + throw new IngressError(ErrorCode.ConfigError, + "`connect_timeout` must be positive"); + } + + if (ct.TotalMilliseconds > maxBackoffMillis) + { + throw new IngressError(ErrorCode.ConfigError, + $"`connect_timeout` must be <= {maxBackoffMillis}ms (~24.8 days)"); + } + } + if (max_batch_rows < 0) { throw new IngressError(ErrorCode.ConfigError, diff --git a/src/net-questdb-client/Qwp/Query/QwpQueryWebSocketClient.cs b/src/net-questdb-client/Qwp/Query/QwpQueryWebSocketClient.cs index c729589..dd3e7d1 100644 --- a/src/net-questdb-client/Qwp/Query/QwpQueryWebSocketClient.cs +++ b/src/net-questdb-client/Qwp/Query/QwpQueryWebSocketClient.cs @@ -34,7 +34,7 @@ namespace QuestDB.Qwp.Query; -internal sealed class QwpQueryWebSocketClient : IQwpQueryClient +internal sealed class QwpQueryWebSocketClient : IQwpQueryClient, QuestDB.Pooling.IPooledQueryClientInner { private static readonly UTF8Encoding StrictUtf8 = QwpConstants.StrictUtf8; private const int InitialReceiveBufferBytes = 64 * 1024; @@ -316,9 +316,10 @@ lastError is null candidate = BuildTransport(addr); QwpServerInfo? info = null; + var connectBudget = _options.EffectiveConnectTimeout; using (var upgradeCts = CancellationTokenSource.CreateLinkedTokenSource(ct)) { - upgradeCts.CancelAfter(_options.auth_timeout_ms); + upgradeCts.CancelAfter(connectBudget); try { await candidate.ConnectAsync(upgradeCts.Token).ConfigureAwait(false); @@ -326,7 +327,7 @@ lastError is null catch (OperationCanceledException) when (upgradeCts.IsCancellationRequested && !ct.IsCancellationRequested) { throw new IngressError(ErrorCode.SocketError, - $"WebSocket upgrade for {addr} exceeded auth_timeout={_options.auth_timeout_ms.TotalMilliseconds}ms"); + $"WebSocket upgrade for {addr} exceeded connect_timeout={connectBudget.TotalMilliseconds}ms"); } } @@ -404,10 +405,34 @@ public void Cancel() { var rid = Interlocked.Read(ref _currentRequestId); if (rid < 0) return; - // Record the exact rid being cancelled. If this thread is pre-empted between the read above - // and here while query `rid` finishes and the next query starts, the stale rid no longer - // matches the running requestId, so the failover-path checks ignore it. - Interlocked.Exchange(ref _cancelTargetRid, rid); + CancelCore(rid); + } + + long QuestDB.Pooling.IPooledQueryClientInner.CurrentRequestId => Interlocked.Read(ref _currentRequestId); + + void QuestDB.Pooling.IPooledQueryClientInner.CancelRequest(long requestId) + { + if (requestId < 0) return; + CancelCore(requestId); + } + + // Test seam: lets tests pin CancelCore's never-regress guarantee on the cancel marker. + internal long CancelTargetRid => Interlocked.Read(ref _cancelTargetRid); + + private void CancelCore(long rid) + { + // Record the exact rid being cancelled. If this thread is pre-empted between the caller's + // read and here while query `rid` finishes and the next query starts, the stale rid no + // longer matches the running requestId, so the failover-path checks ignore it. Never + // regress the marker: rids are monotonic, so a stale cancel arriving late must not + // overwrite a newer query's pending cancel. + long seen; + do + { + seen = Interlocked.Read(ref _cancelTargetRid); + if (rid < seen) break; + } while (Interlocked.CompareExchange(ref _cancelTargetRid, rid, seen) != seen); + if (Volatile.Read(ref _disposed) != 0) return; try @@ -1232,6 +1257,13 @@ private void ThrowIfTerminal() } private void MarkTerminal() => Volatile.Write(ref _terminal, 1); + + /// + /// Pool seam: true once the client is terminal or disposed, so a pooled wrapper can discard + /// rather than re-pool it even on a non-throwing return path. + /// + bool QuestDB.Pooling.IPooledQueryClientInner.IsTerminalOrDisposed => + Volatile.Read(ref _terminal) != 0 || Volatile.Read(ref _disposed) != 0; } #endif diff --git a/src/net-questdb-client/Qwp/QwpColumn.cs b/src/net-questdb-client/Qwp/QwpColumn.cs index 7eb890c..ea8495e 100644 --- a/src/net-questdb-client/Qwp/QwpColumn.cs +++ b/src/net-questdb-client/Qwp/QwpColumn.cs @@ -523,16 +523,6 @@ public void AppendDecimal256(decimal value) AppendDecimalAtSize(QwpTypeCode.Decimal256, value, QwpConstants.Decimal256SizeBytes); } - public void AppendDecimal64(long unscaledValue, byte scale) - { - ValidateDecimalScale(scale, QwpConstants.MaxDecimal64Scale, "Decimal64"); - AppendDecimalFromMantissa( - QwpTypeCode.Decimal64, - new BigInteger(unscaledValue), - scale, - QwpConstants.Decimal64SizeBytes); - } - public void AppendDecimal128(long lo, long hi, byte scale) { ValidateDecimalScale(scale, QwpConstants.MaxDecimal128Scale, "Decimal128"); @@ -587,9 +577,6 @@ private void AppendDecimalAtSize(QwpTypeCode code, decimal value, int sizeBytes) // Fast path: no rescale needed — either this is the first non-null write (which locks the // column scale to this value's scale) or the value's scale already matches the locked scale. // Serialise the 96-bit mantissa straight into FixedData as fixed-width two's-complement. - // BigInteger is a struct, but the slow path still allocates on the heap per value: its uint[] - // backing store (whenever the magnitude is past int range) plus the byte[] that ToByteArray - // returns. A scale mismatch falls back to that BigInteger rescale arithmetic below. if (!DecimalScaleSet || scale == DecimalScale) { AssertOrSetType(code); @@ -602,12 +589,8 @@ private void AppendDecimalAtSize(QwpTypeCode code, decimal value, int sizeBytes) return; } - var mantissa = (new BigInteger((uint)bits[2]) << 64) - | (new BigInteger((uint)bits[1]) << 32) - | new BigInteger((uint)bits[0]); - if (negative) mantissa = -mantissa; - - AppendDecimalFromMantissa(code, mantissa, scale, sizeBytes); + // Scale mismatch: rescale this BCL decimal to the locked column scale (see AppendDecimalRescaled). + AppendDecimalRescaled(code, value, sizeBytes); } // Writes a 96-bit magnitude (three little-endian 32-bit words) with the given sign into FixedData @@ -730,6 +713,68 @@ private void WriteFixedDecimalMantissa(BigInteger mantissa, byte targetScale, in AdvanceNonNull(); } + // System.Decimal tops out at a 96-bit mantissa / scale 28, so a BCL decimal whose locked column + // scale is within that range is rescaled entirely in decimal arithmetic — GC-free, no hand-rolled + // integer math — and serialised via the existing 96-bit two's-complement writer. It defers to the + // BigInteger path only when decimal can't hold the result at the target scale (a locked scale > 28, + // or a magnitude that needs the full 128-/256-bit field). The raw-limb long… APIs always use that + // path; for a BCL decimal it is the rare overflow tail. + private const int MaxBclDecimalScale = 28; + + private void AppendDecimalRescaled(QwpTypeCode code, decimal value, int sizeBytes) + { + var targetScale = DecimalScale; + Span bits = stackalloc int[4]; + decimal.GetBits(value, bits); + var sourceScale = (byte)((bits[3] >> 16) & 0xFF); + var negative = (bits[3] & unchecked((int)0x80000000)) != 0; + + if (targetScale <= MaxBclDecimalScale + && TryRescaleDecimal(value, sourceScale, targetScale, out var scaled)) + { + AssertOrSetType(code); + Span scaledBits = stackalloc int[4]; + decimal.GetBits(scaled, scaledBits); + var scaledNeg = (scaledBits[3] & unchecked((int)0x80000000)) != 0; + AppendDecimal96TwosComplement( + (uint)scaledBits[0], (uint)scaledBits[1], (uint)scaledBits[2], scaledNeg, sizeBytes); + return; + } + + var mantissa = (new BigInteger((uint)bits[2]) << 64) + | (new BigInteger((uint)bits[1]) << 32) + | new BigInteger((uint)bits[0]); + if (negative) mantissa = -mantissa; + AppendDecimalFromMantissa(code, mantissa, sourceScale, sizeBytes); + } + + // Re-expresses `value` at exactly `targetScale` using System.Decimal arithmetic. Up-scaling pads the + // mantissa by adding a zero at the target scale (decimal addition raises the result to the larger + // operand scale); down-scaling first rounds and only succeeds when no digit is lost. Returns false — + // so the caller falls back to BigInteger — when decimal can't hold the result at that scale, which it + // signals by leaving the result scale short of targetScale (it silently keeps a lower scale rather + // than overflow the 96-bit mantissa). + private static bool TryRescaleDecimal(decimal value, byte sourceScale, byte targetScale, out decimal scaled) + { + var aligned = value; + if (sourceScale > targetScale) + { + var rounded = Math.Round(value, targetScale, MidpointRounding.ToEven); + if (rounded != value) + { + scaled = default; + return false; + } + + aligned = rounded; + } + + scaled = aligned + new decimal(0, 0, 0, false, targetScale); + Span bits = stackalloc int[4]; + decimal.GetBits(scaled, bits); + return ((bits[3] >> 16) & 0xFF) == targetScale; + } + /// Appends a DECIMAL64 value coerced to an explicit scale (see ). public void AppendDecimal64(decimal value, byte scale) => AppendDecimalScaled(QwpTypeCode.Decimal64, value, scale, QwpConstants.Decimal64SizeBytes); @@ -791,12 +836,7 @@ private void AppendDecimalScaled(QwpTypeCode code, decimal value, byte targetSca return; } - var mantissa = (new BigInteger((uint)bits[2]) << 64) - | (new BigInteger((uint)bits[1]) << 32) - | new BigInteger((uint)bits[0]); - if (negative) mantissa = -mantissa; - mantissa *= BigInteger.Pow(10, targetScale - rScale); - WriteFixedDecimalMantissa(mantissa, targetScale, sizeBytes); + AppendDecimalRescaled(code, rounded, sizeBytes); } /// Appends a BINARY value as length-prefixed opaque bytes (same wire layout as VARCHAR). diff --git a/src/net-questdb-client/Qwp/QwpConnectStringKeys.cs b/src/net-questdb-client/Qwp/QwpConnectStringKeys.cs index b865491..6050000 100644 --- a/src/net-questdb-client/Qwp/QwpConnectStringKeys.cs +++ b/src/net-questdb-client/Qwp/QwpConnectStringKeys.cs @@ -43,8 +43,15 @@ internal static class QwpConnectStringKeys { "addr", "protocol", "tls_verify", "tls_roots", "tls_roots_password", "username", "user", "password", "pass", "token", - "auth_timeout_ms", + "auth_timeout_ms", "connect_timeout", "zone", "error_inbox_capacity", + // Connection-pool knobs read by the QuestDBClient handle. Protocol-agnostic: accepted + // (and ignored) by a plain Sender on every scheme so a pool connect string also builds + // a sender without tripping the unknown-key check. + "sender_pool_min", "sender_pool_max", + "query_pool_min", "query_pool_max", + "acquire_timeout_ms", "idle_timeout_ms", "max_lifetime_ms", "housekeeper_interval_ms", + "lazy_connect", }; /// diff --git a/src/net-questdb-client/Qwp/QwpConstants.cs b/src/net-questdb-client/Qwp/QwpConstants.cs index 7a5dbad..e1a9977 100644 --- a/src/net-questdb-client/Qwp/QwpConstants.cs +++ b/src/net-questdb-client/Qwp/QwpConstants.cs @@ -185,9 +185,14 @@ internal static class QwpConstants public const int MaxConnSymbolDictEntries = 8 * 1024 * 1024; public const int MaxConnSymbolDictHeapBytes = 256 * 1024 * 1024; - /// Inclusive zstd compression level range. Server clamps anything higher to 9 silently. + /// + /// Inclusive zstd compression level range accepted in the connect string. zstd defines levels + /// 1–22; the QuestDB server clamps anything above 9 to 9 silently. We accept the full 1–22 (like + /// the Java/Go/Rust clients) and pass the value through unchanged in the accept-encoding header, + /// so a single connect string stays portable across clients and the server does the clamping. + /// public const int ZstdLevelMin = 1; - public const int ZstdLevelMax = 9; + public const int ZstdLevelMax = 22; /// Egress upgrade headers. public const string HeaderAcceptEncoding = "X-QWP-Accept-Encoding"; diff --git a/src/net-questdb-client/Qwp/QwpTableBuffer.cs b/src/net-questdb-client/Qwp/QwpTableBuffer.cs index 74f306a..e8af303 100644 --- a/src/net-questdb-client/Qwp/QwpTableBuffer.cs +++ b/src/net-questdb-client/Qwp/QwpTableBuffer.cs @@ -265,12 +265,6 @@ public void AppendDecimal256(ReadOnlySpan columnName, decimal value, byte try { GetOrCreateColumn(columnName)?.AppendDecimal256(value, scale); } catch { CancelCurrentRow(); throw; } } - /// Append a DECIMAL64 value as the unscaled int64 with explicit scale. - public void AppendDecimal64(ReadOnlySpan columnName, long unscaledValue, byte scale) - { - try { GetOrCreateColumn(columnName)?.AppendDecimal64(unscaledValue, scale); } catch { CancelCurrentRow(); throw; } - } - /// Append a DECIMAL128 value: lo = unsigned low 64 bits, hi = signed high 64 bits. public void AppendDecimal128(ReadOnlySpan columnName, long lo, long hi, byte scale) { diff --git a/src/net-questdb-client/Qwp/Sf/QwpCursorSendEngine.cs b/src/net-questdb-client/Qwp/Sf/QwpCursorSendEngine.cs index a3981a4..388bed7 100644 --- a/src/net-questdb-client/Qwp/Sf/QwpCursorSendEngine.cs +++ b/src/net-questdb-client/Qwp/Sf/QwpCursorSendEngine.cs @@ -218,6 +218,23 @@ public long AckedFsn } } + /// + /// True when the ring holds no un-acked frames (AckedFsn >= NextFsn). The pool reads this + /// to hold off idle reaping until in-flight data has been delivered — tearing the engine down + /// while un-acked frames remain frees the ring (in RAM mode, dropping them with no error). A + /// disposed engine reports drained: there is nothing left to protect. + /// + public bool IsFullyDrained + { + get + { + lock (_stateLock) + { + return _disposed || _ackedFsn >= _ring.NextFsn; + } + } + } + public long TotalFramesSent => Volatile.Read(ref _totalFramesSent); public long TotalAcks => Volatile.Read(ref _totalAcks); public long TotalServerErrors => Volatile.Read(ref _totalServerErrors); diff --git a/src/net-questdb-client/Qwp/Sf/QwpOrphanScanner.cs b/src/net-questdb-client/Qwp/Sf/QwpOrphanScanner.cs index 70884a4..f69ca9c 100644 --- a/src/net-questdb-client/Qwp/Sf/QwpOrphanScanner.cs +++ b/src/net-questdb-client/Qwp/Sf/QwpOrphanScanner.cs @@ -54,7 +54,14 @@ internal static class QwpOrphanScanner /// /// The current sender's slot name, never claimed by the scanner. /// - public static IReadOnlyList ClaimOrphans(string sfRoot, string ourSenderId) + /// + /// When non-null, slot dirs named {managedBase}-{i} for i ∈ [0, managedCount) are + /// also skipped: they belong to live or future siblings of a QuestDBClient pool, so only true + /// out-of-family orphans are adopted. Pass null for the single-sender case. + /// + /// Size of the managed slot family (see ). + public static IReadOnlyList ClaimOrphans( + string sfRoot, string ourSenderId, string? managedBase = null, int managedCount = 0) { ArgumentNullException.ThrowIfNull(sfRoot); ArgumentNullException.ThrowIfNull(ourSenderId); @@ -80,7 +87,7 @@ public static IReadOnlyList ClaimOrphans(string sfRoot, string ourS { try { - TryClaim(slotDir, ourSenderId, claimed); + TryClaim(slotDir, ourSenderId, managedBase, managedCount, claimed); } catch (Exception) { @@ -91,7 +98,36 @@ public static IReadOnlyList ClaimOrphans(string sfRoot, string ourS return claimed; } - private static void TryClaim(string slotDir, string ourSenderId, List claimed) + /// + /// True if names a managed pool slot + /// ({managedBase}-{i}, i ∈ [0, managedCount)), which the scanner must not adopt. + /// + internal static bool IsManagedSlot(string senderId, string? managedBase, int managedCount) + { + if (managedBase is null || managedCount <= 0) + { + return false; + } + + var prefix = managedBase + "-"; + if (!senderId.StartsWith(prefix, StringComparison.Ordinal)) + { + return false; + } + + var suffix = senderId.AsSpan(prefix.Length); + if (!int.TryParse(suffix, out var index) || index < 0 || index >= managedCount) + { + return false; + } + + // Pool slots are exactly "{base}-{index}": a lookalike like "default-03" parses to 3 but is + // never allocated by the pool, so treating it as managed would strand it — nothing drains it. + return suffix.Equals(index.ToString(), StringComparison.Ordinal); + } + + private static void TryClaim( + string slotDir, string ourSenderId, string? managedBase, int managedCount, List claimed) { var senderId = new DirectoryInfo(slotDir).Name; if (string.Equals(senderId, ourSenderId, StringComparison.Ordinal)) @@ -99,6 +135,11 @@ private static void TryClaim(string slotDir, string ourSenderId, List max) max = gen; } diff --git a/src/net-questdb-client/Qwp/Sf/QwpSlotLock.cs b/src/net-questdb-client/Qwp/Sf/QwpSlotLock.cs index 28bcfa3..bde2dc8 100644 --- a/src/net-questdb-client/Qwp/Sf/QwpSlotLock.cs +++ b/src/net-questdb-client/Qwp/Sf/QwpSlotLock.cs @@ -84,6 +84,22 @@ public void RefreshHeartbeat() /// Full path of the lock file. public string LockFilePath { get; } + /// + /// True once the OS lock has been released (the backing disposed). + /// A pool consults this after closing a sender to decide whether the slot index is safe to + /// reuse, since a wedged I/O path could in principle leave the handle open. + /// + public bool IsReleased + { + get + { + lock (_stateLock) + { + return _disposed; + } + } + } + /// /// Runs against while holding an /// internal mutex that excludes . Returns false (without diff --git a/src/net-questdb-client/Qwp/Sf/QwpTrackedCursorTransport.cs b/src/net-questdb-client/Qwp/Sf/QwpTrackedCursorTransport.cs index 9079333..70f97e2 100644 --- a/src/net-questdb-client/Qwp/Sf/QwpTrackedCursorTransport.cs +++ b/src/net-questdb-client/Qwp/Sf/QwpTrackedCursorTransport.cs @@ -84,7 +84,7 @@ public async Task ConnectAsync(CancellationToken cancellationToken) { _tracker.RecordTransportError(_hostIndex); throw new IngressError(ErrorCode.SocketError, - $"WebSocket upgrade exceeded auth_timeout={_connectTimeout.TotalMilliseconds}ms"); + $"WebSocket upgrade exceeded connect_timeout={_connectTimeout.TotalMilliseconds}ms"); } catch (QwpIngressRoleRejectedException ex) { diff --git a/src/net-questdb-client/Senders/AbstractSender.cs b/src/net-questdb-client/Senders/AbstractSender.cs index b50b646..b315bc4 100644 --- a/src/net-questdb-client/Senders/AbstractSender.cs +++ b/src/net-questdb-client/Senders/AbstractSender.cs @@ -301,13 +301,6 @@ public ISender ColumnDecimal128(ReadOnlySpan name, decimal value, byte sca public ISender ColumnDecimal256(ReadOnlySpan name, decimal value, byte scale) => AppendIlpDecimalScaled(name, value, scale, QwpConstants.MaxDecimal256Scale, "Decimal256"); - /// - public ISender ColumnDecimal64(ReadOnlySpan name, long unscaledValue, byte scale) - { - ValidateDecimalScale(scale, QwpConstants.MaxDecimal64Scale, "Decimal64"); - return AppendIlpDecimalFromMantissa(name, new BigInteger(unscaledValue), scale); - } - /// public ISender ColumnDecimal128(ReadOnlySpan name, long lo, long hi, byte scale) { diff --git a/src/net-questdb-client/Senders/HttpSender.cs b/src/net-questdb-client/Senders/HttpSender.cs index 5463682..37832f3 100644 --- a/src/net-questdb-client/Senders/HttpSender.cs +++ b/src/net-questdb-client/Senders/HttpSender.cs @@ -150,7 +150,8 @@ private SocketsHttpHandler CreateHandler(string host) } } - handler.ConnectTimeout = Options.auth_timeout; + // Bounds socket connect + TLS handshake (the connect_timeout budget; see EffectiveConnectTimeout). + handler.ConnectTimeout = Options.EffectiveConnectTimeout; handler.PreAuthenticate = true; return handler; } diff --git a/src/net-questdb-client/Senders/ISender.cs b/src/net-questdb-client/Senders/ISender.cs index 65c7b1c..54184b5 100644 --- a/src/net-questdb-client/Senders/ISender.cs +++ b/src/net-questdb-client/Senders/ISender.cs @@ -28,8 +28,14 @@ namespace QuestDB.Senders; /// -/// Interface representing implementations. For ws:: / wss:: -/// senders prefer await using var so the close-time ACK drain doesn't block the caller. +/// Interface representing implementations. +/// +/// Dispose does not send. (and +/// ) are pure resource release: any buffered-but-unsent +/// rows are discarded and no exception is thrown. Call / +/// to hand rows to the transport, and (for delivery confirmation on ws::) +/// / +/// to block until the server has acknowledged them, before disposing. /// public interface ISender : IDisposable, IAsyncDisposable { @@ -98,13 +104,59 @@ ValueTask IAsyncDisposable.DisposeAsync() /// /// /// Only usable outside of a transaction. If there are no pending rows, then this is a no-op. + /// + /// Delivery semantics: HTTP is synchronous (returns once the server has committed the batch). A + /// standalone ws/wss sender drains — it flushes and then blocks until the + /// server acknowledges (bounded by close_flush_timeout_millis, throwing on timeout) — so + /// Send() before Dispose reliably delivers even though Dispose no longer drains. A + /// pooled ws sender (from IQuestDBClient.BorrowSender) instead hands the frame to the + /// ring and returns immediately; the pooled connection ships it asynchronously and delivery is + /// confirmed via / IQuestDBClient.Flush. + /// For an explicit bounded, bool-returning drain on any sender, call Flush. /// - /// When the request fails. + /// When the request fails, or a standalone ws drain times out. public Task SendAsync(CancellationToken ct = default); /// public void Send(CancellationToken ct = default); + /// + /// Drains the sender: flushes any buffered rows (like ) and then blocks until + /// the server has acknowledged every frame this sender has published, or + /// elapses. This is the delivery-confirmation barrier — the non-pooled analogue of the pool-wide + /// drain on IQuestDBClient.Flush. + /// + /// + /// On ws/wss this waits for the ACK watermark to catch up to the published frame + /// sequence. On HTTP/TCP there are no frame-level ACKs, so the flush happens and the method returns + /// true immediately (HTTP's is already synchronous and confirmed). + /// + /// Upper bound on the wait for acknowledgement. + /// Cancels the flush and the wait. + /// true if everything published was acknowledged on return; false on timeout. + /// If the transport has latched a terminal error. + public bool Flush(TimeSpan timeout, CancellationToken ct = default) + { + Send(ct); + return true; + } + + /// + public async Task FlushAsync(TimeSpan timeout, CancellationToken ct = default) + { + await SendAsync(ct).ConfigureAwait(false); + return true; + } + + /// + /// Convenience overload of using + /// as the drain timeout. + /// + public bool Flush(CancellationToken ct = default) => Flush(Options.close_flush_timeout_millis, ct); + + /// + public Task FlushAsync(CancellationToken ct = default) => FlushAsync(Options.close_flush_timeout_millis, ct); + /// /// Set table (measurement) name for the next row. /// Each row may have a different table name within a batch. @@ -485,17 +537,6 @@ public ISender NullableColumn(ReadOnlySpan name, decimal? value) /// Number of fractional digits (0–76). public ISender ColumnDecimal256(ReadOnlySpan name, decimal value, byte scale); - /// - /// Adds a DECIMAL64 column from the raw unscaled int64 mantissa with an explicit - /// . Exposes the full 18-digit range that - /// cannot always represent. Over ILP this is reconstructed as a and - /// requires protocol version 3. - /// - /// The column name. - /// The unscaled integer mantissa; the value is unscaledValue / 10^scale. - /// Number of fractional digits (0–18). - public ISender ColumnDecimal64(ReadOnlySpan name, long unscaledValue, byte scale); - /// /// Adds a DECIMAL128 column from the two two's-complement 64-bit limbs of the unscaled integer: /// is the low 64 bits (unsigned magnitude), the diff --git a/src/net-questdb-client/Senders/QwpWebSocketSender.cs b/src/net-questdb-client/Senders/QwpWebSocketSender.cs index 2319e4f..59f7f86 100644 --- a/src/net-questdb-client/Senders/QwpWebSocketSender.cs +++ b/src/net-questdb-client/Senders/QwpWebSocketSender.cs @@ -24,9 +24,8 @@ #if NET7_0_OR_GREATER -using System.Diagnostics; -using System.Runtime.ExceptionServices; using QuestDB.Enums; +using QuestDB.Pooling; using QuestDB.Qwp; using QuestDB.Qwp.Sf; using QuestDB.Utils; @@ -43,7 +42,7 @@ namespace QuestDB.Senders; /// frames on transient WS failures, and only terminate on permanent errors (auth, upgrade /// reject, protocol violation, or reconnect-budget exhaustion). /// -internal sealed class QwpWebSocketSender : IQwpWebSocketSender +internal sealed class QwpWebSocketSender : IQwpWebSocketSender, IPooledSlotSender, IPooledTransactionalSender, IPooledDrainAwareSender { private const long TicksPerMicrosecond = 10L; private const int EncoderInitialCapacity = 1 << 16; @@ -103,6 +102,28 @@ public QwpWebSocketSender(SenderOptions options) _engine.SetTableEntryHandler(UpdateSeqTxnFromAck); } + /// + /// True when there is no slot lock (RAM mode), or the slot lock the engine owns has been + /// released. The pool reads this after disposing this sender to decide whether the slot index + /// may be reused. See . + /// + public bool IsSlotLockReleased => _slotLock is null || _slotLock.IsReleased; + + /// + /// True while auto-flushed transactional rows are staged server-side awaiting a commit frame. + /// The pool reads this on return: staged rows cannot be rolled back, so the entry must be + /// discarded rather than re-pooled. See . + /// + public bool HasUncommittedDeferredRows => _hasDeferredMessages; + + /// + /// True when the cursor engine's ring holds no un-acked frames. The pool reads this to keep the + /// idle-reap clock from starting until in-flight rows are delivered, so an idle pooled sender is + /// never torn down (freeing its RAM-backed ring) while it still owes un-acked data. See + /// . + /// + public bool IsFullyDrained => _engine.IsFullyDrained; + private static (QwpSlotLock? slotLock, QwpCursorSendEngine engine, QwpBackgroundDrainerPool? pool, QwpSenderErrorDispatcher? dispatcher, QwpConnectionEventDispatcher? eventDispatcher, QwpTlsAuth.CertificateValidator? certValidator) BuildEngineStack(SenderOptions options) { @@ -210,7 +231,9 @@ private static (QwpSlotLock? slotLock, QwpCursorSendEngine engine, QwpBackground pool = new QwpBackgroundDrainerPool( options.max_background_drainers, drainer); - var orphans = QwpOrphanScanner.ClaimOrphans(options.sf_dir!, options.sender_id); + var orphans = QwpOrphanScanner.ClaimOrphans( + options.sf_dir!, options.sender_id, + options.OrphanExcludeManagedBase, options.OrphanExcludeManagedCount); var enqueued = 0; try { @@ -537,14 +560,6 @@ public ISender ColumnDecimal256(ReadOnlySpan name, decimal value, byte sca return this; } - /// - public ISender ColumnDecimal64(ReadOnlySpan name, long unscaledValue, byte scale) - { - ThrowIfTerminal(); - EnsureCurrentTable().AppendDecimal64(name, unscaledValue, scale); - return this; - } - /// public ISender ColumnDecimal128(ReadOnlySpan name, long lo, long hi, byte scale) { @@ -754,19 +769,71 @@ public void AtNanos(long timestampNanos, CancellationToken ct = default) /// public Task SendAsync(CancellationToken ct = default) { + // Standalone (SendAwaitsAck): Send drains (flush + await ACK) so "Send() before Dispose" delivers + // even though Dispose does not drain. Pooled: fast flush-to-ring; the pool ships async and + // delivery is confirmed via IQuestDBClient.Flush(). close_flush_timeout_millis=0 is the documented + // "fast, don't wait" opt-out — a 0ms drain can never observe an ACK, so treat it as flush-to-ring + // rather than a spurious timeout. + if (Options.SendAwaitsAck && Options.close_flush_timeout_millis > TimeSpan.Zero) + { + return SendDrainAsync(ct); + } + ThrowIfTerminal(); EnsureNoRowInProgress(); return FlushAsyncCore(deferCommit: false, ct).AsTask(); } + private async Task SendDrainAsync(CancellationToken ct) + { + if (!await FlushAsync(Options.close_flush_timeout_millis, ct).ConfigureAwait(false)) + { + ThrowSendDrainTimeout(); + } + } + /// public void Send(CancellationToken ct = default) { + if (Options.SendAwaitsAck && Options.close_flush_timeout_millis > TimeSpan.Zero) + { + if (!Flush(Options.close_flush_timeout_millis, ct)) + { + ThrowSendDrainTimeout(); + } + + return; + } + ThrowIfTerminal(); EnsureNoRowInProgress(); FlushSync(deferCommit: false, ct); } + private void ThrowSendDrainTimeout() => + throw new IngressError(ErrorCode.ServerFlushError, + $"Send timed out after {Options.close_flush_timeout_millis.TotalMilliseconds:F0}ms waiting for the " + + "server to acknowledge; data may not have been delivered"); + + /// + public bool Flush(TimeSpan timeout, CancellationToken ct = default) + => FlushCore(timeout, ct).GetAwaiter().GetResult(); + + /// + public Task FlushAsync(TimeSpan timeout, CancellationToken ct = default) + => FlushCore(timeout, ct); + + // Drain = flush buffered rows into the ring (publishing them) then wait for the ACK watermark to reach + // that published frame. Reuses FlushAndGetSequenceAsync so the deferred-commit / commit-frame handling + // stays in one place. + private async Task FlushCore(TimeSpan timeout, CancellationToken ct) + { + ThrowIfTerminal(); + EnsureNoRowInProgress(); + var publishedFsn = await FlushAndGetSequenceAsync(ct).ConfigureAwait(false); + return await AwaitAckedFsnAsync(publishedFsn, timeout, ct).ConfigureAwait(false); + } + private void EnsureNoRowInProgress() { foreach (var t in _tables.Values) @@ -1044,124 +1111,25 @@ public async ValueTask DisposeAsync() await DisposeStackAsync().ConfigureAwait(false); } + // Dispose is pure resource release: no flush, no ACK wait, no throw. Buffered-but-unsent rows are + // discarded. Callers wanting delivery must Send() (hand rows to the ring) and, for confirmation, + // Flush(timeout) (await ACK) BEFORE disposing. Already-sent frames are unaffected: in SF mode they + // persist in the mmap ring and replay on the next run; in RAM mode they ship best-effort until + // teardown. Sync and async teardown are identical — the cursor engine's own Dispose is synchronous + // (it defers a wedged pump-join past its budget), so there is nothing to await. private void DisposeStackSync() { - ExceptionDispatchInfo? toRethrow = null; - try - { - toRethrow = DrainOnClose( - encode: () => FlushSync(deferCommit: false, CancellationToken.None), - wait: () => _engine.FlushAsync(Options.close_flush_timeout_millis).GetAwaiter().GetResult()); - } - finally - { - SfCleanup.Dispose(_drainerPool); - SfCleanup.Dispose(_engine); - SfCleanup.Dispose(_errorDispatcher); - SfCleanup.Dispose(_connectionEventDispatcher); - SfCleanup.Dispose(_certValidator); - } - - toRethrow?.Throw(); + SfCleanup.Dispose(_drainerPool); + SfCleanup.Dispose(_engine); + SfCleanup.Dispose(_errorDispatcher); + SfCleanup.Dispose(_connectionEventDispatcher); + SfCleanup.Dispose(_certValidator); } - private async ValueTask DisposeStackAsync() + private ValueTask DisposeStackAsync() { - ExceptionDispatchInfo? toRethrow = null; - try - { - toRethrow = await DrainOnCloseAsync( - encode: () => FlushAsyncCore(deferCommit: false, CancellationToken.None), - wait: () => new ValueTask(_engine.FlushAsync(Options.close_flush_timeout_millis))).ConfigureAwait(false); - } - finally - { - SfCleanup.Dispose(_drainerPool); - SfCleanup.Dispose(_engine); - SfCleanup.Dispose(_errorDispatcher); - SfCleanup.Dispose(_connectionEventDispatcher); - SfCleanup.Dispose(_certValidator); - } - - toRethrow?.Throw(); - } - - private ExceptionDispatchInfo? DrainOnClose(Action encode, Action wait) - { - var timeoutMs = Options.close_flush_timeout_millis.TotalMilliseconds; - if (timeoutMs <= 0) - { - // Explicit "don't wait" close. Still encode buffered rows into the on-disk ring in SF mode - // (a local append bounded by sf_append_deadline, not a network wait) so they replay after - // restart; best-effort and never throws. In RAM mode the ring is torn down un-sent, so - // there is nothing useful to encode. - if (!string.IsNullOrEmpty(Options.sf_dir)) - { - try { encode(); } - catch { } - } - return null; - } - - try - { - encode(); - wait(); - } - catch (TimeoutException ex) - { - LogCloseFlushTimeoutWarn(timeoutMs); - return CaptureTerminalForRethrow() ?? ExceptionDispatchInfo.Capture(ex); - } - catch { } - - return CaptureTerminalForRethrow(); - } - - private async ValueTask DrainOnCloseAsync(Func encode, Func wait) - { - var timeoutMs = Options.close_flush_timeout_millis.TotalMilliseconds; - if (timeoutMs <= 0) - { - // See DrainOnClose: encode best-effort into the on-disk ring in SF mode even when the - // ack-wait is skipped, so buffered rows survive to replay; never throws. - if (!string.IsNullOrEmpty(Options.sf_dir)) - { - try { await encode().ConfigureAwait(false); } - catch { } - } - return null; - } - - try - { - await encode().ConfigureAwait(false); - await wait().ConfigureAwait(false); - } - catch (TimeoutException ex) - { - LogCloseFlushTimeoutWarn(timeoutMs); - return CaptureTerminalForRethrow() ?? ExceptionDispatchInfo.Capture(ex); - } - catch { } - - return CaptureTerminalForRethrow(); - } - - private ExceptionDispatchInfo? CaptureTerminalForRethrow() - { - var terminal = _engine.TerminalError; - if (terminal is null) return null; - if (_errorDispatcher?.HasDeliveredToCustomHandler == true) return null; - return ExceptionDispatchInfo.Capture(terminal); - } - - private void LogCloseFlushTimeoutWarn(double timeoutMs) - { - Trace.TraceWarning( - "QWP close: close_flush_timeout ({0:F0} ms) expired with un-acked frames pending; pending data {1}.", - timeoutMs, - !string.IsNullOrEmpty(Options.sf_dir) ? "remains on disk (SF mode)" : "is lost (Memory mode)"); + DisposeStackSync(); + return default; } private QwpTableBuffer EnsureCurrentTable() @@ -1322,7 +1290,7 @@ private static Func BuildHostRotatingFactory( }; return new QwpTrackedCursorTransport(new QwpWebSocketTransport(transportOpts), tracker, idx, - options.auth_timeout); + options.EffectiveConnectTimeout); }; } diff --git a/src/net-questdb-client/Senders/TcpSender.cs b/src/net-questdb-client/Senders/TcpSender.cs index 3056720..1b40bc4 100644 --- a/src/net-questdb-client/Senders/TcpSender.cs +++ b/src/net-questdb-client/Senders/TcpSender.cs @@ -25,6 +25,7 @@ // ReSharper disable CommentTypo using System.Buffers.Text; +using System.Linq; using System.Net.Security; using System.Net.Sockets; using System.Security.Cryptography.X509Certificates; @@ -84,9 +85,30 @@ private void Build() var socket = new Socket(AddressFamily.InterNetwork, SocketType.Stream, System.Net.Sockets.ProtocolType.Tcp); NetworkStream? networkStream = null; SslStream? sslStream = null; + // Single deadline spanning socket connect + TLS handshake + ECDSA auth — the total connection + // budget. Disarmed (no token) if misconfigured to a non-positive / out-of-range value. + var connectBudget = Options.EffectiveConnectTimeout; + CancellationTokenSource? connectCts = + connectBudget > TimeSpan.Zero && connectBudget.TotalMilliseconds <= int.MaxValue + ? new CancellationTokenSource(connectBudget) + : null; + var connectToken = connectCts?.Token ?? CancellationToken.None; try { - socket.ConnectAsync(Options.Host, Options.Port).Wait(); + try + { + // .Wait() (not GetResult) preserves the released AggregateException-wrapped contract for + // ordinary connect failures (e.g. connection refused). Only the connect_timeout case is + // unwrapped into a clear IngressError. + socket.ConnectAsync(Options.Host, Options.Port, connectToken).AsTask().Wait(); + } + catch (AggregateException ae) when (connectCts is { IsCancellationRequested: true } + && ae.InnerExceptions.Any(e => e is OperationCanceledException)) + { + throw new IngressError(ErrorCode.SocketError, + $"connect to {Options.Host}:{Options.Port} exceeded connect_timeout={connectBudget.TotalMilliseconds}ms"); + } + networkStream = new NetworkStream(socket, Options.own_socket); Stream dataStream = networkStream; @@ -106,7 +128,16 @@ private void Build() sslOptions.ClientCertificates.Add(Options.client_cert); } - sslStream.AuthenticateAsClient(sslOptions); + try + { + sslStream.AuthenticateAsClientAsync(sslOptions, connectToken).GetAwaiter().GetResult(); + } + catch (OperationCanceledException) when (connectCts is { IsCancellationRequested: true }) + { + throw new IngressError(ErrorCode.TlsError, + $"TLS handshake with {Options.Host}:{Options.Port} exceeded connect_timeout={connectBudget.TotalMilliseconds}ms"); + } + if (!sslStream.IsEncrypted) { throw new IngressError(ErrorCode.TlsError, "Could not establish encrypted connection."); @@ -119,7 +150,9 @@ private void Build() _dataStream = dataStream; if (!string.IsNullOrEmpty(Options.token)) { - var authTimeout = new CancellationTokenSource(); + // The ECDSA exchange is bounded by both the auth-timeout setting and the remaining + // connect budget, whichever fires first. + using var authTimeout = CancellationTokenSource.CreateLinkedTokenSource(connectToken); authTimeout.CancelAfter(Options.auth_timeout); _signatureGenerator = Secp256r1SignatureGenerator.Instance.Value; AuthenticateAsync(authTimeout.Token).AsTask().Wait(authTimeout.Token); @@ -132,6 +165,10 @@ private void Build() sslStream?.Dispose(); throw; } + finally + { + connectCts?.Dispose(); + } } /// diff --git a/src/net-questdb-client/Utils/SenderOptions.cs b/src/net-questdb-client/Utils/SenderOptions.cs index 5dbd549..28d1b85 100644 --- a/src/net-questdb-client/Utils/SenderOptions.cs +++ b/src/net-questdb-client/Utils/SenderOptions.cs @@ -25,6 +25,7 @@ // ReSharper disable CommentTypo +using System.ComponentModel; using System.Data.Common; using System.Reflection; using System.Runtime.CompilerServices; @@ -54,6 +55,11 @@ public record SenderOptions private string _addr = "localhost:9000"; private List _addresses = new(); private TimeSpan _authTimeout = TimeSpan.FromMilliseconds(15000); + // Nullable so "unset" is distinct from any concrete value: a null falls back to the compatibility + // default, and the config binder (which round-trips every writable property) never writes a null + // back — so binding + // from appsettings can't spuriously pin it. + private TimeSpan? _connectTimeout; private AutoFlushType _autoFlush = AutoFlushType.on; private int _autoFlushBytes = int.MaxValue; private bool _autoFlushBytesUserSet; @@ -107,6 +113,15 @@ public record SenderOptions private int _maxBackgroundDrainers = 4; private TimeSpan _pingTimeout = TimeSpan.FromMilliseconds(5000); private TimeSpan _durableAckKeepaliveInterval = TimeSpan.FromMilliseconds(200); + private int _senderPoolMin = 1; + private int _senderPoolMax = 4; + private int _queryPoolMin = 1; + private int _queryPoolMax = 4; + private TimeSpan _acquireTimeout = TimeSpan.FromMilliseconds(5000); + private TimeSpan _idleTimeout = TimeSpan.FromMilliseconds(60000); + private TimeSpan _maxLifetime = TimeSpan.FromMilliseconds(1800000); + private TimeSpan _housekeeperInterval = TimeSpan.FromMilliseconds(5000); + private bool _lazyConnect; private string? _proxy; private string? _zone; private SenderErrorHandler? _errorHandler; @@ -227,7 +242,28 @@ public SenderOptions(string confStr) } ParseMillisecondsWithDefault(nameof(request_timeout), "30000", out _requestTimeout); ParseMillisecondsWithDefault(nameof(retry_timeout), "10000", out _retryTimeout); + // All-protocol: total wall-clock budget for establishing a connection (socket + TLS + WS + // upgrade + auth). Left null when absent so it falls back to the compatibility default, + // keeping the released HTTP connect wiring and the WS upgrade bound byte-compatible. + if (ReadOptionFromBuilder(nameof(connect_timeout)) is not null) + { + ParseMillisecondsWithDefault(nameof(connect_timeout), "15000", out var ct); + _connectTimeout = ct; + } ParseMillisecondsWithDefault(nameof(pool_timeout), "120000", out _poolTimeout); + + // Connection-pool knobs (QuestDBClient handle). Protocol-agnostic; a plain Sender parses + // and ignores them. ValidatePoolOptions enforces the relationships. + ParseIntWithDefault(nameof(sender_pool_min), "1", out _senderPoolMin); + ParseIntWithDefault(nameof(sender_pool_max), "4", out _senderPoolMax); + ParseIntWithDefault(nameof(query_pool_min), "1", out _queryPoolMin); + ParseIntWithDefault(nameof(query_pool_max), "4", out _queryPoolMax); + ParseMillisecondsWithDefault(nameof(acquire_timeout_ms), "5000", out _acquireTimeout); + ParseMillisecondsWithDefault(nameof(idle_timeout_ms), "60000", out _idleTimeout); + ParseMillisecondsWithDefault(nameof(max_lifetime_ms), "1800000", out _maxLifetime); + ParseMillisecondsWithDefault(nameof(housekeeper_interval_ms), "5000", out _housekeeperInterval); + ParseBoolOnOff(nameof(lazy_connect), "off", out _lazyConnect); + ParseEnumWithDefault(nameof(tls_verify), "on", out _tlsVerify); ParseStringWithDefault(nameof(tls_roots), null, out _tlsRoots); ParseStringWithDefault(nameof(tls_roots_password), null, out _tlsRootsPassword); @@ -516,6 +552,15 @@ private void ValidateTimeouts() { if (_authTimeout <= TimeSpan.Zero) throw new IngressError(ErrorCode.ConfigError, $"`auth_timeout` must be > 0; got {_authTimeout.TotalMilliseconds}ms"); + if (_connectTimeout is { } connectTimeout) + { + if (connectTimeout <= TimeSpan.Zero) + throw new IngressError(ErrorCode.ConfigError, $"`connect_timeout` must be > 0; got {connectTimeout.TotalMilliseconds}ms"); + // Armed via CancellationTokenSource(TimeSpan)/CancelAfter and SocketsHttpHandler.ConnectTimeout, + // all of which reject a delay past int.MaxValue ms (~24.8 days). + if (connectTimeout.TotalMilliseconds > int.MaxValue) + throw new IngressError(ErrorCode.ConfigError, $"`connect_timeout` must be ≤ {int.MaxValue}ms (~24.8 days); got {connectTimeout.TotalMilliseconds}ms"); + } if (_requestTimeout <= TimeSpan.Zero) throw new IngressError(ErrorCode.ConfigError, $"`request_timeout` must be > 0; got {_requestTimeout.TotalMilliseconds}ms"); if (_retryTimeout < TimeSpan.Zero) @@ -565,6 +610,53 @@ internal void EnsureValid() ApplyInitialConnectPromotion(); ValidateErrorInboxCapacity(); ValidateBufferSizes(); + ValidatePoolOptions(); + } + + /// + /// Validates the pool knobs. Called from + /// (connect-string path) and re-run by QuestDBClientBuilder.Build after programmatic + /// overrides so an invalid min/max set via builder methods is still caught. + /// + internal void ValidatePoolOptions() + { + if (_senderPoolMin < 0) + throw new IngressError(ErrorCode.ConfigError, $"`sender_pool_min` must be ≥ 0; got {_senderPoolMin}"); + if (_senderPoolMax < 1) + throw new IngressError(ErrorCode.ConfigError, $"`sender_pool_max` must be ≥ 1; got {_senderPoolMax}"); + if (_senderPoolMin > _senderPoolMax) + throw new IngressError(ErrorCode.ConfigError, + $"`sender_pool_min` ({_senderPoolMin}) must be ≤ `sender_pool_max` ({_senderPoolMax})"); + if (_acquireTimeout < TimeSpan.Zero) + throw new IngressError(ErrorCode.ConfigError, $"`acquire_timeout_ms` must be ≥ 0; got {_acquireTimeout.TotalMilliseconds}ms"); + if (_acquireTimeout.TotalMilliseconds > int.MaxValue) + throw new IngressError(ErrorCode.ConfigError, $"`acquire_timeout_ms` must be ≤ {int.MaxValue}ms; got {_acquireTimeout.TotalMilliseconds}ms"); + if (_idleTimeout <= TimeSpan.Zero) + throw new IngressError(ErrorCode.ConfigError, $"`idle_timeout_ms` must be > 0; got {_idleTimeout.TotalMilliseconds}ms"); + if (_maxLifetime <= TimeSpan.Zero) + throw new IngressError(ErrorCode.ConfigError, $"`max_lifetime_ms` must be > 0; got {_maxLifetime.TotalMilliseconds}ms"); + if (_housekeeperInterval < TimeSpan.FromMilliseconds(100)) + throw new IngressError(ErrorCode.ConfigError, $"`housekeeper_interval_ms` must be ≥ 100; got {_housekeeperInterval.TotalMilliseconds}ms"); + if (_housekeeperInterval.TotalMilliseconds > int.MaxValue) + throw new IngressError(ErrorCode.ConfigError, $"`housekeeper_interval_ms` must be ≤ {int.MaxValue}ms; got {_housekeeperInterval.TotalMilliseconds}ms"); + } + + /// + /// Validates the query-pool knobs. Kept separate from + /// (and OUT of the unconditional + /// path) so a plain Sender.New stays lenient about query keys it ignores; only the query + /// pool and QuestDBClientBuilder.Build enforce them. The shared acquire/idle/lifetime/ + /// housekeeper relationships are already checked by . + /// + internal void ValidateQueryPoolOptions() + { + if (_queryPoolMin < 0) + throw new IngressError(ErrorCode.ConfigError, $"`query_pool_min` must be ≥ 0; got {_queryPoolMin}"); + if (_queryPoolMax < 1) + throw new IngressError(ErrorCode.ConfigError, $"`query_pool_max` must be ≥ 1; got {_queryPoolMax}"); + if (_queryPoolMin > _queryPoolMax) + throw new IngressError(ErrorCode.ConfigError, + $"`query_pool_min` ({_queryPoolMin}) must be ≤ `query_pool_max` ({_queryPoolMax})"); } /// @@ -689,6 +781,20 @@ private void ValidateStoreAndForwardOptions() $"`sf_durability` only accepts 'memory' in v1, got `{_sfDurability}`"); } + // Transactional staging is connection-scoped (the server drops staged deferred rows when the + // connection closes), but store-and-forward deliberately outlives connections: it persists + // frames and replays them after reconnects, restarts, and slot re-adoption. A replayed + // FLAG_DEFER_COMMIT frame re-stages rows whose commit decision is lost — possibly abandoned — + // and the next commit on that connection silently publishes them. The two promises are + // irreconcilable, so reject the combination outright. + if (_transaction && !string.IsNullOrEmpty(_sfDir)) + { + throw new IngressError(ErrorCode.ConfigError, + "`transaction=on` cannot be combined with `sf_dir`: store-and-forward replays persisted " + + "frames across connections, which would re-stage and later publish uncommitted " + + "(possibly abandoned) transactional rows"); + } + // Programmatic init's field initializer is the no-SF default (128 MiB); promote when sf_dir // is set and the user didn't pick their own value. Equality-on-128MiB would falsely promote // an explicit user 128 MiB. @@ -1050,15 +1156,36 @@ public string? token /// - /// Timeout for authentication requests. - /// Defaults to 15 seconds. + /// Supported for backward compatibility but not advertised; use + /// instead. When is unset this value is the connection budget, + /// and on TCP it bounds the ECDSA auth exchange. Defaults to 15 seconds. /// + [EditorBrowsable(EditorBrowsableState.Never)] public TimeSpan auth_timeout { get => _authTimeout; set => _authTimeout = value; } + /// + /// Total wall-clock budget for bringing up a usable connection across every layer — TCP + /// socket connect, TLS handshake, WebSocket upgrade, and (TCP) the ECDSA auth exchange. The + /// attempt is aborted if the budget is exceeded. Applies to every transport (HTTP / TCP / WS, + /// ingest and egress). When unset, the effective budget is 15 seconds. Read the value actually + /// applied via . + /// + public TimeSpan? connect_timeout + { + get => _connectTimeout; + set => _connectTimeout = value; + } + + /// + /// The connect budget actually applied at runtime: when set, + /// otherwise the compatibility fallback. + /// + internal TimeSpan EffectiveConnectTimeout => _connectTimeout ?? _authTimeout; + /// /// Specifies a minimum expect network throughput when sending data to QuestDB. /// Defaults to 100 KiB @@ -1163,6 +1290,152 @@ public TimeSpan pool_timeout set => _poolTimeout = value; } + // Pool knobs are a QuestDBClient-handle concern, not part of a sender's serialized identity, so + // they are [JsonIgnore]d out of ToString()/PrintMembers — a plain Sender that parsed them stays + // byte-identical on round-trip. The pool reads them off the parsed options directly; it holds the + // user's original connect string, so it never relies on ToString to carry them. + + /// + /// Minimum number of senders the pool keeps warm. Pre-created + /// at construction and never reaped below. Default 1. + /// + [JsonIgnore] + public int sender_pool_min + { + get => _senderPoolMin; + set => _senderPoolMin = value; + } + + /// + /// Maximum number of senders the pool will create. Borrowers + /// block up to once this many are in use. Default 4. + /// + [JsonIgnore] + public int sender_pool_max + { + get => _senderPoolMax; + set => _senderPoolMax = value; + } + + /// + /// Minimum number of query clients the pool keeps warm (only when a + /// query config is present). Default 1. Set to 0 for lazy first-borrow creation. + /// + [JsonIgnore] + public int query_pool_min + { + get => _queryPoolMin; + set => _queryPoolMin = value; + } + + /// + /// Maximum number of query clients the pool will create. Borrowers + /// block up to once this many are in use. Default 4. + /// + [JsonIgnore] + public int query_pool_max + { + get => _queryPoolMax; + set => _queryPoolMax = value; + } + + /// + /// How long a borrow blocks waiting for a free sender before throwing. Zero means a single + /// non-blocking attempt. Default 5s. + /// + [JsonIgnore] + public TimeSpan acquire_timeout_ms + { + get => _acquireTimeout; + set => _acquireTimeout = value; + } + + /// + /// Idle duration after which the housekeeper reaps a pooled sender (never below + /// ). Default 60s. + /// + [JsonIgnore] + public TimeSpan idle_timeout_ms + { + get => _idleTimeout; + set => _idleTimeout = value; + } + + /// + /// Maximum age after which the housekeeper recycles a pooled sender (never below + /// ). Default 30min. + /// + [JsonIgnore] + public TimeSpan max_lifetime_ms + { + get => _maxLifetime; + set => _maxLifetime = value; + } + + /// + /// Sweep interval of the daemon housekeeper that reaps idle / over-age pooled senders. + /// Default 5s; must be ≥ 100ms. + /// + [JsonIgnore] + public TimeSpan housekeeper_interval_ms + { + get => _housekeeperInterval; + set => _housekeeperInterval = value; + } + + /// + /// Tolerant-startup flag for the handle. When on/true + /// the handle builds even while the server is down: the ingest side connects asynchronously + /// (buffering writes meanwhile) and the read pool defaults to 0 so + /// nothing connects eagerly — a query connects lazily on first borrow once the server is up. + /// Conflicting knobs that force a blocking / fail-fast startup (an explicit + /// initial_connect_retry other than async, or an explicit query_pool_min > 0) + /// are rejected by QuestDBClientBuilder.Build. A plain ignores this + /// key. Default off. + /// + [JsonIgnore] + public bool lazy_connect + { + get => _lazyConnect; + set => _lazyConnect = value; + } + + /// + /// True when initial_connect_retry / initial_connect_mode was set explicitly by the + /// user (connect string or programmatic setter), as opposed to left at its default or promoted + /// internally. Used by the pool facade to detect a lazy_connect conflict without treating + /// the reconnect-tuning promotion as a user choice. + /// + internal bool IsInitialConnectModeExplicit => + _initialConnectModeUserSet || IsKeyExplicit(nameof(initial_connect_retry)); + + /// True when query_pool_min appeared explicitly in this connect string. + internal bool IsQueryPoolMinExplicit => IsKeyExplicit(nameof(query_pool_min)); + + /// True when query_pool_max appeared explicitly in this connect string. + internal bool IsQueryPoolMaxExplicit => IsKeyExplicit(nameof(query_pool_max)); + + /// + /// Set by the QuestDBClient pool on each per-slot sender: the orphan scanner skips slot dirs + /// named {OrphanExcludeManagedBase}-{i} for i ∈ [0, OrphanExcludeManagedCount) so + /// a pooled sender never adopts a slot belonging to a live or future pool sibling. Internal — + /// not a connect-string key, not serialized (the reflection ToString reads public props only). + /// + internal string? OrphanExcludeManagedBase { get; set; } + + /// + internal int OrphanExcludeManagedCount { get; set; } + + /// + /// Whether a WS Send() / SendAsync() drains (flushes then blocks until the server + /// acknowledges) rather than just handing the frame to the ring. Default true for a + /// standalone sender, so "Send() before Dispose" delivers even though Dispose no longer drains. + /// The QuestDBClient pool sets it false on pooled senders: their connection survives the + /// borrow's return and ships asynchronously, and delivery is confirmed via the pool-wide + /// IQuestDBClient.Flush(). Internal — not a connect-string key, not serialized. + /// + internal bool SendAwaitsAck { get; set; } = true; + /// /// If true, requests STATUS_DURABLE_ACK frames from the server via the /// X-QWP-Request-Durable-Ack upgrade header. Off by default. @@ -1176,9 +1449,11 @@ public bool request_durable_ack /// /// Enables WebSocket transactional mode (connect-string key transaction=on). Auto-flush /// ships frames with FLAG_DEFER_COMMIT so the server appends without committing; an - /// explicit Send()/Commit() (or dispose) ships a non-deferred frame that triggers + /// explicit Send()/Commit() ships a non-deferred frame that triggers /// the server-side WAL commit. Lets a producer stage a dataset larger than the server recv - /// buffer and commit it atomically per table. WebSocket-only; off by default. + /// buffer and commit it atomically per table. WebSocket-only; off by default. Mutually + /// exclusive with : store-and-forward replays persisted frames across + /// connections, which cannot honour connection-scoped transactional staging. /// public bool transaction { @@ -1417,9 +1692,10 @@ public SenderErrorPolicy? on_write_error } /// - /// Maximum time to wait for unacked SF frames to drain on Sender.Dispose. - /// Defaults to 60 s, matching the Java client — a wide default so close() does not - /// silently drop unacked rows on slow/backlogged consumers. + /// Default drain timeout used by the no-argument ISender.Flush() / + /// IQuestDBClient.Flush() overloads — the maximum time to wait for un-acked frames to be + /// acknowledged. Defaults to 60 s. Note: Dispose no longer drains (it is pure resource + /// release); call Send() then Flush() explicitly for delivery confirmation. /// public TimeSpan close_flush_timeout_millis {