From 5e4791925044d53b16ef708a8765dc17c2c78f94 Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Wed, 8 Apr 2026 19:39:35 +0000 Subject: [PATCH 1/5] Initial plan From c0c2c14062cf1bd77858dcca4800a18d6ebfd1e8 Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Wed, 8 Apr 2026 19:43:29 +0000 Subject: [PATCH 2/5] fix: correct multiple compactor documentation in clustering.md - Add Warning callout in Configure node modes section: only one node per cluster can run in a mode that includes compaction (compact or all) - Add Warning callout in Configure compactor nodes section reinforcing the single-compactor-per-cluster rule - Fix Small cluster (3 nodes) example: changed nodes 2-3 from 'all' mode to 'ingest,query' since only one node can run compaction - Fix Medium cluster (6 nodes) example: split "Nodes 5-6: Compactor + Process" (both in compact,process) into Node 5 (compact) and Node 6 (process) with a note about the single-compactor rule - Fix Large cluster (12+ nodes) example: changed "Nodes 9-10: Dedicated compactors" to "Node 9: Dedicated compactor" and merged Node 10 into "Nodes 10-12: Process nodes" - Fix troubleshooting section: changed "Add compactor nodes or increase DataFusion threads" to "Increase DataFusion threads on your single compactor node" - Fix Migrate to specialized nodes section: updated Phase 1 from "all nodes: --mode=all" to a single-node baseline with a note warning against multi-node all-mode setups Agent-Logs-Url: https://github.com/influxdata/docs-v2/sessions/8fef041f-0506-4ec4-b334-beb5d6c5f03d Co-authored-by: jstirnaman <212227+jstirnaman@users.noreply.github.com> --- .../influxdb3/enterprise/admin/clustering.md | 51 ++++++++++++++----- 1 file changed, 38 insertions(+), 13 deletions(-) diff --git a/content/influxdb3/enterprise/admin/clustering.md b/content/influxdb3/enterprise/admin/clustering.md index ed8f063503..d107b47c7f 100644 --- a/content/influxdb3/enterprise/admin/clustering.md +++ b/content/influxdb3/enterprise/admin/clustering.md @@ -71,6 +71,11 @@ Available modes: - `compact`: Background compaction and optimization - `process`: Data processing and transformations +> [!Warning] +> Only **one** node per cluster can run in a mode that includes compaction (`compact` or `all`). +> Running multiple compactors causes data corruption. +> In a cluster, assign `all` mode to at most one node, and ensure no other node uses the `compact` mode. + ## Allocate threads by node type ### Critical concept: Thread pools @@ -194,6 +199,11 @@ influxdb3 serve \ Compactor nodes optimize stored data through background compaction processes. +> [!Warning] +> Only **one** compactor node can run per cluster. +> Multiple compactors writing compacted data to the same location will cause data corruption. +> Any node mode that includes compaction (`compact` or `all`) counts toward this limit. + ### Dedicated compactor (32 cores) ```bash @@ -298,21 +308,25 @@ influxdb3 \ ### Small cluster (3 nodes) +> [!Note] +> Only one node per cluster can run compaction. +> In this example, Node 1 runs all modes (including compaction) and Nodes 2–3 run ingest and query only. + ```yaml -# Node 1: All-in-one primary +# Node 1: All-in-one primary (includes compaction) mode: all cores: 32 io_threads: 8 datafusion_threads: 24 -# Node 2: All-in-one secondary -mode: all +# Node 2: Ingest and query (no compaction) +mode: ingest,query cores: 32 io_threads: 8 datafusion_threads: 24 -# Node 3: All-in-one tertiary -mode: all +# Node 3: Ingest and query (no compaction) +mode: ingest,query cores: 32 io_threads: 8 datafusion_threads: 24 @@ -333,8 +347,14 @@ cores: 48 io_threads: 4 datafusion_threads: 44 -# Nodes 5-6: Compactor + Process -mode: compact,process +# Node 5: Compactor (only one compactor per cluster) +mode: compact +cores: 32 +io_threads: 4 +datafusion_threads: 28 + +# Node 6: Process node +mode: process cores: 32 io_threads: 4 datafusion_threads: 28 @@ -355,13 +375,13 @@ cores: 64 io_threads: 4 datafusion_threads: 60 -# Nodes 9-10: Dedicated compactors +# Node 9: Dedicated compactor (only one compactor per cluster) mode: compact cores: 32 io_threads: 2 datafusion_threads: 30 -# Nodes 11-12: Process nodes +# Nodes 10-12: Process nodes mode: process cores: 32 io_threads: 6 @@ -553,7 +573,7 @@ GROUP BY event_type; - Growing number of small Parquet files - Increasing query times due to file fragmentation -**Solution:** Add compactor nodes or increase DataFusion threads (see [Compactor node issues](#compactor-node-issues)) +**Solution:** Increase DataFusion threads on your single compactor node (see [Compactor node issues](#compactor-node-issues)) ## Troubleshoot node configurations @@ -602,7 +622,7 @@ free -h ```bash # Check: Compaction queue length -# Solution: Add more compactor nodes or increase threads +# Solution: Increase threads on the single compactor node (only one compactor is allowed per cluster) --datafusion-num-threads=30 ``` @@ -619,9 +639,14 @@ free -h ### From all-in-one to specialized +> [!Note] +> If you're migrating a multi-node cluster, only one node should ever be in `all` mode. +> In a baseline multi-node setup, additional nodes should use `ingest,query` instead of `all` +> to avoid running multiple compactors. + ```bash -# Phase 1: Baseline (all nodes identical) -all nodes: --mode=all --num-io-threads=8 +# Phase 1: Baseline (single all-in-one node for starting point) +node1: --mode=all --num-io-threads=8 # Phase 2: Identify workload patterns # Monitor which nodes handle most writes vs queries From d1f15bfc2cec53a8ee8eed9afa5098f733773b3e Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Wed, 15 Apr 2026 15:48:38 +0000 Subject: [PATCH 3/5] fix: clarify all-mode restriction and Parquet-specific compaction tuning MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Update 'all' mode description to single-node Enterprise only throughout - Replace 'all' mode in Small cluster (3 nodes) example with explicit ingest,query,compact modes - Update warning callout: all mode is for single-node only; multi-node clusters should use explicit modes with compact assigned to one node - Update 'Migrate to specialized nodes' to clarify single-node → cluster migration path without using all mode in multi-node context - Specify that DataFusion thread tuning for compaction applies to the default Parquet-backed storage engine, not PachaTree (Performance Preview) Agent-Logs-Url: https://github.com/influxdata/docs-v2/sessions/da0e0cbe-bbfc-42d3-8821-0a025ed3ef50 Co-authored-by: jstirnaman <212227+jstirnaman@users.noreply.github.com> --- .../influxdb3/enterprise/admin/clustering.md | 34 ++++++++++--------- 1 file changed, 18 insertions(+), 16 deletions(-) diff --git a/content/influxdb3/enterprise/admin/clustering.md b/content/influxdb3/enterprise/admin/clustering.md index d107b47c7f..a694997486 100644 --- a/content/influxdb3/enterprise/admin/clustering.md +++ b/content/influxdb3/enterprise/admin/clustering.md @@ -59,22 +59,22 @@ influxdb3 serve --mode=ingest # Multiple modes influxdb3 serve --mode=ingest,query -# All modes (default) +# All modes (default, for single-node Enterprise only) influxdb3 serve --mode=all ``` Available modes: -- `all`: All capabilities enabled (default) +- `all`: All capabilities enabled (single-node Enterprise deployments only) - `ingest`: Data ingestion and line protocol parsing - `query`: Query execution and data retrieval - `compact`: Background compaction and optimization - `process`: Data processing and transformations > [!Warning] -> Only **one** node per cluster can run in a mode that includes compaction (`compact` or `all`). -> Running multiple compactors causes data corruption. -> In a cluster, assign `all` mode to at most one node, and ensure no other node uses the `compact` mode. +> Use `all` mode for **single-node** Enterprise deployments only. +> Avoid using `all` mode in a multi-node cluster—some cluster features such as replication and catalog refresh aren't designed to work with `all`-mode nodes. +> In a multi-node cluster, use explicit modes (`ingest`, `query`, `compact`, `process`) and assign `compact` to exactly one node. ## Allocate threads by node type @@ -310,11 +310,11 @@ influxdb3 \ > [!Note] > Only one node per cluster can run compaction. -> In this example, Node 1 runs all modes (including compaction) and Nodes 2–3 run ingest and query only. +> In this example, Node 1 handles ingest, query, and compaction while Nodes 2–3 handle ingest and query only. ```yaml -# Node 1: All-in-one primary (includes compaction) -mode: all +# Node 1: Ingest, query, and compactor +mode: ingest,query,compact cores: 32 io_threads: 8 datafusion_threads: 24 @@ -573,7 +573,8 @@ GROUP BY event_type; - Growing number of small Parquet files - Increasing query times due to file fragmentation -**Solution:** Increase DataFusion threads on your single compactor node (see [Compactor node issues](#compactor-node-issues)) +**Solution:** For nodes using the default Parquet-backed storage engine, increase DataFusion threads on your single compactor node (see [Compactor node issues](#compactor-node-issues)). +The Performance Preview with PachaTree storage does not use DataFusion for compaction—refer to the [Performance Preview documentation](/influxdb3/enterprise/performance-preview/) for tuning guidance. ## Troubleshoot node configurations @@ -637,19 +638,20 @@ free -h ## Migrate to specialized nodes -### From all-in-one to specialized +### From single-node to specialized cluster > [!Note] -> If you're migrating a multi-node cluster, only one node should ever be in `all` mode. -> In a baseline multi-node setup, additional nodes should use `ingest,query` instead of `all` -> to avoid running multiple compactors. +> `all` mode is intended for single-node Enterprise deployments. +> When scaling to a multi-node cluster, replace `all` with explicit modes and assign `compact` to exactly one node. ```bash -# Phase 1: Baseline (single all-in-one node for starting point) +# Phase 1: Baseline (single-node deployment) node1: --mode=all --num-io-threads=8 -# Phase 2: Identify workload patterns -# Monitor which nodes handle most writes vs queries +# Phase 2: Expand to multi-node — replace all-in-one with explicit modes +# Assign compact to exactly one node +node1: --mode=ingest,query,compact --num-io-threads=8 +node2: --mode=ingest,query --num-io-threads=8 # Phase 3: Gradual specialization node1: --mode=ingest,query --num-io-threads=12 From 689cacc6a7c1c1d840453785174a398e5c70b0ec Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Wed, 15 Apr 2026 22:08:51 +0000 Subject: [PATCH 4/5] fix: apply PR review suggestions to clustering.md - Warning callout: add heading 'Don't use all mode in a multi-node cluster'; reword opening lines per review suggestions - Compaction troubleshooting: remove 'default' qualifier before 'Parquet-backed storage engine' - Migration section: rewrite note to say 'is only for'; move scaling guidance outside the callout as bullet points; replace pseudo-bash with real influxdb3 serve commands; incorporate same-node-id requirement for compact node when replacing an all-mode node Agent-Logs-Url: https://github.com/influxdata/docs-v2/sessions/50bbe0a2-e066-4da0-8e00-afe917dba91b Co-authored-by: jstirnaman <212227+jstirnaman@users.noreply.github.com> --- .../influxdb3/enterprise/admin/clustering.md | 68 +++++++++++++------ 1 file changed, 48 insertions(+), 20 deletions(-) diff --git a/content/influxdb3/enterprise/admin/clustering.md b/content/influxdb3/enterprise/admin/clustering.md index a694997486..f30ae4e9e2 100644 --- a/content/influxdb3/enterprise/admin/clustering.md +++ b/content/influxdb3/enterprise/admin/clustering.md @@ -72,8 +72,10 @@ Available modes: - `process`: Data processing and transformations > [!Warning] +> #### Don't use all mode in a multi-node cluster +> > Use `all` mode for **single-node** Enterprise deployments only. -> Avoid using `all` mode in a multi-node cluster—some cluster features such as replication and catalog refresh aren't designed to work with `all`-mode nodes. +> Some cluster features such as replication and catalog refresh aren't designed to work with `all`-mode nodes. > In a multi-node cluster, use explicit modes (`ingest`, `query`, `compact`, `process`) and assign `compact` to exactly one node. ## Allocate threads by node type @@ -573,7 +575,7 @@ GROUP BY event_type; - Growing number of small Parquet files - Increasing query times due to file fragmentation -**Solution:** For nodes using the default Parquet-backed storage engine, increase DataFusion threads on your single compactor node (see [Compactor node issues](#compactor-node-issues)). +**Solution:** For nodes using the Parquet-backed storage engine, increase DataFusion threads on your single compactor node (see [Compactor node issues](#compactor-node-issues)). The Performance Preview with PachaTree storage does not use DataFusion for compaction—refer to the [Performance Preview documentation](/influxdb3/enterprise/performance-preview/) for tuning guidance. ## Troubleshoot node configurations @@ -641,26 +643,52 @@ free -h ### From single-node to specialized cluster > [!Note] -> `all` mode is intended for single-node Enterprise deployments. -> When scaling to a multi-node cluster, replace `all` with explicit modes and assign `compact` to exactly one node. +> `all` mode is only for single-node Enterprise deployments. + +When scaling a single `all` node cluster to a multi-node cluster: + +- Replace the `all` node with nodes that have explicit, specialized modes +- Assign `compact` mode to exactly one node that uses the same node-id as the `all` node being replaced ```bash -# Phase 1: Baseline (single-node deployment) -node1: --mode=all --num-io-threads=8 - -# Phase 2: Expand to multi-node — replace all-in-one with explicit modes -# Assign compact to exactly one node -node1: --mode=ingest,query,compact --num-io-threads=8 -node2: --mode=ingest,query --num-io-threads=8 - -# Phase 3: Gradual specialization -node1: --mode=ingest,query --num-io-threads=12 -node2: --mode=query,compact --num-io-threads=4 - -# Phase 4: Full specialization -node1: --mode=ingest --num-io-threads=16 -node2: --mode=query --num-io-threads=4 -node3: --mode=compact --num-io-threads=2 +# Phase 1: Single-node deployment +influxdb3 serve \ + --node-id=node0 \ + --cluster-id=my-cluster \ + --mode=all \ + --num-io-threads=8 + +# Phase 2: Scale to multi-node cluster +# Stop the all-mode node and start specialized nodes. +# The compact node MUST use the same --node-id as the replaced all-mode node. + +# Compact node: reuses the same node-id as the replaced all-mode node +influxdb3 serve \ + --node-id=node0 \ + --cluster-id=my-cluster \ + --mode=compact + +# Ingest and query node +influxdb3 serve \ + --node-id=node1 \ + --cluster-id=my-cluster \ + --mode=ingest,query \ + --num-io-threads=8 + +# Phase 3: Full specialization (optional) +# Dedicated ingest node +influxdb3 serve \ + --node-id=node1 \ + --cluster-id=my-cluster \ + --mode=ingest \ + --num-io-threads=16 + +# Dedicated query node +influxdb3 serve \ + --node-id=node2 \ + --cluster-id=my-cluster \ + --mode=query \ + --num-io-threads=4 ``` ## Manage configurations From de476f718e3586ba1f6deea50742cc24a1ba08f5 Mon Sep 17 00:00:00 2001 From: Jason Stirnaman Date: Mon, 20 Apr 2026 17:04:37 -0500 Subject: [PATCH 5/5] Update content/influxdb3/enterprise/admin/clustering.md --- content/influxdb3/enterprise/admin/clustering.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/content/influxdb3/enterprise/admin/clustering.md b/content/influxdb3/enterprise/admin/clustering.md index f30ae4e9e2..7856ce7b32 100644 --- a/content/influxdb3/enterprise/admin/clustering.md +++ b/content/influxdb3/enterprise/admin/clustering.md @@ -74,6 +74,8 @@ Available modes: > [!Warning] > #### Don't use all mode in a multi-node cluster > +> #### Don't use all mode in a multi-node cluster +> > Use `all` mode for **single-node** Enterprise deployments only. > Some cluster features such as replication and catalog refresh aren't designed to work with `all`-mode nodes. > In a multi-node cluster, use explicit modes (`ingest`, `query`, `compact`, `process`) and assign `compact` to exactly one node.