-
Notifications
You must be signed in to change notification settings - Fork 323
docs(enterprise): fix contradictory single-compactor constraint in clustering docs #7075
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: master
Are you sure you want to change the base?
Changes from 7 commits
5e47919
c0c2c14
b7b04d4
792af02
a0c6d2e
1e92766
d1f15bf
86c7ba8
689cacc
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
|
@@ -59,18 +59,23 @@ influxdb3 serve --mode=ingest | |||||||||||||
| # Multiple modes | ||||||||||||||
| influxdb3 serve --mode=ingest,query | ||||||||||||||
|
|
||||||||||||||
| # All modes (default) | ||||||||||||||
| # All modes (default, for single-node Enterprise only) | ||||||||||||||
| influxdb3 serve --mode=all | ||||||||||||||
| ``` | ||||||||||||||
|
|
||||||||||||||
| Available modes: | ||||||||||||||
|
|
||||||||||||||
| - `all`: All capabilities enabled (default) | ||||||||||||||
| - `all`: All capabilities enabled (single-node Enterprise deployments only) | ||||||||||||||
| - `ingest`: Data ingestion and line protocol parsing | ||||||||||||||
| - `query`: Query execution and data retrieval | ||||||||||||||
| - `compact`: Background compaction and optimization | ||||||||||||||
| - `process`: Data processing and transformations | ||||||||||||||
|
|
||||||||||||||
| > [!Warning] | ||||||||||||||
| > Use `all` mode for **single-node** Enterprise deployments only. | ||||||||||||||
| > Avoid using `all` mode in a multi-node cluster—some cluster features such as replication and catalog refresh aren't designed to work with `all`-mode nodes. | ||||||||||||||
|
jstirnaman marked this conversation as resolved.
Outdated
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Applied in 689cacc. |
||||||||||||||
| > In a multi-node cluster, use explicit modes (`ingest`, `query`, `compact`, `process`) and assign `compact` to exactly one node. | ||||||||||||||
|
|
||||||||||||||
| ## Allocate threads by node type | ||||||||||||||
|
|
||||||||||||||
| ### Critical concept: Thread pools | ||||||||||||||
|
|
@@ -194,6 +199,11 @@ influxdb3 serve \ | |||||||||||||
|
|
||||||||||||||
| Compactor nodes optimize stored data through background compaction processes. | ||||||||||||||
|
|
||||||||||||||
| > [!Warning] | ||||||||||||||
| > Only **one** compactor node can run per cluster. | ||||||||||||||
| > Multiple compactors writing compacted data to the same location will cause data corruption. | ||||||||||||||
| > Any node mode that includes compaction (`compact` or `all`) counts toward this limit. | ||||||||||||||
|
|
||||||||||||||
| ### Dedicated compactor (32 cores) | ||||||||||||||
|
|
||||||||||||||
| ```bash | ||||||||||||||
|
|
@@ -298,21 +308,25 @@ influxdb3 \ | |||||||||||||
|
|
||||||||||||||
| ### Small cluster (3 nodes) | ||||||||||||||
|
|
||||||||||||||
| > [!Note] | ||||||||||||||
| > Only one node per cluster can run compaction. | ||||||||||||||
| > In this example, Node 1 handles ingest, query, and compaction while Nodes 2–3 handle ingest and query only. | ||||||||||||||
|
|
||||||||||||||
| ```yaml | ||||||||||||||
| # Node 1: All-in-one primary | ||||||||||||||
| mode: all | ||||||||||||||
| # Node 1: Ingest, query, and compactor | ||||||||||||||
| mode: ingest,query,compact | ||||||||||||||
| cores: 32 | ||||||||||||||
| io_threads: 8 | ||||||||||||||
| datafusion_threads: 24 | ||||||||||||||
|
|
||||||||||||||
| # Node 2: All-in-one secondary | ||||||||||||||
| mode: all | ||||||||||||||
| # Node 2: Ingest and query (no compaction) | ||||||||||||||
| mode: ingest,query | ||||||||||||||
| cores: 32 | ||||||||||||||
| io_threads: 8 | ||||||||||||||
| datafusion_threads: 24 | ||||||||||||||
|
|
||||||||||||||
| # Node 3: All-in-one tertiary | ||||||||||||||
| mode: all | ||||||||||||||
| # Node 3: Ingest and query (no compaction) | ||||||||||||||
| mode: ingest,query | ||||||||||||||
| cores: 32 | ||||||||||||||
| io_threads: 8 | ||||||||||||||
| datafusion_threads: 24 | ||||||||||||||
|
|
@@ -333,8 +347,14 @@ cores: 48 | |||||||||||||
| io_threads: 4 | ||||||||||||||
| datafusion_threads: 44 | ||||||||||||||
|
|
||||||||||||||
| # Nodes 5-6: Compactor + Process | ||||||||||||||
| mode: compact,process | ||||||||||||||
| # Node 5: Compactor (only one compactor per cluster) | ||||||||||||||
| mode: compact | ||||||||||||||
| cores: 32 | ||||||||||||||
| io_threads: 4 | ||||||||||||||
| datafusion_threads: 28 | ||||||||||||||
|
|
||||||||||||||
| # Node 6: Process node | ||||||||||||||
| mode: process | ||||||||||||||
| cores: 32 | ||||||||||||||
| io_threads: 4 | ||||||||||||||
| datafusion_threads: 28 | ||||||||||||||
|
|
@@ -355,13 +375,13 @@ cores: 64 | |||||||||||||
| io_threads: 4 | ||||||||||||||
| datafusion_threads: 60 | ||||||||||||||
|
|
||||||||||||||
| # Nodes 9-10: Dedicated compactors | ||||||||||||||
| # Node 9: Dedicated compactor (only one compactor per cluster) | ||||||||||||||
| mode: compact | ||||||||||||||
| cores: 32 | ||||||||||||||
| io_threads: 2 | ||||||||||||||
| datafusion_threads: 30 | ||||||||||||||
|
|
||||||||||||||
| # Nodes 11-12: Process nodes | ||||||||||||||
| # Nodes 10-12: Process nodes | ||||||||||||||
| mode: process | ||||||||||||||
| cores: 32 | ||||||||||||||
| io_threads: 6 | ||||||||||||||
|
|
@@ -553,7 +573,8 @@ GROUP BY event_type; | |||||||||||||
| - Growing number of small Parquet files | ||||||||||||||
| - Increasing query times due to file fragmentation | ||||||||||||||
|
|
||||||||||||||
| **Solution:** Add compactor nodes or increase DataFusion threads (see [Compactor node issues](#compactor-node-issues)) | ||||||||||||||
| **Solution:** For nodes using the default Parquet-backed storage engine, increase DataFusion threads on your single compactor node (see [Compactor node issues](#compactor-node-issues)). | ||||||||||||||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Applied in 689cacc — "default" removed. |
||||||||||||||
| The Performance Preview with PachaTree storage does not use DataFusion for compaction—refer to the [Performance Preview documentation](/influxdb3/enterprise/performance-preview/) for tuning guidance. | ||||||||||||||
|
|
||||||||||||||
| ## Troubleshoot node configurations | ||||||||||||||
|
|
||||||||||||||
|
|
@@ -602,7 +623,7 @@ free -h | |||||||||||||
|
|
||||||||||||||
| ```bash | ||||||||||||||
| # Check: Compaction queue length | ||||||||||||||
| # Solution: Add more compactor nodes or increase threads | ||||||||||||||
| # Solution: Increase threads on the single compactor node (only one compactor is allowed per cluster) | ||||||||||||||
| --datafusion-num-threads=30 | ||||||||||||||
| ``` | ||||||||||||||
|
|
||||||||||||||
|
|
@@ -617,14 +638,20 @@ free -h | |||||||||||||
|
|
||||||||||||||
| ## Migrate to specialized nodes | ||||||||||||||
|
|
||||||||||||||
| ### From all-in-one to specialized | ||||||||||||||
| ### From single-node to specialized cluster | ||||||||||||||
|
|
||||||||||||||
| > [!Note] | ||||||||||||||
| > `all` mode is intended for single-node Enterprise deployments. | ||||||||||||||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Applied in 689cacc. |
||||||||||||||
| > When scaling to a multi-node cluster, replace `all` with explicit modes and assign `compact` to exactly one node. | ||||||||||||||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. it needs to be more than this; when upgrading from an All to multinode, the "new"
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Incorporated in 689cacc. The migration example now shows the compact node in Phase 2 using
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. |
||||||||||||||
|
|
||||||||||||||
| ```bash | ||||||||||||||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. @copilot Fix the notation/syntax in this "bash" example so that it represents real code. Observe the following for all examples scaling from
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. |
||||||||||||||
| # Phase 1: Baseline (all nodes identical) | ||||||||||||||
| all nodes: --mode=all --num-io-threads=8 | ||||||||||||||
| # Phase 1: Baseline (single-node deployment) | ||||||||||||||
| node1: --mode=all --num-io-threads=8 | ||||||||||||||
|
|
||||||||||||||
| # Phase 2: Identify workload patterns | ||||||||||||||
| # Monitor which nodes handle most writes vs queries | ||||||||||||||
| # Phase 2: Expand to multi-node — replace all-in-one with explicit modes | ||||||||||||||
| # Assign compact to exactly one node | ||||||||||||||
| node1: --mode=ingest,query,compact --num-io-threads=8 | ||||||||||||||
| node2: --mode=ingest,query --num-io-threads=8 | ||||||||||||||
|
|
||||||||||||||
| # Phase 3: Gradual specialization | ||||||||||||||
| node1: --mode=ingest,query --num-io-threads=12 | ||||||||||||||
|
|
||||||||||||||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Applied in 689cacc. The warning now has a heading "Don't use all mode in a multi-node cluster" and the opening line matches your suggestion.