diff --git a/docs/user-guide/deployments-administration/configuration.md b/docs/user-guide/deployments-administration/configuration.md index ffb130441..96748f766 100644 --- a/docs/user-guide/deployments-administration/configuration.md +++ b/docs/user-guide/deployments-administration/configuration.md @@ -619,9 +619,9 @@ selector = "round_robin" # Store data in memory, false by default. use_memory_store = false # Whether to enable region failover. -# This feature is only available on GreptimeDB running on cluster mode and -# - Using Remote WAL -# - Using shared storage (e.g., s3). +# This feature is only available on GreptimeDB running on cluster mode with shared storage (e.g., s3) and one of: +# - Remote WAL +# - Local WAL with `allow_region_failover_on_local_wal = true` (may lead to data loss during failover) enable_region_failover = false ## The delay before starting region failure detection. ## This delay helps prevent Metasrv from triggering unnecessary region failovers before all Datanodes are fully started. @@ -769,7 +769,7 @@ timeout = "3s" | `store_addrs` | Array | `["127.0.0.1:2379"]` | Store server address. Configure the address based on your backend type, for example:
- Use `"127.0.0.1:2379"` to connect to etcd
- Use `"password=password dbname=postgres user=postgres host=localhost port=5432"` to connect to postgres
- Use `"mysql://user:password@ip:port/dbname"` to connect to mysql | | `selector` | String | `lease_based` | Datanode selector type.
- `lease_based` (default value).
- `load_based`
For details, see [Selector](/contributor-guide/metasrv/selector.md) | | `use_memory_store` | Bool | `false` | Store data in memory. | -| `enable_region_failover` | Bool | `false` | Whether to enable region failover.
This feature is only available on GreptimeDB running on cluster mode and
- Using Remote WAL
- Using shared storage (e.g., s3). | +| `enable_region_failover` | Bool | `false` | Whether to enable region failover.
This feature is only available on GreptimeDB running on cluster mode with shared storage (e.g., s3) and one of:
- Remote WAL
- Local WAL with `allow_region_failover_on_local_wal = true` (may lead to data loss during failover). | | `region_failure_detector_initialization_delay` | String | `10m` | The delay before starting region failure detection. This delay helps prevent Metasrv from triggering unnecessary region failovers before all Datanodes are fully started. Especially useful when the cluster is not deployed with GreptimeDB Operator and maintenance mode is not enabled. | | `allow_region_failover_on_local_wal` | Bool | `false` | Whether to allow region failover on local WAL.
**This option is not recommended to be set to true, because it may lead to data loss during failover.** | | `node_max_idle_time` | String | `24hours` | Max allowed idle time before removing node info from metasrv memory. Nodes that haven't sent heartbeats for this duration will be considered inactive and removed. | diff --git a/docs/user-guide/deployments-administration/manage-data/region-failover.md b/docs/user-guide/deployments-administration/manage-data/region-failover.md index 4f5e00b00..8421ac78a 100644 --- a/docs/user-guide/deployments-administration/manage-data/region-failover.md +++ b/docs/user-guide/deployments-administration/manage-data/region-failover.md @@ -1,17 +1,17 @@ --- keywords: [region failover, GreptimeDB, data recovery, Kafka WAL, read amplification, distributed mode, region migration, recovery time, shared storage, configuration] -description: Covers enabling and understanding region failover in GreptimeDB, which allows for the recovery of regions from failures without data loss, and discusses the impact of read amplification on recovery time. +description: Covers enabling and understanding region failover in GreptimeDB, including prerequisites, WAL-related data loss risks, and the impact of read amplification on recovery time with Kafka WAL. --- # Region Failover -Region Failover provides the ability to recover regions from region failures without losing data. This is implemented via [Region Migration](/user-guide/deployments-administration/manage-data/region-migration.md). +Region Failover provides the ability to recover regions from region failures. With Kafka WAL (Remote WAL) and shared storage, Region Failover can recover regions without losing data. This is implemented via [Region Migration](/user-guide/deployments-administration/manage-data/region-migration.md). ## Enable the Region Failover This feature is only available on GreptimeDB running on distributed mode and -- Using Kafka WAL (Remote WAL) or Local WAL (Enable region failover on local WAL may lead to data loss during failover) +- Using Kafka WAL (Remote WAL) or Local WAL with `allow_region_failover_on_local_wal=true` (enabling region failover on Local WAL may lead to data loss during failover) - Using [shared storage](/user-guide/deployments-administration/configuration.md#storage-options) (e.g., AWS S3) If you want to enable region failover on local WAL, you need to set `allow_region_failover_on_local_wal=true` in [metasrv](/user-guide/deployments-administration/configuration.md#metasrv-only-configuration) configuration file. It's not recommended to enable this option, because it may lead to data loss. @@ -25,7 +25,7 @@ To enable region failover via GreptimeDB Operator, you can refer to [Common Helm ## The recovery time of Region Failover -The recovery time of Region Failover depends on: +The recovery time of Region Failover with Kafka WAL (Remote WAL) depends on: - number of regions per Topic. - the Kafka cluster read throughput performance. @@ -47,14 +47,14 @@ For example, configure 128 topics for [metasrv](/user-guide/deployments-administ

(Figure1: recovery Region 3 need to read redundant data 7 times larger than the actual size)

-A simple model to estimate the read amplification factor (replay data size/actual data size): +A simple model to estimate the redundant read amplification factor (redundant replay data size/actual data size): - For a single topic, if we try to replay all regions that belong to the topic, then the amplification factor would be 7+6+...+1 = 28 times. (The Region WAL data distribution is shown in the Figure 1. Replaying Region 3 will read 7 times redundant data larger than the actual size; Region 6 will read 6 times, and so on) - When recovering 100 regions (requiring about 13 topics), the amplification factor is approximately 28 \* 13 = 364 times. Assuming we have 100 regions to recover, and the actual data size of all regions is 0.5GB, the following table shows the replay data size based on the number of regions per topic. -| Number of regions per Topic | Number of topics required for 100 Regions | Single topic read amplification factor | Total reading amplification factor | Replay data size (GB) | +| Number of regions per Topic | Number of topics required for 100 Regions | Single topic redundant read factor | Total redundant read factor | Replay data size (GB) | | --------------------------- | ----------------------------------------- | -------------------------------------- | ---------------------------------- | ---------------- | | 1 | 100 | 0 | 0 | 0.5 | | 2 | 50 | 1 | 50 | 25.5 | @@ -79,5 +79,4 @@ The following table shows the recovery time of 100 regions under different read In the above example, we calculated the recovery time based on the number of Regions contained in each Topic for reference. We have calculated the recovery time under different Number of regions per Topic configuration for reference. In actual scenarios, the read amplification may be larger than this model. -If you are very sensitive to recovery time, we recommend that each region have its topic(i.e., Number of regions per Topic is 1). - +If you use Kafka WAL and are very sensitive to recovery time, we recommend that each region have its own topic (i.e., Number of regions per Topic is 1). diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/user-guide/deployments-administration/configuration.md b/i18n/zh/docusaurus-plugin-content-docs/current/user-guide/deployments-administration/configuration.md index 0c66fc8d1..121a72fac 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/user-guide/deployments-administration/configuration.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/user-guide/deployments-administration/configuration.md @@ -611,9 +611,9 @@ selector = "lease_based" # 将数据存储在内存中,默认值为 false。 use_memory_store = false # 是否启用 region failover。 -# 该功能仅适用于运行在集群模式下的 GreptimeDB,并且 -# - 使用 Remote WAL -# - 使用共享存储(例如 s3)。 +# 该功能仅适用于以集群模式运行并使用共享存储(例如 s3)的 GreptimeDB,并且 WAL 需要满足以下条件之一: +# - Remote WAL +# - Local WAL 且设置 `allow_region_failover_on_local_wal = true`(故障转移期间可能导致数据丢失) enable_region_failover = false ## 设置启动 region 故障检测的延迟时间。 ## 该延迟有助于避免在所有 Datanode 尚未完全启动时,Metasrv 过早启动 region 故障检测,从而导致不必要的 region failover。 @@ -743,7 +743,7 @@ timeout = "3s" | `store_addrs` | Array | `["127.0.0.1:2379"]` | 元数据服务地址,默认值为 `["127.0.0.1:2379"]`。支持配置多个服务地址,格式为 `["ip1:port1","ip2:port2",...]`。默认使用 Etcd 作为元数据后端。
根据你的存储服务器类型配置地址,例如:
- 使用 `"127.0.0.1:2379"` 连接到 etcd
- 使用 `"password=password dbname=postgres user=postgres host=localhost port=5432"` 连接到 postgres
- 使用 `"mysql://user:password@ip:port/dbname"` 连接到 mysql | | `selector` | String | `lease_based` | 创建新表时选择 datanode 的负载均衡策略,详见 [选择器](/contributor-guide/metasrv/selector.md)。 | | `use_memory_store` | Boolean | `false` | 仅用于在没有 etcd 集群时的测试,将数据存储在内存中,默认值为 `false`。 | -| `enable_region_failover` | Bool | `false` | 是否启用 region failover。
该功能仅在以集群模式运行的 GreptimeDB 上可用,并且
- 使用远程 WAL
- 使用共享存储(如 s3)。 | +| `enable_region_failover` | Bool | `false` | 是否启用 region failover。
该功能仅适用于以集群模式运行并使用共享存储(如 s3)的 GreptimeDB,并且 WAL 需要满足以下条件之一:
- Remote WAL
- Local WAL 且设置 `allow_region_failover_on_local_wal = true`(故障转移期间可能导致数据丢失)。 | | `region_failure_detector_initialization_delay` | String | `10m` | 设置启动 region 故障检测的延迟时间。该延迟有助于避免在所有 Datanode 尚未完全启动时,Metasrv 过早启动 region 故障检测,从而导致不必要的 region failover。尤其适用于未通过 GreptimeDB Operator 部署的集群,此时可能未正确启用集群维护模式,提前检测可能会引发误判。 | | `allow_region_failover_on_local_wal` | Bool | false | 是否允许在本地 WAL 上进行 region failover。
**此选项不建议设置为 true,因为这可能会在故障转移期间导致数据丢失。** | | `node_max_idle_time` | String | `24hours` | 从 metasrv 内存中删除节点信息前允许的最大空闲时间。超过该时间未发送心跳的节点将被视为不活跃并被删除。 | diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/user-guide/deployments-administration/manage-data/region-failover.md b/i18n/zh/docusaurus-plugin-content-docs/current/user-guide/deployments-administration/manage-data/region-failover.md index aaa9180bb..563a017fc 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/user-guide/deployments-administration/manage-data/region-failover.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/user-guide/deployments-administration/manage-data/region-failover.md @@ -1,18 +1,18 @@ --- keywords: [Region Failover, 故障恢复, 恢复时间, 共享存储, Kafka WAL] -description: 介绍 Region Failover 功能及其在 GreptimeDB 中的应用,包括开启 Region Failover、恢复时间和改进恢复时间的建议等内容。 +description: 介绍 Region Failover 功能及其在 GreptimeDB 中的应用,包括开启 Region Failover、WAL 相关的数据丢失风险、恢复时间和改进恢复时间的建议等内容。 --- # Region Failover -Region Failover 提供了在不丢失数据的情况下从 Region 故障中恢复的能力。这是通过 [Region 迁移](/user-guide/deployments-administration/manage-data/region-migration.md) 实现的。 +Region Failover 提供了从 Region 故障中恢复的能力。使用 Kafka WAL (Remote WAL) 和共享存储时,Region Failover 可以在不丢失数据的情况下恢复 Region。这是通过 [Region 迁移](/user-guide/deployments-administration/manage-data/region-migration.md) 实现的。 ## 开启 Region Failover 该功能仅在 GreptimeDB 集群模式下可用,并且需要满足以下条件 -- 使用 Kafka WAL (Remote WAL) 或 Local WAL (在本地 WAL 上启用 Region Failover ,在 Failover 过程中可能会导致数据丢失) +- 使用 Kafka WAL (Remote WAL),或者使用 Local WAL 并设置 `allow_region_failover_on_local_wal=true`(在 Local WAL 上启用 Region Failover,Failover 过程中可能会导致数据丢失) - 使用[共享存储](/user-guide/deployments-administration/configuration.md#storage-options) (例如:AWS S3) 如果想要在本地 WAL 上启用 Region Failover,需要设置 `allow_region_failover_on_local_wal=true` 在 [metasrv](/user-guide/deployments-administration/configuration.md#metasrv-only-configuration) 配置文件中。不建议启用此选项,因为它可能会导致数据丢失。 @@ -28,7 +28,7 @@ Region Failover 提供了在不丢失数据的情况下从 Region 故障中恢 ## Region Failover 的恢复用时 -Region Failover 的恢复时间取决于: +使用 Kafka WAL (Remote WAL) 时,Region Failover 的恢复时间取决于: - 每个 Topic 的 region 数量 - Kafka 集群的读取吞吐性能 @@ -49,14 +49,14 @@ Region Failover 的恢复时间取决于:

(图 1:恢复 Region 3 需要读取比实际大小大 7 倍的冗余数据)

-估算读取放大倍数(重放数据大小/实际数据大小)的简单模型: +估算冗余读取放大倍数(冗余重放数据大小/实际数据大小)的简单模型: - 对于单个 Topic,如果我们尝试重放属于该 Topic 的所有 Region,那么放大倍数将是 7+6+...+1 = 28 倍。(图 1 显示了 Region WAL 数据分布。重放 Region 3 将读取约为实际大小 7 倍的数据;重放 Region 6 将读取约为实际大小 6 倍的数据,以此类推) - 在恢复 100 个 Region 时(需要大约 13 个 Topic),放大倍数大约为 28 \* 13 = 364 倍。 假设要恢复 100 个 Region,所有 Region 的实际数据大小是 0.5 GB,下表根据每个 Topic 的 Region 数量展示了数据重放的总量。 -| 每个 Topic 的 Region 数量 | 100 个 Region 所需 Topic 数量 | 单个 Topic 读放大系数 | 总读放大系数 | 重放数据大小(GB) | +| 每个 Topic 的 Region 数量 | 100 个 Region 所需 Topic 数量 | 单个 Topic 冗余读取系数 | 总冗余读取系数 | 重放数据大小(GB) | | ------------------------- | ----------------------------- | --------------------- | ------------ | ------------------ | | 1 | 100 | 0 | 0 | 0.5 | | 2 | 50 | 1 | 50 | 25.5 | @@ -79,4 +79,4 @@ Region Failover 的恢复时间取决于: 在上文中我们根据不同的每个 Topic 包含的 Region 数量计算了恢复时间以供参考。 在实际场景中,读取放大的现象可能会比这个模型更为严重。 -如果您对恢复时间非常敏感,我们建议每个 Region 都有自己的 Topic(即,每个 Topic 包含的 Region 数量为 1)。 +如果使用 Kafka WAL 且对恢复时间非常敏感,我们建议每个 Region 都有自己的 Topic(即,每个 Topic 包含的 Region 数量为 1)。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-1.0/user-guide/deployments-administration/configuration.md b/i18n/zh/docusaurus-plugin-content-docs/version-1.0/user-guide/deployments-administration/configuration.md index 7f7e9c7fb..5680ab7e2 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-1.0/user-guide/deployments-administration/configuration.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-1.0/user-guide/deployments-administration/configuration.md @@ -596,9 +596,9 @@ selector = "lease_based" # 将数据存储在内存中,默认值为 false。 use_memory_store = false # 是否启用 region failover。 -# 该功能仅适用于运行在集群模式下的 GreptimeDB,并且 -# - 使用 Remote WAL -# - 使用共享存储(例如 s3)。 +# 该功能仅适用于以集群模式运行并使用共享存储(例如 s3)的 GreptimeDB,并且 WAL 需要满足以下条件之一: +# - Remote WAL +# - Local WAL 且设置 `allow_region_failover_on_local_wal = true`(故障转移期间可能导致数据丢失) enable_region_failover = false ## 设置启动 region 故障检测的延迟时间。 ## 该延迟有助于避免在所有 Datanode 尚未完全启动时,Metasrv 过早启动 region 故障检测,从而导致不必要的 region failover。 @@ -728,7 +728,7 @@ timeout = "3s" | `store_addrs` | Array | `["127.0.0.1:2379"]` | 元数据服务地址,默认值为 `["127.0.0.1:2379"]`。支持配置多个服务地址,格式为 `["ip1:port1","ip2:port2",...]`。默认使用 Etcd 作为元数据后端。
根据你的存储服务器类型配置地址,例如:
- 使用 `"127.0.0.1:2379"` 连接到 etcd
- 使用 `"password=password dbname=postgres user=postgres host=localhost port=5432"` 连接到 postgres
- 使用 `"mysql://user:password@ip:port/dbname"` 连接到 mysql | | `selector` | String | `lease_based` | 创建新表时选择 datanode 的负载均衡策略,详见 [选择器](/contributor-guide/metasrv/selector.md)。 | | `use_memory_store` | Boolean | `false` | 仅用于在没有 etcd 集群时的测试,将数据存储在内存中,默认值为 `false`。 | -| `enable_region_failover` | Bool | `false` | 是否启用 region failover。
该功能仅在以集群模式运行的 GreptimeDB 上可用,并且
- 使用远程 WAL
- 使用共享存储(如 s3)。 | +| `enable_region_failover` | Bool | `false` | 是否启用 region failover。
该功能仅适用于以集群模式运行并使用共享存储(如 s3)的 GreptimeDB,并且 WAL 需要满足以下条件之一:
- Remote WAL
- Local WAL 且设置 `allow_region_failover_on_local_wal = true`(故障转移期间可能导致数据丢失)。 | | `region_failure_detector_initialization_delay` | String | `10m` | 设置启动 region 故障检测的延迟时间。该延迟有助于避免在所有 Datanode 尚未完全启动时,Metasrv 过早启动 region 故障检测,从而导致不必要的 region failover。尤其适用于未通过 GreptimeDB Operator 部署的集群,此时可能未正确启用集群维护模式,提前检测可能会引发误判。 | | `allow_region_failover_on_local_wal` | Bool | false | 是否允许在本地 WAL 上进行 region failover。
**此选项不建议设置为 true,因为这可能会在故障转移期间导致数据丢失。** | | `node_max_idle_time` | String | `24hours` | 从 metasrv 内存中删除节点信息前允许的最大空闲时间。超过该时间未发送心跳的节点将被视为不活跃并被删除。 | diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-1.0/user-guide/deployments-administration/manage-data/region-failover.md b/i18n/zh/docusaurus-plugin-content-docs/version-1.0/user-guide/deployments-administration/manage-data/region-failover.md index aaa9180bb..563a017fc 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-1.0/user-guide/deployments-administration/manage-data/region-failover.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-1.0/user-guide/deployments-administration/manage-data/region-failover.md @@ -1,18 +1,18 @@ --- keywords: [Region Failover, 故障恢复, 恢复时间, 共享存储, Kafka WAL] -description: 介绍 Region Failover 功能及其在 GreptimeDB 中的应用,包括开启 Region Failover、恢复时间和改进恢复时间的建议等内容。 +description: 介绍 Region Failover 功能及其在 GreptimeDB 中的应用,包括开启 Region Failover、WAL 相关的数据丢失风险、恢复时间和改进恢复时间的建议等内容。 --- # Region Failover -Region Failover 提供了在不丢失数据的情况下从 Region 故障中恢复的能力。这是通过 [Region 迁移](/user-guide/deployments-administration/manage-data/region-migration.md) 实现的。 +Region Failover 提供了从 Region 故障中恢复的能力。使用 Kafka WAL (Remote WAL) 和共享存储时,Region Failover 可以在不丢失数据的情况下恢复 Region。这是通过 [Region 迁移](/user-guide/deployments-administration/manage-data/region-migration.md) 实现的。 ## 开启 Region Failover 该功能仅在 GreptimeDB 集群模式下可用,并且需要满足以下条件 -- 使用 Kafka WAL (Remote WAL) 或 Local WAL (在本地 WAL 上启用 Region Failover ,在 Failover 过程中可能会导致数据丢失) +- 使用 Kafka WAL (Remote WAL),或者使用 Local WAL 并设置 `allow_region_failover_on_local_wal=true`(在 Local WAL 上启用 Region Failover,Failover 过程中可能会导致数据丢失) - 使用[共享存储](/user-guide/deployments-administration/configuration.md#storage-options) (例如:AWS S3) 如果想要在本地 WAL 上启用 Region Failover,需要设置 `allow_region_failover_on_local_wal=true` 在 [metasrv](/user-guide/deployments-administration/configuration.md#metasrv-only-configuration) 配置文件中。不建议启用此选项,因为它可能会导致数据丢失。 @@ -28,7 +28,7 @@ Region Failover 提供了在不丢失数据的情况下从 Region 故障中恢 ## Region Failover 的恢复用时 -Region Failover 的恢复时间取决于: +使用 Kafka WAL (Remote WAL) 时,Region Failover 的恢复时间取决于: - 每个 Topic 的 region 数量 - Kafka 集群的读取吞吐性能 @@ -49,14 +49,14 @@ Region Failover 的恢复时间取决于:

(图 1:恢复 Region 3 需要读取比实际大小大 7 倍的冗余数据)

-估算读取放大倍数(重放数据大小/实际数据大小)的简单模型: +估算冗余读取放大倍数(冗余重放数据大小/实际数据大小)的简单模型: - 对于单个 Topic,如果我们尝试重放属于该 Topic 的所有 Region,那么放大倍数将是 7+6+...+1 = 28 倍。(图 1 显示了 Region WAL 数据分布。重放 Region 3 将读取约为实际大小 7 倍的数据;重放 Region 6 将读取约为实际大小 6 倍的数据,以此类推) - 在恢复 100 个 Region 时(需要大约 13 个 Topic),放大倍数大约为 28 \* 13 = 364 倍。 假设要恢复 100 个 Region,所有 Region 的实际数据大小是 0.5 GB,下表根据每个 Topic 的 Region 数量展示了数据重放的总量。 -| 每个 Topic 的 Region 数量 | 100 个 Region 所需 Topic 数量 | 单个 Topic 读放大系数 | 总读放大系数 | 重放数据大小(GB) | +| 每个 Topic 的 Region 数量 | 100 个 Region 所需 Topic 数量 | 单个 Topic 冗余读取系数 | 总冗余读取系数 | 重放数据大小(GB) | | ------------------------- | ----------------------------- | --------------------- | ------------ | ------------------ | | 1 | 100 | 0 | 0 | 0.5 | | 2 | 50 | 1 | 50 | 25.5 | @@ -79,4 +79,4 @@ Region Failover 的恢复时间取决于: 在上文中我们根据不同的每个 Topic 包含的 Region 数量计算了恢复时间以供参考。 在实际场景中,读取放大的现象可能会比这个模型更为严重。 -如果您对恢复时间非常敏感,我们建议每个 Region 都有自己的 Topic(即,每个 Topic 包含的 Region 数量为 1)。 +如果使用 Kafka WAL 且对恢复时间非常敏感,我们建议每个 Region 都有自己的 Topic(即,每个 Topic 包含的 Region 数量为 1)。 diff --git a/versioned_docs/version-1.0/user-guide/deployments-administration/configuration.md b/versioned_docs/version-1.0/user-guide/deployments-administration/configuration.md index aa91c5b89..8b08d2880 100644 --- a/versioned_docs/version-1.0/user-guide/deployments-administration/configuration.md +++ b/versioned_docs/version-1.0/user-guide/deployments-administration/configuration.md @@ -604,9 +604,9 @@ selector = "round_robin" # Store data in memory, false by default. use_memory_store = false # Whether to enable region failover. -# This feature is only available on GreptimeDB running on cluster mode and -# - Using Remote WAL -# - Using shared storage (e.g., s3). +# This feature is only available on GreptimeDB running on cluster mode with shared storage (e.g., s3) and one of: +# - Remote WAL +# - Local WAL with `allow_region_failover_on_local_wal = true` (may lead to data loss during failover) enable_region_failover = false ## The delay before starting region failure detection. ## This delay helps prevent Metasrv from triggering unnecessary region failovers before all Datanodes are fully started. @@ -754,7 +754,7 @@ timeout = "3s" | `store_addrs` | Array | `["127.0.0.1:2379"]` | Store server address. Configure the address based on your backend type, for example:
- Use `"127.0.0.1:2379"` to connect to etcd
- Use `"password=password dbname=postgres user=postgres host=localhost port=5432"` to connect to postgres
- Use `"mysql://user:password@ip:port/dbname"` to connect to mysql | | `selector` | String | `lease_based` | Datanode selector type.
- `lease_based` (default value).
- `load_based`
For details, see [Selector](/contributor-guide/metasrv/selector.md) | | `use_memory_store` | Bool | `false` | Store data in memory. | -| `enable_region_failover` | Bool | `false` | Whether to enable region failover.
This feature is only available on GreptimeDB running on cluster mode and
- Using Remote WAL
- Using shared storage (e.g., s3). | +| `enable_region_failover` | Bool | `false` | Whether to enable region failover.
This feature is only available on GreptimeDB running on cluster mode with shared storage (e.g., s3) and one of:
- Remote WAL
- Local WAL with `allow_region_failover_on_local_wal = true` (may lead to data loss during failover). | | `region_failure_detector_initialization_delay` | String | `10m` | The delay before starting region failure detection. This delay helps prevent Metasrv from triggering unnecessary region failovers before all Datanodes are fully started. Especially useful when the cluster is not deployed with GreptimeDB Operator and maintenance mode is not enabled. | | `allow_region_failover_on_local_wal` | Bool | `false` | Whether to allow region failover on local WAL.
**This option is not recommended to be set to true, because it may lead to data loss during failover.** | | `node_max_idle_time` | String | `24hours` | Max allowed idle time before removing node info from metasrv memory. Nodes that haven't sent heartbeats for this duration will be considered inactive and removed. | diff --git a/versioned_docs/version-1.0/user-guide/deployments-administration/manage-data/region-failover.md b/versioned_docs/version-1.0/user-guide/deployments-administration/manage-data/region-failover.md index 4f5e00b00..8421ac78a 100644 --- a/versioned_docs/version-1.0/user-guide/deployments-administration/manage-data/region-failover.md +++ b/versioned_docs/version-1.0/user-guide/deployments-administration/manage-data/region-failover.md @@ -1,17 +1,17 @@ --- keywords: [region failover, GreptimeDB, data recovery, Kafka WAL, read amplification, distributed mode, region migration, recovery time, shared storage, configuration] -description: Covers enabling and understanding region failover in GreptimeDB, which allows for the recovery of regions from failures without data loss, and discusses the impact of read amplification on recovery time. +description: Covers enabling and understanding region failover in GreptimeDB, including prerequisites, WAL-related data loss risks, and the impact of read amplification on recovery time with Kafka WAL. --- # Region Failover -Region Failover provides the ability to recover regions from region failures without losing data. This is implemented via [Region Migration](/user-guide/deployments-administration/manage-data/region-migration.md). +Region Failover provides the ability to recover regions from region failures. With Kafka WAL (Remote WAL) and shared storage, Region Failover can recover regions without losing data. This is implemented via [Region Migration](/user-guide/deployments-administration/manage-data/region-migration.md). ## Enable the Region Failover This feature is only available on GreptimeDB running on distributed mode and -- Using Kafka WAL (Remote WAL) or Local WAL (Enable region failover on local WAL may lead to data loss during failover) +- Using Kafka WAL (Remote WAL) or Local WAL with `allow_region_failover_on_local_wal=true` (enabling region failover on Local WAL may lead to data loss during failover) - Using [shared storage](/user-guide/deployments-administration/configuration.md#storage-options) (e.g., AWS S3) If you want to enable region failover on local WAL, you need to set `allow_region_failover_on_local_wal=true` in [metasrv](/user-guide/deployments-administration/configuration.md#metasrv-only-configuration) configuration file. It's not recommended to enable this option, because it may lead to data loss. @@ -25,7 +25,7 @@ To enable region failover via GreptimeDB Operator, you can refer to [Common Helm ## The recovery time of Region Failover -The recovery time of Region Failover depends on: +The recovery time of Region Failover with Kafka WAL (Remote WAL) depends on: - number of regions per Topic. - the Kafka cluster read throughput performance. @@ -47,14 +47,14 @@ For example, configure 128 topics for [metasrv](/user-guide/deployments-administ

(Figure1: recovery Region 3 need to read redundant data 7 times larger than the actual size)

-A simple model to estimate the read amplification factor (replay data size/actual data size): +A simple model to estimate the redundant read amplification factor (redundant replay data size/actual data size): - For a single topic, if we try to replay all regions that belong to the topic, then the amplification factor would be 7+6+...+1 = 28 times. (The Region WAL data distribution is shown in the Figure 1. Replaying Region 3 will read 7 times redundant data larger than the actual size; Region 6 will read 6 times, and so on) - When recovering 100 regions (requiring about 13 topics), the amplification factor is approximately 28 \* 13 = 364 times. Assuming we have 100 regions to recover, and the actual data size of all regions is 0.5GB, the following table shows the replay data size based on the number of regions per topic. -| Number of regions per Topic | Number of topics required for 100 Regions | Single topic read amplification factor | Total reading amplification factor | Replay data size (GB) | +| Number of regions per Topic | Number of topics required for 100 Regions | Single topic redundant read factor | Total redundant read factor | Replay data size (GB) | | --------------------------- | ----------------------------------------- | -------------------------------------- | ---------------------------------- | ---------------- | | 1 | 100 | 0 | 0 | 0.5 | | 2 | 50 | 1 | 50 | 25.5 | @@ -79,5 +79,4 @@ The following table shows the recovery time of 100 regions under different read In the above example, we calculated the recovery time based on the number of Regions contained in each Topic for reference. We have calculated the recovery time under different Number of regions per Topic configuration for reference. In actual scenarios, the read amplification may be larger than this model. -If you are very sensitive to recovery time, we recommend that each region have its topic(i.e., Number of regions per Topic is 1). - +If you use Kafka WAL and are very sensitive to recovery time, we recommend that each region have its own topic (i.e., Number of regions per Topic is 1).