diff --git a/docs/whpg/7x/admin_guide/performance/wlmgmt/about-resgroups-changes.md b/docs/whpg/7x/admin_guide/performance/wlmgmt/about-resgroups-changes.md index 2e0dff1..daa6cd8 100644 --- a/docs/whpg/7x/admin_guide/performance/wlmgmt/about-resgroups-changes.md +++ b/docs/whpg/7x/admin_guide/performance/wlmgmt/about-resgroups-changes.md @@ -11,7 +11,7 @@ The following table compares the main concepts of resource management and how ea | ------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Concurrency | Managed at the transaction level | Managed at the transaction level | | CPU | Specify percentage of CPU resources or the number of CPU cores; uses Linux Control Groups | Specify percentage of CPU resources or the number of CPU cores, as well as set upper limits per resource group; uses Linux Control Groups | -| Memory | Managed at the transaction level, with enhanced allocation and tracking; users cannot over-subscribe | Managed at the transaction level, with enhanced allocation and tracking; users can over-subscribe; simpler and more convenient configuration | +| Memory | Managed at the transaction level, with enhanced allocation and tracking; users cannot over-subscribe | `MEMORY_QUOTA` and `CONCURRENCY` shape a query's planned memory allocation only, and WarehousePG doesn't enforce a per-resource-group memory limit at the segment level; actual memory usage is governed by the shared per-segment pool that `gp_vmem_protect_limit` defines, so users can over-subscribe | | Disk I/O | None | Limit the maximum read/write disk I/O throughput, and maximum read/write I/O operations per second | | Users | Limits are applied to `SUPERUSER` and non-admin users alike
There are two default resource groups: `admin_group` and `default_group` | Limits are applied to `SUPERUSER`, non-admin users, and system processes of non-user classes
There are three default resource groups: `admin_group`, `default_group`, and `system_group` | | Queueing | Queue when no slot is available or not enough available memory | Queue when no slot is available | @@ -19,7 +19,13 @@ The following table compares the main concepts of resource management and how ea | Limit Bypass | Limits are not enforced on `SET`, `RESET`, and `SHOW` commands | Limits are not enforced on `SET`, `RESET`, and `SHOW` commands. Additionally, certain queries may be configured to bypass the concurrency limit | | External Components | Manage PL/Container CPU and memory resources | Manage PL/Container CPU resources | -> **Note** Disk I/O limits are only available when you use Linux Control Groups v2. See [Configuring and Using Resource Groups](workload_mgmt_resgroups.md#topic71717999) for more information. +::: info Note +Disk I/O limits are only available when you use Linux Control Groups v2. See [Configuring and Using Resource Groups](workload_mgmt_resgroups.md#topic71717999) for more information. +::: + +::: info Note +WarehousePG 7 removed the direct per-group memory attributes that WarehousePG 6 used (`MEMORY_LIMIT`, `MEMORY_SHARED_QUOTA`, and `MEMORY_SPILL_RATIO`), and memory management is delegated to `gp_vmem_protect_limit` at the segment level. See [Memory Limits](workload_mgmt_resgroups.md#topic8339717) for more information. +::: diff --git a/docs/whpg/7x/admin_guide/performance/wlmgmt/index.md b/docs/whpg/7x/admin_guide/performance/wlmgmt/index.md index 0e20c5d..8e92cfc 100644 --- a/docs/whpg/7x/admin_guide/performance/wlmgmt/index.md +++ b/docs/whpg/7x/admin_guide/performance/wlmgmt/index.md @@ -19,7 +19,7 @@ The following table summarizes some of the differences between resource queues a | ------------------- | ----------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------- | | Concurrency | Defines the number of query slots available at a time | Defines the number of transaction slots available at a time | | CPU | Specify query priority | Specify percentage of CPU resources or the specific number of CPU cores; uses Linux Control Groups | -| Memory | Managed at the queue and operator level; users can over-subscribe | Managed at the transaction level, with enhanced allocation and tracking; users can over-subscribe | +| Memory | Managed at the queue and operator level; users can over-subscribe | `MEMORY_QUOTA` and `CONCURRENCY` shape a query's planned memory allocation only; actual memory is enforced at the segment level by `gp_vmem_protect_limit`, so users can over-subscribe | | Users | Limits are applied only to non-admin users | Limits are applied to `SUPERUSER`, non-admin users, and system processes of non-user classes | | Disk I/O | None | Limit the maximum read/write disk I/O throughput, and maximum read/write I/O operations per second | | Queueing | Queue when no slot available or not enough available memory | Queue only when no slot is available | @@ -27,4 +27,11 @@ The following table summarizes some of the differences between resource queues a | Limit Bypass | Limits are not enforced for `SUPERUSER` roles and certain operators and functions | Limits are not enforced on `SET`, `RESET`, and `SHOW` commands. Additionally, certain queries may be configured to bypass the concurrency limit | | External Components | None | Manage PL/Container CPU resources | +- **[About Changes to Resource Groups in WarehousePG 7](about-resgroups-changes.md)** + Compares resource group-based resource management between WarehousePG 6 and WarehousePG 7. +- **[Using Resource Groups](workload_mgmt_resgroups.md)** + Configuring and using resource groups to manage CPU, memory, concurrency, and disk I/O. +- **[Using Resource Queues](workload_mgmt.md)** + Configuring and using resource queues to prioritize and allocate resources to queries. + **Parent topic:** [Managing Performance](../index.md) diff --git a/docs/whpg/7x/admin_guide/performance/wlmgmt/workload_mgmt_resgroups.md b/docs/whpg/7x/admin_guide/performance/wlmgmt/workload_mgmt_resgroups.md index bb926de..88c3428 100644 --- a/docs/whpg/7x/admin_guide/performance/wlmgmt/workload_mgmt_resgroups.md +++ b/docs/whpg/7x/admin_guide/performance/wlmgmt/workload_mgmt_resgroups.md @@ -11,6 +11,8 @@ WarehousePG uses Linux-based control groups for CPU resource management, and Run When using resource groups to control resources like CPU cores, review the Hyperthreading note in [Hardware and Network](../../../install_guide/platform-requirements.md#hardware-requirements). +For guidance on configuring resource groups to manage memory effectively, see [Memory and Resource Management with Resource Groups](../../../best_practices/resgroups.md) in the WarehousePG best practices guide. + **Parent topic:** [Managing Resources](index.md) @@ -40,10 +42,12 @@ When you create a resource group, you provide a set of limits that determine the | CPU_WEIGHT | The scheduling priority of the resource group. | [1 - 500] | 100 | | CPUSET | The specific CPU logical core (or logical thread in hyperthreading) reserved for this resource group. | It depends on system core configuration | -1 | | IO_LIMIT | The limit for the maximum read/write disk I/O throughput, and maximum read/write I/O operations per second. Set the value on a per-tablespace basis. | [2 - 4294967295 or `max`] | -1 | -| MEMORY_QUOTA | The memory limit value specified for the resource group. | Integer (MB) | -1 (not set, use `statement_mem` as the memory limit for a single query) | +| MEMORY_QUOTA | The memory value used, together with `CONCURRENCY`, to calculate a query's planned memory allocation for the resource group. `MEMORY_QUOTA` doesn't cap the memory that the resource group's queries can consume on a segment. | Integer (MB) | -1 (not set, use `statement_mem` as the memory limit for a single query) | | MIN_COST | The minimum cost of a query plan to be included in the resource group. | Integer | 0 | -> **Note** Resource limits are not enforced on `SET`, `RESET`, and `SHOW` commands. +::: info Note +Resource limits are not enforced on `SET`, `RESET`, and `SHOW` commands. +::: @@ -84,9 +88,13 @@ When you set one of the allocation modes for a resource group, WarehousePG deact WarehousePG uses the server configuration parameter [gp_resource_group_cpu_limit](../../../ref_guide/config_params/guc-list.md#gp_resource_group_cpu_limit) to identify the maximum percentage of system CPU resources to allocate to resource groups on each WarehousePG segment node. The remaining unreserved CPU resources are used for the operating system kernel and WarehousePG daemons. The amount of CPU available to WarehousePG queries per host is then divided equally among each segment on the WarehousePG node. -> **Note** The default `gp_resource_group_cpu_limit` value may not leave sufficient CPU resources if you are running other workloads on your WarehousePG cluster nodes, so be sure to adjust this server configuration parameter accordingly. +::: info Note +The default `gp_resource_group_cpu_limit` value may not leave sufficient CPU resources if you are running other workloads on your WarehousePG cluster nodes, so be sure to adjust this server configuration parameter accordingly. +::: -> **Caution** Avoid setting `gp_resource_group_cpu_limit` to a value higher than .9. Doing so may result in high workload queries taking near all CPU resources, potentially starving WarehousePG auxiliary processes. +::: warning Caution +Avoid setting `gp_resource_group_cpu_limit` to a value higher than .9. Doing so may result in high workload queries taking near all CPU resources, potentially starving WarehousePG auxiliary processes. +::: @@ -106,7 +114,9 @@ When you assign CPU cores to `CPUSET` groups, consider the following: Resource groups that you configure with `CPUSET` have a higher priority on CPU resources. The maximum CPU resource usage percentage for all resource groups configured with `CPUSET` on a segment host is the number of CPU cores reserved divided by the number of all CPU cores, multiplied by 100. -> **Note** You must configure `CPUSET` for a resource group *after* you have enabled resource group-based resource management for your WarehousePG cluster with the [gp_resource_manager](../../../ref_guide/config_params/guc-list.md#gp_resource_manager) server configuration parameter. +::: info Note +You must configure `CPUSET` for a resource group *after* you have enabled resource group-based resource management for your WarehousePG cluster with the [gp_resource_manager](../../../ref_guide/config_params/guc-list.md#gp_resource_manager) server configuration parameter. +::: @@ -141,38 +151,25 @@ Roles in `test` have a CPU ratio of 10/(10+30+10+10)=16%. However, as the hard l ### Memory Limits -When you enable resource groups, memory usage is managed at the WarehousePG segment and resource group levels. You can also manage memory at the transaction level. See [WarehousePG Memory Overview](../wlmgmt_intro.md) to estimate how much memory each WarehousePG segment has available to use. This will help you estimate how much memory to assign to the resource groups. - -The amount of memory allocated to a query is determined by the following parameters: +When you enable resource groups, `MEMORY_QUOTA` and `CONCURRENCY` shape a query's planned memory allocation, but WarehousePG doesn't enforce a per-resource-group memory limit at the segment level. Instead, all queries on a segment, regardless of resource group membership, draw from a single shared memory pool for that segment. See [WarehousePG Memory Overview](../wlmgmt_intro.md) to estimate how much memory each WarehousePG segment has available to use. This helps you estimate how much memory to assign to the resource groups. -The parameter `MEMORY_QUOTA` of a resource group sets the maximum amount of memory reserved for this resource group on a segment. This determines the total amount of memory that all worker processes for a query can consume on the segment host during query execution. The amount of memory allotted to a query is the group memory limit divided by the group concurrency limit: `MEMORY_QUOTA` / `CONCURRENCY`. +The amount of memory a query plans to use is determined by the following parameters, in order of precedence: -If a query requires a large amount of memory, you may use the server configuration parameter [gp_resgroup_memory_query_fixed_mem](../../../ref_guide/config_params/guc-list.md#gp_resgroup_memory_query_fixed_mem) to set a fixed memory amount for the query at the session level. This parameter overrides and can surpass the allocated memory of the resource group. +- [gp_resgroup_memory_query_fixed_mem](../../../ref_guide/config_params/guc-list.md#gp_resgroup_memory_query_fixed_mem): When you set this server configuration parameter at the session level, WarehousePG uses it directly as the query's planned memory allocation, bypassing the resource group's `MEMORY_QUOTA` and `CONCURRENCY` settings. Use it when a query requires more memory than its resource group would otherwise plan for. +- `MEMORY_QUOTA` / `CONCURRENCY`: When you don't set `gp_resgroup_memory_query_fixed_mem`, WarehousePG divides the resource group's `MEMORY_QUOTA` by its `CONCURRENCY` limit to determine the query's planned memory allocation at the planning phase. `MEMORY_QUOTA` doesn't reserve or cap memory for the resource group on a segment. It's an input to the query planner, not an enforced limit on the worker processes that execute the query. +- [statement_mem](../../../ref_guide/config_params/guc-list.md#statement_mem): When the resource group doesn't have a `MEMORY_QUOTA` set, the planned allocation defaults to `statement_mem`. -WarehousePG allocates memory for an incoming query using the `gp_resgroup_memory_query_fixed_mem` value, if set, to bypass the resource group settings. Otherwise, it uses `MEMORY_QUOTA` / `CONCURRENCY` as the memory allocated for the query. If `MEMORY_QUOTA` is not set, the value for the query memory allocation defaults to [statement_mem](../../../ref_guide/config_params/guc-list.md#statement_mem). +Actual memory enforcement happens at the segment level, not through these planning parameters. The [gp_vmem_protect_limit](../../../ref_guide/config_params/guc-list.md#gp_vmem_protect_limit) parameter sets the size of the memory pool shared by all queries on a segment, and the [runaway_detector_activation_percent](../../../ref_guide/config_params/guc-list.md#runaway_detector_activation_percent) parameter sets the percentage of that pool WarehousePG allows before it cancels queries. This tracking is reactive: WarehousePG accounts for memory after a query has already allocated it, so a query can briefly exceed its planned allocation before the Runaway Detector cancels it. When segment memory usage exceeds `gp_vmem_protect_limit` \* `runaway_detector_activation_percent`, WarehousePG cancels queries starting with the one that consumes the most memory, until usage falls back below the threshold. This behavior is the same whether your cluster uses cgroup v1 or cgroup v2 for resource groups. For all queries, if there is not enough memory in the system, they spill to disk. When the limit [gp_workfile_limit_files_per_query](../../../ref_guide/config_params/guc-list.md#gp_workfile_limit_files_per_query) is reached, WarehousePG generates an out of memory (OOM) error. -For example, consider a resource group named `adhoc` with `MEMORY_QUOTA`set to 1.5 GB and `CONCURRENCY` set to 3. By default, each statement submitted to the group is allocated 500 MB of memory. Now consider the following series of events: - -1. User `ADHOC_1` submits query `Q1`, overriding `gp_resgroup_memory_query_fixed_mem` to 800MB. The `Q1` statement is admitted into the system. - -2. User `ADHOC_2` submits query `Q2`, using the default 500MB. - -3. With `Q1` and `Q2` still running, user `ADHOC3` submits query `Q3`, using the default 500MB. - - Queries `Q1` and `Q2` have used 1300MB of the group's 1500MB. However, if there is enough system memory available for query `Q3` in the segment at that time, it can run normally. - -4. User `ADHOC4` submits query `Q4`, using `gp_resgroup_memory_query_fixed_mem` set to 700 MB. - - Query `Q4` runs immediately as it bypasses the resource group limits. - There are some special usage considerations regarding memory limits: -- If you set the configuration parameters `gp_resource_group_bypass` or `gp_resource_group_bypass_catalog_query` to bypass the resource group limits, the memory limit for the query takes the value of `statement_mem`. -- When (`MEMORY_QUOTA` / `CONCURRENCY`) < `statement_mem`, WarehousePG uses `statement_mem` as the fixed amount of memory allocated by query. +- Because memory is enforced at the segment level rather than per resource group, a resource group's queries can consume more memory than `MEMORY_QUOTA` suggests, as long as the shared per-segment pool has capacity. +- If you set the configuration parameters `gp_resource_group_bypass` or `gp_resource_group_bypass_catalog_query` to bypass the resource group limits, the planned memory allocation for the query takes the value of `statement_mem`. +- When (`MEMORY_QUOTA` / `CONCURRENCY`) < `statement_mem`, WarehousePG uses `statement_mem` as the fixed amount of planned memory for the query. - The maximum value of `statement_mem` is capped at [max_statement_mem](../../../ref_guide/config_params/guc-list.md#max_statement_mem). -- Queries whose plan cost is less than the limit `MIN_COST` use a memory quota of `statement_mem`. +- Queries whose plan cost is less than the limit `MIN_COST` use a planned memory quota of `statement_mem`. @@ -180,7 +177,9 @@ There are some special usage considerations regarding memory limits: WarehousePG leverages Linux control groups to implement disk I/O limits. The parameter `IO_LIMIT` limits the maximum read/write disk I/O throughput, and the maximum read/write I/O operations per second for the queries assigned to a specific resource group. It allocates bandwidth, ensures the use of high-priority resource groups, and avoids excessive use of disk bandwidth. The value of the parameter is set on a per-tablespace basis. -> **Note** Disk I/O limits are only available when you use Linux Control Groups v2. See [Configuring and Using Resource Groups](#topic71717999) for more information. +::: info Note +Disk I/O limits are only available when you use Linux Control Groups v2. See [Configuring and Using Resource Groups](#topic71717999) for more information. +::: When you limit disk I/O you specify: @@ -429,7 +428,9 @@ ALTER RESOURCE GROUP rgroup1 SET CPUSET '1;2,4'; ALTER RESOURCE GROUP sales SET IO_LIMIT 'tablespace1:wbps=2000,wiops=2000;tablespace2:rbps=2024,riops=2024'; ``` -> **Note** You cannot set or alter the `CONCURRENCY` value for the `admin_group` to zero (0). +::: info Note +You cannot set or alter the `CONCURRENCY` value for the `admin_group` to zero (0). +::: You can rename a resource group using the `name` option: @@ -584,7 +585,9 @@ SELECT pg_cancel_backend(31905); You can provide an optional message in a second argument to `pg_cancel_backend()` to indicate to the user why the process was cancelled. -> **Note** Do not use an operating system `KILL` command to cancel any WarehousePG process. +::: info Note +Do not use an operating system `KILL` command to cancel any WarehousePG process. +::: @@ -592,7 +595,9 @@ You can provide an optional message in a second argument to `pg_cancel_backend() A user with WarehousePG superuser privileges can run the `gp_toolkit.pg_resgroup_move_query()` function to move a running query from one resource group to another, without stopping the query. Use this function to expedite a long-running query by moving it to a resource group with a higher resource allotment or availability. -> **Note** You can move only an active or running query to a new resource group. You cannot move a queued or pending query that is in an idle state due to concurrency or memory limits. +::: info Note +You can move only an active or running query to a new resource group. You cannot move a queued or pending query that is in an idle state due to concurrency or memory limits. +::: `pg_resgroup_move_query()` requires the process id (pid) of the running query, as well as the name of the resource group to which you want to move the query. The signature of the function follows: diff --git a/docs/whpg/7x/best_practices/resgroups.md b/docs/whpg/7x/best_practices/resgroups.md index 9bd86db..476f529 100644 --- a/docs/whpg/7x/best_practices/resgroups.md +++ b/docs/whpg/7x/best_practices/resgroups.md @@ -7,6 +7,8 @@ Managing WarehousePG resources with resource groups. Memory, CPU, and concurrent transaction management have a significant impact on performance in a WarehousePG cluster. Resource groups are a newer resource management scheme that enforce memory, CPU, and concurrent transaction limits in WarehousePG. +For conceptual information about configuring and using resource groups, see [Using Resource Groups](../admin_guide/performance/wlmgmt/workload_mgmt_resgroups.md). + - [Configuring Memory for WarehousePG](#section_r52_rbl_zt) - [Memory Considerations when using Resource Groups](#toolowmem) - [Configuring Resource Groups](#configuring_rg) @@ -64,8 +66,8 @@ WarehousePG resource groups provide a powerful mechanism for managing the worklo - A transaction submitted by any WarehousePG role with `SUPERUSER` privileges runs under the default resource group named `admin_group`. Keep this in mind when scheduling and running WarehousePG administration utilities. - Ensure that you assign each non-admin role a resource group. If you do not assign a resource group to a role, queries submitted by the role are handled by the default resource group named `default_group`. - Use the `CONCURRENCY` resource group parameter to limit the number of active queries that members of a particular resource group can run concurrently. -- Use the `MEMORY_QUOTA` parameter to control the maximum amount of memory that queries running in the resource group can consume. -- WarehousePG assigns unreserved memory (100 - (sum of all resource group `MEMORY_QUOTA`s) to a global shared memory pool. This memory is available to all queries on a first-come, first-served basis. +- Use the `MEMORY_QUOTA` parameter, together with `CONCURRENCY`, to size a query's planned memory allocation during query planning. `MEMORY_QUOTA` doesn't cap the memory that a resource group's queries can consume on a segment. +- All queries on a segment, regardless of resource group, draw memory from a single shared per-segment memory pool. The [gp_vmem_protect_limit](../ref_guide/config_params/guc-list.md#gp_vmem_protect_limit) parameter controls the size of this pool, and the Runaway Detector cancels queries when segment memory usage exceeds `gp_vmem_protect_limit` \* `runaway_detector_activation_percent`. - Alter resource groups dynamically to match the real requirements of the group for the workload and the time of day. - Use the `gp_toolkit` views to examine resource group resource usage and to monitor how the groups are working. @@ -87,6 +89,8 @@ The default resource group for database transactions initiated by WarehousePG `S Certain WarehousePG administrative utilities may use more than one `CONCURRENCY` slot at runtime, such as `gpbackup` that you invoke with the `--jobs` option. If the utility(s) you run require more concurrent transactions than that configured for `admin_group`, consider temporarily increasing the group's `MEMORY_QUOTA` and `CONCURRENCY` values to meet the utility's requirement, making sure to return these parameters back to their original settings when the utility completes. -> **Note** Memory allocation changes that you initiate with `ALTER RESOURCE GROUP` may not take affect immediately due to resource consumption by currently running queries. Be sure to alter resource group parameters in advance of your maintenance window. +::: info Note +Memory allocation changes that you initiate with `ALTER RESOURCE GROUP` may not take affect immediately due to resource consumption by currently running queries. Be sure to alter resource group parameters in advance of your maintenance window. +::: **Parent topic:** [WarehousePG Best Practices](index.md)