diff --git a/sites/docs/src/content/docs/contributing/contribute-existing-pipelines.md b/sites/docs/src/content/docs/contributing/contribute-existing-pipelines.md index 279d84749f..8dd761a81e 100644 --- a/sites/docs/src/content/docs/contributing/contribute-existing-pipelines.md +++ b/sites/docs/src/content/docs/contributing/contribute-existing-pipelines.md @@ -224,16 +224,18 @@ This adds your parameters to `nextflow_schema.json` with descriptions and valida ### Resource requirements -Define process resource requirements (CPUs, memory, time) in `conf/base.config` using `withLabel:` selectors: +Define process resource requirements (CPUs, memory, time) in `conf/base.config` using `withLabel:` selectors. The nf-core template ships axis-decomposed labels - pick at most one label per axis (CPU, memory, time) and combine them on each module: ```groovy -withLabel: process_low { - cpus = 2 - memory = 4.GB +withLabel:process_cpus_low { + cpus = { 2 * task.attempt } +} +withLabel:process_mem_low { + memory = { 1.GB * task.attempt } } ``` -Use standardized labels from the [nf-core pipeline template](https://github.com/nf-core/tools/blob/master/nf_core/pipeline-template/conf/base.config). +Use standardized labels from the [nf-core pipeline template](https://github.com/nf-core/tools/blob/main/nf_core/pipeline-template/conf/base.config). The full label catalogue and the deprecation status of the older bundled labels (`process_low`, `process_medium`, `process_high`) are documented in [Resource requirements](/docs/specifications/components/modules/resource-requirements). Reference resources dynamically in process blocks: ```groovy diff --git a/sites/docs/src/content/docs/developing/migration-guides/process-axis-labels.md b/sites/docs/src/content/docs/developing/migration-guides/process-axis-labels.md new file mode 100644 index 0000000000..2e449b23fa --- /dev/null +++ b/sites/docs/src/content/docs/developing/migration-guides/process-axis-labels.md @@ -0,0 +1,95 @@ +--- +title: Migrating to axis-decomposed resource labels +subtitle: Move pipelines, modules and site configs from bundled to axis-decomposed labels +description: How to update modules, pipelines and site configs from the bundled resource labels (process_low etc.) to the axis-decomposed labels (process_cpus_*, process_mem_*, process_time_*). +shortTitle: Migrating resource labels +--- + +The bundled resource labels (`process_low`, `process_medium`, `process_high`, `process_long`, `process_high_memory`) are being replaced by axis-decomposed labels (`process_cpus_*`, `process_mem_*`, `process_time_*`) that tune CPU, memory and time independently. Bundled labels still work and emit a lint warning; they are removed after a two-release deprecation window. + +The full scheme is defined in [Resource requirements](/docs/specifications/components/modules/resource-requirements); the rationale is in [nf-core/proposals#139](https://github.com/nf-core/proposals/issues/139). + +## Why migrate + +- Most tools scale on only one axis; bundled labels over-provision the others, often by hundreds of times. +- Common shapes like "many CPUs, little memory" or "moderate CPUs, high memory" cannot be expressed cleanly with bundled labels. +- Pipelines have been working around this with locally invented labels (`process_high_cpu` etc.), which diverge across the ecosystem. + +## Default mapping + +Start from this table. The `process_low` row is opinionated: its 12 GB legacy default is closer to `process_mem_medium` than `process_mem_low`, so module authors should sanity-check the choice rather than apply it mechanically. + +| Legacy label | CPU | Memory | Time | +| --------------------- | --------------------- | -------------------- | --------------------- | +| `process_single` | `process_cpus_single` | (template default) | (template default) | +| `process_low` | `process_cpus_low` | `process_mem_medium` | (template default) | +| `process_medium` | `process_cpus_medium` | `process_mem_medium` | `process_time_medium` | +| `process_high` | `process_cpus_high` | `process_mem_high` | `process_time_long` | +| `process_long` | (unchanged) | (unchanged) | `process_time_long` | +| `process_high_memory` | (unchanged) | `process_mem_high` | (unchanged) | +| `process_low_memory` | (unchanged) | `process_mem_low` | (unchanged) | + +A `process_high` module becomes: + +```diff groovy title="main.nf" + process FOO { +- label 'process_high' ++ label 'process_cpus_high' ++ label 'process_mem_high' ++ label 'process_time_long' + ... + } +``` + +A `process_medium` + `process_high_memory` stack becomes: + +```diff groovy title="main.nf" + process FOO { +- label 'process_medium' +- label 'process_high_memory' ++ label 'process_cpus_medium' ++ label 'process_mem_high' ++ label 'process_time_medium' + ... + } +``` + +Omit any axis label that matches the template default; that is the canonical way to say "this axis does not need to be tuned". + +## Modules and pipelines + +1. Replace each legacy label on the process with the axis labels from the table, adjusting per the tool's real scaling profile. +2. Apply the same change to any `conf/base.config` overrides in the pipeline, and drop pipeline-local label definitions that just duplicate the template. +3. Run `nf-core modules lint` to confirm the new labels are recognised. + +The bundled-label `withLabel:` blocks in the template's `base.config` are removed by the nf-core template sync; they should not be edited by hand. + +## Institutional site configs + +Sites that override `withLabel: process_*` need parallel overrides for the new axis labels. The default translation is the mapping table above; institutions with custom tiering may want different values per axis. + +A `sed` recipe to seed the CPU axis from existing overrides (always review the diff before committing): + +```bash +sed -E ' + s/withLabel: ?process_low\b/withLabel:process_cpus_low/; + s/withLabel: ?process_medium\b/withLabel:process_cpus_medium/; + s/withLabel: ?process_high\b/withLabel:process_cpus_high/; +' conf/site.config +``` + +The recipe rewrites the CPU axis only. Memory and time overrides do not translate one-to-one and need review by hand. + +## Deprecation timeline + +| Stage | Status | +| -------- | ------------------------------------------------------------------------------------------------- | +| Now | Both schemes supported. Lint warns on bundled labels. | +| +1 minor | Both schemes supported. nf-core/modules and nf-core/configs migrated. | +| +2 minor | Bundled labels removed from the template. Pipelines that still use them must define them locally. | + +## See also + +- [Resource requirements](/docs/specifications/components/modules/resource-requirements) +- RFC: [nf-core/proposals#139](https://github.com/nf-core/proposals/issues/139) +- Template change: [nf-core/tools#4265](https://github.com/nf-core/tools/pull/4265) diff --git a/sites/docs/src/content/docs/running/configuration/nextflow-for-your-system.md b/sites/docs/src/content/docs/running/configuration/nextflow-for-your-system.md index 8c23a50347..0bee7e08b6 100644 --- a/sites/docs/src/content/docs/running/configuration/nextflow-for-your-system.md +++ b/sites/docs/src/content/docs/running/configuration/nextflow-for-your-system.md @@ -21,7 +21,7 @@ Pipelines are coded to configure tools to use available resources when possible Not all tools support dynamic resource configuration. ::: -Most nf-core pipelines use process labels to define resource requirements for each module, as shown in this base configuration example: +Most nf-core pipelines use process labels to define resource requirements for each module. Newer pipelines (built from the nf-core template that ships axis-decomposed labels) split the requirement across three independent axes - CPU, memory and time - so each can be tuned independently: ```groovy process { @@ -30,24 +30,38 @@ process { memory: 256.GB, time: 24.h ] - withLabel:process_low { - cpus = { 2 * task.attempt } - memory = { 14.GB * task.attempt } - time = { 6.h * task.attempt } + withLabel:process_cpus_low { + cpus = { 2 * task.attempt } } - withLabel:process_medium { + withLabel:process_cpus_medium { cpus = { 6 * task.attempt } - memory = { 42.GB * task.attempt } - time = { 8.h * task.attempt } } - withLabel:process_high { + withLabel:process_cpus_high { cpus = { 12 * task.attempt } - memory = { 84.GB * task.attempt } - time = { 10.h * task.attempt } + } + withLabel:process_mem_low { + memory = { 1.GB * task.attempt } + } + withLabel:process_mem_medium { + memory = { 12.GB * task.attempt } + } + withLabel:process_mem_high { + memory = { 72.GB * task.attempt } + } + withLabel:process_time_short { + time = { 1.h * task.attempt } + } + withLabel:process_time_medium { + time = { 8.h * task.attempt } + } + withLabel:process_time_long { + time = { 20.h * task.attempt } } } ``` +Older pipelines use bundled labels (`process_low`, `process_medium`, `process_high`) that set all three axes at once. The bundled labels remain functional but are scheduled for deprecation; see the [resource labels migration guide](/docs/developing/migration-guides/process-axis-labels). + The `resourceLimits` list sets the absolute maximum resources any pipeline job can request (typically matching your machine's maximum available resources). The label blocks define the initial default resources each pipeline job requests. When a job runs out of memory, most nf-core pipelines will attempt to retry the job and increase the resource request up to the `resourceLimits` maximum. @@ -58,16 +72,18 @@ When a job runs out of memory, most nf-core pipelines will attempt to retry the Copy only the labels you want to change into your custom configuration file, not all labels. ::: -To set a fixed memory allocation for all large tasks across most nf-core pipelines (without increases during retries), add this to a custom Nextflow configuration file: +To set a fixed memory allocation for all memory-intensive tasks across most nf-core pipelines (without increases during retries), add this to a custom Nextflow configuration file: ```groovy process { - withLabel:process_high { + withLabel:process_mem_high { memory = 200.GB } } ``` +For pipelines that still use bundled labels, target `process_high` instead. + :::tip To find the default labels and resources of the pipeline you want to optimise, go to its GitHub repository and look at the file `conf/base.config`. ::: diff --git a/sites/docs/src/content/docs/specifications/components/modules/resource-requirements.md b/sites/docs/src/content/docs/specifications/components/modules/resource-requirements.md index 4443a78d71..9e942d3093 100644 --- a/sites/docs/src/content/docs/specifications/components/modules/resource-requirements.md +++ b/sites/docs/src/content/docs/specifications/components/modules/resource-requirements.md @@ -10,22 +10,53 @@ The keywords "MUST", "MUST NOT", "SHOULD", etc. are to be interpreted as describ ## Use of labels in modules -An appropriate resource `label` MUST be provided for the module as listed in the [nf-core pipeline template](https://github.com/nf-core/tools/blob/main/nf_core/pipeline-template/conf/base.config). +An appropriate set of resource `label` directives MUST be provided for the module as listed in the [nf-core pipeline template](https://github.com/nf-core/tools/blob/main/nf_core/pipeline-template/conf/base.config). -The template (since nf-core/tools:4.1.0) defines two kinds of labels: +The template defines two label schemes. New and updated modules SHOULD use the axis-decomposed labels; the bundled labels are retained for backwards compatibility and are scheduled for deprecation. -- **Bundled labels** set CPU, memory and time together. Use exactly one per module: `process_single`, `process_low`, `process_medium`, or `process_high`. -- **Modifier labels** override a single axis and SHOULD be stacked on top of a bundled label: `process_long` (extends time), `process_low_memory` (drops memory), `process_high_memory` (raises memory). +### Axis-decomposed labels (preferred) + +Three independent axes cover CPU, memory and time. A module picks at most one label per axis and combines them to describe its resource shape: + +| Axis | Labels (low to high) | | | | +| ------ | --------------------- | -------------------- | --------------------- | ------------------- | +| CPU | `process_cpus_single` | `process_cpus_low` | `process_cpus_medium` | `process_cpus_high` | +| Memory | | `process_mem_low` | `process_mem_medium` | `process_mem_high` | +| Time | | `process_time_short` | `process_time_medium` | `process_time_long` | + +A CPU-bound but memory-light short job declares: + +```groovy title="main.nf" +process FOO { + label 'process_cpus_high' + label 'process_mem_low' + label 'process_time_short' + ... +} +``` + +A module MAY omit a label on an axis it does not need to tune. Axes without an explicit label receive the per-process defaults defined at the top of `base.config`. + +A module MUST NOT declare more than one label on the same axis. `process_cpus_low` and `process_cpus_high` together is invalid and is rejected by the linter. + +### Bundled labels (legacy) + +The original label set ties CPU, memory and time together. Use exactly one per module: `process_single`, `process_low`, `process_medium`, or `process_high`. Two modifier labels override a single axis and SHOULD be stacked on top of a bundled label: `process_long` (extends time), `process_low_memory` (drops memory), `process_high_memory` (raises memory). + +These labels remain functional but emit a lint warning, and SHOULD be migrated to the axis-decomposed scheme. See the [migration guide](/docs/developing/migration-guides/process-axis-labels) for the legacy-to-axis mapping table and a recipe for site-config maintainers. + +### Label precedence When a process matches more than one `withLabel:` block, Nextflow applies them in the order they appear in the config file. The order of the `label` directives in the process itself does not affect precedence. -The nf-core template defines bundled labels first and modifier labels after, so modifier labels always overwrite the corresponding bundled value. + +The nf-core template defines bundled labels first, axis-decomposed labels next, and the legacy modifier labels (`process_long`, `process_low_memory`, `process_high_memory`) last. Stacking a modifier on top of a bundled label resolves to the modifier's value on the axis it overrides, and the bundled values on the others. ## Source of multiple threads or cores value If the tool supports multi-threading, provide the appropriate parameter using the Nextflow `task` variable. For example, `--threads $task.cpus`. -If the tool does not support multi-threading, consider `process_single` unless large amounts of RAM are required. +If the tool does not support multi-threading, consider `process_cpus_single` unless large amounts of RAM are required. ## Specifying multiple threads for piped commands