diff --git a/.bumpversion.cfg b/.bumpversion.cfg index f717811..39452ae 100644 --- a/.bumpversion.cfg +++ b/.bumpversion.cfg @@ -1,5 +1,5 @@ [bumpversion] -current_version = 0.7.0 +current_version = 0.7.1 commit = True tag = False parse = (?P\d+)\.(?P\d+)\.(?P[a-z0-9+]+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 7eaeb7e..e19cb51 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -15,6 +15,24 @@ Initial release of umccr/sash, created with the [nf-core](https://nf-co.re/) tem ### `Deprecated` +## [Unreleased] + +### Fixed + +- PAVE_SOMATIC MNV guard: `bcftools view -h | grep -q` could receive `SIGPIPE` on large real VCF headers, which under `pipefail` silently skipped MNV filtering even when `MNVTAG` was present, feeding unfiltered variants into PAVE 1.8 ([sash#66](https://github.com/umccr/sash/issues/66)) + +## [0.7.1] - 2026-07-17 + +### Fixed + +- Bump bolt to 0.3.2: PCGR now gracefully skips hypermutated samples whose variant count still exceeds the per-chunk ceiling ([sash#52](https://github.com/umccr/sash/issues/52)), and PCGR chunk runs disable MSI/TMB estimates that were meaningless on partial VCFs + +### Dependencies + +| Tool | Old | New | +| ---- | ----- | ----- | +| bolt | 0.3.1 | 0.3.2 | + ## [0.7.0] - 2026-05-13 ### Added @@ -38,14 +56,13 @@ Initial release of umccr/sash, created with the [nf-core](https://nf-co.re/) tem ### Fixed -- Bump bolt to 0.3.2: PCGR now gracefully skips hypermutated samples whose variant count still exceeds the per-chunk ceiling ([sash#52](https://github.com/umccr/sash/issues/52)), and PCGR chunk runs disable MSI/TMB estimates that were meaningless on partial VCFs - Migrate sigrap containers from personal `docker.io/qclayssen/sigrap:0.3.0-dev-8` build to official `ghcr.io/umccr/sigrap:0.3.0` ### Dependencies | Tool | Old | New | | --------- | -------- | -------- | -| bolt | 0.2.17 | 0.3.2 | +| bolt | 0.2.17 | 0.3.1 | | sigrap | — | 0.3.0 | | PCGR | — | v2.2.5 | | PCGR data | 20220203 | 20250314 | diff --git a/docs/adr.md b/docs/adr.md index f4a058b..a97af24 100644 --- a/docs/adr.md +++ b/docs/adr.md @@ -68,6 +68,30 @@ Negative: - pcgrr HTML skip: `sigven/pcgr`, `pcgrr/R/main.R` ~line 954 - sash #52, bolt #26, bolt #35 +### The `Hypermutated` flag in the Cancer Report is a coarser, separate signal + +The `Hypermutated: TRUE/FALSE` row curators see in the gpgr Cancer Report QC table is **not** the same thing as the PCGR tiered-filtering cascade described above. A `TRUE` result does not by itself mean clinically-prioritised filtering (steps 2–3 of the cascade) actually ran against that sample's variants. + +**How the flag is computed** (bolt ≥ 0.3.0, `bolt/workflows/smlv_somatic/report.py::count_variant_process()`): + +```python +counts['is_hypermutated'] = counts['dragen'] > constants.MAX_SOMATIC_VARIANTS +``` + +`dragen` is the raw DRAGEN variant count — **PASS and non-PASS included** (only novel SAGE calls are excluded). gpgr (≥ commit `f51919d`) reads this flag directly rather than re-deriving it; this replaced an earlier, less reliable inference based on `sage != annotated` count divergence. + +**Why a `TRUE` flag doesn't necessarily mean TMB was affected.** The variant-count-limiting cascade in `bolt smlv_somatic annotate` (`annotate.py::select_variants()`) runs independently of, and _before_, `select_pcgr_variants()`. It tries progressively more aggressive reductions and returns as soon as the count is under `MAX_SOMATIC_VARIANTS`: + +1. Exclude non-PASS variants (retaining hotspots) — FILTER tag `max_variants_non_pass`. If this alone brings the count under the limit, **the cascade stops here**. +2. Exclude common gnomAD population variants — FILTER tag `max_variants_gnomad`. +3. Restrict to the cancer gene panel region — FILTER tag `max_variants_non_cancer_genes`. + +Because `is_hypermutated` is derived purely from the raw (PASS + non-PASS) count, a sample with a lot of non-PASS noise (e.g. FFPE artifacts) can trip the flag while only needing step 1 above — i.e. ordinary PASS-only filtering, identical in effect to what any non-hypermutated sample gets via the separate `BOLT_SMLV_SOMATIC_FILTER` module. In that case PCGR annotates the full PASS(+hotspot) variant set and the resulting TMB is not an underestimate; no cancer-gene-panel-only restriction occurred. + +**Real case, verified directly from the published VCF** (2026-08-03, `L2600475__L2600476`): `dragen: 531,556; sage: 531,556; annotated: 163,996; filter_pass: 62,883`. Pulling `smlv_somatic/annotate/L2600475.annotations.vcf.gz` and counting FILTER tags confirms this precisely — `max_variants_non_pass` appears on 367,560 records (531,556 − 367,560 = 163,996 = `annotated`), and `max_variants_gnomad`/`max_variants_non_cancer_genes` appear on **zero** records. The cascade stopped after step 1. The sample was flagged hypermutated, but PCGR's TMB was computed from PASS-only variants with no region-based restriction — i.e. the flag was a false positive with respect to TMB accuracy. + +**Caveat for curators:** the flag alone cannot distinguish "raw non-PASS noise tripped the threshold, only ordinary PASS filtering applied" from "PCGR received a clinically-prioritised, region-restricted subset". Confirming which occurred currently requires checking the `annotated`/`filter_pass` counts in `{tumor_id}.somatic.variant_counts_process.json`, or grepping the FILTER column of `{tumor_id}.annotations.vcf.gz` for `max_variants_gnomad`/`max_variants_non_cancer_genes` as above — there is no single boolean for this yet. Changing `is_hypermutated` itself to be based on a PASS-only count (e.g. `filter_pass`, already computed in the same function) has been raised as a follow-up fix in `bolt` (Slack `#genomics-technical`, 2026-08-03 thread on `C025TLC7D`). + --- ## ADR #2: SV Caller Replacement — GRIDSS/GRIPSS → eSVee diff --git a/docs/details.md b/docs/details.md index a3f30ca..80587a5 100644 --- a/docs/details.md +++ b/docs/details.md @@ -609,7 +609,7 @@ These are populated from the DRAGEN mapping and coverage summary files ingested - **SV prioritisation**: genes on this list raise a variant's priority tier in the `simple_sv_annotation` step ([Somatic Structural Variants → Prioritization](#somatic-structural-variants)) - **Somatic small variant annotation**: the gene region BED (`umccr_cancer_genes.gene_regions.bed`) and CDS region BED are passed to `bolt smlv_somatic annotate` and `bolt smlv_somatic report` ([modules/local/bolt/smlv_somatic/annotate/main.nf](modules/local/bolt/smlv_somatic/annotate/main.nf), [modules/local/bolt/smlv_somatic/report/main.nf](modules/local/bolt/smlv_somatic/report/main.nf)) -- **Hypermutated sample handling**: when variant counts exceed 500,000, variants lacking clinical impact or hotspot annotations are filtered until the threshold is met; the cancer gene list defines what "clinical impact" means in this context +- **Hypermutated sample handling**: when variant counts exceed 450,000 (`MAX_SOMATIC_VARIANTS`), variants lacking clinical impact or hotspot annotations are filtered until the threshold is met; the cancer gene list defines what "clinical impact" means in this context. See [ADR #1](adr.md) for the full cascade and the caveat that the `Hypermutated` flag itself is computed separately, on raw (PASS + non-PASS) counts. In [v0.6.4](https://github.com/umccr/sash/blob/main/CHANGELOG.md), the CDKN2A MANE Plus Clinical transcript `ENST00000579755` was added, fixing a gap in CDKN2A CDS coverage. @@ -717,7 +717,7 @@ A: Rescue is performed by BOLT using SAGE hotspot calls layered onto the DRAGEN ### Q: How are hypermutated samples handled in the current version, and is there any impact on derived metrics such as TMB or MSI? -A: When the post-filter variant count exceeds PCGR's 500,000-variant limit, bolt progressively reduces the variant set before passing it to PCGR. The published filtered VCF and PURPLE-derived TMB/MSI are unaffected. See [ADR #1](adr.md) for the full rationale, implementation details, and history of the cancer report hypermutated flag revert. +A: When the post-filter variant count exceeds bolt's 450,000-variant cap (`MAX_SOMATIC_VARIANTS`, set below PCGR's own 500,000-variant limit as a safety margin), bolt progressively reduces the variant set before passing it to PCGR. The published filtered VCF and PURPLE-derived TMB/MSI are unaffected. See [ADR #1](adr.md) for the full rationale, implementation details, and history of the cancer report hypermutated flag revert — including the caveat that the `Hypermutated` flag itself is computed from the raw (PASS + non-PASS) DRAGEN count, so a `TRUE` result does not necessarily mean region-based filtering was applied to the variants PCGR actually annotated. ### Q: How are we handling non-standard chromosomes if present in the input VCFs (ALTs, chrM, etc)? diff --git a/modules/local/bolt/smlv_somatic/report/main.nf b/modules/local/bolt/smlv_somatic/report/main.nf index 18818ca..3ac0e45 100644 --- a/modules/local/bolt/smlv_somatic/report/main.nf +++ b/modules/local/bolt/smlv_somatic/report/main.nf @@ -19,9 +19,9 @@ process BOLT_SMLV_SOMATIC_REPORT { tuple val(meta), path("output/*.bcftools_stats.txt") , emit: bcftools_stats tuple val(meta), path("output/*.variant_counts_type.yaml") , emit: counts_type tuple val(meta), path("output/*.variant_counts_process.json") , emit: counts_process - tuple val(meta), path("output/pcgr/*.pcgr.grch38.pass.vcf.gz"), emit: pcgr_pass_vcf - path 'output/pcgr/' , emit: pcgr_dir - path "output/*.pcgr.grch38.html" , emit: pcgr_report + tuple val(meta), path("output/pcgr/*.pcgr.grch38.pass.vcf.gz"), emit: pcgr_pass_vcf, optional: true + path 'output/pcgr/' , emit: pcgr_dir , optional: true + path "output/*.pcgr.grch38.html" , emit: pcgr_report , optional: true path 'versions.yml' , emit: versions when: @@ -52,7 +52,12 @@ process BOLT_SMLV_SOMATIC_REPORT { --threads ${task.cpus} \\ --output_dir output/ - mv output/pcgr/${meta.tumor_id}.pcgr.grch38.html output/ + # NOTE(QC): PCGR is skipped (and output/pcgr/*.html never created) when the + # sample has too many PASS variants to trim below MAX_SOMATIC_VARIANTS + # (sash #59 / ADR-001) -- guard so the process still exits 0 in that case. + if [[ -f output/pcgr/${meta.tumor_id}.pcgr.grch38.html ]]; then + mv output/pcgr/${meta.tumor_id}.pcgr.grch38.html output/ + fi cat <<-END_VERSIONS > versions.yml "${task.process}": @@ -61,16 +66,20 @@ process BOLT_SMLV_SOMATIC_REPORT { """ stub: - """ + def pcgr_skipped = smlv_vcf.name.contains('hypermutated') + def pcgr_stub_lines = pcgr_skipped ? '' : """ mkdir -p output/pcgr/ - touch output/pcgr/pcgr.stub + touch output/pcgr/${meta.tumor_id}.pcgr.grch38.pass.vcf.gz + touch output/${meta.tumor_id}.pcgr.grch38.html + """ + """ + mkdir -p output/ touch output/af_tumor.txt touch output/af_tumor_keygenes.txt touch output/${meta.tumor_id}.somatic.variant_counts_type.yaml touch output/${meta.tumor_id}.somatic.variant_counts_process.json touch output/${meta.tumor_id}.somatic.bcftools_stats.txt - touch output/pcgr/${meta.tumor_id}.pcgr.grch38.pass.vcf.gz - touch output/${meta.tumor_id}.pcgr.grch38.html + ${pcgr_stub_lines} echo -e '${task.process}:\\n stub: noversions\\n' > versions.yml """ } diff --git a/modules/local/bolt/smlv_somatic/report/tests/fixtures/hypermutated/hypermutated.dragen.vcf.gz b/modules/local/bolt/smlv_somatic/report/tests/fixtures/hypermutated/hypermutated.dragen.vcf.gz new file mode 100644 index 0000000..e69de29 diff --git a/modules/local/bolt/smlv_somatic/report/tests/fixtures/hypermutated/hypermutated.filters.vcf.gz b/modules/local/bolt/smlv_somatic/report/tests/fixtures/hypermutated/hypermutated.filters.vcf.gz new file mode 100644 index 0000000..e69de29 diff --git a/modules/local/bolt/smlv_somatic/report/tests/fixtures/hypermutated/hypermutated.vcf.gz b/modules/local/bolt/smlv_somatic/report/tests/fixtures/hypermutated/hypermutated.vcf.gz new file mode 100644 index 0000000..e69de29 diff --git a/modules/local/bolt/smlv_somatic/report/tests/fixtures/hypermutated/hypermutated.vcf.gz.tbi b/modules/local/bolt/smlv_somatic/report/tests/fixtures/hypermutated/hypermutated.vcf.gz.tbi new file mode 100644 index 0000000..e69de29 diff --git a/modules/local/bolt/smlv_somatic/report/tests/fixtures/hypermutated/purple.purity.tsv b/modules/local/bolt/smlv_somatic/report/tests/fixtures/hypermutated/purple.purity.tsv new file mode 100644 index 0000000..e69de29 diff --git a/modules/local/bolt/smlv_somatic/report/tests/fixtures/normal/normal.dragen.vcf.gz b/modules/local/bolt/smlv_somatic/report/tests/fixtures/normal/normal.dragen.vcf.gz new file mode 100644 index 0000000..e69de29 diff --git a/modules/local/bolt/smlv_somatic/report/tests/fixtures/normal/normal.filters.vcf.gz b/modules/local/bolt/smlv_somatic/report/tests/fixtures/normal/normal.filters.vcf.gz new file mode 100644 index 0000000..e69de29 diff --git a/modules/local/bolt/smlv_somatic/report/tests/fixtures/normal/normal.vcf.gz b/modules/local/bolt/smlv_somatic/report/tests/fixtures/normal/normal.vcf.gz new file mode 100644 index 0000000..e69de29 diff --git a/modules/local/bolt/smlv_somatic/report/tests/fixtures/normal/normal.vcf.gz.tbi b/modules/local/bolt/smlv_somatic/report/tests/fixtures/normal/normal.vcf.gz.tbi new file mode 100644 index 0000000..e69de29 diff --git a/modules/local/bolt/smlv_somatic/report/tests/fixtures/normal/purple.purity.tsv b/modules/local/bolt/smlv_somatic/report/tests/fixtures/normal/purple.purity.tsv new file mode 100644 index 0000000..e69de29 diff --git a/modules/local/bolt/smlv_somatic/report/tests/fixtures/real_run/pcgr_data_dir_placeholder/.gitkeep b/modules/local/bolt/smlv_somatic/report/tests/fixtures/real_run/pcgr_data_dir_placeholder/.gitkeep new file mode 100644 index 0000000..e69de29 diff --git a/modules/local/bolt/smlv_somatic/report/tests/fixtures/real_run/vep_dir_placeholder/.gitkeep b/modules/local/bolt/smlv_somatic/report/tests/fixtures/real_run/vep_dir_placeholder/.gitkeep new file mode 100644 index 0000000..e69de29 diff --git a/modules/local/bolt/smlv_somatic/report/tests/main.nf.test b/modules/local/bolt/smlv_somatic/report/tests/main.nf.test new file mode 100644 index 0000000..27d86e0 --- /dev/null +++ b/modules/local/bolt/smlv_somatic/report/tests/main.nf.test @@ -0,0 +1,84 @@ +nextflow_process { + + name "Test Process BOLT_SMLV_SOMATIC_REPORT" + script "modules/local/bolt/smlv_somatic/report/main.nf" + process "BOLT_SMLV_SOMATIC_REPORT" + config "modules/local/bolt/smlv_somatic/report/tests/nextflow.config" + tag "modules/local/bolt/smlv_somatic/report" + + // Stub tests validate that the optional: true PCGR outputs (pcgr_pass_vcf, + // pcgr_dir, pcgr_report) are correctly present/absent per sample, and that + // the process still succeeds when they're absent (sash #59 / ADR-001). + // Branching is driven by the input VCF filename, mirroring PAVE's + // fixture-directory-name convention. + + test("stub | normal sample — emits pcgr_pass_vcf/pcgr_dir/pcgr_report") { + + options "-stub" + + when { + process { + """ + def vcf = file("${projectDir}/modules/local/bolt/smlv_somatic/report/tests/fixtures/normal/normal.vcf.gz") + def filters_vcf = file("${projectDir}/modules/local/bolt/smlv_somatic/report/tests/fixtures/normal/normal.filters.vcf.gz") + def dragen_vcf = file("${projectDir}/modules/local/bolt/smlv_somatic/report/tests/fixtures/normal/normal.dragen.vcf.gz") + def purity = file("${projectDir}/modules/local/bolt/smlv_somatic/report/tests/fixtures/normal/purple.purity.tsv") + input[0] = [[id: 'TEST_01', tumor_id: 'TUMOR', normal_id: 'NORMAL'], vcf, filters_vcf, dragen_vcf, purity] + input[1] = [] + input[2] = [] + input[3] = [] + input[4] = [] + input[5] = [] + input[6] = [] + """ + } + } + + then { + assert process.success + assert process.out.pcgr_pass_vcf.size() == 1 + assert process.out.pcgr_dir.size() == 1 + assert process.out.pcgr_report.size() == 1 + assert process.out.af_global.size() == 1 + assert process.out.bcftools_stats.size() == 1 + assert process.out.counts_type.size() == 1 + assert process.out.counts_process.size() == 1 + assert process.out.versions.size() == 1 + } + } + + test("stub | hypermutated sample — PCGR skipped, process still succeeds") { + + options "-stub" + + when { + process { + """ + def vcf = file("${projectDir}/modules/local/bolt/smlv_somatic/report/tests/fixtures/hypermutated/hypermutated.vcf.gz") + def filters_vcf = file("${projectDir}/modules/local/bolt/smlv_somatic/report/tests/fixtures/hypermutated/hypermutated.filters.vcf.gz") + def dragen_vcf = file("${projectDir}/modules/local/bolt/smlv_somatic/report/tests/fixtures/hypermutated/hypermutated.dragen.vcf.gz") + def purity = file("${projectDir}/modules/local/bolt/smlv_somatic/report/tests/fixtures/hypermutated/purple.purity.tsv") + input[0] = [[id: 'TEST_02', tumor_id: 'TUMOR', normal_id: 'NORMAL'], vcf, filters_vcf, dragen_vcf, purity] + input[1] = [] + input[2] = [] + input[3] = [] + input[4] = [] + input[5] = [] + input[6] = [] + """ + } + } + + then { + assert process.success + assert process.out.pcgr_pass_vcf.size() == 0 + assert process.out.pcgr_dir.size() == 0 + assert process.out.pcgr_report.size() == 0 + assert process.out.af_global.size() == 1 + assert process.out.bcftools_stats.size() == 1 + assert process.out.counts_type.size() == 1 + assert process.out.counts_process.size() == 1 + assert process.out.versions.size() == 1 + } + } +} diff --git a/modules/local/bolt/smlv_somatic/report/tests/main_real.nf.test b/modules/local/bolt/smlv_somatic/report/tests/main_real.nf.test new file mode 100644 index 0000000..52109d3 --- /dev/null +++ b/modules/local/bolt/smlv_somatic/report/tests/main_real.nf.test @@ -0,0 +1,85 @@ +nextflow_process { + + name "Test Process BOLT_SMLV_SOMATIC_REPORT (real, non-stub)" + script "modules/local/bolt/smlv_somatic/report/main.nf" + process "BOLT_SMLV_SOMATIC_REPORT" + config "modules/local/bolt/smlv_somatic/report/tests/nextflow_docker.config" + tag "modules/local/bolt/smlv_somatic/report" + tag "real" + tag "docker" + + // Real (non-stub) execution against the ghcr.io/umccr/bolt:0.3.2-pcgr container, + // exercising the actual `script:` block -- including the `mv output/pcgr/*.html` + // guard fix (sash #59 / ADR-001) that stub-mode tests structurally cannot reach, + // since -stub bypasses `script:` entirely. Caught a real bug this way earlier in + // this investigation: the mv was previously unconditional and crashed the whole + // process when PCGR was skipped, defeating the optional:true outputs. + // + // Input VCF is a synthetic ~550k-PASS-variant VCF (real REF/ALT vs the genome + // fasta, all records forced INFO/SAGE_HOTSPOT so select_pcgr_variants() has + // nothing droppable and genuinely fails to trim below MAX_SOMATIC_VARIANTS -- + // see ~/workspace/synthetic_hypermutated/generate_synthetic_vcf_v2.py). + // + // NOT reproducible elsewhere as-is: every non-fixture path below is a hardcoded + // absolute path into this machine's ~/workspace (or ~/Documents/data), none of + // it committed to the repo -- the synthetic VCF, its generator script, and the + // real refdata (genome fasta, GIAB regions, cancer genes bed, a real sample's + // purity/dragen-vcf) all live outside version control. This is a local, manual + // diagnostic test, not part of the default nf-test suite and not CI-runnable; + // it will fail with missing-file errors on any other machine. Requires Docker. + // Prerequisites to rerun: + // 1. Docker running, `docker pull ghcr.io/umccr/bolt:0.3.2-pcgr` (~7.6GB) + // 2. Genome fasta+fai (GRCh38_full_analysis_set_plus_decoy_hla, ~3.2GB) and + // the GIAB high-confidence regions bed staged locally + // 3. UMCCR somatic cancer-gene-panel regions bed (umccr_reference_data) + // 4. A real sample's purple purity TSV + a small real PASS VCF to stand in + // for --vcf_dragen_fp (content doesn't need to match the synthetic VCF -- + // only used for a stats/count comparison, not gating logic) + // 5. The synthetic VCF regenerated via generate_synthetic_vcf_v2.py against + // any real sample's PASS VCF + the genome fasta above + // If none of that is available, run modules/local/bolt/smlv_somatic/report/tests/ + // main.nf.test instead (the -stub equivalent) -- it needs none of the above but + // can't exercise the real `script:` block this test is specifically for. + + test("real | hypermutated sample — PCGR skipped, process still succeeds") { + + options "" + + when { + process { + """ + def vcf = file("/Users/quentinclayssen/workspace/synthetic_hypermutated/synthetic_550k_v3.sorted.vcf.gz") + def filters_vcf = file("/Users/quentinclayssen/workspace/synthetic_hypermutated/synthetic_550k_v3_filters.sorted.vcf.gz") + def dragen_vcf = file("/Users/quentinclayssen/workspace/sash_070_vs_older/0.7.0/L2600127_L2600126/smlv_somatic/filter/L2600127.pass.vcf.gz") + def purity = file("/Users/quentinclayssen/workspace/sash_070_vs_older/0.7.0/L2600127_L2600126/purple/L2600127.purple.purity.tsv") + def vep_dir = file("${projectDir}/modules/local/bolt/smlv_somatic/report/tests/fixtures/real_run/vep_dir_placeholder") + def pcgr_data_dir = file("${projectDir}/modules/local/bolt/smlv_somatic/report/tests/fixtures/real_run/pcgr_data_dir_placeholder") + def cancer_genes = file("/Users/quentinclayssen/Documents/data/reference_data/umccr_reference_data/v2--0/gene_panels/somatic/umccr_cancer_genes.gene_regions.bed") + def giab_regions = file("/Users/quentinclayssen/workspace/reference_data/annotations/HG001_GRCh38_GIAB_highconf_CG-IllFB-IllGATKHC-Ion-10X-SOLID_CHROM1-X_v.3.3.2_highconf_nosomaticdel_noCENorHET7.bed.gz") + def genome_fasta = file("/Users/quentinclayssen/workspace/reference_data/genomes/GRCh38_full_analysis_set_plus_decoy_hla.fa") + def genome_fai = file("/Users/quentinclayssen/workspace/reference_data/genomes/GRCh38_full_analysis_set_plus_decoy_hla.fa.fai") + + input[0] = [[id: 'TEST_REAL', tumor_id: 'L2600127', normal_id: 'L2600126'], vcf, filters_vcf, dragen_vcf, purity] + input[1] = pcgr_data_dir + input[2] = vep_dir + input[3] = cancer_genes + input[4] = giab_regions + input[5] = genome_fasta + input[6] = genome_fai + """ + } + } + + then { + assert process.success + assert process.out.pcgr_pass_vcf.size() == 0 + assert process.out.pcgr_dir.size() == 0 + assert process.out.pcgr_report.size() == 0 + assert process.out.af_global.size() == 1 + assert process.out.bcftools_stats.size() == 1 + assert process.out.counts_type.size() == 1 + assert process.out.counts_process.size() == 1 + assert process.out.versions.size() == 1 + } + } +} diff --git a/modules/local/bolt/smlv_somatic/report/tests/nextflow.config b/modules/local/bolt/smlv_somatic/report/tests/nextflow.config new file mode 100644 index 0000000..f1ce121 --- /dev/null +++ b/modules/local/bolt/smlv_somatic/report/tests/nextflow.config @@ -0,0 +1,15 @@ +manifest { + nextflowVersion = '!>=23.04.0' +} + +process { + resourceLimits = [ + memory: 16.GB, + cpus: 4, + time: 1.h + ] + withName: 'BOLT_SMLV_SOMATIC_REPORT' { + memory = 8.GB + cpus = 2 + } +} diff --git a/modules/local/bolt/smlv_somatic/report/tests/nextflow_docker.config b/modules/local/bolt/smlv_somatic/report/tests/nextflow_docker.config new file mode 100644 index 0000000..e78693a --- /dev/null +++ b/modules/local/bolt/smlv_somatic/report/tests/nextflow_docker.config @@ -0,0 +1,25 @@ +// Used only by main_real.nf.test (the real, non-stub, container-execution test) -- +// separate from tests/nextflow.config (the -stub suite) because it needs +// docker.enabled and has no bearing on the fast default test run. See +// main_real.nf.test for prerequisites; this test is local-only, not CI-runnable. + +manifest { + nextflowVersion = '!>=23.04.0' +} + +docker { + enabled = true + userEmulation = true +} + +process { + resourceLimits = [ + memory: 16.GB, + cpus: 4, + time: 1.h + ] + withName: 'BOLT_SMLV_SOMATIC_REPORT' { + memory = 8.GB + cpus = 2 + } +} diff --git a/modules/local/pave/somatic/main.nf b/modules/local/pave/somatic/main.nf index 7a301eb..b6cc927 100644 --- a/modules/local/pave/somatic/main.nf +++ b/modules/local/pave/somatic/main.nf @@ -29,7 +29,8 @@ process PAVE_SOMATIC { def xmx_mod = task.ext.xmx_mod ?: 0.75 """ - if bcftools view -h ${vcf} | grep -q '##INFO= header.txt + if grep -q '##INFO= header.txt + if grep -q '##INFO=&2 + fail=1 +else + echo "PASS: with_mnvtag fixture -> filtered" +fi + +work2="${tmp_dir}/without_mnvtag" +mkdir -p "${work2}" +result=$(run_guard "${without_mnvtag_vcf}" "${work2}") +if [[ "${result}" != "unfiltered" ]]; then + echo "FAIL: without_mnvtag fixture expected 'unfiltered', got '${result}'" >&2 + fail=1 +else + echo "PASS: without_mnvtag fixture -> unfiltered" +fi + +# Reproduce the original SIGPIPE race: a large header (thousands of ##contig lines, like a +# real hg38 VCF) with MNVTAG declared early, well before the pipe buffer fills up. +large_header_vcf="${tmp_dir}/large_header.vcf" +{ + echo '##fileformat=VCFv4.2' + echo '##INFO=' + for i in $(seq 1 20000); do + echo "##contig=" + done + echo -e '#CHROM\tPOS\tID\tREF\tALT\tQUAL\tFILTER\tINFO\tFORMAT\tTUMOR' + echo -e 'chr1\t100\t.\tA\tT\t60\tPASS\tDP=30;MNVTAG=.\tGT\t0/1' +} > "${large_header_vcf}" +bgzip -f "${large_header_vcf}" +tabix -p vcf "${large_header_vcf}.gz" + +work3="${tmp_dir}/large_header" +mkdir -p "${work3}" +result=$(run_guard "${large_header_vcf}.gz" "${work3}") +if [[ "${result}" != "filtered" ]]; then + echo "FAIL: large synthetic header expected 'filtered' (SIGPIPE/pipefail race), got '${result}'" >&2 + fail=1 +else + echo "PASS: large synthetic header -> filtered (no SIGPIPE race)" +fi + +exit "${fail}" diff --git a/nextflow.config b/nextflow.config index 5b45f02..317cfe5 100644 --- a/nextflow.config +++ b/nextflow.config @@ -175,7 +175,7 @@ manifest { description = """Cancer WGTS post-processing pipeline""" mainScript = 'main.nf' nextflowVersion = '!>=22.10.6' - version = '0.7.0' + version = '0.7.1' doi = 'https://doi.org/10.5281/zenodo.15833493' } diff --git a/workflows/sash.nf b/workflows/sash.nf index b9d0564..2e6559e 100644 --- a/workflows/sash.nf +++ b/workflows/sash.nf @@ -361,6 +361,7 @@ workflow SASH { // // channel: [ meta_vcf2maf, report_pcgr_pass_vcf ] + // Filter handles the case where PCGR was skipped (>450k variants) and pcgr_pass_vcf was not emitted ch_smlv_somatic_report_pcgr_pass_vcf_out = WorkflowSash.restoreMeta(BOLT_SMLV_SOMATIC_REPORT.out.pcgr_pass_vcf, ch_inputs) .map { meta, vcf -> def meta_vcf2maf = [ @@ -371,6 +372,7 @@ workflow SASH { ] return [meta_vcf2maf, vcf] } + .filter { _meta, vcf -> vcf } VCF2MAF( ch_smlv_somatic_report_pcgr_pass_vcf_out, diff --git a/workflows/tests/vcf2maf_input_filter/main.nf b/workflows/tests/vcf2maf_input_filter/main.nf new file mode 100644 index 0000000..b71250c --- /dev/null +++ b/workflows/tests/vcf2maf_input_filter/main.nf @@ -0,0 +1,29 @@ +// Test-only workflow: replicates the VCF2MAF input channel chain from +// workflows/sash.nf (the restoreMeta + .map + .filter block downstream of +// BOLT_SMLV_SOMATIC_REPORT.out.pcgr_pass_vcf) so it can be exercised in +// isolation, since this logic is inline in sash.nf and not otherwise +// importable. See modules/local/bolt/smlv_somatic/report/tests/main.nf.test +// for the empirical confirmation that an optional:true output that wasn't +// produced yields zero channel items for that sample (not a null entry). + +workflow VCF2MAF_INPUT_FILTER { + take: + ch_pcgr_pass_vcf // channel: [ meta_bolt, vcf ] — only emitted for samples where PCGR ran + ch_inputs // channel: [ meta ] — one entry per sample, regardless of PCGR skip + + main: + ch_out = WorkflowSash.restoreMeta(ch_pcgr_pass_vcf, ch_inputs) + .map { meta, vcf -> + def meta_vcf2maf = [ + key: meta.id, + id: meta.id, + tumor_id: meta.tumor_id, + normal_id: meta.normal_id, + ] + return [meta_vcf2maf, vcf] + } + .filter { _meta, vcf -> vcf } + + emit: + out = ch_out +} diff --git a/workflows/tests/vcf2maf_input_filter/main.nf.test b/workflows/tests/vcf2maf_input_filter/main.nf.test new file mode 100644 index 0000000..20958d0 --- /dev/null +++ b/workflows/tests/vcf2maf_input_filter/main.nf.test @@ -0,0 +1,96 @@ +nextflow_workflow { + + name "Test Workflow VCF2MAF_INPUT_FILTER" + script "workflows/tests/vcf2maf_input_filter/main.nf" + workflow "VCF2MAF_INPUT_FILTER" + config "workflows/tests/vcf2maf_input_filter/nextflow.config" + tag "workflows/tests/vcf2maf_input_filter" + + // Replicates workflows/sash.nf's VCF2MAF input channel chain in isolation + // (sash #59 / ADR-001). Confirmed empirically via the BOLT_SMLV_SOMATIC_REPORT + // stub test that a sample with PCGR skipped (optional:true output not produced) + // yields zero channel items for that sample — not a [meta, null] entry. So this + // mocks ch_pcgr_pass_vcf with only ONE entry (the non-skipped sample) while + // ch_inputs carries metas for BOTH samples, matching real per-process-output vs. + // per-sample-input channel shapes. + + test("hypermutated sample dropped, normal sample passes through with correct meta_vcf2maf") { + + when { + workflow { + """ + def vcf = file("${projectDir}/workflows/tests/vcf2maf_input_filter/pcgr_pass.vcf.gz") + + input[0] = Channel.of( + [[key: 'SAMPLE_NORMAL', id: 'SAMPLE_NORMAL', tumor_id: 'TUMOR_A', normal_id: 'NORMAL_A'], vcf] + ) + input[1] = Channel.of( + [id: 'SAMPLE_NORMAL', tumor_id: 'TUMOR_A', normal_id: 'NORMAL_A'], + [id: 'SAMPLE_HYPERMUTATED', tumor_id: 'TUMOR_B', normal_id: 'NORMAL_B'], + ) + """ + } + } + + then { + assert workflow.success + assert workflow.out.out.size() == 1 + + with(workflow.out.out.get(0)) { + def meta_vcf2maf = get(0) + assert meta_vcf2maf.key == 'SAMPLE_NORMAL' + assert meta_vcf2maf.id == 'SAMPLE_NORMAL' + assert meta_vcf2maf.tumor_id == 'TUMOR_A' + assert meta_vcf2maf.normal_id == 'NORMAL_A' + } + } + } + + // Three-sample variant: rules out sample mis-keying in restoreMeta's .cross() + // join — e.g. an off-by-one or first-match-wins bug that could let the wrong + // meta pair with the wrong vcf when more than two samples share a run. + // Two samples ran PCGR (SAMPLE_A, SAMPLE_C), one was skipped (SAMPLE_B, the + // middle entry in ch_inputs — deliberately not first/last to catch positional + // bugs); ch_pcgr_pass_vcf only has entries for A and C. + + test("three samples, middle one hypermutated — correct pairing, no mis-keying") { + + when { + workflow { + """ + def vcf_a = file("${projectDir}/workflows/tests/vcf2maf_input_filter/pcgr_pass.vcf.gz") + def vcf_c = file("${projectDir}/workflows/tests/vcf2maf_input_filter/pcgr_pass_2.vcf.gz") + + input[0] = Channel.of( + [[key: 'SAMPLE_A', id: 'SAMPLE_A', tumor_id: 'TUMOR_A', normal_id: 'NORMAL_A'], vcf_a], + [[key: 'SAMPLE_C', id: 'SAMPLE_C', tumor_id: 'TUMOR_C', normal_id: 'NORMAL_C'], vcf_c], + ) + input[1] = Channel.of( + [id: 'SAMPLE_A', tumor_id: 'TUMOR_A', normal_id: 'NORMAL_A'], + [id: 'SAMPLE_B', tumor_id: 'TUMOR_B', normal_id: 'NORMAL_B'], + [id: 'SAMPLE_C', tumor_id: 'TUMOR_C', normal_id: 'NORMAL_C'], + ) + """ + } + } + + then { + assert workflow.success + assert workflow.out.out.size() == 2 + + def by_key = workflow.out.out.collectEntries { [(it[0].key): it] } + assert by_key.keySet() == ['SAMPLE_A', 'SAMPLE_C'] as Set + + with(by_key['SAMPLE_A']) { + assert get(0).tumor_id == 'TUMOR_A' + assert get(0).normal_id == 'NORMAL_A' + assert get(1).toString().endsWith('pcgr_pass.vcf.gz') + } + with(by_key['SAMPLE_C']) { + assert get(0).tumor_id == 'TUMOR_C' + assert get(0).normal_id == 'NORMAL_C' + assert get(1).toString().endsWith('pcgr_pass_2.vcf.gz') + } + } + } +} diff --git a/workflows/tests/vcf2maf_input_filter/nextflow.config b/workflows/tests/vcf2maf_input_filter/nextflow.config new file mode 100644 index 0000000..cee2e5f --- /dev/null +++ b/workflows/tests/vcf2maf_input_filter/nextflow.config @@ -0,0 +1,4 @@ +manifest { + name = 'nextflow_workflow' + nextflowVersion = '!>=23.04.0' +} diff --git a/workflows/tests/vcf2maf_input_filter/pcgr_pass.vcf.gz b/workflows/tests/vcf2maf_input_filter/pcgr_pass.vcf.gz new file mode 100644 index 0000000..e69de29 diff --git a/workflows/tests/vcf2maf_input_filter/pcgr_pass_2.vcf.gz b/workflows/tests/vcf2maf_input_filter/pcgr_pass_2.vcf.gz new file mode 100644 index 0000000..e69de29 diff --git a/workflows/tests/vcf2maf_input_filter/pcgr_pass_3.vcf.gz b/workflows/tests/vcf2maf_input_filter/pcgr_pass_3.vcf.gz new file mode 100644 index 0000000..e69de29