Release/0.7.1

.bumpversion.cfg

@@ -1,5 +1,5 @@
 [bumpversion]
-current_version = 0.7.0
+current_version = 0.7.1
 commit = True
 tag = False
 parse = (?P<major>\d+)\.(?P<minor>\d+)\.(?P<patch>[a-z0-9+]+)

CHANGELOG.md

@@ -15,6 +15,24 @@ Initial release of umccr/sash, created with the [nf-core](https://nf-co.re/) tem
 
 ### `Deprecated`
 
+## [Unreleased]
+
+### Fixed
+
+- PAVE_SOMATIC MNV guard: `bcftools view -h | grep -q` could receive `SIGPIPE` on large real VCF headers, which under `pipefail` silently skipped MNV filtering even when `MNVTAG` was present, feeding unfiltered variants into PAVE 1.8 ([sash#66](https://github.com/umccr/sash/issues/66))
+
+## [0.7.1] - 2026-07-17
+
+### Fixed
+
+- Bump bolt to 0.3.2: PCGR now gracefully skips hypermutated samples whose variant count still exceeds the per-chunk ceiling ([sash#52](https://github.com/umccr/sash/issues/52)), and PCGR chunk runs disable MSI/TMB estimates that were meaningless on partial VCFs
+
+### Dependencies
+
+| Tool | Old   | New   |
+| ---- | ----- | ----- |
+| bolt | 0.3.1 | 0.3.2 |
+
 ## [0.7.0] - 2026-05-13
 
 ### Added
@@ -38,14 +56,13 @@ Initial release of umccr/sash, created with the [nf-core](https://nf-co.re/) tem
 
 ### Fixed
 
-- Bump bolt to 0.3.2: PCGR now gracefully skips hypermutated samples whose variant count still exceeds the per-chunk ceiling ([sash#52](https://github.com/umccr/sash/issues/52)), and PCGR chunk runs disable MSI/TMB estimates that were meaningless on partial VCFs
 - Migrate sigrap containers from personal `docker.io/qclayssen/sigrap:0.3.0-dev-8` build to official `ghcr.io/umccr/sigrap:0.3.0`
 
 ### Dependencies
 
 | Tool      | Old      | New      |
 | --------- | -------- | -------- |
-| bolt      | 0.2.17   | 0.3.2    |
+| bolt      | 0.2.17   | 0.3.1    |
 | sigrap    | —        | 0.3.0    |
 | PCGR      | —        | v2.2.5   |
 | PCGR data | 20220203 | 20250314 |

docs/adr.md

@@ -68,6 +68,30 @@ Negative:
 - pcgrr HTML skip: `sigven/pcgr`, `pcgrr/R/main.R` ~line 954
 - sash #52, bolt #26, bolt #35
 
+### The `Hypermutated` flag in the Cancer Report is a coarser, separate signal
+
+The `Hypermutated: TRUE/FALSE` row curators see in the gpgr Cancer Report QC table is **not** the same thing as the PCGR tiered-filtering cascade described above. A `TRUE` result does not by itself mean clinically-prioritised filtering (steps 2–3 of the cascade) actually ran against that sample's variants.
+
+**How the flag is computed** (bolt ≥ 0.3.0, `bolt/workflows/smlv_somatic/report.py::count_variant_process()`):
+
+```python
+counts['is_hypermutated'] = counts['dragen'] > constants.MAX_SOMATIC_VARIANTS
+```
+
+`dragen` is the raw DRAGEN variant count — **PASS and non-PASS included** (only novel SAGE calls are excluded). gpgr (≥ commit `f51919d`) reads this flag directly rather than re-deriving it; this replaced an earlier, less reliable inference based on `sage != annotated` count divergence.
+
+**Why a `TRUE` flag doesn't necessarily mean TMB was affected.** The variant-count-limiting cascade in `bolt smlv_somatic annotate` (`annotate.py::select_variants()`) runs independently of, and _before_, `select_pcgr_variants()`. It tries progressively more aggressive reductions and returns as soon as the count is under `MAX_SOMATIC_VARIANTS`:
+
+1. Exclude non-PASS variants (retaining hotspots) — FILTER tag `max_variants_non_pass`. If this alone brings the count under the limit, **the cascade stops here**.
+2. Exclude common gnomAD population variants — FILTER tag `max_variants_gnomad`.
+3. Restrict to the cancer gene panel region — FILTER tag `max_variants_non_cancer_genes`.
+
+Because `is_hypermutated` is derived purely from the raw (PASS + non-PASS) count, a sample with a lot of non-PASS noise (e.g. FFPE artifacts) can trip the flag while only needing step 1 above — i.e. ordinary PASS-only filtering, identical in effect to what any non-hypermutated sample gets via the separate `BOLT_SMLV_SOMATIC_FILTER` module. In that case PCGR annotates the full PASS(+hotspot) variant set and the resulting TMB is not an underestimate; no cancer-gene-panel-only restriction occurred.
+
+**Real case, verified directly from the published VCF** (2026-08-03, `L2600475__L2600476`): `dragen: 531,556; sage: 531,556; annotated: 163,996; filter_pass: 62,883`. Pulling `smlv_somatic/annotate/L2600475.annotations.vcf.gz` and counting FILTER tags confirms this precisely — `max_variants_non_pass` appears on 367,560 records (531,556 − 367,560 = 163,996 = `annotated`), and `max_variants_gnomad`/`max_variants_non_cancer_genes` appear on **zero** records. The cascade stopped after step 1. The sample was flagged hypermutated, but PCGR's TMB was computed from PASS-only variants with no region-based restriction — i.e. the flag was a false positive with respect to TMB accuracy.
+
+**Caveat for curators:** the flag alone cannot distinguish "raw non-PASS noise tripped the threshold, only ordinary PASS filtering applied" from "PCGR received a clinically-prioritised, region-restricted subset". Confirming which occurred currently requires checking the `annotated`/`filter_pass` counts in `{tumor_id}.somatic.variant_counts_process.json`, or grepping the FILTER column of `{tumor_id}.annotations.vcf.gz` for `max_variants_gnomad`/`max_variants_non_cancer_genes` as above — there is no single boolean for this yet. Changing `is_hypermutated` itself to be based on a PASS-only count (e.g. `filter_pass`, already computed in the same function) has been raised as a follow-up fix in `bolt` (Slack `#genomics-technical`, 2026-08-03 thread on `C025TLC7D`).
+
 ---
 
 ## ADR #2: SV Caller Replacement — GRIDSS/GRIPSS → eSVee

docs/details.md

@@ -609,7 +609,7 @@ These are populated from the DRAGEN mapping and coverage summary files ingested
 
 - **SV prioritisation**: genes on this list raise a variant's priority tier in the `simple_sv_annotation` step ([Somatic Structural Variants → Prioritization](#somatic-structural-variants))
 - **Somatic small variant annotation**: the gene region BED (`umccr_cancer_genes.gene_regions.bed`) and CDS region BED are passed to `bolt smlv_somatic annotate` and `bolt smlv_somatic report` ([modules/local/bolt/smlv_somatic/annotate/main.nf](modules/local/bolt/smlv_somatic/annotate/main.nf), [modules/local/bolt/smlv_somatic/report/main.nf](modules/local/bolt/smlv_somatic/report/main.nf))
-- **Hypermutated sample handling**: when variant counts exceed 500,000, variants lacking clinical impact or hotspot annotations are filtered until the threshold is met; the cancer gene list defines what "clinical impact" means in this context
+- **Hypermutated sample handling**: when variant counts exceed 450,000 (`MAX_SOMATIC_VARIANTS`), variants lacking clinical impact or hotspot annotations are filtered until the threshold is met; the cancer gene list defines what "clinical impact" means in this context. See [ADR #1](adr.md) for the full cascade and the caveat that the `Hypermutated` flag itself is computed separately, on raw (PASS + non-PASS) counts.
 
 In [v0.6.4](https://github.com/umccr/sash/blob/main/CHANGELOG.md), the CDKN2A MANE Plus Clinical transcript `ENST00000579755` was added, fixing a gap in CDKN2A CDS coverage.
 
@@ -717,7 +717,7 @@ A: Rescue is performed by BOLT using SAGE hotspot calls layered onto the DRAGEN
 
 ### Q: How are hypermutated samples handled in the current version, and is there any impact on derived metrics such as TMB or MSI?
 
-A: When the post-filter variant count exceeds PCGR's 500,000-variant limit, bolt progressively reduces the variant set before passing it to PCGR. The published filtered VCF and PURPLE-derived TMB/MSI are unaffected. See [ADR #1](adr.md) for the full rationale, implementation details, and history of the cancer report hypermutated flag revert.
+A: When the post-filter variant count exceeds bolt's 450,000-variant cap (`MAX_SOMATIC_VARIANTS`, set below PCGR's own 500,000-variant limit as a safety margin), bolt progressively reduces the variant set before passing it to PCGR. The published filtered VCF and PURPLE-derived TMB/MSI are unaffected. See [ADR #1](adr.md) for the full rationale, implementation details, and history of the cancer report hypermutated flag revert — including the caveat that the `Hypermutated` flag itself is computed from the raw (PASS + non-PASS) DRAGEN count, so a `TRUE` result does not necessarily mean region-based filtering was applied to the variants PCGR actually annotated.
 
 ### Q: How are we handling non-standard chromosomes if present in the input VCFs (ALTs, chrM, etc)?
 

modules/local/bolt/smlv_somatic/report/main.nf

@@ -19,9 +19,9 @@ process BOLT_SMLV_SOMATIC_REPORT {
     tuple val(meta), path("output/*.bcftools_stats.txt")          , emit: bcftools_stats
     tuple val(meta), path("output/*.variant_counts_type.yaml")    , emit: counts_type
     tuple val(meta), path("output/*.variant_counts_process.json") , emit: counts_process
-    tuple val(meta), path("output/pcgr/*.pcgr.grch38.pass.vcf.gz"), emit: pcgr_pass_vcf
-    path 'output/pcgr/'                                           , emit: pcgr_dir
-    path "output/*.pcgr.grch38.html"                              , emit: pcgr_report
+    tuple val(meta), path("output/pcgr/*.pcgr.grch38.pass.vcf.gz"), emit: pcgr_pass_vcf, optional: true
+    path 'output/pcgr/'                                           , emit: pcgr_dir    , optional: true
+    path "output/*.pcgr.grch38.html"                              , emit: pcgr_report , optional: true
     path 'versions.yml'                                           , emit: versions
 
     when:
@@ -52,7 +52,12 @@ process BOLT_SMLV_SOMATIC_REPORT {
         --threads ${task.cpus} \\
         --output_dir output/
 
-    mv output/pcgr/${meta.tumor_id}.pcgr.grch38.html output/
+    # NOTE(QC): PCGR is skipped (and output/pcgr/*.html never created) when the
+    # sample has too many PASS variants to trim below MAX_SOMATIC_VARIANTS
+    # (sash #59 / ADR-001) -- guard so the process still exits 0 in that case.
+    if [[ -f output/pcgr/${meta.tumor_id}.pcgr.grch38.html ]]; then
+        mv output/pcgr/${meta.tumor_id}.pcgr.grch38.html output/
+    fi
 
     cat <<-END_VERSIONS > versions.yml
     "${task.process}":
@@ -61,16 +66,20 @@ process BOLT_SMLV_SOMATIC_REPORT {
     """
 
     stub:
-    """
+    def pcgr_skipped = smlv_vcf.name.contains('hypermutated')
+    def pcgr_stub_lines = pcgr_skipped ? '' : """
     mkdir -p output/pcgr/
-    touch output/pcgr/pcgr.stub
+    touch output/pcgr/${meta.tumor_id}.pcgr.grch38.pass.vcf.gz
+    touch output/${meta.tumor_id}.pcgr.grch38.html
+    """
+    """
+    mkdir -p output/
     touch output/af_tumor.txt
     touch output/af_tumor_keygenes.txt
     touch output/${meta.tumor_id}.somatic.variant_counts_type.yaml
     touch output/${meta.tumor_id}.somatic.variant_counts_process.json
     touch output/${meta.tumor_id}.somatic.bcftools_stats.txt
-    touch output/pcgr/${meta.tumor_id}.pcgr.grch38.pass.vcf.gz
-    touch output/${meta.tumor_id}.pcgr.grch38.html
+    ${pcgr_stub_lines}
     echo -e '${task.process}:\\n  stub: noversions\\n' > versions.yml
     """
 }

modules/local/bolt/smlv_somatic/report/tests/main.nf.test

@@ -0,0 +1,84 @@
+nextflow_process {
+
+    name "Test Process BOLT_SMLV_SOMATIC_REPORT"
+    script "modules/local/bolt/smlv_somatic/report/main.nf"
+    process "BOLT_SMLV_SOMATIC_REPORT"
+    config "modules/local/bolt/smlv_somatic/report/tests/nextflow.config"
+    tag "modules/local/bolt/smlv_somatic/report"
+
+    // Stub tests validate that the optional: true PCGR outputs (pcgr_pass_vcf,
+    // pcgr_dir, pcgr_report) are correctly present/absent per sample, and that
+    // the process still succeeds when they're absent (sash #59 / ADR-001).
+    // Branching is driven by the input VCF filename, mirroring PAVE's
+    // fixture-directory-name convention.
+
+    test("stub | normal sample — emits pcgr_pass_vcf/pcgr_dir/pcgr_report") {
+
+        options "-stub"
+
+        when {
+            process {
+                """
+                def vcf = file("${projectDir}/modules/local/bolt/smlv_somatic/report/tests/fixtures/normal/normal.vcf.gz")
+                def filters_vcf = file("${projectDir}/modules/local/bolt/smlv_somatic/report/tests/fixtures/normal/normal.filters.vcf.gz")
+                def dragen_vcf = file("${projectDir}/modules/local/bolt/smlv_somatic/report/tests/fixtures/normal/normal.dragen.vcf.gz")
+                def purity = file("${projectDir}/modules/local/bolt/smlv_somatic/report/tests/fixtures/normal/purple.purity.tsv")
+                input[0] = [[id: 'TEST_01', tumor_id: 'TUMOR', normal_id: 'NORMAL'], vcf, filters_vcf, dragen_vcf, purity]
+                input[1] = []
+                input[2] = []
+                input[3] = []
+                input[4] = []
+                input[5] = []
+                input[6] = []
+                """
+            }
+        }
+
+        then {
+            assert process.success
+            assert process.out.pcgr_pass_vcf.size() == 1
+            assert process.out.pcgr_dir.size() == 1
+            assert process.out.pcgr_report.size() == 1
+            assert process.out.af_global.size() == 1
+            assert process.out.bcftools_stats.size() == 1
+            assert process.out.counts_type.size() == 1
+            assert process.out.counts_process.size() == 1
+            assert process.out.versions.size() == 1
+        }
+    }
+
+    test("stub | hypermutated sample — PCGR skipped, process still succeeds") {
+
+        options "-stub"
+
+        when {
+            process {
+                """
+                def vcf = file("${projectDir}/modules/local/bolt/smlv_somatic/report/tests/fixtures/hypermutated/hypermutated.vcf.gz")
+                def filters_vcf = file("${projectDir}/modules/local/bolt/smlv_somatic/report/tests/fixtures/hypermutated/hypermutated.filters.vcf.gz")
+                def dragen_vcf = file("${projectDir}/modules/local/bolt/smlv_somatic/report/tests/fixtures/hypermutated/hypermutated.dragen.vcf.gz")
+                def purity = file("${projectDir}/modules/local/bolt/smlv_somatic/report/tests/fixtures/hypermutated/purple.purity.tsv")
+                input[0] = [[id: 'TEST_02', tumor_id: 'TUMOR', normal_id: 'NORMAL'], vcf, filters_vcf, dragen_vcf, purity]
+                input[1] = []
+                input[2] = []
+                input[3] = []
+                input[4] = []
+                input[5] = []
+                input[6] = []
+                """
+            }
+        }
+
+        then {
+            assert process.success
+            assert process.out.pcgr_pass_vcf.size() == 0
+            assert process.out.pcgr_dir.size() == 0
+            assert process.out.pcgr_report.size() == 0
+            assert process.out.af_global.size() == 1
+            assert process.out.bcftools_stats.size() == 1
+            assert process.out.counts_type.size() == 1
+            assert process.out.counts_process.size() == 1
+            assert process.out.versions.size() == 1
+        }
+    }
+}

modules/local/bolt/smlv_somatic/report/tests/main_real.nf.test

@@ -0,0 +1,85 @@
+nextflow_process {
+
+    name "Test Process BOLT_SMLV_SOMATIC_REPORT (real, non-stub)"
+    script "modules/local/bolt/smlv_somatic/report/main.nf"
+    process "BOLT_SMLV_SOMATIC_REPORT"
+    config "modules/local/bolt/smlv_somatic/report/tests/nextflow_docker.config"
+    tag "modules/local/bolt/smlv_somatic/report"
+    tag "real"
+    tag "docker"
+
+    // Real (non-stub) execution against the ghcr.io/umccr/bolt:0.3.2-pcgr container,
+    // exercising the actual `script:` block -- including the `mv output/pcgr/*.html`
+    // guard fix (sash #59 / ADR-001) that stub-mode tests structurally cannot reach,
+    // since -stub bypasses `script:` entirely. Caught a real bug this way earlier in
+    // this investigation: the mv was previously unconditional and crashed the whole
+    // process when PCGR was skipped, defeating the optional:true outputs.
+    //
+    // Input VCF is a synthetic ~550k-PASS-variant VCF (real REF/ALT vs the genome
+    // fasta, all records forced INFO/SAGE_HOTSPOT so select_pcgr_variants() has
+    // nothing droppable and genuinely fails to trim below MAX_SOMATIC_VARIANTS --
+    // see ~/workspace/synthetic_hypermutated/generate_synthetic_vcf_v2.py).
+    //
+    // NOT reproducible elsewhere as-is: every non-fixture path below is a hardcoded
+    // absolute path into this machine's ~/workspace (or ~/Documents/data), none of
+    // it committed to the repo -- the synthetic VCF, its generator script, and the
+    // real refdata (genome fasta, GIAB regions, cancer genes bed, a real sample's
+    // purity/dragen-vcf) all live outside version control. This is a local, manual
+    // diagnostic test, not part of the default nf-test suite and not CI-runnable;
+    // it will fail with missing-file errors on any other machine. Requires Docker.
+    // Prerequisites to rerun:
+    //   1. Docker running, `docker pull ghcr.io/umccr/bolt:0.3.2-pcgr` (~7.6GB)
+    //   2. Genome fasta+fai (GRCh38_full_analysis_set_plus_decoy_hla, ~3.2GB) and
+    //      the GIAB high-confidence regions bed staged locally
+    //   3. UMCCR somatic cancer-gene-panel regions bed (umccr_reference_data)
+    //   4. A real sample's purple purity TSV + a small real PASS VCF to stand in
+    //      for --vcf_dragen_fp (content doesn't need to match the synthetic VCF --
+    //      only used for a stats/count comparison, not gating logic)
+    //   5. The synthetic VCF regenerated via generate_synthetic_vcf_v2.py against
+    //      any real sample's PASS VCF + the genome fasta above
+    // If none of that is available, run modules/local/bolt/smlv_somatic/report/tests/
+    // main.nf.test instead (the -stub equivalent) -- it needs none of the above but
+    // can't exercise the real `script:` block this test is specifically for.
+
+    test("real | hypermutated sample — PCGR skipped, process still succeeds") {
+
+        options ""
+
+        when {
+            process {
+                """
+                def vcf = file("/Users/quentinclayssen/workspace/synthetic_hypermutated/synthetic_550k_v3.sorted.vcf.gz")
+                def filters_vcf = file("/Users/quentinclayssen/workspace/synthetic_hypermutated/synthetic_550k_v3_filters.sorted.vcf.gz")
+                def dragen_vcf = file("/Users/quentinclayssen/workspace/sash_070_vs_older/0.7.0/L2600127_L2600126/smlv_somatic/filter/L2600127.pass.vcf.gz")
+                def purity = file("/Users/quentinclayssen/workspace/sash_070_vs_older/0.7.0/L2600127_L2600126/purple/L2600127.purple.purity.tsv")
+                def vep_dir = file("${projectDir}/modules/local/bolt/smlv_somatic/report/tests/fixtures/real_run/vep_dir_placeholder")
+                def pcgr_data_dir = file("${projectDir}/modules/local/bolt/smlv_somatic/report/tests/fixtures/real_run/pcgr_data_dir_placeholder")
+                def cancer_genes = file("/Users/quentinclayssen/Documents/data/reference_data/umccr_reference_data/v2--0/gene_panels/somatic/umccr_cancer_genes.gene_regions.bed")
+                def giab_regions = file("/Users/quentinclayssen/workspace/reference_data/annotations/HG001_GRCh38_GIAB_highconf_CG-IllFB-IllGATKHC-Ion-10X-SOLID_CHROM1-X_v.3.3.2_highconf_nosomaticdel_noCENorHET7.bed.gz")
+                def genome_fasta = file("/Users/quentinclayssen/workspace/reference_data/genomes/GRCh38_full_analysis_set_plus_decoy_hla.fa")
+                def genome_fai = file("/Users/quentinclayssen/workspace/reference_data/genomes/GRCh38_full_analysis_set_plus_decoy_hla.fa.fai")
+
+                input[0] = [[id: 'TEST_REAL', tumor_id: 'L2600127', normal_id: 'L2600126'], vcf, filters_vcf, dragen_vcf, purity]
+                input[1] = pcgr_data_dir
+                input[2] = vep_dir
+                input[3] = cancer_genes
+                input[4] = giab_regions
+                input[5] = genome_fasta
+                input[6] = genome_fai
+                """
+            }
+        }
+
+        then {
+            assert process.success
+            assert process.out.pcgr_pass_vcf.size() == 0
+            assert process.out.pcgr_dir.size() == 0
+            assert process.out.pcgr_report.size() == 0
+            assert process.out.af_global.size() == 1
+            assert process.out.bcftools_stats.size() == 1
+            assert process.out.counts_type.size() == 1
+            assert process.out.counts_process.size() == 1
+            assert process.out.versions.size() == 1
+        }
+    }
+}