Release 0.3.2 (pending full-pipeline validation)

.bumpversion.cfg

@@ -1,5 +1,5 @@
 [bumpversion]
-current_version = 0.3.1
+current_version = 0.3.2
 commit = True
 tag = False
 parse = (?P<major>\d+)\.(?P<minor>\d+)\.(?P<patch>[a-z0-9+]+)

.github/workflows/build.yaml

@@ -18,7 +18,7 @@ jobs:
         uses: actions/checkout@v4
 
       - name: Free Disk Space (Ubuntu)
-        uses: jlumbroso/free-disk-space@main
+        uses: jlumbroso/free-disk-space@v1.3.0
         with:
           # this might remove tools that are actually needed,
           # if set to "true" but frees about 6 GB

CHANGELOG.md

@@ -1,24 +1,51 @@
 # bolt changelog
 
+## 0.3.2
+
+- Fix: `rescue.py`'s SAGE VCF header-consistency check compared the DRAGEN input VCF's headers against the expected SAGE header descriptions instead of the actual SAGE VCF's — the check ran but validated nothing (introduced [24](https://github.com/umccr/bolt/pull/24)). Now checks `sage_vcf_fp`. First-ever test coverage added for `rescue.py` and for `util.check_annotation_headers`
+- Fix: hypermutated `report` samples (tiered-selection path) never disabled `--estimate_msi`/`--estimate_tmb` — TMB/MSI were computed on a trimmed variant subset with no indication in the report. Now passes `disable_estimates=True` when `select_pcgr_variants` trims input, matching `run_somatic_chunk`'s existing chunked-input behavior
+- Fix: `get_annotations_vcf` crashed with `AssertionError` on a duplicate PCGR VCF key (variant mapping to multiple transcripts) — now logs a warning and keeps the first entry ([34](https://github.com/umccr/bolt/pull/34))
+- Fix: `collect_pcgr_annotation_data` / `collect_cpsr_annotation_data` had the same duplicate-key `AssertionError` on their TSV paths — somatic path now keeps the most actionable tier on a duplicate, germline/CPSR path keeps the first entry; both warn instead of crashing
+- Test: `TestGetAnnotationsVcf`, `TestCollectPcgrAnnotationData`, `TestCollectCpsrAnnotationData` — duplicate-key regression coverage for all three annotation-collection paths
+- Test: `TestMergeVcfFiles` — bcftools-guarded integration test proving `merge_vcf_files` (`bcftools merge -m all`) losslessly reassembles PCGR sites-only chunk VCFs (no loss/dup, position-sorted, indexed output). Locks the sites-only invariant that keeps `bcftools merge` safe for the hypermutated chunk-merge path; documented with a `NOTE` in `merge_vcf_files`
+- [32](https://github.com/umccr/bolt/pull/32) - Fix `PCGR_MUTATION_HOTSPOT=.` (dot placeholder) treated as truthy in retention check — was preventing tiered filtering from running for any sample with >450k PASS variants ([sash#52](https://github.com/umccr/sash/issues/52))
+- [32](https://github.com/umccr/bolt/pull/32) - Graceful PCGR skip when `select_pcgr_variants` cannot cap variants to `MAX_SOMATIC_VARIANTS` — logs warning and continues without cancer report; non-PCGR outputs still publish ([sash#52](https://github.com/umccr/sash/issues/52))
+- Fix: disable `--estimate_msi`/`--estimate_tmb` in chunked PCGR annotation runs — estimates on partial VCFs are not meaningful
+- Fix: pin `jlumbroso/free-disk-space` CI action to `v1.3.0` (was `@main`)
+- Test: `TestSelectPcgrVariants` — 8 integration tests covering tiered trimming, hotspot retention, PANEL retention, NONCODING-first drop order, and the `PCGR_MUTATION_HOTSPOT=.` regression
+- Test: `TestEntrySkipsPcgrOnOverflow` — 2 tests: entry() skips PCGR on `RuntimeError` from unresolvable overflow; entry() calls PCGR normally when within limit
+- Test: `TestSelectPcgrVariantsRaisesOnUnresolvableOverflow` — asserts `RuntimeError` when retained variants alone exceed `MAX_SOMATIC_VARIANTS`
+- Test: `TestRunSomaticCommandArgs` — 2 tests: `--estimate_signatures` absent from all `run_somatic` commands; `disable_estimates=True` suppresses `--estimate_msi`/`--estimate_tmb`
+- Test: `TestRunSomaticChunkArgMapping.test_disable_estimates_passed_to_run_somatic` — `run_somatic_chunk` passes `disable_estimates=True` to every `run_somatic` call
+
 ## 0.3.1
 
 - Fix `ModuleNotFoundError: No module named 'pkg_resources'` in `bolt:0.3.0-multiqc` — add `setuptools <81` to conda env
 - Fix `merge_vcf_files` producing wrong output filename — `Path.with_suffix()` was stripping `.pass` component; use explicit path concatenation instead
 - Fix VCF writers not closed in `transfer_annotations_somatic` and `transfer_annotations_germline` — BGZip output could be truncated
 - Fix `split_vcf` writing uncompressed plain `.vcf` chunks — now uses `.vcf.gz` with `wz` mode
 - Fix `PCGR_ACTIONABILITY_TIER` VCF header description — updated to match stored short-form values (`1`,`2`,`3`,`4`,`N`)
-- Add regression test for chunk file compression (`test_chunks_are_gzipped`)
+- Fix `split_vcf` chunks not tabix-indexed, causing PCGR to fail reading them
+- Fix `build.yaml` and `Dockerfile.pcgr` build issues
+- Remove unused `logging` import and fix `PCGR_MAX_SOMATIC_VARIANTS` header description in `constants.py`
+- Bump `r-gpgr` to 2.3.1 in `Dockerfile.gpgr`
+- Add CI smoke tests to catch Docker image startup failures before push
+- [31](https://github.com/umccr/bolt/pull/31) - Drop `--estimate_signatures` from PCGR somatic invocation — signature analysis comes from gpgr/sigrap downstream; keep `--estimate_msi`/`--estimate_tmb` ([sash#57](https://github.com/umccr/sash/issues/57))
+- Test: regression test for chunk file compression (`test_chunks_are_gzipped`)
+- Test: regression test for chunk tabix indexing (`test_chunks_are_tabix_indexed`)
+- Test: `TestTierOrdering` — 3 tests verifying `PCGR_TIERS_FILTERING` uses short forms (`N`,`4`,`3`,`2`,`1`) and NONCODING precedes TIER_1 in `get_ordering()`
+- Test: `TestSplitVcf` — 4 tests: chunking above/below limit, `.vcf.gz` compression, `.tbi` indexing
+- Test: `TestSelectPcgrVariants` — initial 6 integration tests for tiered trimming logic
+- Test: `TestGetVariantFilterData`, `TestDetermineFilter`, `TestGetImpacts` — 14 unit tests covering variant attribute extraction and filter-category determination
+- Test: `TestCountVariantProcess` — 4 tests: `is_hypermutated` flag, DRAGEN count, SAGE_NOVEL exclusion, annotation-filter exclusion
+- Test: `TestRunSomaticChunkArgMapping` — asserts `pcgr_conda` is not shifted into `pcgr_threads` position on positional arg mapping
 
 ## 0.3.0
 
 - [28](https://github.com/umccr/bolt/pull/28) - gpgr version bump to 2.2.12 for cancer report hypermutated flag fix
 
 - [17](https://github.com/umccr/bolt/pull/17) - change dragen HRD file optional
-
 - [14](https://github.com/umccr/bolt/pull/14) - gpgr version bump to 2.2.0
-
 - [3](https://github.com/scwatts/bolt/pull/3) - Improve PCGR / CPSR argument handling
-
 - [6](https://github.com/umccr/bolt/pull/6) - Change oncoanalyser v2.0.0 update, with switch sv caller from GRIPSS to eSVee
-
 - [9](https://github.com/umccr/bolt/pull/9) Add hypermutation sample handling

TESTING.md

@@ -0,0 +1,74 @@
+# Testing
+
+## Running tests
+
+Canonical (matches CI):
+
+```bash
+python -m unittest discover tests/ --buffer
+```
+
+Also supported:
+
+```bash
+python -m pytest tests/ -v
+```
+
+Single test:
+
+```bash
+python -m pytest tests/test_smlv_somatic_filter.py::TestSmlvSomaticFilter::test_min_af_filter -v
+```
+
+Tests in `tests/` are pure Python + in-memory `cyvcf2` logic and must run
+without any bioinformatics binary installed (no `bcftools`, `pcgr`, `cpsr`,
+`vcfanno`, `snpEff`, `gpgr`, VEP). Functions that shell out to those tools are
+either mocked/patched in tests or are not unit-tested (see below). The one
+exception is `TestMergeVcfFiles`, an integration test that exercises the real
+`merge_vcf_files` → `bcftools merge` path; it is guarded with
+`@unittest.skipUnless(shutil.which('bcftools'), ...)`, so it runs in the conda
+CI env and skips cleanly (never fails) where `bcftools` is absent.
+
+## Test Coverage
+
+| Module | Status | Functions covered | Test file |
+|---|---|---|---|
+| `bolt/util.py` | Partial | `get_vcf_header_entry`, `get_vcf_header_line`, `get_qualified_vcf_annotation`, `add_vcf_header_entry`, `merge_tsv_files`, `merge_vcf_files` (bcftools-guarded lossless/sorted integration test), `check_annotation_headers` | `tests/test_util.py` |
+| `bolt/common/pcgr.py` | Partial | `get_ordering`, `get_impacts`, `determine_filter`, `get_variant_filter_data`, `split_vcf`, `run_somatic_chunk` (arg-mapping regression) | `tests/test_pcgr.py` |
+| `bolt/common/pcgr.py` | Partial | `parse_genomic_change`, `get_impacts_higher`, `get_annotation_entry_tsv`, `compile_annotation_data`, `annotate_record`, `get_annotations_vcf` (duplicate-key regression), `collect_pcgr_annotation_data` (duplicate-key tier resolution), `collect_cpsr_annotation_data` (duplicate-key regression) | `tests/test_pcgr_annotation.py` |
+| `bolt/workflows/smlv_somatic/filter.py` | Partial | `set_filter_data` | `tests/test_smlv_somatic_filter.py` |
+| `bolt/workflows/smlv_somatic/report.py` | Partial | `select_pcgr_variants`, `count_variant_process`, `entry` overflow handling, `entry` `disable_estimates` branching | `tests/test_smlv_somatic_report.py` |
+| `bolt/workflows/smlv_somatic/rescue.py` | Partial | `annotate_existing_sage_calls` (SAGE VCF header-consistency check only) | `tests/test_smlv_somatic_rescue.py` |
+
+"Partial" means the module has meaningful test coverage for its pure/testable logic,
+but not every function in the file is tested (see below for what is excluded and why).
+
+## Untested / not unit-testable in CI
+
+These require live bioinformatics binaries (bcftools, PCGR/CPSR, vcfanno, snpEff,
+gpgr) or full end-to-end subprocess orchestration, so they are excluded from the
+unit test suite:
+
+| Function/module | Reason |
+|---|---|
+| `bolt/util.py: count_vcf_records` | Shells out to `bcftools view` |
+| `bolt/util.py: execute_command` | Spawns real subprocesses via `/bin/bash` |
+| `bolt/common/pcgr.py: prepare_vcf_somatic` / `prepare_vcf_germline` | Shells out to `bcftools index`/`bcftools view`/`bcftools annotate` |
+| `bolt/common/pcgr.py: run_somatic` / `run_somatic_chunk` (execution path) / `run_germline` | Invoke `pcgr`/`cpsr` CLI directly |
+| `bolt/common/pcgr.py: transfer_annotations_somatic` / `transfer_annotations_germline` | Depend on real PCGR/CPSR TSV+VCF output files |
+| `bolt/common/pcgr.py: merging_pcgr_files` | Wraps `merge_vcf_files`/`merge_tsv_files` (bcftools-dependent) |
+| `bolt/workflows/smlv_somatic/annotate.py` | Orchestrates vcfanno + PON + PCGR subprocess pipeline; no unit tests |
+| `bolt/workflows/smlv_somatic/rescue.py` (all functions except the header check) | SAGE hotspot recall orchestrates `bcftools isec`/`concat`/`annotate` subprocesses end-to-end; no unit tests |
+| `bolt/workflows/smlv_somatic/prepare.py` | bcftools-based VCF prep; no unit tests |
+| `bolt/workflows/smlv_germline/prepare.py` | bcftools-based panel region selection; no unit tests |
+| `bolt/workflows/smlv_germline/report.py` | bcftools stats + CPSR report generation; no unit tests |
+| `bolt/workflows/sv_somatic/annotate.py` | snpEff subprocess annotation; no unit tests |
+| `bolt/workflows/sv_somatic/prioritise.py` | Wraps `external/prioritize_sv.py`; no unit tests |
+| `bolt/workflows/other/cancer_report.py` | Invokes `gpgr.R canrep` subprocess; no unit tests |
+| `bolt/workflows/other/multiqc_report.py` | Invokes `multiqc` subprocess; no unit tests |
+| `bolt/workflows/other/purple_baf_plot.py` | Invokes `circos` subprocess; no unit tests |
+| `bolt/external/prioritize_sv.py` | Vendored third-party AstraZeneca SV annotation logic; no unit tests |
+
+Contributions raising coverage for these are welcome, but will require mocking
+subprocess calls (as done in `tests/test_smlv_somatic_report.py` for
+`select_pcgr_variants`) or dedicated fixture VCFs/TSVs under `tests/fixtures/`.

bolt/common/pcgr.py

@@ -146,7 +146,8 @@ def run_somatic(input_fp, pcgr_refdata_dir, vep_dir, output_dir, chunk_nbr=None,
         f'--control_af_tag NORMAL_AF',
         f'--genome_assembly grch38',
         f'--assay WGS',
-        *([] if disable_estimates else ['--estimate_signatures', '--estimate_msi', '--estimate_tmb']),
+        # NOTE: --estimate_signatures dropped (umccr/sash#57); keep msi/tmb.
+        *([] if disable_estimates else ['--estimate_msi', '--estimate_tmb']),
         f'--vcfanno_n_proc {vcfanno_threads}',
         f'--vep_n_forks {vep_forks}',
         f'--vep_pick_order biotype,rank,appris,tsl,ccds,canonical,length,mane_plus_clinical,mane_select',
@@ -352,6 +353,25 @@ def transfer_annotations_germline(input_fp, normal_name, cpsr_dir, output_dir):
     output_fh.close()
 
 
+# Lower value = more clinically actionable. Used to resolve duplicate PCGR TSV
+# rows for the same variant (PCGR can emit one row per transcript mapping).
+_TIER_ORDER = {'1': 0, '2': 1, '3': 2, '4': 3, 'N': 4}
+
+
+def _normalise_tier(raw_tier):
+    # Normalize PCGR actionability tier to simple values: '1','2','3','4','N'
+    tier_norm = (raw_tier or '').strip().replace('_', ' ').upper()
+    if tier_norm in ('TIER 1', 'TIER1', '1'):
+        return '1'
+    elif tier_norm in ('TIER 2', 'TIER2', '2'):
+        return '2'
+    elif tier_norm in ('TIER 3', 'TIER3', '3'):
+        return '3'
+    elif tier_norm in ('TIER 4', 'TIER4', '4'):
+        return '4'
+    return 'N'
+
+
 def collect_pcgr_annotation_data(tsv_fp, vcf_fp, info_field_map):
     # Gather all annotations from TSV
     data_tsv = dict()
@@ -360,23 +380,19 @@ def collect_pcgr_annotation_data(tsv_fp, vcf_fp, info_field_map):
     with open_fn(tsv_fp, 'rt') as tsv_fh:
         for record in csv.DictReader(tsv_fh, delimiter='\t'):
             key, record_ann = get_annotation_entry_tsv(record, info_field_map)
-            assert key not in data_tsv
-
-            # Normalize PCGR actionability tier to simple values: '1','2','3','4','N'
-            raw_tier = (record.get('ACTIONABILITY_TIER') or '').strip()
-            tier_norm = raw_tier.replace('_', ' ').upper()
-            if tier_norm in ('TIER 1','TIER1','1'):
-                tier_val = '1'
-            elif tier_norm in ('TIER 2','TIER2','2'):
-                tier_val = '2'
-            elif tier_norm in ('TIER 3','TIER3','3'):
-                tier_val = '3'
-            elif tier_norm in ('TIER 4','TIER4','4'):
-                tier_val = '4'
-            else:
-                tier_val = 'N'
+            tier_val = _normalise_tier(record.get('ACTIONABILITY_TIER'))
             record_ann[constants.VcfInfo.PCGR_ACTIONABILITY_TIER] = tier_val
 
+            # NOTE(QC): PCGR can emit multiple TSV rows for the same variant when it
+            # maps to multiple transcripts. Keep the most actionable entry (lowest
+            # tier number) rather than asserting. See https://github.com/umccr/bolt/pull/34
+            if key in data_tsv:
+                existing_tier = data_tsv[key][constants.VcfInfo.PCGR_ACTIONABILITY_TIER]
+                if _TIER_ORDER[tier_val] >= _TIER_ORDER[existing_tier]:
+                    logger.warning(f'Duplicate PCGR TSV key {key}: keeping tier {existing_tier}, skipping tier {tier_val}')
+                    continue
+                logger.warning(f'Duplicate PCGR TSV key {key}: replacing tier {existing_tier} with more actionable tier {tier_val}')
+
             # Store annotation data
             data_tsv[key] = record_ann
 
@@ -404,7 +420,15 @@ def collect_cpsr_annotation_data(tsv_fp, vcf_fp, info_field_map):
             record['ALT'] = re_result.group('alt')
 
             key, record_ann = get_annotation_entry_tsv(record, info_field_map)
-            assert key not in data_tsv
+
+            # NOTE(QC): CPSR can emit multiple TSV rows for the same variant when it
+            # maps to multiple transcripts (same class of issue as the PCGR somatic
+            # TSV path above). CPSR rows carry no actionability tier to break ties
+            # on, so keep the first entry and warn. See https://github.com/umccr/bolt/pull/34
+            if key in data_tsv:
+                logger.warning(f'Duplicate CPSR TSV key {key}: keeping first entry')
+                continue
+
             data_tsv[key] = record_ann
 
     # Gather annotations from VCF
@@ -414,18 +438,14 @@ def collect_cpsr_annotation_data(tsv_fp, vcf_fp, info_field_map):
     return compile_annotation_data(data_tsv, data_vcf)
 
 def parse_genomic_change(genomic_change):
-    """
-    Parse a genomic change string, e.g., "3:g.41224645T>C"
-    Returns a tuple: (chrom, pos, ref, alt)
-    """
-    # Regular expression for the format "chrom:g.posRef>Alt"
+    # Format: "chrom:g.posRef>Alt" e.g. "3:g.41224645T>C"
     pattern = r'^(?P<chrom>\w+):g\.(?P<pos>\d+)(?P<ref>\w+)>(?P<alt>\w+)$'
     match = re.match(pattern, genomic_change)
     if not match:
         raise ValueError(f"Format not recognized: {genomic_change}")
-
-    # Get values and format as needed
-    chrom = f"chr{match.group('chrom')}"
+    chrom = match.group('chrom')
+    if not chrom.startswith('chr'):
+        chrom = f'chr{chrom}'
     pos = int(match.group('pos'))
     ref = match.group('ref')
     alt = match.group('alt')
@@ -439,7 +459,14 @@ def get_annotations_vcf(vcf_fp, info_field_map):
         assert len(record.ALT) == 1
         [alt] = record.ALT
         key = (f'chr{record.CHROM}', record.POS, record.REF, alt)
-        assert key not in data_vcf
+
+        # NOTE(QC): PCGR can emit duplicate variant entries in its output VCF when a
+        # variant maps to multiple transcripts. Keep the first entry and warn rather
+        # than asserting, consistent with the TSV-side dedup.
+        # https://github.com/umccr/bolt/pull/34
+        if key in data_vcf:
+            logger.warning(f'Duplicate PCGR VCF key {key}: keeping first entry')
+            continue
 
         data_vcf[key] = dict()
         for info_dst, info_src in info_field_map.items():
@@ -505,7 +532,7 @@ def annotate_record(record, annotations, *, allow_missing=False):
         if allow_missing:
             return record
         else:
-            assert key not in annotations
+            assert False, f'Missing annotation key: {key}'
 
     # Transfer annotations
     for info_enum, v in annotations[key].items():
@@ -514,11 +541,6 @@ def annotate_record(record, annotations, *, allow_missing=False):
     return record
 
 def split_vcf(input_vcf, output_dir, *, max_variants=None):
-    """
-    Splits a VCF file into multiple chunks, each containing up to max_variants variants.
-    Each chunk includes the VCF header.
-    Ensures no overlapping positions between chunks.
-    """
     if max_variants is None:
         max_variants = constants.MAX_SOMATIC_VARIANTS
     elif max_variants <= 0:
@@ -566,27 +588,25 @@ def split_vcf(input_vcf, output_dir, *, max_variants=None):
 def run_somatic_chunk(vcf_chunks, pcgr_data_dir, vep_dir, output_dir, pcgr_output_dir, max_threads, pcgr_conda, pcgrr_conda):
     pcgr_tsv_files = []
     pcgr_vcf_files = []
-
-    # Process each chunk sequentially
+
     for chunk_number, vcf_file in enumerate(vcf_chunks, start=1):
-        pcgr_tsv_fp, pcgr_vcf_fp = run_somatic(vcf_file, pcgr_data_dir, vep_dir, pcgr_output_dir, chunk_nbr=chunk_number, threads=max_threads, pcgr_conda=pcgr_conda, pcgrr_conda=pcgrr_conda)
+        pcgr_tsv_fp, pcgr_vcf_fp = run_somatic(vcf_file, pcgr_data_dir, vep_dir, pcgr_output_dir, chunk_nbr=chunk_number, threads=max_threads, pcgr_conda=pcgr_conda, pcgrr_conda=pcgrr_conda, disable_estimates=True)
         if pcgr_tsv_fp:
             pcgr_tsv_files.append(pcgr_tsv_fp)
         if pcgr_vcf_fp:
             pcgr_vcf_files.append(pcgr_vcf_fp)
-    
+
     merged_vcf_fp, merged_tsv_fp = merging_pcgr_files(output_dir, pcgr_vcf_files, pcgr_tsv_files)
     return merged_tsv_fp, merged_vcf_fp
 
+
 def merging_pcgr_files(output_dir, pcgr_vcf_files, pcgr_tsv_files):
     pcgr_dir = pathlib.Path(output_dir) / 'pcgr'
     pcgr_dir.mkdir(exist_ok=True)
 
-    # Merge all TSV files into a single file in the pcgr directory
     merged_tsv_fp = pcgr_dir / "nosampleset.pcgr_acmg.grch38.snvs_indels.tiers.tsv.gz"
     util.merge_tsv_files(pcgr_tsv_files, merged_tsv_fp)
 
-    # Step 5: Merge all VCF files into a single file in the pcgr directory
     merged_vcf_path = pcgr_dir / "nosampleset.pcgr.grch38.pass"
     if len(pcgr_vcf_files) == 1:
         # NOTE(QC): bcftools merge requires 2+ inputs; with a single chunk there is