feat: migrate publishDir to workflow output definition (pilot, 15/19) - #65
feat: migrate publishDir to workflow output definition (pilot, 15/19)#65qclayssen wants to merge 2 commits into
Conversation
|
Pushed `652c433` applying the confirmed simplification from the review: dropped the 4 redundant `_raw` emit channels (`purple_dir_raw`, `somatic_raw`, `germline_raw`, `plot_dir_raw`). Cross-checked against Nextflow's own docs/tutorial for the workflow output system before making this change — there's no precedent there for a parallel "raw meta" channel; closures just read whatever field the channel's existing meta carries. Combined with the earlier empirical proof that `restoreMeta()`'s join means the restored channel's `meta.id` always equals what `meta.key` would be on the raw channel, this confirmed the simplification was safe: switched `main.nf`'s `purple_dir`/`linx_germline_annotation_dir`/`linx_somatic_annotation_dir`/`linx_somatic_plot_dir` path closures to read `meta.id` off the already-existing restored channels instead. Also renamed `linxreport_raw` → `linxreport` for clarity — that one wasn't actually redundant (LINXREPORT's output never goes through `restoreMeta()`, so it was always the only channel available), just misleadingly named after the same pattern. `nextflow lint`: 0/0. `nf-test`: 11/13 (unchanged). `--help` parses clean. |
652c433 to
cf64145
Compare
run.sh's production invocation passes a bare --monochrome_logs flag (no value), which Nextflow's CLI parser sets as the String "true" before schema-based param coercion runs. Nextflow 26.04.6's stricter static method dispatch no longer silently coerces that into NfcoreTemplate.logColours(Boolean), so every #64/#65 run failed at launch before submitting a single process. Two independent crash sites, same root cause: - NfcoreTemplate.logColours(): relaxed the Boolean-typed signature and coerce internally. - NfcoreSchema.cleanParameters(): schema validation separately rejected the String "true" against the schema's declared boolean type; added a matching String->Boolean cast alongside the existing MemoryUnit/ Duration/LinkedHashMap casts. Verified locally under NXF_VER=26.04.6: --help and -stub-run with a bare --monochrome_logs flag both clear params-summary and schema validation with no MissingMethodException or ValidationException (previously reproduced and fixed only as an unpushed, instance-local workaround on i-05550e6eaf958e606, branch fix/64-monochrome-logs-coercion).
…(pilot)
Migrates the 15 "simple" publishDir/saveAs blocks (pure subdir+filename
formulas, no filename-content routing) to Nextflow's workflow output
definition (publish:/output{}), leaving the 4 filename-routing processes
(BOLT_SMLV_SOMATIC_REPORT, BOLT_SMLV_GERMLINE_REPORT,
BOLT_OTHER_CANCER_REPORT, BOLT_OTHER_MULTIQC_REPORT) on legacy publishDir.
Adds small pass-through `emit:` plumbing (process-scoped meta, needed for
`meta.key`) through purple_calling/linx_annotation/linx_plotting ->
workflow SASH -> UMCCR_SASH -> entry workflow, where the new publish:/
output{} blocks live. Path formulas were validated against live nextflow
runs (26.04.6) to byte-for-byte reproduce prior publishDir/saveAs output
layouts, including directory-flattening edge cases.
Code review + Nextflow docs research confirmed purple_dir_raw,
somatic_raw, germline_raw, and plot_dir_raw were unnecessary: each
subworkflow's already-restored emit (via WorkflowSash.restoreMeta())
carries a meta.id that's provably identical to what meta.key would be
on the raw channel, since restoreMeta joins ch_inputs on that same id.
Official Nextflow workflow-output docs/tutorial show no precedent for
a parallel "raw meta" channel pattern -- closures just read whatever
field the channel's existing meta already carries.
- purple_calling.nf / linx_annotation.nf: drop the _raw emits, keep
only the restored ones.
- linx_plotting.nf: drop plot_dir_raw (redundant); rename
linxreport_raw -> linxreport (not actually redundant -- LINXREPORT's
output never goes through restoreMeta, so this was always the only
channel, just misleadingly named after the same pattern).
- workflows/sash.nf: point SASH's emit block at the restored channels.
- main.nf: switch the corresponding output{} path closures from
meta.key to meta.id to match.
nextflow lint: 0/0. nf-test: 11/13 (unchanged, 2 pre-existing failures
unrelated to this change). --help parses clean.
cf64145 to
65f90a6
Compare
Summary
Pilots migrating sash's
publishDirblocks to Nextflow's new workflow output definition (publish:section + top-leveloutput {}block), which came out of preview and is stable as of Nextflow 25.10.0 — no feature flag needed on the 26.04.6 this branch already runs (built on top of #64).publishDiris being deprecated in favor of this.Migrates 15 of sash's 19
publishDirblocks inconf/modules.config. Verified path-identical to the currentpublishDir/saveAsoutput layout via ~10 standalone empirical test pipelines, since the new syntax's directory/subdir-prefix semantics aren't well documented and differ subtly from legacysaveAs.Base branch is
refactor/nextflow-strict-syntax(#64) since this depends on the Nextflow 26.04.6 bump — rebase ontomainonce that merges.What changed
conf/modules.config: removedpublishDirfrom the 15 migratedwithName:blocks.main.nf: added thepublish:section and a new top-leveloutput {}block on the entryworkflow {}(Nextflow only permitspublish:in the unnamed entry workflow).workflows/sash.nf/main.nf(UMCCR_SASH): addedemit:blocks passing the 18 named channels needed by the publish paths up from the subworkflows to the entry workflow.subworkflows/local/{purple_calling,linx_annotation,linx_plotting}.nf: added small additive_rawemits alongside each subworkflow's existing emit, exposing the process-scopedmeta(needed formeta.keyin publish paths). No logic changes.nextflow.config:workflow.output.mode = params.publish_dir_mode, the new system's equivalent of the old per-blockmode:.What's deliberately NOT migrated (4 of 19)
BOLT_SMLV_SOMATIC_REPORT,BOLT_SMLV_GERMLINE_REPORT,BOLT_OTHER_CANCER_REPORT,BOLT_OTHER_MULTIQC_REPORTremain onpublishDirinconf/modules.config, unchanged. All four do filename-pattern-based routing/renaming inside theirsaveAsclosure — e.g.BOLT_SMLV_SOMATIC_REPORTpulls one specific*.pcgr.grch38.htmlout of a directory of PCGR output files and renames it to${meta.key}/${meta.tumor_id}.pcgr.html, while everything else in that same process's output goes to a different subpath (${meta.key}/smlv_somatic/report/).BOLT_SMLV_GERMLINE_REPORTdoes the same for CPSR HTML;BOLT_OTHER_CANCER_REPORT/BOLT_OTHER_MULTIQC_REPORTdo it for the cancer/multiqc reports.The new workflow-output system's
path {}closures operate on a channel's structured value (meta, file ->), not on arbitrary filenames matched against a directory the waysaveAsdoes. Reproducing this exactly requires splitting each of these 4 processes' ownoutput:block into separate named emit channels first (e.g. one channel for "the html report", one for "the rest of the output directory") — a change to the module.nffiles themselves, not just config/workflow wiring. That's a materially bigger and riskier change than this pilot (touches process-level output declarations for report-generating processes whose output is what clinicians/curators actually review), so it's left as follow-up work, tracked here rather than attempted in this PR.publishDirremains fully functional (deprecated, not removed) in the meantime, so nothing breaks by leaving these 4 as-is.Verification
nextflow lint . -exclude .git -exclude work -exclude .nf-test -exclude nf-test.config -o concise— 0 errors, 0 warnings.nextflow run . --help— parses cleanly.NXF_SYNTAX_PARSER=v1 nf-test test— 11/13 passing, matching the known-good baseline on refactor: Nextflow 26.04.6 + strict syntax migration #64 exactly (2 pre-existing failures, unrelated).nextflow run . -profile test,docker -stub-run— compiles the newpublish:/output {}wiring with no structural error (fails only on the pre-existingconf/test.configmissing-fixture gap tracked in refactor: Nextflow 26.04.6 + strict syntax migration #64, not on anything from this PR).docs/testing-validation.mdtier 3) is the actual sign-off before this merges.Follow-up (separate work, not this PR)
publishDirblocks.main's, byte-for-byte on the migrated paths.