Add DeepMAsED for deep learning evaluation of metagenomic de novo assemblies by SkyLexS · Pull Request #1051 · nf-core/mag

SkyLexS · 2026-06-17T19:19:03Z

PR checklist

…cker

…/shm limit

Release 5.4.2

nf-core-bot · 2026-06-17T19:19:49Z

Warning

Newer version of the nf-core template is available.

Your pipeline is using an old version of the nf-core template: 3.5.2.
Please update your pipeline to the latest version.

For more documentation on how to update your pipeline, please see the nf-core documentation and Synchronisation documentation.

jfy133

Initial pass, but I leave the proper review to @dialvarezs :)

jfy133 · 2026-06-18T12:56:05Z

+        cpus   = { 12 * task.attempt }
+        memory = { 120.GB * task.attempt }
+        time   = { 96.h * task.attempt }


Default resource requests should be moved to base.config

jfy133 · 2026-06-18T12:56:19Z

+        cpus   = { 12 * task.attempt }
+        memory = { 120.GB * task.attempt }
+        time   = { 96.h * task.attempt }
+        ext.args = params.deepmased_features_gzip ? '--gzip' : ''


At a minimum should be wrapped in a closure, see other sfor examples

jfy133 · 2026-06-18T12:56:59Z

+        memory = { 120.GB * task.attempt }
+        time   = { 96.h * task.attempt }
+        ext.args = params.deepmased_features_gzip ? '--gzip' : ''
+        containerOptions = '--shm-size=8g'


Again something that should be in base.config ,but I worry about hardcoding such a thing - what is that exactly?

jfy133 · 2026-06-18T12:57:10Z

+        memory = { 24.GB * task.attempt }
+        time   = { 4.h * task.attempt }
+        ext.args = [ params.deepmased_cpu_only ? '--cpu-only' : '', "--seed ${params.deepmased_predict_seed}" ].join(' ').trim()
+        containerOptions = '--shm-size=8g'


Same comments as above

jfy133 · 2026-06-18T12:58:44Z

+Per-contig assembly error detection is also performed using [DeepMAsED](https://github.com/leylabmpi/DeepMAsED), a deep learning tool that predicts misassemblies from read alignment features. DeepMAsED runs in two sequential steps:
+
+1. **`DeepMAsED features`** (`DEEPMASED_FEATURES`): Extracts alignment-based features from the BAM file and assembly FASTA for each contig. This step is I/O intensive and is the most time-consuming part of the analysis, especially for large assemblies.
+2. **`DeepMAsED predict`** (`DEEPMASED_PREDICT`): Runs the pre-trained deep learning model on the feature tables to produce a per-contig misassembly score (0 = correctly assembled, 1 = likely misassembly).
+
+The two steps are implemented as separate modules, allowing Nextflow to resume from `predict` if `features` has already completed successfully. DeepMAsED only runs on short-read assemblies (MEGAHIT, SPAdes).
+
+The following parameters control DeepMAsED execution:
+
+| Parameter                   | Description                                                                                                           |
+| --------------------------- | --------------------------------------------------------------------------------------------------------------------- |
+| `--skip_deepmased`          | Skip DeepMAsED entirely (both features and predict)                                                                   |
+| `--skip_deepmased_predict`  | Run only the features step, skip prediction                                                                           |
+| `--skip_deepmased_features` | **Not allowed** unless `--skip_deepmased_predict` is also set — DeepMAsED predict requires features output            |
+| `--deepmased_cpu_only`      | Run DeepMAsED predict in CPU-only mode. Default: `true` (recommended for HPC environments without GPU)                |
+| `--deepmased_features_gzip` | Gzip feature tables produced by the features step. Useful for large assemblies to reduce disk usage. Default: `false` |
+| `--deepmased_predict_seed`  | Random seed for numpy in DeepMAsED predict. Set for reproducible results. Default: `12`                               |


I do'nt think any of thesea re related to repdocubility (the section it is in). I also feel it is a lot of partly unnecessary detail that can be derived from the tool's docs itself? At a minimum (if you want to keep them), move to the end under the 'Note on....' sections at teh bottom of this page.

jfy133 · 2026-06-18T13:03:22Z

+            ch_versions = ch_versions.mix(DEEPMASED_FEATURES.out.versions_deepmased.ifEmpty([]))
+            ch_versions = ch_versions.mix(DEEPMASED_FEATURES.out.versions_setuptools.ifEmpty([]))


Suggested change

ch_versions = ch_versions.mix(DEEPMASED_FEATURES.out.versions_deepmased.ifEmpty([]))

ch_versions = ch_versions.mix(DEEPMASED_FEATURES.out.versions_setuptools.ifEmpty([]))

Not needed when the module is using topics for version reporting

jfy133 · 2026-06-18T13:03:47Z

+                ch_versions = ch_versions.mix(DEEPMASED_PREDICT.out.versions_deepmased.ifEmpty([]))
+                ch_versions = ch_versions.mix(DEEPMASED_PREDICT.out.versions_setuptools.ifEmpty([]))


Suggested change

ch_versions = ch_versions.mix(DEEPMASED_PREDICT.out.versions_deepmased.ifEmpty([]))

ch_versions = ch_versions.mix(DEEPMASED_PREDICT.out.versions_setuptools.ifEmpty([]))

Not needed when the module is using topics for version reporting

jfy133 · 2026-06-18T13:04:05Z

 - [#1041](https://github.com/nf-core/mag/pull/1041) - Refined and corrected unclear section of metromap (by @jfy133)
 - [#1044](https://github.com/nf-core/mag/pull/1044) - Add new `--gtdbtk_place_species` parameter (by @dialvarezs)
 - [#1047](https://github.com/nf-core/mag/issues/1007) - Add `--gtdbtk_single_job` to run GTDB-Tk classification for all bins in a single job (requested by @sarah-shah-bioinf, by @dialvarezs)
+- Add DeepMAsED assembly error detection to the MAG workflow as two sequential steps (`features` and `predict`) for short-read assemblies.


Missing PR number and your name

jfy133 · 2026-06-18T13:04:34Z

+- Update DeepMAsED module tests to the current `sanitizeOutput(process.out)` snapshot pattern and regenerate snapshots.
+- Update version aggregation compatibility to support topic-based module version tuples.


Suggested change

- Update DeepMAsED module tests to the current `sanitizeOutput(process.out)` snapshot pattern and regenerate snapshots.

- Update version aggregation compatibility to support topic-based module version tuples.

The aggregation support should already be in through the template? Is that not the case? (and the cause of the stuff at the top)?

jfy133 · 2026-06-18T13:04:58Z

 - [#1021](https://github.com/nf-core/mag/pull/1021) - Prevent execution of `gtdbtk/summary` when no bins pass QC (reported by @jfy133, fix by @dialvarezs)
 - [#1031](https://github.com/nf-core/mag/pull/1031) - Fix hybrid co-assembly with SPAdes (short & long reads with `--coassemble_group`) (fix by @d4straub)
 - [#1049](https://github.com/nf-core/mag/pull/1049) - Fix publishing issue with `gtdbtk/classifywf` (by @dialvarezs)
+- Validate DeepMAsED skip-parameter dependencies early, preventing invalid combinations where `predict` would run without `features` outputs.


Suggested change

- Validate DeepMAsED skip-parameter dependencies early, preventing invalid combinations where `predict` would run without `features` outputs.

Not necessary as DeepMAsED is not in any past release ;)

… the nftignore because deepmased uses Deep Learning TensorFlow and when you run predictions for the same input the output could be of by one decimal and fail the test

jfy133 · 2026-06-24T07:42:31Z

Still have test various failures due to variation in the snapshot...

SkyLexS and others added 14 commits March 17, 2026 09:07

Initial implementation of deep mased

caf4707

Fix DEEPMASED features: use -p 1 to avoid multiprocessing crash in Do…

7b1aa6e

…cker

Set DEEPMASED time limit to 48h

9fcc1d2

DEEPMASED: restore -p task.cpus, add --shm-size=8g to fix Docker /dev…

7a7c780

…/shm limit

DEEPMASED: set 12 CPUs, 72GB RAM

408d2b7

increasing time and resources for deepmased

589b38c

Merge pull request nf-core#1005 from nf-core/dev

5dabb01

Release 5.4.2

Split DEEPMASED into DEEPMASED_FEATURES and DEEPMASED_PREDICT

9c2480b

removed hardcodded parameter

20bda3e

removed hardcodded parameter

b146ca8

Merge branch 'nf-core:main' into deep_mased_imp

4ceb899

added optional parameters in the module.config

996f2d8

sincroninzing modules and mag

33a9052

syncronised with modules

0d9a431

SkyLexS requested review from d4straub, dialvarezs, jfy133, muabnezor and prototaxites as code owners June 17, 2026 19:19

SkyLexS and others added 4 commits June 17, 2026 22:21

Merge branch 'dev' into deep_mased_imp

ea43a6b

fixing lint and test snapshots

5b21b5d

fixing conflicts

169e582

regenerated the test snapshots

8c5e2b4

jfy133 changed the title ~~Deep mased imp~~ Add DeepMAsED for deep learning evaluation of metagenomic de novo assemblies Jun 18, 2026

jfy133 reviewed Jun 18, 2026

View reviewed changes

SkyLexS and others added 3 commits June 21, 2026 02:26

rerun the tests and moved config to base config also edited the md files

d5a61b8

Merge branch 'dev' into deep_mased_imp

32a8edb

modified a bit the tests and dealt with deepmased added some files in…

742ed7e

… the nftignore because deepmased uses Deep Learning TensorFlow and when you run predictions for the same input the output could be of by one decimal and fail the test

SkyLexS and others added 5 commits July 12, 2026 16:48

Merge branch 'dev' into deep_mased_imp

45461a9

added PR number on changelog

9773bca

resolving linting errors

d5b38e4

fixing linting errors

f491aed

fixing test snapshots and removing undeterministic tests

17c85f1

dialvarezs mentioned this pull request Jul 14, 2026

[tracker] new tools #1022

Open

SkyLexS and others added 7 commits July 14, 2026 10:16

fixing more tests

0a0d96c

Merge branch 'dev' into deep_mased_imp

101a8e2

fixing test snapshots

a2e83e6

Merge branch 'dev' into deep_mased_imp

1284788

fixing snapshots

bc79433

Merge branch 'dev' into deep_mased_imp

34b95e8

fixing snapshots

41edf6e

		ch_versions = ch_versions.mix(DEEPMASED_FEATURES.out.versions_deepmased.ifEmpty([]))
		ch_versions = ch_versions.mix(DEEPMASED_FEATURES.out.versions_setuptools.ifEmpty([]))

		- Update DeepMAsED module tests to the current `sanitizeOutput(process.out)` snapshot pattern and regenerate snapshots.
		- Update version aggregation compatibility to support topic-based module version tuples.

Uh oh!

Conversation

SkyLexS commented Jun 17, 2026

PR checklist

Uh oh!

nf-core-bot commented Jun 17, 2026

Uh oh!

jfy133 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jfy133 commented Jun 24, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants