refactor: virtual-stain --init/--copy with Nextflow subworkflow - #259
Closed
aofei-liu wants to merge 5 commits into
Closed
refactor: virtual-stain --init/--copy with Nextflow subworkflow#259aofei-liu wants to merge 5 commits into
aofei-liu wants to merge 5 commits into
Conversation
4 tasks
aofei-liu
force-pushed
the
refactor-reconstruct
branch
from
June 1, 2026 22:08
2b07aa8 to
64ce8fe
Compare
aofei-liu
force-pushed
the
refactor-virtual-stain
branch
from
June 1, 2026 22:08
b3ece23 to
ee75ce4
Compare
aofei-liu
force-pushed
the
refactor-reconstruct
branch
from
June 4, 2026 02:18
eaa861d to
e4120c5
Compare
aofei-liu
force-pushed
the
refactor-virtual-stain
branch
from
June 4, 2026 02:19
d30e05c to
a7f854b
Compare
aofei-liu
force-pushed
the
refactor-reconstruct
branch
from
June 8, 2026 20:51
a8d8bdd to
7015763
Compare
aofei-liu
force-pushed
the
refactor-virtual-stain
branch
from
June 8, 2026 20:51
fe5f52d to
9c2a486
Compare
…workflow Add --init and --copy modes to the unified `biahub virtual-stain` CLI: - --init: creates output plate with prediction channels from config, copies per-position OME metadata, emits RESOURCES: line - --copy: copies viscy prediction from temp per-position zarr into the output plate, then cleans up the temp zarr The predict config (-c) is now shared across all modes (replaces the old --predict-config-filepath). The -i option is optional (not needed for --copy mode). Nextflow module (virtual_stain.nf) wires the subworkflow: init → preprocess (viscy) → fan-out (predict + copy) × N positions Viscy preprocess/predict remain as external commands called directly from Nextflow processes — only init and copy are biahub CLI commands. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…irtual_stain Derive position_keys from input_position_dirpaths and read shape/scale from the first position directly, matching the pattern used by deskew, flat-field, reconstruct, and track. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
virtual_stain_wf now receives input_zarr, output_zarr, config as explicit val parameters. Temp zarr and log paths derived from output_zarr's parent directory. viscy_cmd() stays as-is. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
aofei-liu
force-pushed
the
refactor-reconstruct
branch
from
June 8, 2026 20:56
7015763 to
0fb0627
Compare
aofei-liu
force-pushed
the
refactor-virtual-stain
branch
from
June 8, 2026 20:56
9c2a486 to
16c0d9c
Compare
3 tasks
3 tasks
Collaborator
ieivanov
added a commit
that referenced
this pull request
Jun 23, 2026
* feat(nextflow): integrate virtual-stain into mantis-v2 pipeline Recreate the Nextflow virtual-stain integration against the current in-process cytoland CLI (post-#267), replacing the #259 approach which was built on the old `viscy predict` + `--copy` temp-zarr flow. - New `nextflow/modules/virtual_stain.nf`: init → preprocess → fan-out per-position GPU prediction. Per-position work is a single `biahub virtual-stain --cluster debug` call (no temp zarr, no --copy). - `viscy preprocess` runs over the whole input plate to compute the normalization statistics the model reads via read_norm_meta. - Both `biahub virtual-stain` and `viscy` run under biahub's `stain` extra (cytoland → viscy-utils provides the `viscy` console script). - Add a `gpu` process label (queue=gpu) to nextflow.config. - Wire virtual_stain_wf after reconstruct in mantis-v2.nf. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix biahub virtual-stain mem request * refactor: unify per-position resource requests across step CLIs Every step CLI (deskew, flat-field, apply-inv-tf, virtual-stain) now emits its per-position resource request through a single shared helper, biahub.cli.utils.echo_resources, as a JSON payload: RESOURCES:{"cpus": 16, "mem_gb": 64, "time_min": 480} The same (cpus, total mem_gb, time_min) values feed both the CLI's own slurm_* submission args and, via parse_resources in nextflow, the Nextflow per-position task directives — so the SLURM fan-out and the Nextflow fan-out can no longer request different resources. - Add echo_resources() and parse the JSON in common.nf::parse_resources, which now returns time_min alongside cpus/mem_gb. - Emit memory as a TOTAL (mem_gb) and request it via slurm_mem rather than slurm_mem_per_cpu, fixing virtual-stain's per-cpu over-request. - Wire each CLI's slurm_time from time_min; drop the hardcoded `time` literals in the run_* processes for `meta.time_min * task.attempt`. - virtual-stain computes time_min before the init_only return so --init emits it. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * payload bugfix * debug viscy predict * refactor: rename time_min to time_minutes for clarity Avoids confusion with "minimum time"; renames the resource payload key and Nextflow meta field consistently across both languages. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * update iohub dep --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
--initand--copymodes tobiahub virtual-stainCLI, absorbinginit-virtual-stainandcopy-virtual-stainfromnf.py--initcreates the output plate with<channel>_predictionchannels from predict config, copies per-position OME metadata, emitsRESOURCES:line--copycopies viscy prediction data from a temp per-position zarr into the output plate, then removes the temp zarr-coption (replaces old--predict-config-filepath)-iis now optional (not needed for--copymode)nextflow/modules/virtual_stain.nfsubworkflow: init → preprocess (viscy) → fan-out (predict + copy) × N positionsChained PR — base is
refactor-reconstruct(#258).Test plan
test_virtual_stain_cli_init_only,test_virtual_stain_cli_copy)--initagainst real data — channels/shape/scale match reference exactly, emittedRESOURCES:10 120list_positions (1/1) → init (1/1) → preprocess (1/1) → run_virtual_stain (4/4)completed in 20m 56s, zero failuresTier 3 output verification
['nuclei_prediction', 'membrane_prediction'], shape=(5, 2, 86, 1600, 1370)[1.0, 1.0, 0.17, 0.1133, 0.1133]3-virtual-stainwas produced from a reconstruct output with a duplicate-channel bug (see below)Data fixes (outside this PR)
2-reconstruct: had duplicate identical channels (Phase3D_recon_recon,Phase3D_recon). Fixed to singlePhase3Dchannel. Old store backed up.predict.yml:ckpt_pathpointed to a stale path; fixed to include the4gpu_bf16_bs16_to_ep7/subdirectory.🤖 Generated with Claude Code