image-sims workflow: campaign hardening - #287
Open
cailmdaley wants to merge 3 commits into
Open
Conversation
Collaborator
Author
|
Added 36dd99b: the m-bias step hardcoded a — Claude (Fable) on behalf of Cail |
cailmdaley
force-pushed
the
imsims-workflow-upstream
branch
from
August 31, 2026 00:51
790363a to
99f1ca9
Compare
cailmdaley
changed the title
cailmdaley
force-pushed
the
imsims-workflow-upstream
branch
from
August 31, 2026 00:55
99f1ca9 to
3f1bbcd
Compare
The ShapePipe pipeline stage's ngmix ini (which owns METACAL_PSF) is resolved from the shapepipe worktree's example/cfis_image_sims via run_job's self-computed config_dir, so nothing in the run config could select a different ini -- an A/B on the metacal reconvolution PSF needed a separate worktree per arm. Add an optional operational key `pipeline_config_dir`. When set it is passed to run_job as `-c DIR`, redirecting the pipeline stage's whole ShapePipe config tree (ngmix ini included) at a per-arm copy; when empty (the default) no `-c` is passed and run_job self-computes exactly as before, so runs predating this key reproduce bit-exact. Merge/final_cat still read the top-level cfis symlink (config_dir), untouched. Candidate upstream PR for the image-sims orchestration branch. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XSWPiisH6y7NFFbjHqthuC
ImageSimMBias.load_catalogs hardcoded the '_grid_' infix between branch
name and run number, so blended arms ({branch}_{num} dirs) hit
FileNotFoundError on a '{branch}_grid_{num}' path that never existed.
Derive the suffix from sims_type (defaulting to grid for backward
compat and the synthetic-recovery tests), mirroring the workflow's own
_SUFFIX contract. Emit sims_type into the generated m_bias_config, and
bump im_mbias mem to 8G since blended catalogues are ~5x grid size and
OOM the snakemake 1000M default.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HubFhoU6bH8ij6hcTWxw7s
…lization
Add an optional `exp_num` config key so the exposure input can be pulled from a
different SKiLLS realization than the tiles. Tiles stay on `{branch}_{num}`;
exposures resolve `{branch}_{exp_num}` via a new SIM_EXP_BASE mapping, consumed
by rule im_init's `sim_exp` symlink source. Unset `exp_num` defaults to `num`,
reproducing prior behaviour bit-for-bit. Guarded to non-grid sims (grid names
embed sims_type; the pathology this addresses is a blended-family concern).
Motivation: the blended-family realizations _1.._4 carry stale cross-linked
SP_exp (grid-sims exposures symlinked in, signal-free); _5 is the clean all-real
exposure set. This reproduces Fabian's production split (exposures from _5, tiles
from _1), unblocking the blended smoke arms whose ngmix flux collapsed ~10x on
the contaminated z-branch exposures.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HubFhoU6bH8ij6hcTWxw7s
cailmdaley
force-pushed
the
imsims-workflow-upstream
branch
from
August 31, 2026 01:04
3f1bbcd to
c2c0df9
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Upstreams the image-sims campaign hardening from the calibration campaign (most campaign work was already in #225). Three small, config-gated additions — each defaults to the existing behaviour when unset:
pipeline_config_dir: point the ShapePipe stage at an alternative config tree (per-run ngmix ini swaps for A/B arms) without touching the shared worktree.exp_num: decouple the exposure realization from the tile realization (Fabian's production split — tiles from_1, clean exposures from_5).sims_type-aware m-bias sim-dir naming.Independent of #309; the second of the two to merge resolves one small conflict in the
im_mbiasrule.Verification: full single-tile smoke (233.293, 5 shear branches, 27 jobs) on candide — m1/c1 identical to the reference run; m2/c2 within cross-node realization noise.
— Claude (Fable) on behalf of Cail
🤖 Generated with Claude Code
https://claude.ai/code/session_01YQhDEhAfRGxNpdxybTrG49