Persist per-exposure PSF products before reclamation (exp_persist) - #879
Open
cailmdaley wants to merge 5 commits into
Open
Persist per-exposure PSF products before reclamation (exp_persist)#879cailmdaley wants to merge 5 commits into
cailmdaley wants to merge 5 commits into
Conversation
persist_exp.py copies one exposure's named PSF products off /scratch onto the persistent root and records what went, with sizes. The threat it answers is the 60-day purge, not clean_exposure: run_dir is scratch and products_dir is /project, so the only way a per-exposure product outlives its campaign is to leave the filesystem. Exempting files from reclamation would not have done it. The search is recursive beneath the PSF chain's four module output dirs, because setools writes into mask/, rand_split/, new_cat/, plot/ and stat/ rather than flat -- so the config's patterns stay plain file names and the layout stays ours. A pattern that matches nothing is a recorded warning (setools rejects sparse CCDs); nothing matching at all is a failure, since a green manifest over an empty copy is what would let reclamation delete an unsaved exposure. config.yaml's persist_exp: defaults to validation_psf-*.fits -- the psfex_interp VALIDATION catalogue, the rho/tau statistics input, the minimum. The opt-in candidates are documented there with what each buys; sizes are still to be measured. PSFEx residuals and XML are not candidates as the chain stands: the committed default.psfex sets CHECKIMAGE_TYPE NONE and WRITE_XML N. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
One rule per exposure, output = ONE manifest on the persistent root at <products_dir>/exp/<shard>/<exp>/manifests/exp_persist.json. Not a directory() output: what we want written down is which files were copied and how big each was, and a directory attests only that a directory exists. Byte-stable, so a no-op rerun does not move an mtime clean_exposure reads. Its own rule rather than a cp on the end of exp_psf, and that is the whole point: the keep list rides on params, so adding a pattern reruns seconds of copying instead of four hours of PSF fitting per exposure. A localrule, by the arithmetic that made exp_star_cat one -- a few MB of cp, ~20k of them at DR6 scale, each shorter than the scheduling latency that would submit it. The mid-chain grouping constraint does not bite: its neighbours are exp_psf (too heavy to fuse) and clean_exposure (local already). clean_exposure gains the manifest as an input, so a store is never reclaimed before its keepers have left scratch -- conditional only on there being a keep list, since "keep nothing" must not become a dependency on a rule that would fail for having nothing to copy. rule all requests the persist manifests DIRECTLY, not only through clean_exposure: the purge takes the store whether or not clean: is on, so hanging the copy off reclamation alone would lose everything in a clean:false campaign. Cleaned exposures are excluded -- their exp_psf manifest is gone, so asking would rebuild the chain from VOS, and a tombstone already means the copy happened. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
… omits it run_report disk-scans the scratch run_dir, and exp_persist's manifest is the one exposure manifest that lives on products_dir instead -- the placement that makes it survive clean_exposure. Listed in EXP_STAGES it would read as "not run" for every exposure in the campaign, so it is deliberately absent, with the reason on the line. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…e copies Inodes, not bytes, bind on /project (~1 M-file group quota): smk-m2 measured ~200 loose files per exposure with all candidates on — 25k for 64 tiles, ~2 M at DR6 scale, for 7 GB. persist_exp.py now writes <products_dir>/exp/<shard>/ <exp>/psf/<exp>.tar (uncompressed, flat members, deterministic: ownership zeroed, sorted, tmp-cmp-mv so a no-op rerun keeps the mtime) and the manifest lists every member. Manifest path, rule wiring and params are unchanged. config.yaml's candidate table carries the smk-m2 per-exposure sizes; psfex_cat and star_stat are marked unmeasured (no live store held them). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015qtLUV3bVLPV5p6un7aTFR
cailmdaley
marked this pull request as ready for review
September 3, 2026 00:16
An orphaned tar.tmp on /project is an inode nothing revisits — the leak the tar design exists to avoid. try/finally around both tmp writes. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015qtLUV3bVLPV5p6un7aTFR
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #852. Every campaign so far has zero surviving PSF star products:
clean_exposurereclaims the whole exposure store (scratch, purged anyway) and onlyfinal_catis copied toproducts_dir. So no campaign can producefull_starcat(rho/tau input).Change. A rule
exp_persistbetweenexp_psfandclean_exposurepacks the files named by a new config listpersist_exp:(globs, matched recursively underrun_sp_exp_SxSePsfPi/*/output/) into one uncompressed tar per exposure,<products_dir>/exp/<shard>/<exp>/psf/<exp>.tar, and writes one manifest on the persistent root listing the members. One tar rather than loose copies because inodes, not bytes, bind on /project: with all candidates on, loose copies are ~200 files per exposure (~2 M at DR6 scale against a ~1 M-file group quota) for ~7 GB. FITS members read straight from the tar viatarfile+BytesIO. Default is the minimum,validation_psf-*.fits; the config comment documents the opt-in candidates with measured per-exposure sizes.rule allrequests the manifests directly so aclean: falsecampaign persists too; cleaned exposures are excluded so no chain is rebuilt from VOS. The keep list rides onparams, so editing it re-packs without re-running the 4 h PSF job — the reason this is a separate rule and not atarinexp_psf.No merging: stacking the persisted
validation_psftree intofull_starcatis sp_validation's job.Verified. The loose-copy form ran end-to-end on smk-m2 (64 tiles, 127 exposures, all candidates on): all 127 stores tombstoned, 127/127 manifests, 25 211 files / 7.15 GB survived — the count that motivated the tar. The tar form is exercised on a fixture (byte-stable rerun, member readback, name-collision and no-match failures) and by dry-run (one
exp_persistper exposure betweenexp_psfandclean_exposure); its first real run is masking-unification's next campaign.🤖 Generated with Claude Code
https://claude.ai/code/session_015qtLUV3bVLPV5p6un7aTFR