Notes from running a three-part lc analysis end to end on Perlmutter (lc 0.5.0rc2, direct mode — landlock unavailable there). Each item below cost real time to diagnose; none is documented today. Suggested home for the user-facing prose: the lightcone skill's diagnosis.md "HPC and venues" section, with the runtime-side items fixed or surfaced by lc itself.
Blocking
- Project location: GPFS home breaks commits. With the project on
/global/homes, git-annex's sqlite keysdb returns disk I/O error whenever lc commits an output from a compute node (Repeatedly unable to open sqlite database .git/annex/keysdb/db). The commit phase loops for 10–25 min and dies; not one materialize commit ever succeeded there. On Lustre ($PSCRATCH) commits are instant. → Document: projects live on $PSCRATCH (and note the purge policy — annexed result bytes need a durable remote before publishing).
- uv cannot lock its cache from compute nodes.
Could not acquire lock for ~/.cache/uv … Unknown error 524 (flock on GPFS). lc drives uv internally, so materialize fails on the node. → UV_CACHE_DIR=/tmp/uvcache (or lc could default the cache to a node-local path when it detects Slurm).
- Memory cgroup.
srun -c 128 on a 256-thread node grants half the node's memory; a recipe with parallel prepare_sim OOM-killed the whole step. → --exclusive --mem=0; worth stating in the refusal that lc materialize prints for login nodes.
- Declared-input hashing. Declaring a 200 GB directory as a
source: made lc status hash it for >27 min per invocation. → Guidance to declare only the files/shells a recipe actually reads (and perhaps a warning when a source exceeds some size).
Confusing but benign
lc launches its dask worker via srun --overlap; every invocation ends with *** STEP … CANCELLED … DUE TO SIGNAL Killed *** and srun: error: … task 0: Killed. It is lc's own teardown, but it reads as an external kill (we chased a phantom process sweeper for an afternoon). A quieter teardown, or one line in the docs.- A login-node
srun client is fragile (arbiter/timeouts). The robust pattern was salloc … --no-shell, then ssh <node> 'nohup script &' with the script self-logging to $PSCRATCH; or sbatch -q regular (the interactive QOS refuses sbatch). Because outputs are committed at the end of an invocation, long chains should be split into short lc materialize <targets> stages so a kill loses one stage, not all.
Recovery
- After any killed run: stale
.git/index.lock and .git/annex/keysdb/db-shm must be removed by hand before the next invocation; untracked results/ then trips the "uncommitted changes" refusal, whose git clean remedy is the right one.
- A materialize sitting at 0 % CPU with
/proc/<pid>/wchan = cl_sync_io_wait is a Lustre outage, not an lc hang — worth a diagnosis-table row.
Related: #207 (nested-spec execution), LightconeResearch/agent-skills#37 (git layout for composed analyses).
Notes from running a three-part
lcanalysis end to end on Perlmutter (lc 0.5.0rc2, direct mode — landlock unavailable there). Each item below cost real time to diagnose; none is documented today. Suggested home for the user-facing prose: the lightcone skill'sdiagnosis.md"HPC and venues" section, with the runtime-side items fixed or surfaced bylcitself.Blocking
/global/homes, git-annex's sqlite keysdb returnsdisk I/O errorwheneverlccommits an output from a compute node (Repeatedly unable to open sqlite database .git/annex/keysdb/db). The commit phase loops for 10–25 min and dies; not one materialize commit ever succeeded there. On Lustre ($PSCRATCH) commits are instant. → Document: projects live on$PSCRATCH(and note the purge policy — annexed result bytes need a durable remote before publishing).Could not acquire lock for ~/.cache/uv … Unknown error 524(flock on GPFS).lcdrives uv internally, so materialize fails on the node. →UV_CACHE_DIR=/tmp/uvcache(or lc could default the cache to a node-local path when it detects Slurm).srun -c 128on a 256-thread node grants half the node's memory; a recipe with parallelprepare_simOOM-killed the whole step. →--exclusive --mem=0; worth stating in the refusal thatlc materializeprints for login nodes.source:madelc statushash it for >27 min per invocation. → Guidance to declare only the files/shells a recipe actually reads (and perhaps a warning when a source exceeds some size).Confusing but benign
lclaunches its dask worker viasrun --overlap; every invocation ends with*** STEP … CANCELLED … DUE TO SIGNAL Killed ***andsrun: error: … task 0: Killed. It is lc's own teardown, but it reads as an external kill (we chased a phantom process sweeper for an afternoon). A quieter teardown, or one line in the docs.srunclient is fragile (arbiter/timeouts). The robust pattern wassalloc … --no-shell, thenssh <node> 'nohup script &'with the script self-logging to$PSCRATCH; orsbatch -q regular(the interactive QOS refusessbatch). Because outputs are committed at the end of an invocation, long chains should be split into shortlc materialize <targets>stages so a kill loses one stage, not all.Recovery
.git/index.lockand.git/annex/keysdb/db-shmmust be removed by hand before the next invocation; untrackedresults/then trips the "uncommitted changes" refusal, whosegit cleanremedy is the right one./proc/<pid>/wchan=cl_sync_io_waitis a Lustre outage, not an lc hang — worth a diagnosis-table row.Related: #207 (nested-spec execution), LightconeResearch/agent-skills#37 (git layout for composed analyses).