Skip to content

Adds lazy loading functionality to SpectrumList - #1338

Open
havok2063 wants to merge 13 commits into
astropy:mainfrom
havok2063:lazyloader
Open

Adds lazy loading functionality to SpectrumList#1338
havok2063 wants to merge 13 commits into
astropy:mainfrom
havok2063:lazyloader

Conversation

@havok2063

@havok2063 havok2063 commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

This PR splits off the lazy-loading and string label features from #1303 into a separate PR so they can be merged earlier than the Roman data loaders. The SSC Roman spectral data format is still changing. Once those crystallize into a final form, I'll update that PR to focus only on the Roman data loaders.

The PR updates SpectrumList to

  • allow for lazy loading of Spectrum objects in the list
  • apply a mapping of an alternate string label to index the list by

I've implemented lazy data loaders for the JWST x1d/c1d multi format and some of the SDSS data products. With the lazy loading+label feature, Jdaviz could now more directly rely on specutils to read in data products, and handle the nuances of particular files, rather than Jdaviz having to codify all cases separately. For the JWST files, the string labels are created as

  • [hdu_ext_name]_[idx] for non-table data, e.g. EXTRACT1D_1
  • hduX_[row_idx] for table data without a SOURCEID column, e.g. hdu1_1
  • hduX_source_[sourceid] for table data with a SOURCEID column, e.g hdu1_source_123

This is because the string labels need to be unique for each item in a spectrum list.

Note: the sphinx docs are the same as the old PR, and still reference Roman data loaders.

SDSS example of lazy loading + alt id

This uses the extension name as alt string labels.

sdss='spec-015004-59190-4400696424.fits'

s = SpectrumList.read(sdss, format='SDSS-V spec', lazy_load=True)
s
lazy list: 0 items loaded; access an index to load a spectrum:
['COADD', 'MJD_EXP_59190-00', 'MJD_EXP_59190-01', 'MJD_EXP_59190-02']

s['COADD']
<Spectrum(flux=[-35847924.0 ... -1.905903697013855] 1e-17 erg / (Angstrom s cm2) (shape=(4648,), mean=-102516.42969 1e-17 erg / (Angstrom s cm2)); spectral_axis=<SpectralAxis [ 3566.9744  3567.795   3568.6177 ... 10394.412  10396.809  10399.206 ] Angstrom> (length=4648); uncertainty=InverseVariance)>

s
lazy list: 1 items loaded; access an index to load a spectrum:
[<Spectrum(flux=[-35847924.0 ... -1.905903697013855] 1e-17 erg / (Angstrom s cm2) (shape=(4648,), mean=-102516.42969 1e-17 erg / (Angstrom s cm2)); spectral_axis=<SpectralAxis [ 3566.9744  3567.795   3568.6177 ... 10394.412  10396.809  10399.206 ] Angstrom> (length=4648); uncertainty=InverseVariance)>, 
'MJD_EXP_59190-00', 'MJD_EXP_59190-01', 'MJD_EXP_59190-02']

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant