chore(deps): move to pyramids 0.60 and cleopatra 0.37, and repair the notebooks the upgrade exposed by MAfarrag · Pull Request #1197 · serapeum-org/earthlens

MAfarrag · 2026-09-09T17:17:24Z

Description

Move the GIS stack to pyramids-gis 0.60.0 and cleopatra 0.37.0, and fix the defects that re-running the
example notebooks on the new stack exposed.

The upgrade is additive — no call site changed

The changelog for 0.60 lists two Fix entries, which is not enough to merge on, so the API was diffed rather than
trusted. A full public-surface snapshot of both versions (167 → 170 modules, 554 → 560 classes) was compared:

	result
removed modules	none
removed class methods / properties	none
removed top-level names	7, all in private modules (`base._coverage`, `base._raster_meta`, `base._utils`, `dataset.engines.analysis`, `dataset.ops.io`, `feature.bbox`, `netcdf.engines.interop`)
changed signatures	16, all purely additive

The private-module names are internal helpers relocated into three new private modules by pyramids' #1084
consolidation. earthlens references none of them — the single DriverNotExistError hit is a comment in
soilgrids/backend.py, and the one genuinely private import earthlens does make,
from pyramids._io import extract_from_gz, still resolves with an unchanged signature.

The 16 signature changes are new keyword-only parameters with behaviour-preserving defaults:
is_no_data / inside_domain gained atol=1e-08, DatasetCollection.apply / crop gained compute=True, and
decode_cf_time's unit widened to str | bytes | None.

atol on is_no_data was the one change that could silently alter no-data masking, which every notebook depends
on, so it was executed rather than reasoned about: the same six cases — exact -9999, float drift, a 0.0 fill
against 1e-9, exact zero, NaN fill, -128 — return identical results on both versions. It exposes numpy's
existing default, it does not change it.

cleopatra moves 0.35 → 0.37 for ColorScaling.log(), which 0.35 lacked.

Notebook defects the rerun exposed

Six ecmwf notebooks were broken on main and had been for some time. aggregate_netcdf names each window
<cds_variable>_<dataset>_<freq>_<date>.tif because _output_stem appends the dataset id, but these globbed
<variable>_<freq>_*.tif and so matched nothing — every run died on an empty np.stack or min(). Three sibling
notebooks in the same folder already used the correct <variable>_*_<freq>_*.tif form, which is what these now
use. Unrelated to the upgrade; they fail identically on 0.59.

Three plots misrepresented their data.

asf/opera_rtc_backscatter displayed a hardcoded -25..0 dB, which clipped 28.9% of samples to solid black
while nothing in the scene came near the top of the ramp. The range now comes from the scene's own 1st/99.9th
percentiles (-34.5..-7.0 dB for the bundled granule) and is printed. The granule is a diagonal swath in a
north-up grid — 61% of the file is nodata — so it is cropped to its valid extent, taking fill from 38.8% to
60.9%. The 85-character filename left the title, where it collided with the top-mounted x tick labels.
bathymetry/05_shaded_relief ran vert_exag=1.0 over a seafloor ~180 km wide and 3.8 km deep, about 1:47. The
hillshade had nothing to model and the map came out featureless, so the rift valley and transform faults the
prose promises never appeared. The ramp was also a land-elevation one painting the shallowest seafloor green in
an AOI with zero cells above sea level.
bathymetry/08_emodnet_wcs_deep_dive hid its EarthLens calls behind fetch_depth, which also flattened the
distinction the cell exists to teach: gebco_2020 declares its 32767 fill, the two EMODnet products declare
nothing.

Filed upstream, not worked around silently

feat(styling): add a continuous equalising colour scale, and let callers pass their own norm cleopatra#343 — no continuous equalising colour scale. On the bathymetry tile a linear norm
gives the deepest half of the seafloor 31% of the ramp and the sparse shallow tail 36%. Measured against every
alternative, a CDF-backed FuncNorm is the only one that equalises (unevenness 0.2 against 55.8 for linear,
57.4 for power, 90.8 for asinh) — and ColorScaling cannot express it. boundary equalises but is discrete and
would posterise a hillshade. There is no escape hatch either: plot(norm=…) is rejected and color= requires a
ColorScaling.
fix(array_glyph): a multi-line title overlaps the top x tick labels when figsize is set cleopatra#345 — x tick labels render on the top spine while the title uses matplotlib's default
pad, so any multi-line title overlaps them. Reproduced on 0.37, which is why the ASF notebook has to reach into
glyph.ax to move them.

Issues

No tracking issue — the upgrade was requested directly. The two gaps it surfaced are filed upstream in cleopatra
(#343, #345) and are not blocked on this PR.

Type of change

Check relevant points.

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to not work as expected)
This change requires a documentation update

How Has This Been Tested?

Full non-e2e suite on 0.60 — 11642 passed, 65 skipped, 210 deselected in 9m04s, zero failures,
including the nwp test that had been flaking on the previous branch. That run predates the main merge; the
post-merge re-run is in flight and this line will be updated with its result.
API surface diffed, not assumed — both versions introspected and compared module by module and member by
member; the table above is that output.
The one behavioural risk executed — is_no_data compared across six no-data shapes on both versions,
identical results.
uv lock --check passes, so CI's --locked gate is satisfied. The lock was regenerated with
--refresh-package for both packages, recording the complete 23-wheel set plus sdist rather than the partial
set a stale cache produces.
Notebooks executed against live services, which is how all four defects above were found. Each edited
notebook was re-run end to end and its figure inspected.
The remaining notebooks have not all been re-run on 0.37 — a full pass reached 138/267 under cleopatra 0.35
before the upgrade, and restarting it is follow-up work, not a blocker for a dependency floor.

Checklist:

updated version number in pyproject.toml.
added changes to docs/change-log.md.
updated the latest version in README file.
I have added tests that prove my fix is effective or that my feature works.
New and existing unit tests pass locally with my changes.
documentation are updated.

No version bump and no change-log entry: this raises a dependency floor and repairs four example notebooks. The
notebooks are the documentation being updated, and the existing suite already covers the library surface the
upgrade touches — which is the point of the API diff above.

pyramids 0.60 is additive for earthlens: the full public API surface was diffed against 0.59 and nothing earthlens calls was removed or narrowed. The seven names that did disappear are all in private modules the package does not import, and the sixteen changed signatures only gained keyword arguments with behaviour-preserving defaults. The non-e2e suite is green. cleopatra 0.37 brings ColorScaling.log(), absent from the pinned 0.35.

aggregate_netcdf names each window <cds_variable>_<dataset>_<freq>_<date>.tif because _output_stem appends the dataset id, but six notebooks globbed <variable>_<freq>_*.tif and so matched nothing: each run died on an empty np.stack or min(). Three sibling notebooks in the same folder already used the <variable>_*_<freq>_*.tif form, which is what these now use.

asf/opera_rtc_backscatter: the display range was hardcoded to -25..0 dB, clipping 28.9% of samples to solid black while nothing reached the top of the ramp; it now comes from the scene's own 1st/99.9th percentiles. The granule is a diagonal swath in a north-up grid and 61% nodata, so it is cropped to its valid extent, and the 85-character filename is off the title where it collided with the top-mounted tick labels. bathymetry/05_shaded_relief: vert_exag was 1.0 on a seafloor ~180 km wide and 3.8 km deep, so the hillshade had nothing to model and the map came out featureless; the ramp was also a land-elevation one that painted the shallowest seafloor green. bathymetry/08_emodnet_wcs_deep_dive: the fetch_depth and _depth_span helpers hid the EarthLens calls the notebook exists to teach, and flattened the difference between a source that declares its fill and two that do not.

…yramids-0.60

…stem The earlier form globbed <variable>_*_<freq>_*.tif. That wildcard was meant to stand in for the dataset id, but it also matches a longer variable name sharing the same prefix: snow_depth_*_1MS_*.tif picked up snow_depth_water_equivalent as well, stacking 18 rasters where 9 were expected and dying on the shape mismatch. Each glob now names its dataset explicitly, so a prefix cannot run past the variable it belongs to. All six notebooks in the folder were audited for the same class of bug.

Ten example notebooks opened their downloads with xarray. That duplicates what pyramids already does and puts a second array API in front of readers of a repo whose GIS backend is pyramids. They now go through NetCDF.read_file / get_variable / get_group and Dataset, matching every other example. Two defects the conversion exposed are fixed here rather than carried over: - pyramids returns the raw CF-packed store instead of applying scale_factor and add_offset (serapeum-org/pyramids#1124), so the packing is applied explicitly. Without it the wave notebook reported a peak significant wave height of 1246 m rather than 12.5 m, and the chlorophyll one was out by four orders of magnitude. Both executed cleanly while being wrong. - nc.lat / nc.lon return pixel indices, not coordinates, so a nearest-cell lookup written against them lands on the corner cell. Point lookups now use Variable.rowcol, cross-checked against geotransform arithmetic. Every converted notebook was re-executed against the live service and its numbers sanity-checked in physical units. Three of the ten could not be confirmed end to end because their upstream products are unavailable right now (cmems/seaice_arctic_thickness, earthdata/pace_ocean_colour, showcase/hurricane_harvey_rainfall); their conversions follow the same shape as the seven that were verified.

global_wave_significant_height plotted its series against date-only labels, so all eight steps of a day collapsed onto a single x position and the 3-hourly cadence the notebook exists to demonstrate was lost to a zig-zag. It now asks get_time_variable for the hour and plots a real datetime axis. The map gains Natural Earth land and coastlines, so its white patches read as Ireland and Iberia rather than missing data, and 1 m contour banding, so the storm's structure is legible instead of a smooth ramp. Both panels are now one figure, which the surrounding prose already claimed. The window moves to 2014-02-07..10 so the storm's rise, peak and decay all fall inside it; the old window opened six hours before the peak and spent fifteen of its seventeen steps decaying. atmosphere_pressure_levels keeps its raw-quiver overlay, but now names cleopatra#346 as the blocker instead of pyramids#1128. The latter was filed against the wrong component: the axes-clearing happens in cleopatra's glyph base and reproduces with no pyramids in the call stack.

The TIGGE cell reached for NetCDF, which is the raster container, and got a bare gdal.MDArray back because the file has no spatial dimensions -- ECMWF serves the native reduced-Gaussian representation as a `values` dimension with latitude and longitude as coordinate variables. The cell then called ReadAsArray() and GetUnit() on it, putting raw GDAL in a docs notebook, which the no-direct-GDAL rule exists to prevent. pyramids already had the right container and it was never used: LabeledDataset reads the `values` dimension together with its CF coordinate variables and returns a dataframe of the 22 points. NetCDF is for raster-shaped CF, UgridDataset for a UGRID mesh, LabeledDataset for labelled point data; the surrounding prose now says so. Having real coordinates also fixes the figure. It plotted temperature against the bare `values` index, because the index was all that was reachable; it is now a cleopatra ScatterGlyph at true lon/lat, which shows the reduced-Gaussian rows -- constant latitude, longitude spacing varying per row -- that the index plot hid. pyramids#1126 is rewritten to the real defect: get_variable should raise and name LabeledDataset rather than leak a GDAL object into caller code.

…efects All three were reported as upstream-data outages. Only one was. hurricane_harvey_rainfall asked for a `Grid` group that daily IMERG does not have. The V07 daily granules are flat NetCDF-4 with `precipitation` at the root; the `Grid` group belongs to the monthly HDF5 product, and the call was carried over from it in the xarray conversion. Storm total over the Texas box is 861 mm. That also contradicted the prose, which promised a bullseye over 1 m, so the text now says what the data says: IMERG's 0.1 degree cells peak near 860 mm, while Harvey's quoted 1.5 m is a point gauge total that an 11 km cell averages away. pace_ocean_colour requested PACE_OCI_L3M_CHL, which does not exist -- not "no granules for that date": zero collections and zero granules unfiltered. Chlorophyll ships inside the Level-3 mapped biogeochemistry collection, PACE_OCI_L3M_BGC, beside poc, pic and carbon_phyto. One date there returns six granules (daily, 8-day and monthly, each at 0.1 degree and 4 km, 314 MB in total), so the notebook now selects the 8 MB daily 0.1 degree file by name instead of indexing the list. Its markdown also claimed each step was wrapped in try/except when the code has none; that text is gone. seaice_arctic_thickness computed its date as now minus 45 days. The product's time axis ends 2026-04-12 and has not advanced, so the probe fell outside the coverage and the subset was rejected with CoordinatesOutOfDatasetBounds. The date is now pinned inside the window, which a docs example needs anyway to be reproducible. Two further defects were hiding behind that failure: the read indexed `[0]` into an array pyramids had already collapsed to 2-D, leaving a single row, and the field was scaled by a `scale_factor` the variable does not declare. The map was also drawn on the subset's lat/lon grid, which renders the basin as a 15:1 strip; it is reprojected to EPSG:3413 at 25 km, the conventional sea-ice spacing, which is both a real polar map and small enough to plot. Not changed, and worth a look separately: the earthdata catalog still maps PACE_OCI_L3M_CHL_31 to a short_name that CMR no longer serves, and the CMEMS catalog reports the sea-ice product as running to present when it stops at 2026-04-12.

… aoi= Six notebooks built a `dict(lat_lim=..., lon_lim=...)` constant, splatted it into one constructor, and never touched it again. The indirection bought nothing and cost something specific: in a cell whose whole job is to show what a request looks like, two of the parameters were hidden behind `**BBOX` while every other one was visible, so the bounding box could not be read at the call site at all. A typo'd key inside the dict also arrives as an unexpected keyword with no line number pointing at the literal, and no tooling can check a splatted dict against the signature. The keywords now sit at the call site. `heatwave/european_heatwave_2026` keeps its `EUROPE` dict, which is splatted once *and* subscripted twice -- there the name carries a shared value and earns its place. The three cmems notebooks also move from `lat_lim` / `lon_lim` to the `aoi=` channel, which their four siblings in the same folder already used; the folder is now consistent. `aoi` is [min_lon, min_lat, max_lon, max_lat] -- the GeoJSON / STAC W, S, E, N order documented on `normalize_aoi`. The showcase and soilgrids three keep `lat_lim` / `lon_lim`, because that is their own folder's convention: soilgrids has no `aoi=` in any notebook, and lat_lim dominates showcases. Moving them would trade one inconsistency for another. Re-expressing coordinates is the part that can silently go wrong, so the three cmems notebooks were re-run against the live service with their cached downloads deleted first -- the download filename is derived from the dataset id, not the bbox, so a stale file would have been reused and hidden a flipped box. The subset grids come back identical (40x36, 13x13, 17x17) along with every recorded value: SLA 0.007..0.211 m, sla mean 0.0874, adt mean 0.1305; nearest levels 18.5 / 92.3 / 453.9 m; chl 0.3306..0.7441. The other three are textual moves of identical keywords and values, with no coordinate change to verify.

The discharge field was drawn on bare axes, so the river network had no geographic anchor -- a reader could see the Danube and its tributaries but not which countries they run through. Natural Earth borders at 50 m now draw over the field. Which layers to use is decided by the data, not by habit. Every cell in this raster carries a value (land reads 0 m3 s-1 rather than nodata), so the field is opaque and a filled `land` or `relief` layer underneath would be completely hidden -- only line features drawn on top are visible. `rivers` is deliberately left off for the opposite reason: the raster already is the river network, and a second, differently-derived one over it would invite the two to be compared. Values are unchanged: grid 90 x 120, discharge 0.0 / 4133.8 m3 s-1.

…rature animation The notebook builds a per-day "%b %d" label for every frame and its own prose promises the day is recorded as a frame label, but the animation call never passed it through -- cube.plot() had no animation_axis_values, so animate() fell back to a bare frame index and every frame read "Date = <n>" instead of a real date. Passing the computed `labels` list fixes it; re-running the notebook shows "Date = Jun 21" at the same frame that previously read "Date = 20".

… the frame The vertical "2m temperature (degC)" label ran past the figure's right edge -- confirmed by measuring its bounding box, which extended 8px past the 1200px canvas -- because the colorbar's own axes sits close enough to the figure boundary that its rotated label has nowhere to go. vmin/vmax also moved to 0/38 to match the range asked for. subplots_adjust looked like the obvious fix but isn't safe here: it recomputes the whole subplot geometry rather than nudging one axes, and it pulled the map's own left-edge latitude labels into negative (off-canvas) territory as a side effect -- reproduced and measured, not assumed. labelpad alone also isn't enough on its own: the tick numbers and the label are already touching, so pulling the label inward to fit the canvas just walks it into the ticks instead. The isolated fix is to move only the colorbar's own axes left by a small figure-fraction via cbar.ax.set_position(...) -- the map axes (and every label on it) is never touched, confirmed identical (54.8px) before and after. Re-rendered against the full 70-frame animation: the label now clears the frame edge and the ticks with room on both sides, everything else in the frame is pixel-for-pixel the same.

… CDS ceiling END was 2026-08-31; extended to 2026-09-07, the last day CDS's reanalysis-era5-land actually serves as of this run (day 08 onward: "does not match any constraint entry"). A request past that ceiling does not fail loudly -- it succeeds and silently returns nothing, so the ceiling needs re-checking, not assuming, each time this constant moves. The single combined download call also had to split in two. CDS builds one uniform day=[01..31] list applied across every month named in a request, so a request spanning into a month still being published is rejected in full the moment that list contains an unpublished day -- confirmed live: one request for the whole 2026-05-15..09-07 span returns zero files, even though May through August are each complete. The fetch now issues one request for the complete past months and a second, separately bounded request for the partial current month, then merges the results -- generalised from END's own month rather than a hardcoded date, so extending this again later does not need the same fix reapplied by hand. Re-run end to end: 116 daily tiles (17+30+31+31+7, no gaps), the last frame reads "Date = 06 Sep", and the branding/colorbar layout from the last two fixes is unchanged.

…..END The cache check only asks whether raw_kelvin_dir is non-empty, not whether it covers the currently requested window, so a stale cache from a narrower run gets served silently -- exactly what happened extending this notebook to September: the old 70-day cache would have been reused under the new 116-day END with no error. Reparse each cached file's own date and compare the set against the calendar days START/END actually ask for; mismatch raises instead of running quietly on wrong data. Verified both directions: passes silently against the current correct 116-file cache, and against a synthetic 70-file stale cache raises with the exact gap ("covers 05-15..07-23 (70 days) but START/END ask for 05-15..09-07 (116 days)").

Superseded by european_heatwave_summer_2026.ipynb, which covers the same domain and variable at a longer window (mid-May through the current CDS ceiling, against the earlier notebook's six weeks) and through the ecmwf/CDS backend in one request rather than a per-day Earth Engine loop. Also removes the dangling mkdocs.yml nav entry, and rewrites the two spots in european_heatwave_summer_2026's own prose that named this file by way of comparison, so neither the docs nav nor the surviving notebook's own text points at something that no longer exists.

sonarqubecloud · 2026-09-14T19:52:18Z

Quality Gate passed

Issues
1 New issue
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

MAfarrag added 16 commits September 9, 2026 01:16

Merge branch 'main' of github.com:serapeum-org/earthlens into chore/p…

13b355c

…yramids-0.60

Navigation Menu

chore(deps): move to pyramids 0.60 and cleopatra 0.37, and repair the notebooks the upgrade exposed - #1197

chore(deps): move to pyramids 0.60 and cleopatra 0.37, and repair the notebooks the upgrade exposed#1197
MAfarrag wants to merge 16 commits into
mainfrom
chore/pyramids-0.60

MAfarrag commented Sep 9, 2026

Uh oh!

sonarqubecloud Bot commented Sep 14, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

MAfarrag commented Sep 9, 2026

Description

The upgrade is additive — no call site changed

Notebook defects the rerun exposed

Filed upstream, not worked around silently

Issues

Type of change

How Has This Been Tested?

Checklist:

Uh oh!

sonarqubecloud Bot commented Sep 14, 2026

Quality Gate passed

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant