Skip to content

feat(netcdf): make the container a mapping, and add the xarray-compatible aliases - #1141

Open
MAfarrag wants to merge 4 commits into
mainfrom
feat/netcdf-mapping-and-aliases
Open

MAfarrag wants to merge 4 commits into
mainfrom
feat/netcdf-mapping-and-aliases

Conversation

@MAfarrag

@MAfarrag MAfarrag commented Sep 14, 2026

Copy link
Copy Markdown
Member

Description

Tasks T1, T2 and T16 of planning/xarray/missing-functionality-plan.md — the Tier 1 ergonomics
batch. nc.variables has been a mapping for a while, but the container itself was not: nc["t2m"],
"t2m" in nc, len(nc) and list(nc) all failed, which is the first thing a reader arriving from xarray
types. Every member added here delegates to variables, so there is one enumeration and one refusal message
rather than a second list that can drift.

T1 — the mapping protocol on the container. __getitem__, __contains__, __iter__, __len__, get,
keys, values and items. __iter__ yields data variables only, matching variables and
variable_names, and the docstring says so — this is the variables-shaped collision the plan flagged
(xarray's Dataset.variables includes coordinates; this class's does not).

T2 — the xarray-compatible aliases. data_vars, sizes, attrs, and a new coords built from
get_dimension_values so the storage-order contract stays in one place.

T16 — cheap introspection. dtypes, nbytes and info(buf=None). None of the three reads a pixel.

Three things look like inconsistencies and are pinned by tests so they are not "fixed" later:

  • nc["nope"] raises KeyError where nc.get_variable("nope") raises ValueError. The mapping protocol
    needs KeyError for in, get and dict(nc) to work at all, and get_variable predates the mapping and
    cannot start raising KeyError without breaking callers.
  • dims is a mapping of name to length, as xarray's Dataset.dims is, and is therefore the same object
    as dimension_sizes rather than an alias of dimension_names, which is a list. This is the decision the
    plan asked to be made explicitly; aliasing a mapping name onto a list would let nc.dims["time"] return a
    list index silently. dimension_names keeps its name and its list.
  • nbytes counts data variables only, where xarray's counts its coordinates too, and is computed as
    rows * columns * band_count * itemsize — the band axis is the whole flattened non-spatial stack, so that
    product is the cube and not a plane. A cube far larger than memory can therefore still be sized.

Five places this diverged from the plan's spec, all recorded in planning/xarray/missing-functionality-plan.md
§3:

  1. The plan's T1 parity assertion does not hold universally. sorted(nc) == sorted(nc.to_xarray().data_vars)
    holds on 23 of the repo's 26 stores and fails on three, so it is asserted as a sweep over the 23 plus one
    named test per store where it fails. None of the three is a defect on either side — see How Has This Been
    Tested?
    below.
  2. info reads each variable's axes from the store, not from the subset. get_variable renames a
    variable's y dimension to the window it was cut with (subset_lat_127_-1_128), which is not a name the
    file has, so printing the subset's dimension_names would have put a fabricated axis in the summary.
    _variable_dim_names(rg, name) is used instead. That rename is a pre-existing wart this PR only works
    around; it is worth its own issue.
  3. coords omits an unindexed dimension rather than mapping it to None. The plan said "over
    dimension_names", which would have included the CMIP store's bnds and every UGRID count dimension. A
    None value makes nc.coords[name].shape fail for a caller iterating the mapping, and xarray drops them
    too, so a UGRID store's coords is {}.
  4. dims took the plan's recommendation. Its "alias dimension_names to nothing" is satisfied by leaving
    that member alone rather than by removing anything.
  5. dtypes reports the first band's type, not the per-band list. A raster subset reports dtype as one
    entry per band (all the same, since a band is a slice of one array); returning that list stringified would
    have passed a key-set comparison and been useless.

One consequence worth a decision, deliberately not made here. NetCDF now has __iter__ and
__getitem__, so np.asarray(nc) builds an array of the variable names where it previously gave a 0-d
object array, and np.asarray(variable) gives an empty float64 array. bool(nc) is unaffected —
Dataset.__bool__ still refuses. Nothing in the package or in the 8524 tests run below depends on either, and
np.asarray on a container was never a supported operation; but the class already refuses one ambiguous
coercion in __bool__, and a matching __array__ pointing at read_array() would close this one. Left out
of this PR as scope creep.

No new dependencies.

Issues

Type of change

Check relevant points.

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • This change requires a documentation update

How Has This Been Tested?

Two new modules, 150 tests.

tests/netcdf/structure/test_container_mapping_and_aliases.py — 123 tests, marked core. Sweeps seven
stores, one per shape the new members have to survive: a plain 4-D CF cube, a container declaring dimensions
its variables do not all use, a packed 2-D store whose variables report a renamed subset_y_... axis, a
container holding only LabeledArrays, a grouped store with group/var names, a curvilinear store that reads
more names than it enumerates, and a single-variable store.

  • The delegation is total — nc[name] and nc.variables[name] agree in type and shape on every name of
    every store; list(nc) == variable_names including order; dict(nc) round-trips; 3 in nc and
    None in nc are ordinary False rather than a TypeError.
  • The curvilinear asymmetry holds from the container: nc.get_variable("lat_rho") reads the array,
    "lat_rho" not in nc, and nc["lat_rho"] refuses with the message that names the accessor which works.
  • KeyError / ValueError split, and dims being a mapping rather than dimension_names, are each
    pinned from both directions.
  • coords[name] equals get_dimension_values(name) for every entry; bnds is absent on the CMIP store;
    a UGRID store's coords is {}.
  • dtypes, nbytes and info all answer with read_array monkeypatched to raise, so a
    sum(v.read_array().nbytes) implementation cannot pass. nbytes is also pinned to one hand-computed
    number (4 * 3 * 5 * 6 * 8) so the sum is not merely self-consistent.
  • info names every dimension and every variable, writes to sys.stdout by default, and reports each
    variable's own axes — float32 area(lat, lon) beside float32 ua(time, plev, lat, lon), with no
    subset_ anywhere in the output.

tests/netcdf/parity/test_container_names_match_xarray.py — 27 tests, marked interop, run by the
interop-tests job.

  • sorted(nc) == sorted(nc.to_xarray().data_vars) on the 23 stores where it holds, with a test tying the
    sweep to the fixture directory so the exclusion list cannot grow silently.
  • The grouped store: to_xarray flattens 8 group/var names to group_var because a Dataset is one
    flat namespace, and skips 20 variables because a Dataset holds one size per dimension name while each
    flight group has its own recNum. Both are asserted through the warnings to_xarray raises, not
    through the counts alone — the counts would also match an export that dropped them silently.
  • The GOES store exports DQF and the UGRID store exports face_node_connectivity, neither of which CF
    classification puts in variable_names. Both are asserted to remain reachable through get_variable.

Regression run, on this branch:

  • pytest tests/netcdf tests/dataset -m "not plot"8524 passed, 82 skipped, 0 failures (11m30s).
    Run because adding __iter__ and __len__ to a core class changes protocol membership; isinstance
    was checked against Mapping, Sequence and the package's _ArrayLikeProto (all still False; only
    Iterable flips), and bool(nc), v + 1, v * 2 and v > v were checked by hand.
  • pytest --doctest-modules src/pyramids/netcdf/netcdf.py — 46 passed, 9 skipped. Every new member
    carries executable doctests.
  • ruff format and ruff check at the version pinned in .pre-commit-config.yaml (0.15.22) — clean; no
    line over 120 characters.

Checklist:

  • updated version number in pyproject.toml. — handled by commitizen on release
  • added changes to History.rst. — the changelog is commitizen-generated
  • updated the latest version in README file. — no version change in this PR
  • I have added tests that prove my fix is effective or that my feature works.
  • New and existing unit tests pass locally with my changes.
  • documentation are updated. — every new member carries a full Google-style docstring with doctests; the
    plan document records the divergences

…ible aliases

`nc.variables` has been a mapping for a while, but the container itself was not:
`nc["t2m"]`, `"t2m" in nc`, `len(nc)` and `list(nc)` all failed, so the first thing
an xarray user types did not work. Each new member delegates to `variables`, so
there is one enumeration and one refusal message, not two.

- add `__getitem__`, `__contains__`, `__iter__`, `__len__`, `get`, `keys`,
  `values` and `items` on `NetCDF`
- add the read-only aliases `data_vars`, `sizes` and `attrs`
- add `coords`, built from `get_dimension_values` so the storage-order contract
  stays in one place
- add `dtypes`, `nbytes` and `info(buf=None)`

`__getitem__` raises `KeyError` where `get_variable` raises `ValueError`; the
mapping protocol needs `KeyError` for `in`, `get` and `dict(nc)` to behave, so the
two spellings differ deliberately.

`dims` is a **mapping** of name to length, as xarray's `Dataset.dims` is, and is
therefore the same object as `dimension_sizes` rather than an alias of
`dimension_names`, which is a list. Aliasing a mapping name onto a list would let
`nc.dims["time"]` silently return a list index.

`nbytes` is computed from each variable's shape and dtype and reads no pixels, so a
cube larger than memory can still be sized. It counts data variables only, where
xarray's counts its coordinates too.
…rospection

Two modules. The first sweeps seven stores — a plain 4-D CF cube, a container
declaring dimensions its variables do not all use, a packed 2-D store whose
variables report a renamed `subset_y_...` axis, a container holding only
`LabeledArray`s, a grouped store with `group/var` names, a curvilinear store that
reads more names than it enumerates, and a single-variable store — and asserts that
every new member delegates totally rather than enumerating a second time.

Three things are pinned because they look like inconsistencies and will otherwise
be "fixed":

- `nc["nope"]` raises `KeyError` where `nc.get_variable("nope")` raises `ValueError`
- `nc.dims` is a mapping and is therefore not `nc.dimension_names`
- `dtypes`, `nbytes` and `info` all answer with `read_array` patched to raise, so a
  `sum(v.read_array().nbytes)` implementation cannot pass

The second asserts `sorted(nc) == sorted(nc.to_xarray().data_vars)` on the 23 stores
where it holds, and gives the three where it does not a named test each: the grouped
store, where an xarray `Dataset` is one flat namespace and holds one size per
dimension name, so 8 names are flattened and 20 variables are skipped — both warned
about; and the GOES and UGRID stores, where the export carries an array CF
classification leaves out of `variable_names`. All three remain reachable through
`get_variable`.
A coverage pass showed every new body exercised, but two kinds of scenario were
missing. Both are now pinned.

A variable subset is a `NetCDF` too, so every new member is reachable on it, and
the answers follow from the members they delegate to being *container* concepts:
`variable_names` is empty, so `len`, `list`, `dict`, `dtypes` and `nbytes` all
report nothing — `nc["temperature"].nbytes` is 0 for a variable that plainly holds
2880 bytes. `dimension_sizes` needs a root group a subset does not have, so `dims`
is `{}` while `dimension_names` still reports four names from the cache built with
the subset. Each is documented on the property it surprises, because the natural
reading of `variable.nbytes` is not what it answers.

That last one settles which member `coords` iterates: `dimension_names`, not
`dims`. Keying off `dims` would throw away the axes a subset can still read, so
`set(coords) <= set(dimension_names)` is the invariant, and the test now asserts
that rather than the `dimension_sizes` version that only held on containers.

Also:

- pin the `"unknown"` arm of `_variable_dtype`, unreachable from any fixture since
  no store has a zero-band variable, through a stub
- pin the `LabeledArray` arm of both sizing helpers directly
- assert `values()` and `items()` return the same objects `__getitem__` does, not
  merely the same count
- assert a returned mapping or list can be mutated without reaching the container
- assert `info`'s section order and that an attribute-less container still closes
- drop the dead `types.get(name, "unknown")` fallback in `info`; it iterates the
  same `variable_names` that built the mapping
…sserting equality

Most examples read `nc.data_vars == nc.variable_names` -> `True`, or
`nc.sizes == nc.dims == nc.dimension_sizes` -> `True`. That confirms the
delegation and teaches nothing: a reader cannot see what any of them returns, and
the doctest passes just as well if both sides are wrong together.

Every example now prints a real value and then does something with it — the names
of a five-variable store and the band counts they reach, the index a pressure
level sits at, the megabytes a store would cost to read, the variables with more
than one band. `info` prints its whole summary, with the tabs expanded so the
example is readable.

Also:

- give `_variable_dtype` and `_variable_nbytes` examples; both branch on the
  variable kind and `_variable_dtype` has a documented `"unknown"` return, so
  neither is self-evident from the signature
- add a second example to each member that had only one
- add `See Also` to `__contains__`, `__iter__`, `__len__`, `keys`, `values`,
  `items` and `get`, which had none

48 doctests pass.
@sonarqubecloud

Copy link
Copy Markdown

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant