zcollection creates a lot of files

[robin-cls]

Brief

zcollection creates a lot of files. This can lead to a decreased mean size for the stored objects.

Context

Our altimetry data is stored in a zcollection over a S3 bucket. We use the fsspec abstraction in order for zcollection to properly manage the objects in the bucket.

Until now, this behavior was satisfying. However, the crew maintaining the buckets is setting stricter requirements for their usage. The main requirement is that the mean size of the stored objects must be over a given threshold.

For the stored data, we have a mean chunk of ~30MiB, which coincides with the threshold. However, because zcollection relies on zarr to store the arrays, it creates two additionnal metadata files (.zarray and .zattrs), which automatically decrease the mean size of the objects to 10MiB.

What we would like

Is it possible for zcollection to diminish the number of files it produces in its storage ?

[robin-cls]

Here’s a concise, GitHub‑ready comment that captures the analysis and proposed directions clearly and neutrally, without over‑committing on implementation details:

---

On the high file count issue

The large number of files mainly comes from a combination of factors inherent to the current layout and usage patterns:

• Zarr v2 metadata model: multiple metadata files per group/array (.zgroup, .zarray, .zattrs, optional .zmetadata)
• Chunking strategy: for chunked arrays, each chunk maps to a file, so chunk shape has a direct impact on file count
• Partitioning strategy: many small temporal/spatial partitions multiply both metadata and chunk files
• Long-lived static data: old partitions remain stored with layouts optimized for frequent updates, even when they become read-only

Possible mitigations / improvements to reduce the number of files:

• Switching to Zarr v3:
  • One metadata file per node (zarr.json)
  • Built-in consolidated metadata
  • Support for sharding, which can significantly reduce the number of chunk files for multi-chunked partitions
• Chunk choice:
  • Chunk sizes and shapes strongly influence the number of data files; better defaults or guidance could help
• Partition compaction:
  • Fewer partitions ⇒ fewer groups and metadata files
  • Especially relevant for older or less frequently accessed data
• Container storage (e.g. ZIP store):
  • Using a ZIP store for older, static data reduces file count at the cost of slower reads, which is acceptable for archive datasets

Potential ideas that could be explored:

1. Zarr v3 migration for new (and possibly existing) collections
2. Migration tool for old, static collections to move partitions into ZIP stores
3. Alternative migration tool to convert old collections into a single Zarr group
4. Repartitioning tool to help merge existing partitions (e.g. daily → weekly/monthly)

These approaches are not mutually exclusive and could fit into a broader data‑lifecycle strategy (hot vs cold data).