Support storage_options in Parquet metadata reads (cudf.io.parquet.read_parquet_metadata)

[praateekmahajan]

Summary

cudf.io.parquet.read_parquet_metadata does not accept storage_options, unlike cudf.read_parquet. This prevents callers from reading footer metadata from authenticated S3-compatible/object-store inputs using the same credentials and endpoint configuration as the data read.

This also affects users of the RAPIDS 26.08 bulk helper pylibcudf.io.parquet_metadata.read_parquet_footers: pylibcudf.io.SourceInfo accepts sources, but there is no storage_options parameter to resolve authenticated remote paths. The available workaround is to depend on private cuDF I/O path-resolution helpers or materialize sources through fsspec.

Reproducer

import cudf

cudf.io.parquet.read_parquet_metadata(
    "s3://bucket/data.parquet",
    storage_options={
        "key": "...",
        "secret": "...",
        "client_kwargs": {"endpoint_url": "https://s3-compatible.example"},
    },
)

On cuDF 26.08.00 this raises:

TypeError: read_parquet_metadata() got an unexpected keyword argument 'storage_options'

The current signature is:

read_parquet_metadata(filepath_or_buffer)

Expected behavior

Accept storage_options=None and resolve remote inputs consistently with cudf.read_parquet, while retaining bulk input support and input ordering for read_parquet_footers/metadata reads.

Environment

• cuDF 26.08.00
• pylibcudf 26.08 bindings
• NVIDIA NeMo Curator nightly 2026-08-27

This is needed to replace per-file PyArrow footer reads in NeMo Curator's KMeans/Pairwise grouping without losing support for authenticated remote and S3-compatible datasets.

[praateekmahajan]

Also IIUC the code currently opens all files at once (and can run into FD limit). I wonder if that leads to an issue on lustre where so many concurrent file reads might be an issue?