diff --git a/docs/guides/migration/index.md b/docs/guides/migration/index.md index f32c0ffc..5e21e8c7 100644 --- a/docs/guides/migration/index.md +++ b/docs/guides/migration/index.md @@ -5,6 +5,7 @@ :hidden: v1-v2 +v2-v3 ``` ## Versioning policy and breaking changes diff --git a/docs/guides/migration/v2-v3.md b/docs/guides/migration/v2-v3.md new file mode 100644 index 00000000..4cb47a86 --- /dev/null +++ b/docs/guides/migration/v2-v3.md @@ -0,0 +1,181 @@ +# Migrating from v2 to v3 + +Dataframely v3 focuses on trimming down surface area that turned out to be error-prone -- most notably the built-in +I/O and serialization functionality -- and on aligning the remaining API more closely with `polars`. + +## Improvements + +### Canonical Arrow interoperability + +Dataframely {class}`~dataframely.Schema` now implements the +[Arrow PyCapsule interface](https://arrow.apache.org/docs/format/CDataInterface/PyCapsuleInterface.html). This provides +a "canonical" Arrow representation of a schema that can be consumed by any package supporting the interface +(e.g. `polars`, `pyarrow`, ...) without dataframely having to implement a bespoke conversion method for each of them. +As a consequence, obtaining a `polars` or `pyarrow` schema is now done through the respective package's own +constructor. For a schema `MySchema`, code needs to change as follows: + +- `MySchema.to_polars_schema()` becomes `polars.Schema(MySchema)` +- `MySchema.to_pyarrow_schema()` becomes `pyarrow.schema(MySchema)` + +As a result, dataframely does not rely on `pyarrow` as an optional dependency anymore. + +### Refined `dy.Categorical` + +{class}`~dataframely.Categorical` now accepts a `categories` argument, allowing to pass a `pl.Categories` object. +Alternatively, a data type may be passed, which automatically infers `name` and `namespace` of the `pl.Categories` +object as the column and schema names, respectively, scoping the categorical to the specific column of a single schema. +If `categories` is not specified, the behavior is unchanged. + +```python +import polars as pl +import dataframely as dy + +class MySchema(dy.Schema): + # Column-scoped categories with a specific physical backing type + a = dy.Categorical(pl.UInt16) + # Explicitly shared categories dictionary + b = dy.Categorical(pl.Categories(name="shared", namespace="my_namespace")) +``` + +## Breaking Changes + +### Most I/O and serialization functionality is removed + +This is the headline change of v3 (see [#367](https://github.com/Quantco/dataframely/issues/367)). In v2, dataframely's +I/O methods serialized a schema into the storage backend's metadata (e.g. parquet metadata) and inspected that metadata +on read to decide whether validation was necessary. This approach caused a number of hard-to-solve issues: + +- The serialized schema could mismatch the current schema for many benign reasons (e.g. a new polars version or even a + _more lenient_ constraint), silently triggering an expensive re-validation and emitting warnings even when nothing + was actually wrong. +- Reading files required reading the metadata first. For a large number of files and/or files on S3, the round-trip + time introduced significant overhead -- a `scan_parquet` on a collection could take seconds without performing any + computation. +- Metadata cannot guarantee primary key constraints across more than one file, so the "trust the metadata" path was + efficient but potentially incorrect. + +For these reasons, dataframely v3 removes the bulk of I/O and serialization functionality. This makes user code more +explicit: read data with the corresponding `polars` function and call {meth}`~dataframely.Schema.validate` (or +{meth}`~dataframely.Collection.validate`) yourself when validation is required. All of the removed methods emitted +{class}`DeprecationWarning`s in the latest v2 releases. + +#### Schema I/O methods are removed + +All I/O methods on {class}`~dataframely.Schema` are removed. Replace them with the corresponding `polars` functions and +call {meth}`~dataframely.Schema.validate` explicitly if you need validation: + +| Removed method | Replacement | +| ---------------------- | ----------------------------------------- | +| `Schema.write_parquet` | `polars.DataFrame.write_parquet` | +| `Schema.sink_parquet` | `polars.LazyFrame.sink_parquet` | +| `Schema.read_parquet` | `polars.read_parquet` + `Schema.validate` | +| `Schema.scan_parquet` | `polars.scan_parquet` + `Schema.validate` | +| `Schema.write_delta` | `polars.DataFrame.write_delta` | +| `Schema.read_delta` | `polars.read_delta` + `Schema.validate` | +| `Schema.scan_delta` | `polars.scan_delta` + `Schema.validate` | + +For example: + +```python +# Before (v2) +df = MySchema.read_parquet("data.parquet") + +# After (v3) +import polars as pl +df = MySchema.validate(pl.read_parquet("data.parquet"), cast=True) +``` + +#### Collection I/O is limited to parquet without validation + +On {class}`~dataframely.Collection`, only `{read,scan,write,sink}_parquet` remain, so you can conveniently store and +retrieve collections. The `deltalake` methods (`write_delta`, `read_delta`, `scan_delta`) are removed. + +Additionally, {meth}`~dataframely.Collection.read_parquet` and {meth}`~dataframely.Collection.scan_parquet` no longer +inspect metadata, no longer run validation implicitly, and do not support partitioned files. The `validation` parameter +is removed; these methods now behave like the previous `validation="skip"`. If you need validation, call +{meth}`~dataframely.Collection.validate` explicitly after reading: + +```python +# Before (v2) -- validation ran implicitly (or with validation="warn"/"allow") +collection = MyCollection.read_parquet("dir") + +# After (v3) -- reading never validates; validate explicitly if required +collection = MyCollection.read_parquet("dir") +collection = MyCollection.validate(collection.to_dict(), cast=True) +``` + +#### `FailureInfo` loses its `deltalake` I/O + +Much like for collections, {class}`~dataframely.FailureInfo` retains `write_parquet`, `sink_parquet`, `read_parquet`, +and `scan_parquet` (now without any metadata handling), but its `write_delta`, `read_delta`, and `scan_delta` methods +are removed. + +#### Serialization functions are removed + +Because schemas and collections are no longer serialized into storage metadata, the serialization machinery is removed +entirely. The following are no longer available: + +- `Schema.serialize` and `dy.deserialize_schema` +- `Collection.serialize` and `dy.deserialize_collection` +- `dy.read_parquet_metadata_schema` and `dy.read_parquet_metadata_collection` +- The `dy.Validation` type and the `dy.DeserializationError` exception + +### `eager` is replaced by "lazy-in-lazy-out" and `lazy` + +The `eager: bool` parameter on {meth}`~dataframely.Schema.validate`/{meth}`~dataframely.Schema.filter` and +{meth}`~dataframely.Collection.validate`/{meth}`~dataframely.Collection.filter` is removed in favor of behavior that is +more consistent with `polars` (see [#372](https://github.com/Quantco/dataframely/issues/372)). + +**For schemas**, validation and filtering now follow "lazy-in-lazy-out"/"eager-in-eager-out": if you pass a +`pl.DataFrame`, you get an eager result back (raising immediately on failure); if you pass a `pl.LazyFrame`, you get a +lazy result back (which fails to `collect` if validation does not pass). There is no `eager` parameter anymore -- to +obtain a lazy result, simply pass a lazy frame: + +```python +# Before (v2) +lf = MySchema.validate(df, eager=False) + +# After (v3) +lf = MySchema.validate(df.lazy()) +``` + +**For collections**, the `eager` parameter is replaced by a `lazy: bool = False` parameter (mirroring +`sink_parquet`). By default, collection validation/filtering collects the results; set `lazy=True` to keep them lazy. +Note that `lazy=True` cannot be used on a collection that defines at least one eager member. + +```python +# Before (v2) +result = MyCollection.validate(data, eager=False) + +# After (v3) +result = MyCollection.validate(data, lazy=True) +``` + +### Group rules are replaced by `over` expressions + +The `group_by` parameter of {func}`~dataframely.rule` is removed. Rules that operated on groups of rows should now use +a `polars` [`over`](https://docs.pola.rs/api/python/stable/reference/expressions/api/polars.Expr.over.html) expression +instead. This streamlines the public API as described in [#316](https://github.com/Quantco/dataframely/issues/316). + +To migrate, drop the `group_by` argument and wrap the returned expression in `.over(...)` with the same columns: + +```python +# Before (v2) +class HouseSchema(dy.Schema): + zip_code = dy.String(nullable=False, min_length=3) + + @dy.rule(group_by=["zip_code"]) + def minimum_zip_code_count(cls) -> pl.Expr: + return pl.len() >= 2 + +# After (v3) +class HouseSchema(dy.Schema): + zip_code = dy.String(nullable=False, min_length=3) + + @dy.rule() + def minimum_zip_code_count(cls) -> pl.Expr: + return (pl.len() >= 2).over("zip_code") +``` + +If you previously used the literal `group_by="primary_key"` to reference the schema's primary key dynamically, you can +now do that by accessing `cls.primary_key()` inside the `over` expression.