docs: Add v3 migration guide

docs/guides/migration/index.md

@@ -5,6 +5,7 @@
 :hidden:
 
 v1-v2
+v2-v3
 ```
 
 ## Versioning policy and breaking changes

docs/guides/migration/v2-v3.md

@@ -0,0 +1,152 @@
+# Migrating from v2 to v3
+
+Dataframely v3 focuses on trimming down surface area that turned out to be error-prone -- most notably the built-in
+I/O and serialization functionality -- and on aligning the remaining API more closely with `polars`.
+
+## Improvements
+
+### Canonical Arrow interoperability
+
+Dataframely {class}`~dataframely.Schema` now implements the
+[Arrow PyCapsule interface](https://arrow.apache.org/docs/format/CDataInterface/PyCapsuleInterface.html). This provides
+a "canonical" Arrow representation of a schema that can be consumed by any package supporting the interface
+(e.g. `polars`, `pyarrow`, ...) without dataframely having to implement a bespoke conversion method for each of them.
+As a consequence, obtaining a `polars` or `pyarrow` schema is now done through the respective package's own
+constructor. For a schema `MySchema`, code needs to change as follows:
+
+- `MySchema.to_polars_schema()` becomes `polars.Schema(MySchema)`
+- `MySchema.to_pyarrow_schema()` becomes `pyarrow.schema(MySchema)`
+
+As a result, dataframely does not rely on `pyarrow` as an optional dependency anymore.
+
+### Refined `dy.Categorical`
+
+{class}`~dataframely.Categorical` now accepts a `categories` argument, allowing to pass a `pl.Categories` object.
+Alternatively, a data type may be passed, which automatically infers `name` and `namespace` of the `pl.Categories`
+object as the column and schema names, respectively, scoping the categorical to the specific column of a single schema.
+If `categories` is not specified, the behavior is unchanged.
+
+```python
+import polars as pl
+import dataframely as dy
+
+class MySchema(dy.Schema):
+    # Column-scoped categories with a specific physical backing type
+    a = dy.Categorical(pl.UInt16)
+    # Explicitly shared categories dictionary
+    b = dy.Categorical(pl.Categories(name="shared", namespace="my_namespace"))
+```
+
+## Breaking Changes
+
+### Most I/O and serialization functionality is removed
+
+This is the headline change of v3 (see [#367](https://github.com/Quantco/dataframely/issues/367)). In v2, dataframely's
+I/O methods serialized a schema into the storage backend's metadata (e.g. parquet metadata) and inspected that metadata
+on read to decide whether validation was necessary. This approach caused a number of hard-to-solve issues:
+
+- The serialized schema could mismatch the current schema for many benign reasons (e.g. a new polars version or even a
+  _more lenient_ constraint), silently triggering an expensive re-validation and emitting warnings even when nothing
+  was actually wrong.
+- Reading files required reading the metadata first. For a large number of files and/or files on S3, the round-trip
+  time introduced significant overhead -- a `scan_parquet` on a collection could take seconds without performing any
+  computation.
+- Metadata cannot guarantee primary key constraints across more than one file, so the "trust the metadata" path was
+  efficient but potentially incorrect.
+
+For these reasons, dataframely v3 removes the bulk of I/O and serialization functionality. This makes user code more
+explicit: read data with the corresponding `polars` function and call {meth}`~dataframely.Schema.validate` (or
+{meth}`~dataframely.Collection.validate`) yourself when validation is required. All of the removed methods emitted
+{class}`DeprecationWarning`s in the latest v2 releases.
+
+#### Schema I/O methods are removed
+
+All I/O methods on {class}`~dataframely.Schema` are removed. Replace them with the corresponding `polars` functions and
+call {meth}`~dataframely.Schema.validate` explicitly if you need validation:
+
+| Removed method         | Replacement                               |
+| ---------------------- | ----------------------------------------- |
+| `Schema.write_parquet` | `polars.DataFrame.write_parquet`          |
+| `Schema.sink_parquet`  | `polars.LazyFrame.sink_parquet`           |
+| `Schema.read_parquet`  | `polars.read_parquet` + `Schema.validate` |
+| `Schema.scan_parquet`  | `polars.scan_parquet` + `Schema.validate` |
+| `Schema.write_delta`   | `polars.DataFrame.write_delta`            |
+| `Schema.read_delta`    | `polars.read_delta` + `Schema.validate`   |
+| `Schema.scan_delta`    | `polars.scan_delta` + `Schema.validate`   |
+
+For example:
+
+```python
+# Before (v2)
+df = MySchema.read_parquet("data.parquet")
+
+# After (v3)
+import polars as pl
+df = MySchema.validate(pl.read_parquet("data.parquet"), cast=True)
+```
+
+#### Collection I/O is limited to parquet without validation
+
+On {class}`~dataframely.Collection`, only `{read,scan,write,sink}_parquet` remain, so you can conveniently store and
+retrieve collections. The `deltalake` methods (`write_delta`, `read_delta`, `scan_delta`) are removed.
+
+Additionally, {meth}`~dataframely.Collection.read_parquet` and {meth}`~dataframely.Collection.scan_parquet` no longer
+inspect metadata, no longer run validation implicitly, and do not support partitioned files. The `validation` parameter
+is removed; these methods now behave like the previous `validation="skip"`. If you need validation, call
+{meth}`~dataframely.Collection.validate` explicitly after reading:
+
+```python
+# Before (v2) -- validation ran implicitly (or with validation="warn"/"allow")
+collection = MyCollection.read_parquet("dir")
+
+# After (v3) -- reading never validates; validate explicitly if required
+collection = MyCollection.read_parquet("dir")
+collection = MyCollection.validate(collection.to_dict(), cast=True)
+```
+
+#### `FailureInfo` loses its `deltalake` I/O
+
+Much like for collections, {class}`~dataframely.FailureInfo` retains `write_parquet`, `sink_parquet`, `read_parquet`,
+and `scan_parquet` (now without any metadata handling), but its `write_delta`, `read_delta`, and `scan_delta` methods
+are removed.
+
+#### Serialization functions are removed
+
+Because schemas and collections are no longer serialized into storage metadata, the serialization machinery is removed
+entirely. The following are no longer available:
+
+- `Schema.serialize` and `dy.deserialize_schema`
+- `Collection.serialize` and `dy.deserialize_collection`
+- `dy.read_parquet_metadata_schema` and `dy.read_parquet_metadata_collection`
+- The `dy.Validation` type and the `dy.DeserializationError` exception
+
+### `eager` is replaced by "lazy-in-lazy-out" and `lazy`
+
+The `eager: bool` parameter on {meth}`~dataframely.Schema.validate`/{meth}`~dataframely.Schema.filter` and
+{meth}`~dataframely.Collection.validate`/{meth}`~dataframely.Collection.filter` is removed in favor of behavior that is
+more consistent with `polars` (see [#372](https://github.com/Quantco/dataframely/issues/372)).
+
+**For schemas**, validation and filtering now follow "lazy-in-lazy-out"/"eager-in-eager-out": if you pass a
+`pl.DataFrame`, you get an eager result back (raising immediately on failure); if you pass a `pl.LazyFrame`, you get a
+lazy result back (which fails to `collect` if validation does not pass). There is no `eager` parameter anymore -- to
+obtain a lazy result, simply pass a lazy frame:
+
+```python
+# Before (v2)
+lf = MySchema.validate(df, eager=False)
+
+# After (v3)
+lf = MySchema.validate(df.lazy())
+```
+
+**For collections**, the `eager` parameter is replaced by a `lazy: bool = False` parameter (mirroring
+`sink_parquet`). By default, collection validation/filtering collects the results; set `lazy=True` to keep them lazy.
+Note that `lazy=True` cannot be used on a collection that defines at least one eager member.
+
+```python
+# Before (v2)
+result = MyCollection.validate(data, eager=False)
+
+# After (v3)
+result = MyCollection.validate(data, lazy=True)
+```