fix(dataset): stream sub-batches through merge_columns without 2GiB concat

[jan-exa]

Summary

merge_columns/add_columns fails with Offset overflow error when one updater read batch carries >2GiB of variable-width data (32-bit binary/string offsets): add_columns_from_stream concatenated all stream batches covering a read batch into a single Arrow RecordBatch before handing it to the updater. Observed in production on a ~200k-row fragment whose document_proto column totals ~2.74GiB.

Fix: never concatenate on the v2+ path.

• New Updater::update_batches(Vec<RecordBatch>) alongside the unchanged update:
  • validates sum(rows) == source batch rows;
  • runs DeletionRestorer::restore per sub-batch (restore is incremental over row addresses, so blanks land at identical global positions as the single-concat path);
  • adapts each sub-batch and passes the whole slice to GenericWriter::write — no concat anywhere;
  • legacy (v1) files still delegate to the single-batch update via concat, since DeletionRestorer validates whole row-groups there and v1 row groups can't reach 2GiB.
• add_columns_from_stream now calls updater.update_batches(batches) instead of concat_batches + update.
• add_blanks on an empty batch (fully-deleted read batch) now synthesizes null placeholder rows when all fields are nullable (previously always NotSupported); non-nullable schemas keep the original "run compaction" error.

Purely a write-path change: committed schema, storage version, and reader-visible bytes are unchanged.

Testing

• New deletion-alignment equivalence test: v2 dataset with deletions at read-batch boundaries and a fully-deleted read batch, merged via a multi-batch stream; scan asserted value-exact for binary/string/fixed-width columns incl. nulls.
• update_batches error cases (empty list, row-count mismatch) and legacy-path delegation test.
• add_blanks non-nullable empty-batch regression test.
• #[ignore] >2GiB regression test (test_add_columns_binary_column_over_2gib, ~5GiB RAM): 1,100 x 2MiB binary values streamed in 10-row batches into a single fragment; merge_columns succeeds, dataset.validate() passes, and a full scan verifies every value. Run manually: passed (1 passed; 0 failed, 14.7s). Reproduces the exact production failure shape (previously: Offset overflow error: 2737806570 at schema_evolution.rs:672).
• cargo fmt --all, cargo clippy --all --tests --benches -- -D warnings, targeted cargo test -p lance all pass.

Link to Devin session: https://app.devin.ai/sessions/7cd5f358690a4d7cb4fbe13d6b5e4ad5
Requested by: @jan-exa

[devin-ai-integration]

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

• Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
• Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

• Disable automatic comment, CI, and merge conflict monitoring