fix(dataset): use empty/null blanks when restoring deleted rows in merge - #18
Merged
devin-ai-integration[bot] merged 3 commits intoAug 22, 2026
Conversation
Co-Authored-By: Jan van der Vegt <jan@exa.ai>
Co-Authored-By: Jan van der Vegt <jan@exa.ai>
Co-Authored-By: Jan van der Vegt <jan@exa.ai>
🤖 Devin AI EngineerI'll be helping with this pull request! Here's what you should know: ✅ I will automatically:
Note: I can only respond to comments from users who have write access to this repository. ⚙️ Control Options:
|
devin-ai-integration
Bot
changed the base branch from
main
to
exa/v9.0.1-txn-perf
August 22, 2026 06:48
devin-ai-integration
Bot
merged commit Aug 22, 2026
77c029e
into
exa/v9.0.1-txn-perf
5 checks passed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
add_blanks(used byUpdater/merge_columnsto re-align batches with a fragment's deleted rows) filled every deleted-row position by duplicating the batch's first value via atakeselection vector (the old// For simplicity, we just use the first value for deleted rowsTODO). On heavily-deleted fragments with fat variable-width values this overflows Arrow's 2 GiB 32-bit offset limit — observed in production backfill on a fragment with 191,752 of 195,879 rows deleted (LanceError(Arrow): Failed to add blanks: Offset overflow error).Fix: interleave the real data with a 1-row type-appropriate blank instead of duplicating row 0:
Output buffers no longer scale with deleted-row count (regression test: 1 MiB first value + 100,000 blanks, asserts
value_data().len()stays 1 MiB).Write-path only and reader-equivalent: blank rows sit at deleted positions and are always masked by the deletion vector, so scans never see them —
test_merge_preserves_visible_rows_with_deletionsasserts visible rows are byte-identical before/after a merge over a fragment with deletions. Committed schema and on-disk format are unchanged.Non-nullable types with no constructible blank (e.g. dictionaries) now return a descriptive
NotSupportederror instead of silently duplicating the first value.Verified:
cargo fmt,clippy --all --tests --benches -D warnings, updater/schema-evolution/merge test suites (65 tests) all green.Link to Devin session: https://app.devin.ai/sessions/7cd5f358690a4d7cb4fbe13d6b5e4ad5
Requested by: @jan-exa