Skip to content

fix(dataset): use empty/null blanks when restoring deleted rows in merge - #18

Merged
devin-ai-integration[bot] merged 3 commits into
exa/v9.0.1-txn-perffrom
devin/1787377124-fix-add-blanks-offset-overflow
Aug 22, 2026
Merged

fix(dataset): use empty/null blanks when restoring deleted rows in merge#18
devin-ai-integration[bot] merged 3 commits into
exa/v9.0.1-txn-perffrom
devin/1787377124-fix-add-blanks-offset-overflow

Conversation

@jan-exa

@jan-exa jan-exa commented Aug 22, 2026

Copy link
Copy Markdown

Summary

add_blanks (used by Updater/merge_columns to re-align batches with a fragment's deleted rows) filled every deleted-row position by duplicating the batch's first value via a take selection vector (the old // For simplicity, we just use the first value for deleted rows TODO). On heavily-deleted fragments with fat variable-width values this overflows Arrow's 2 GiB 32-bit offset limit — observed in production backfill on a fragment with 191,752 of 195,879 rows deleted (LanceError(Arrow): Failed to add blanks: Offset overflow error).

Fix: interleave the real data with a 1-row type-appropriate blank instead of duplicating row 0:

// before: selection_vector.push(0)  → take() duplicates row 0's full bytes per blank
// after:  indices.push((1, 0))      → interleave([batch_column, blank_array], indices)
fn blank_array(data_type, nullable, len) -> ArrayRef
// nullable → new_null_array; non-nullable → zero/empty per type
// (primitives 0, Utf8/Binary "", recursive for Struct/List/FixedSizeList/Map/FixedSizeBinary)

Output buffers no longer scale with deleted-row count (regression test: 1 MiB first value + 100,000 blanks, asserts value_data().len() stays 1 MiB).

Write-path only and reader-equivalent: blank rows sit at deleted positions and are always masked by the deletion vector, so scans never see them — test_merge_preserves_visible_rows_with_deletions asserts visible rows are byte-identical before/after a merge over a fragment with deletions. Committed schema and on-disk format are unchanged.

Non-nullable types with no constructible blank (e.g. dictionaries) now return a descriptive NotSupported error instead of silently duplicating the first value.

Verified: cargo fmt, clippy --all --tests --benches -D warnings, updater/schema-evolution/merge test suites (65 tests) all green.

Link to Devin session: https://app.devin.ai/sessions/7cd5f358690a4d7cb4fbe13d6b5e4ad5
Requested by: @jan-exa

devin-ai-integration Bot and others added 3 commits August 22, 2026 06:31
Co-Authored-By: Jan van der Vegt <jan@exa.ai>
Co-Authored-By: Jan van der Vegt <jan@exa.ai>
Co-Authored-By: Jan van der Vegt <jan@exa.ai>
@devin-ai-integration

Copy link
Copy Markdown

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

  • Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
  • Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

  • Disable automatic comment, CI, and merge conflict monitoring

@github-actions github-actions Bot added the bug Something isn't working label Aug 22, 2026
@devin-ai-integration
devin-ai-integration Bot changed the base branch from main to exa/v9.0.1-txn-perf August 22, 2026 06:48
@devin-ai-integration
devin-ai-integration Bot merged commit 77c029e into exa/v9.0.1-txn-perf Aug 22, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant