Skip to content

perf(transaction): backport O(M+N) fragment matching in build_manifest onto v9.0.1 - #11

Merged
ryantqiu merged 2 commits into
exa/v9.0.1from
exa/v9.0.1-txn-perf
Aug 14, 2026
Merged

perf(transaction): backport O(M+N) fragment matching in build_manifest onto v9.0.1#11
ryantqiu merged 2 commits into
exa/v9.0.1from
exa/v9.0.1-txn-perf

Conversation

@devin-ai-integration

Copy link
Copy Markdown

Summary

Backports upstream lance-format#8210 (commit 14c5f3cc, first released only in v11 betas) onto the stable v9.0.1 tag, so the Exa fork can ship the fix in a 9.0.1+exa.1 pylance wheel — the version silk_node's master locks now pin.

Transaction::build_manifest for Update/Delete matched updated fragments against existing fragments with a per-fragment linear scan (updated_fragments.iter().find(|uf| uf.id == f.id)), i.e. O(M×N). On Orion column-backfill commits touching 4096–8192 fragments against a ~1M-fragment table this dominates commit time (and repeats on every conflict retry). The backport replaces it with:

let removed_ids: HashSet<u64> = removed_fragment_ids.iter().copied().collect();
let mut updated_by_id: HashMap<u64, &Fragment> = HashMap::with_capacity(updated_fragments.len());
for fragment in updated_fragments {
    updated_by_id.entry(fragment.id).or_insert(fragment); // first-wins, same as .find()
}

Semantics are unchanged: same first-match duplicate behavior, same overlay carry-forward and RewriteColumns tombstoning, output ordering still follows the existing-fragments iteration. One conflict hunk from the upstream commit (reusing the maps in original_overlaid_frags) was dropped because v9.0.1 doesn't have that code path; the backported test needed merged_generations instead of upstream's compacted_sstables field.

Also bumps python/Cargo.toml to 9.0.1+exa.1 so the fork wheel is distinguishable from PyPI 9.0.1.

Verified: cargo test -p lance --lib dataset::transaction:: — 60/60 pass (incl. the two new tests from lance-format#8210); cargo fmt clean; built abi3 x86_64 + aarch64 manylinux wheels and smoke-tested a real LanceOperation.Update commit.

Supersedes #10 (same backport onto v7.0.0, made obsolete by the silk_node pylance 9.x bump).

Link to Devin session: https://app.devin.ai/sessions/9589ce7177cd42afbae9db44fc352d5b
Requested by: @ryantqiu

@devin-ai-integration

Copy link
Copy Markdown
Author

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

  • Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
  • Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

  • Disable automatic comment, CI, and merge conflict monitoring

@ryantqiu
ryantqiu merged commit 57f591c into exa/v9.0.1 Aug 14, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants