Skip to content

perf: make Rewrite transaction fragment handling O(n) instead of O(groups × fragments) - #24

Merged
tanish-exa merged 2 commits into
exa/v9.0.1-txn-perffrom
devin/1787704701-txn-perf-exa6
Aug 26, 2026
Merged

perf: make Rewrite transaction fragment handling O(n) instead of O(groups × fragments)#24
tanish-exa merged 2 commits into
exa/v9.0.1-txn-perffrom
devin/1787704701-txn-perf-exa6

Conversation

@tanish-exa

@tanish-exa tanish-exa commented Aug 26, 2026

Copy link
Copy Markdown

Summary

handle_rewrite_fragments rebuilt each rewrite group by scanning the whole fragment vector, so a compaction commit cost O(groups × fragments). On a manifest with ~4.5M fragments and 128 groups per batch that dominates commit wall time. It now indexes once and rebuilds once:

let id_to_pos: HashMap<u64, usize> = ...;      // one pass over fragments
for group in groups {
    let positions = group.old_fragments.map(|f| id_to_pos[&f.id]);  // O(1) each
    removed[pos] = true;                        // mark, don't splice
    replacements.insert(min(positions), group.new_fragments);
}
fragments = rebuild(fragments, removed, replacements);  // one pass

Contiguous groups keep their replacements at the original position; non-contiguous groups append, preserving the previous ordering semantics.

Second commit adds the safety guard the position-indexed form needs: two groups replacing the same fragment id used to silently overwrite one replacement entry (the old scan-and-splice form couldn't express that state). It's now a CommitConflict, with a regression test.

Targets the exa/v9.0.1-txn-perf release line — the same change landed on main as #22 and was reverted by #23, since production wheels are cut from this branch. The 9.0.1+exa.6 version bump follows in its own PR, as with #20#21.

Link to Devin session: https://app.devin.ai/sessions/7d75a309bb144583a14b8704bbdb28e3
Requested by: @tanish-exa

tanish-exa and others added 2 commits August 26, 2026 00:38
…oups * fragments)

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
@devin-ai-integration

Copy link
Copy Markdown

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

  • Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
  • Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

  • Disable automatic comment, CI, and merge conflict monitoring

@github-actions

Copy link
Copy Markdown

ACTION NEEDED
Lance follows the Conventional Commits specification for release automation.

The PR title and description are used as the merge commit message. Please update your PR title and description to match the specification.

For details on the error please inspect the "PR Title Check" action.

@devin-ai-integration
devin-ai-integration Bot force-pushed the devin/1787704701-txn-perf-exa6 branch from 5a65483 to c6c6492 Compare August 26, 2026 00:40
@devin-ai-integration devin-ai-integration Bot changed the title perf: O(n) Rewrite transaction fragment handling + version pylance 9.0.1+exa.6 perf: make Rewrite transaction fragment handling O(n) instead of O(groups × fragments) Aug 26, 2026
@tanish-exa
tanish-exa merged commit b7b64cb into exa/v9.0.1-txn-perf Aug 26, 2026
2 of 4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant