fix(index): refresh PQ storage row ids after fragment-reuse remap

[xuanyu-z]

Summary

Fixes a correctness bug where IVF_PQ (and IVF_HNSW_PQ) return stale row addresses after a deferred compaction (defer_index_remap), and adds correctness coverage across the deferred-compaction lifecycle for the vector index types.

ProductQuantizationStorage::new remaps its PQ codes through the fragment-reuse index but left the row_ids field bound to the pre-remap addresses. So once an index is loaded from a version carrying a fragment-reuse index, an IVF_PQ / IVF_HNSW_PQ query returns compacted-away row addresses and the scanner take fails with:

The input to a take operation specified fragment id N but this fragment does not exist in the dataset

This happens even for merge-only compaction and is only observable when the query fetches row content (a take) — the existing test_read_ivf_pq_index_v3_with_defer_index_remap projects no columns (counts row ids without taking), so it didn't catch it. IVF_FLAT/SQ/RQ already refresh their row ids correctly. The fix refreshes row_ids from the remapped batch alongside pq_code.

Test coverage added

Parameterized helpers that, unlike the existing test_read_*_with_defer_index_remap tests, project and fetch row content so stale addresses surface:

• check_vector_defer_compaction — fragment-reuse window correctness, with and without materialized deletions.
• check_vector_remap_and_trim — merge-only, then physical remap_column_index + cleanup_frag_reuse_index, asserting the reuse index trims to zero versions and results stay consistent.

Covered (passing): IVF_FLAT, IVF_SQ, IVF_RQ, IVF_PQ across window/deletions/remap+trim; IVF_HNSW_SQ/PQ merge-only; a scalar bitmap deletion case.

Two known gaps are intentionally not turned into perpetually-ignored tests (they're noted in code comments instead): IVF_HNSW_* and the inverted/FTS index desync under materialized deletions because their positional internal structures (HNSW graph node ids; FTS num_tokens[doc_id]) are not realigned when the fragment-reuse drop removes rows. The HNSW case is the deferred #3993; the FTS case is a separate follow-up.

Testing

cargo test -p lance --lib defer_compaction   # 6 passed
cargo test -p lance --lib remap_and_trim      # 6 passed

[jackye1995]

looks good to me, pending rebase

[codecov]

Codecov Report

❌ Patch coverage is 99.73190% with 1 line in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
rust/lance/src/dataset/optimize.rs	99.73%	1 Missing ⚠️

📢 Thoughts on this report? Let us know!