Skip to content

enhance: scope collection load and state reads - #25

Merged
sunby merged 1 commit into
codex/load-1m-segments-pr-stack-rebased-qv-workfrom
codex/scoped-collection-load-reads
Sep 9, 2026
Merged

enhance: scope collection load and state reads#25
sunby merged 1 commit into
codex/load-1m-segments-pr-stack-rebased-qv-workfrom
codex/scoped-collection-load-reads

Conversation

@sunby

@sunby sunby commented Sep 9, 2026

Copy link
Copy Markdown
Owner

Collection-scoped load and state reads currently rebuild global config or shard snapshots after unrelated updates. This change limits those reads to the requested collections, so their in-memory work scales with the selected configs and shards.

  • Add atomic point reads and scoped load-config snapshots, retaining both collection and store versions without refreshing the full cached snapshot.
  • Resolve Balancer trigger scope before capturing configs. Full listing, full reconciliation, conservative fallback, and cluster-wide row totals retain their existing behavior.
  • Use scoped reads for LoadCollection admission, ShowLoadCollections/ShowLoadPartitions, GetQueryViewLoadInfo, and the load-progress path used by GetLoadState. Preserve current-replica filtering and the RPC's global version semantics.
  • Document the read contract and existing manager-reclamation behavior. Add regressions for concurrent updates, release/reload, missing IDs, replica filtering, manager removal, and scoped/full triggers.

Related issue: this fork has issues disabled, so an associated issue could not be created.

Validation

  • Full loadmgr, coordview, and balancer package tests passed with -race -tags dynamic,test -gcflags='all=-N -l'.
  • Focused QueryCoord tests passed with the same flags: scoped load/progress reads, runtime recovery/notifications, segment-load-info watch sessions, and load/release ACK callbacks.
  • golangci-lint --new-from-rev=HEAD passed for all four changed packages, including the staged new tests. run_clang_format.sh and git diff --check passed.
  • make lint-fix was run but is blocked by an existing base-branch typecheck error: internal/metastore/kv/querycoord/kv_catalog_test.go:372:15: undefined: mocks. Formatter edits outside this change were reverted.

In-memory benchmark

Median of three 300 ms runs on Go 1.26.5, darwin/arm64, Apple M5, with -tags dynamic,test -gcflags='all=-N -l'. Each iteration invalidates the full snapshot and reads one collection with one replica/shard. Persistence, RPCs, and concurrent contention are excluded.

Loaded collections/shards Config full (ms) Config point (ns) Config scoped (ns) Shard full + filter (ms) Shard scoped + filter (ns)
10,000 1.229 48.13 363.4 0.508 236.8
50,000 6.817 47.91 330.0 2.609 245.2
150,000 37.669 48.38 317.1 12.203 240.4

Point reads allocate zero bytes; scoped config and shard reads allocate 608 B and 352 B respectively at every size. These results validate the scoped in-memory cost; end-to-end UAT latency improvement has not been measured or deployed.

Signed-off-by: sunby <sunbingyi1992@gmail.com>
@sunby
sunby merged commit 78e8212 into codex/load-1m-segments-pr-stack-rebased-qv-work Sep 9, 2026
10 of 13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant