Skip to content

enhance: run automatic collection loads without task queues - #15

Merged
sunby merged 2 commits into
codex/load-1m-segments-pr-stack-rebased-qv-workfrom
fix-proxy-auto-load-dql
Sep 10, 2026
Merged

enhance: run automatic collection loads without task queues#15
sunby merged 2 commits into
codex/load-1m-segments-pr-stack-rebased-qv-workfrom
fix-proxy-auto-load-dql

Conversation

@sunby

@sunby sunby commented Sep 4, 2026

Copy link
Copy Markdown
Owner

Automatic loads queued behind a saturated DQL main pool can block the shared dispatcher from reaching requery subtasks. Execute the real loadCollectionTask lifecycle directly from ensureCollectionReady, including timestamp allocation, validation, and the coordinator RPC, without entering a Proxy task queue.

Explicit LoadCollection remains on DDL. Same-collection singleflight, privilege checks, shared-load lifetime, and readiness waits are preserved. Automatic loads for different collections no longer consume Proxy queue capacity.

Validation on this Mac with matching native libraries:

  • Compiled the full Proxy package and passed the selected 21 auto-load/readiness tests plus 8 subtests, including five runs with the race detector. Tests use -tags dynamic,test -gcflags='all=-N -l'.
  • The previous queued implementation fails the saturated-DQL regression at the load deadline; the coordinator LoadCollection RPC is never reached.
  • Proxy lint reports 0 issues; run_clang_format.sh and git diff --check completed successfully.
  • Ran make lint-fix; full-repository lint remains blocked by the existing undefined: mocks error in internal/metastore/kv/querycoord/kv_catalog_test.go:372. The unfiltered Proxy and E2E suites were not run.

Related: #13 (Issues are disabled in this fork).

Signed-off-by: sunby <sunbingyi1992@gmail.com>
Signed-off-by: sunby <sunbingyi1992@gmail.com>
@sunby sunby changed the title enhance: schedule automatic loads with DQL enhance: run automatic collection loads without task queues Sep 8, 2026
@sunby
sunby merged commit fa54d82 into codex/load-1m-segments-pr-stack-rebased-qv-work Sep 10, 2026
10 of 13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant