enhance: add event-driven stats discovery - #20

Summary

Add opt-in event-driven discovery for DataCoord TextIndexJob and JsonKeyIndexJob. Metadata publication records bounded dirty keys; the inspector rechecks current metadata before submission. Keep the existing task format, eligibility rules, scheduler/worker retries and cleanup behavior.

• Deduplicate segment/subjob keys, preserve updates received while a key is in flight, and retry admission/dependency failures without requiring a new business event.
• Promote overflow into bounded collection/global reconciliation scopes. Stream existing metadata with a shared scan budget, including tombstones, without materializing a full segment snapshot or adding another full-size index.
• Notify after successful persistence and cache publication, covering segment/schema changes, compaction, deletion and task cleanup. Reconcile on startup and relevant configuration changes; unregister callbacks on stop.
• Reuse field rules and successful file-resource lookups within a processing batch. Avoid allocating a field-helper map for match checks.
• Add bounded-label discovery metrics, behavioral/failure tests and allocation benchmarks.

Only two public startup-level settings are added:

dataCoord:
  statsInspector:
    discoveryMode: poll
    reconcileInterval: 600

poll remains the default. shadow keeps legacy submission and makes the new path read-only. event replaces periodic legacy discovery with events plus bounded reconciliation. IndexInspector, Load/Search, WAL protocols, Sort/BM25 scheduling and unrelated GC optimizations are outside this PR.

Base and references

• Base: sunby/milvus:codex/load-1m-segments-pr-stack-rebased-qv-work at 7e32a5ca9b7e472f1563668e497595a1f57b0d7d. Only the StatsInspector patch is replayed on this squashed base; no already-merged stack changes are included.
• Related issue: [Enhancement]: [Collection Load] is slow for 1000x1000 recovery milvus-io/milvus#48867 (million-segment scalability context only; this PR does not claim to fix QueryNode recovery or reopen/close that issue).
• Design and implementation record: https://github.com/sunby/milvus/blob/codex/stats-inspector-event-driven/docs/design-docs/design_docs/20260908-stats-inspector-event-driven-plan.md

Verification

Rerun on the submission base:

• DataCoord discovery/queue tests and the existing StatsInspector/StatsTaskMeta suites passed with -race -tags dynamic,test -gcflags="all=-N -l" -count=3 (16.039 seconds). Coverage includes overflow, missed notifications, in-flight generations, capacity/allocator/persistence/resource failures, schema/config changes, cleanup, compaction variants, cancellation and persisted-before-enqueue recovery.
• Targeted metrics/paramtable/field-helper/merr tests passed with the required tags and compiler flags.
• Changed-code lint passed for DataCoord and the affected pkg packages against the submission base, with zero issues.
• run_clang_format.sh completed using clang-format 15.0.7 in an isolated verification worktree. No C/C++ changes or unrelated formatting changes are included in this PR.
• Full make lint-fix ran gofumpt/gci, then failed root-module type checking at internal/metastore/kv/querycoord/kv_catalog_test.go:372: undefined: mocks. This file has the same blob in the base and head; pkg/client stages were not reached. Unrelated auto-format changes are excluded. The pinned gci version misclassifies the standard-library iter package; the committed import grouping passes the current changed-code lint.
• git diff --check passed.

Earlier same-build local microbenchmarks at 100,000 unchanged segments measured 10,624,400 B per legacy discovery pass versus 0 B for an empty event-consumer pass. Starting a streaming cursor, reading its first entry and stopping used 232 B. These measure local allocation paths, not whole-process allocation or online GC/search latency improvement.

Remaining validation / adversarial review

Draft: keep the default poll mode until acceptance is complete.

• Full DataCoord/shared-package regression is not green. Earlier runs reproduced failures on the unmodified implementation base: timestamp and DISKANN assertions, missing persistence/mocks in external-refresh/schema-bump/meta-reload tests, QueryView configuration defaults and old Proxy metric label arity. These unrelated failures are not fixed here.
• Full make test-go was not run; production-build end-to-end acceptance remains pending. Source-traced producer coverage and the targeted failure tests above do not replace all import/restore/worker/storage fault-injection tests.
• Validate large-scale convergence, continuous metadata updates, saturation and multi-collection fairness under realistic load. The conservative internal scan budget is 128 entries per 100 ms: one million entries needs at least roughly 13 minutes before additional backpressure/dependency cost. The 10 ms consumption slice is soft and does not preempt an individual RPC.
• Same-load CPU/alloc pprof and load+search p95/p99 A/B remain pending. No deployment, online configuration change or online performance-gain claim is part of this PR.