Skip to content

feat: add optional Milvus vector backend - #373

Open
zc277584121 wants to merge 2 commits into
volcengine:mainfrom
zc277584121:feat/milvus-vector-backend
Open

feat: add optional Milvus vector backend#373
zc277584121 wants to merge 2 commits into
volcengine:mainfrom
zc277584121:feat/milvus-vector-backend

Conversation

@zc277584121

Copy link
Copy Markdown

Description

Add Milvus as an opt-in vector storage backend while keeping ChromaDB as the default.

The backend uses pymilvus.MilvusClient with a shared uri and optional token configuration for Milvus Lite, remote Milvus servers, and Zilliz Cloud. It preserves MineContext's per-context-type collections and todo deduplication lifecycle, uses explicit schemas with COSINE AUTOINDEX, and reconstructs complete ProcessedContext records with optional vectors.

Additional behavior includes:

  • lazy factory registration and an actionable missing-extra error;
  • optional pymilvus[milvus-lite] dependency metadata;
  • batch and single upsert, retrieval, pagination, filtering, search, deletion, and count support;
  • higher-is-better similarity scores, including the Milvus Lite 3.0 COSINE compatibility case;
  • schema and index validation for existing collections;
  • a clear Windows error for local Milvus Lite paths while allowing remote HTTP endpoints;
  • commented configuration and English usage documentation.

This is backward compatible and does not change the default storage configuration.

Tests

  • PYTHONPATH=. uv run --no-project --python 3.11 --with pytest --with pydantic --with loguru --with pyyaml --with 'pymilvus[milvus-lite]>=3.0.0' pytest tests/test_milvus_backend.py -q6 passed
  • Real Milvus Lite coverage uses a fresh temporary database and verifies initialization, explicit schemas, collection reuse checks, batch and single upsert, retrieval with and without vectors, pagination, scalar and range filters, score ordering, deletion, counts, and the todo deduplication lifecycle.
  • uvx pre-commit run black --files ... — passed
  • uvx pre-commit run isort --files ... — passed
  • uvx ruff check --select E9,F63,F7,F82 ... — passed
  • uv lock --check — resolved successfully
  • Core storage import check without the Milvus extra — passed
  • A scoped Zilliz Cloud run reached the endpoint but could not initialize the required eight collections because the database is limited to five collections. The temporary collection created by the attempt was removed.

Signed-off-by: Cheney Zhang <chen.zhang@zilliz.com>
Signed-off-by: Cheney Zhang <chen.zhang@zilliz.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant