Skip to content

[milvus] Support component deployment groups - #306

Open
chyezh wants to merge 1 commit into
zilliztech:masterfrom
chyezh:codex/component-deployment-groups
Open

chyezh wants to merge 1 commit into
zilliztech:masterfrom
chyezh:codex/component-deployment-groups

Conversation

@chyezh

@chyezh chyezh commented Jul 14, 2026

Copy link
Copy Markdown
Contributor

What this PR does / why we need it:

This PR adds generic deployment group support for selected Milvus components so operators can split workloads into multiple Kubernetes Deployments, for example by availability zone or other user-defined labels.

Changes included:

  • Adds groups support for Proxy, DataNode, QueryNode, and StreamingNode.
  • Keeps MixCoord as a single Deployment while allowing custom labels on that Deployment and its pods.
  • Adds a reserved milvus.io/deployment-group selector label to avoid overlapping Deployment selectors.
  • Supports group-level replicas, labels, annotations, extraEnv, nodeSelector, affinity, tolerations, and topologySpreadConstraints.
  • Renders per-group HPAs for Proxy, DataNode, and QueryNode when HPA is enabled.
  • Keeps the legacy replicaResourceGroups path compatible for QueryNode and StreamingNode.
  • Documents deployment groups and how to inject Milvus resource-group env vars explicitly through extraEnv.

Validation:

  • helm lint charts/milvus
  • helm template defaulttest charts/milvus
  • helm template grouptest charts/milvus ... groups ...
  • helm template legacytest charts/milvus --set replicaResourceGroups[0]=rg-a --set replicaResourceGroups[1]=rg-b
  • helm template badtest charts/milvus --set proxy.groups[0].replicas=1 verified the missing group name error
  • git diff --check

Checklist

  • Chart Version bumped
  • Variables are documented in the README.md
  • Title of the PR starts with chart name (e.g. [mychartname])
  • PR only contains changes for one chart

@sre-ci-robot

Copy link
Copy Markdown
Collaborator

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: chyezh
To complete the pull request process, please assign haorenfsa after the PR has been reviewed.
You can assign the PR to them by writing /assign @haorenfsa in a comment when ready.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@mergify mergify Bot added the ci-passed label Jul 14, 2026
chyezh added a commit to chyezh/milvus-helm that referenced this pull request Jul 14, 2026
Signed-off-by: chyezh <chyezh@outlook.com>
@mergify mergify Bot removed the ci-passed label Jul 14, 2026
@chyezh
chyezh marked this pull request as ready for review July 14, 2026 02:55
@chyezh

chyezh commented Jul 14, 2026

Copy link
Copy Markdown
Contributor Author

QA Verification Summary

I verified this PR in the qa-milvus namespace with Helm release yezhen-dg.

What was validated

  • proxy.groups, dataNode.groups, queryNode.groups, and streamingNode.groups can render multiple independent Deployments.
  • Each Deployment group can independently configure replicas, labels, extraEnv, and scheduling through nodeSelector.
  • mixCoordinator remains a single Deployment and supports custom labels.
  • QueryNode and StreamingNode resource group labels are decoupled from Helm deployment groups and are controlled through MILVUS_SERVER_LABEL_RESOURCE_GROUP.

Runtime examples verified:

  • querynode-new-a: MILVUS_SERVER_LABEL_RESOURCE_GROUP=new_rg_a
  • querynode-new-b: MILVUS_SERVER_LABEL_RESOURCE_GROUP=new_rg_b
  • streamingnode-new-a: MILVUS_SERVER_LABEL_RESOURCE_GROUP=new_rg_a
  • streamingnode-new-b: MILVUS_SERVER_LABEL_RESOURCE_GROUP=new_rg_b

Proxy and DataNode were also split into separate AZ-style deployment groups with independent labels and env vars.

Blue-green flow verified

I validated a multi-replica to multi-replica blue-green flow. Runtime load and streaming settings were changed through etcd dynamic config, not through user.yaml.

Sequence:

  1. Blue: replicaNumber=2, resourceGroups=old_rg_a,old_rg_b, streaming.primaryResourceGroup=old_rg_a
  2. Overlap: replicaNumber=4, resourceGroups=old_rg_a,old_rg_b,new_rg_a,new_rg_b
  3. Green: replicaNumber=2, resourceGroups=new_rg_a,new_rg_b, streaming.primaryResourceGroup=new_rg_a
  4. Removed old QueryNode and StreamingNode deployment groups with a Helm upgrade.

Final state:

  • Old QueryNode/StreamingNode Deployments were removed.
  • New QueryNode/StreamingNode Deployments stayed Ready.
  • /management/replica/loadconfig/compliance returned {"state":"Ready"}.
  • new_rg_a and new_rg_b each had one loaded replica for the test collection.
  • old_rg_a and old_rg_b had no nodes and no loaded replicas.

Workload result

A continuous search/query workload ran through baseline, overlap expansion, dynamic config switch, green cutover, and old deployment removal.

Final workload summary:

{"count": 1860, "errors": 0, "elapsed": 467.11}

Error log:

/tmp/yezhen_bg_errors.log: 0 lines

No serviceability or routing errors were observed, including:

  • channel distribution is not serviceable
  • channel not available
  • no available shard leader
  • partial-result errors

Compatibility check

I also checked the old paths:

  • When groups are not configured, the default render path is unchanged except for chart/app/image version metadata.
  • The existing replicaResourceGroups compatibility path still renders per-RG QueryNode and StreamingNode Deployments and writes the corresponding generated load config.

I added a documentation update to clarify that groups and replicaResourceGroups should not be mixed in the same release, because groups control the rendered Deployments while replicaResourceGroups still writes generated Milvus load config.

@mergify mergify Bot added ci-passed and removed ci-passed labels Jul 14, 2026
Signed-off-by: chyezh <chyezh@outlook.com>
@chyezh
chyezh force-pushed the codex/component-deployment-groups branch from 2f6daef to d39d820 Compare July 30, 2026 01:57
@mergify mergify Bot removed the ci-passed label Jul 30, 2026
Comment thread charts/milvus/values.yaml
# Optional deployment groups. When set, one Deployment is rendered per group.
# Group-level replicas, labels, annotations, extraEnv, nodeSelector, affinity,
# tolerations, and topologySpreadConstraints override or extend this component.
groups: []

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could the relationship between replicaResourceGroups and groups be documented? It would be good to state this explicitly in the guide / values comments.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants