Skip to content

[milvus] Support component deployment groups #306

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
chyezh wants to merge 1 commit into
base: master
Choose a base branch
from
Open

[milvus] Support component deployment groups #306

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions charts/milvus/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -439,6 +439,7 @@ The following table lists the configurable parameters of the Milvus Proxy compon
|-------------------------------------------|---------------------------------------------------------|---------------|
| `proxy.enabled` | Enable or disable Milvus Proxy Deployment | `true` |
| `proxy.replicas` | Desired number of Milvus Proxy pods | `1` |
| `proxy.groups` | Optional deployment groups for rendering multiple Proxy Deployments | `[]` |
| `proxy.resources` | Resource requests/limits for the Milvus Proxy pods | `{}` |
| `proxy.nodeSelector` | Node labels for Milvus Proxy pods assignment | `{}` |
| `proxy.affinity` | Affinity settings for Milvus Proxy pods assignment | `{}` |
Expand Down Expand Up @@ -468,6 +469,7 @@ The following table lists the configurable parameters of the Milvus Query Node c
|-------------------------------------------|-----------------------------------------------|---------------------------------------------------------|
| `queryNode.enabled` | Enable or disable Milvus Query Node component | `true` |
| `queryNode.replicas` | Desired number of Milvus Query Node pods | `1` |
| `queryNode.groups` | Optional deployment groups for rendering multiple Query Node Deployments | `[]` |
| `queryNode.resources` | Resource requests/limits for the Milvus Query Node pods | `{}` |
| `queryNode.nodeSelector` | Node labels for Milvus Query Node pods assignment | `{}` |
| `queryNode.affinity` | Affinity settings for Milvus Query Node pods assignment | `{}` |
Expand Down Expand Up @@ -496,6 +498,7 @@ The following table lists the configurable parameters of the Milvus Data Node co
|-------------------------------------------|-----------------------------------------------|---------------------------------------------------------|
| `dataNode.enabled` | Enable or disable Data Node component | `true` |
| `dataNode.replicas` | Desired number of Data Node pods | `1` |
| `dataNode.groups` | Optional deployment groups for rendering multiple Data Node Deployments | `[]` |
| `dataNode.resources` | Resource requests/limits for the Milvus Data Node pods | `{}` |
| `dataNode.nodeSelector` | Node labels for Milvus Data Node pods assignment | `{}` |
| `dataNode.affinity` | Affinity settings for Milvus Data Node pods assignment | `{}` |
Expand All @@ -518,6 +521,7 @@ The following table lists the configurable parameters of the Milvus Mix Coordina
|-------------------------------------------|-----------------------------------------------|---------------------------------------------------------|
| `mixCoordinator.enabled` | Enable or disable Mix Coordinator component | `true` |
| `mixCoordinator.replicas` | Desired number of Mix Coordinator pods | `1` |
| `mixCoordinator.labels` | Additional labels for the single Mix Coordinator Deployment and pods | `{}` |
| `mixCoordinator.resources` | Resource requests/limits for the Milvus Mix Coordinator pods | `{}` |
| `mixCoordinator.nodeSelector` | Node labels for Milvus Mix Coordinator pods assignment | `{}` |
| `mixCoordinator.affinity` | Affinity settings for Milvus Mix Coordinator pods assignment | `{}` |
Expand All @@ -544,6 +548,7 @@ The following table lists the configurable parameters of the Milvus Streaming No
|-------------------------------------------|---------------------------------------------------------|---------------|
| `streamingNode.enabled` | Enable or disable Streaming Node component | `true` |
| `streamingNode.replicas` | Desired number of Streaming Node pods | `1` |
| `streamingNode.groups` | Optional deployment groups for rendering multiple Streaming Node Deployments | `[]` |
| `streamingNode.resources` | Resource requests/limits for the Milvus Streaming Node pods | `{}` |
| `streamingNode.nodeSelector` | Node labels for Milvus Streaming Node pods assignment | `{}` |
| `streamingNode.affinity` | Affinity settings for Milvus Streaming Node pods assignment | `{}` |
Expand Down Expand Up @@ -695,6 +700,7 @@ The following table lists the configurable parameters of the Woodpecker componen
- [Node Selector Configuration Guide](../../docs/node-selector-configuration-guide.md)
- [Affinity Configuration Guide](../../docs/affinity-configuration-guide.md)
- [Tolerations Configuration Guide](../../docs/tolerations-configuration-guide.md)
- [Deployment Groups and Resource Group Guide](../../docs/deployment-groups-resource-group-guide.md)

#### Important Configuration Considerations

Expand Down
92 changes: 76 additions & 16 deletions charts/milvus/templates/datanode-deployment.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,20 +1,38 @@
{{- if and .Values.dataNode.enabled .Values.cluster.enabled }}
{{/* Datanode deployment template. Accepts merged context with optional .group. */}}
{{- define "milvus.datanode.deployment" -}}
{{- $groupName := "" -}}
{{- if .group }}
{{- $groupName = required "dataNode.groups[].name is required" .group.name -}}
{{- end }}
apiVersion: apps/v1
kind: Deployment
metadata:
name: {{ template "milvus.datanode.fullname" . }}
name: {{ template "milvus.datanode.fullname" . }}{{- if $groupName }}-{{ $groupName }}{{- end }}
namespace: {{ .Release.Namespace }}
labels:
{{ include "milvus.labels" . | indent 4 }}
component: "datanode"
{{- with .group.labels }}
{{ toYaml . | indent 4 }}
{{- end }}
{{- if $groupName }}
milvus.io/deployment-group: {{ $groupName | quote }}
{{- end }}

{{ include "milvus.ud.labels" . | indent 4 }}
annotations:
{{ include "milvus.ud.annotations" . | indent 4 }}
{{- with .group.annotations }}
{{ toYaml . | indent 4 }}
{{- end }}

spec:
{{- if ge (int .Values.dataNode.replicas) 0 }}
replicas: {{ .Values.dataNode.replicas }}
{{- $replicas := .Values.dataNode.replicas -}}
{{- if and .group (hasKey .group "replicas") -}}
{{- $replicas = .group.replicas -}}
{{- end }}
{{- if ge (int $replicas) 0 }}
replicas: {{ $replicas }}
{{- end }}
{{- with .Values.dataNode.strategy }}
strategy:
Expand All @@ -24,11 +42,20 @@ spec:
matchLabels:
{{ include "milvus.matchLabels" . | indent 6 }}
component: "datanode"
{{- if $groupName }}
milvus.io/deployment-group: {{ $groupName | quote }}
{{- end }}
template:
metadata:
labels:
{{ include "milvus.matchLabels" . | indent 8 }}
component: "datanode"
{{- with .group.labels }}
{{ toYaml . | indent 8 }}
{{- end }}
{{- if $groupName }}
milvus.io/deployment-group: {{ $groupName | quote }}
{{- end }}
{{ include "milvus.ud.labels" . | indent 8 }}
annotations:
{{- if .Values.dataNode.profiling.enabled }}
Expand All @@ -38,6 +65,9 @@ spec:
{{- end }}
{{- if .Values.dataNode.annotations }}
{{- toYaml .Values.dataNode.annotations | nindent 8 }}
{{- end }}
{{- with .group.annotations }}
{{- toYaml . | nindent 8 }}
{{- end }}
checksum/config: {{ include (print $.Template.BasePath "/configmap.yaml") . | sha256sum }}
{{ include "milvus.ud.annotations" . | indent 8 }}
Expand Down Expand Up @@ -95,6 +125,9 @@ spec:
{{- if .Values.dataNode.extraEnv }}
{{- toYaml .Values.dataNode.extraEnv | nindent 8 }}
{{- end }}
{{- if .group.extraEnv }}
{{- toYaml .group.extraEnv | nindent 8 }}
{{- end }}
ports:
- name: datanode
containerPort: 21124
Expand Down Expand Up @@ -152,37 +185,53 @@ spec:
{{- toYaml .Values.volumeMounts | nindent 8 }}
{{- end}}

{{- if and (.Values.nodeSelector) (not .Values.dataNode.nodeSelector) }}
{{- $nodeSelector := .Values.dataNode.nodeSelector -}}
{{- if and .group .group.nodeSelector -}}
{{- $nodeSelector = .group.nodeSelector -}}
{{- end }}
{{- if and (.Values.nodeSelector) (not $nodeSelector) }}
nodeSelector:
{{ toYaml .Values.nodeSelector | indent 8 }}
{{- end }}
{{- if .Values.dataNode.nodeSelector }}
{{- if $nodeSelector }}
nodeSelector:
{{ toYaml .Values.dataNode.nodeSelector | indent 8 }}
{{ toYaml $nodeSelector | indent 8 }}
{{- end }}
{{- $affinity := .Values.dataNode.affinity -}}
{{- if and .group .group.affinity -}}
{{- $affinity = .group.affinity -}}
{{- end }}
{{- if and (.Values.affinity) (not .Values.dataNode.affinity) }}
{{- if and (.Values.affinity) (not $affinity) }}
affinity:
{{ toYaml .Values.affinity | indent 8 }}
{{- end }}
{{- if .Values.dataNode.affinity }}
{{- if $affinity }}
affinity:
{{ toYaml .Values.dataNode.affinity | indent 8 }}
{{ toYaml $affinity | indent 8 }}
{{- end }}
{{- if and (.Values.tolerations) (not .Values.dataNode.tolerations) }}
{{- $tolerations := .Values.dataNode.tolerations -}}
{{- if and .group .group.tolerations -}}
{{- $tolerations = .group.tolerations -}}
{{- end }}
{{- if and (.Values.tolerations) (not $tolerations) }}
tolerations:
{{ toYaml .Values.tolerations | indent 8 }}
{{- end }}
{{- if .Values.dataNode.tolerations }}
{{- if $tolerations }}
tolerations:
{{ toYaml .Values.dataNode.tolerations | indent 8 }}
{{ toYaml $tolerations | indent 8 }}
{{- end }}
{{- $topologySpreadConstraints := .Values.dataNode.topologySpreadConstraints -}}
{{- if and .group .group.topologySpreadConstraints -}}
{{- $topologySpreadConstraints = .group.topologySpreadConstraints -}}
{{- end }}
{{- if and (.Values.topologySpreadConstraints) (not .Values.dataNode.topologySpreadConstraints) }}
{{- if and (.Values.topologySpreadConstraints) (not $topologySpreadConstraints) }}
topologySpreadConstraints:
{{ toYaml .Values.topologySpreadConstraints | indent 8 }}
{{- end }}
{{- if .Values.dataNode.topologySpreadConstraints }}
{{- if $topologySpreadConstraints }}
topologySpreadConstraints:
{{ toYaml .Values.dataNode.topologySpreadConstraints | indent 8 }}
{{ toYaml $topologySpreadConstraints | indent 8 }}
{{- end }}
{{- if and (.Values.securityContext) (not .Values.dataNode.securityContext) }}
securityContext:
Expand Down Expand Up @@ -210,4 +259,15 @@ spec:
{{- if .Values.volumes }}
{{- toYaml .Values.volumes | nindent 6 }}
{{- end}}
{{- end -}}

{{- if and .Values.dataNode.enabled .Values.cluster.enabled }}
{{- if .Values.dataNode.groups }}
{{- range $group := .Values.dataNode.groups }}
---
{{ include "milvus.datanode.deployment" (merge (dict "group" $group) $) }}
{{- end }}
{{- else }}
{{ include "milvus.datanode.deployment" (merge (dict "group" (dict)) .) }}
{{- end }}
{{- end }}
26 changes: 22 additions & 4 deletions charts/milvus/templates/datanode-hpa.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,16 +1,23 @@
{{- if and .Values.dataNode.enabled .Values.dataNode.hpa.enabled }}
{{- define "milvus.datanode.hpa" -}}
{{- $groupName := "" -}}
{{- if .group }}
{{- $groupName = required "dataNode.groups[].name is required" .group.name -}}
{{- end }}
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: {{ template "milvus.datanode.fullname" . }}-hpa
name: {{ template "milvus.datanode.fullname" . }}{{- if $groupName }}-{{ $groupName }}{{- end }}-hpa
namespace: {{ .Release.Namespace }}
labels:
component: "datanode"
{{- if $groupName }}
milvus.io/deployment-group: {{ $groupName | quote }}
{{- end }}
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: {{ template "milvus.datanode.fullname" . }}
name: {{ template "milvus.datanode.fullname" . }}{{- if $groupName }}-{{ $groupName }}{{- end }}
minReplicas: {{ .Values.dataNode.hpa.minReplicas | default 1 }}
maxReplicas: {{ .Values.dataNode.hpa.maxReplicas | default 10 }}
metrics:
Expand All @@ -20,4 +27,15 @@ spec:
target:
type: Utilization
averageUtilization: {{ .Values.dataNode.hpa.cpuUtilization | default 40 }}
{{- end }}
{{- end -}}

{{- if and .Values.dataNode.enabled .Values.dataNode.hpa.enabled }}
{{- if .Values.dataNode.groups }}
{{- range $group := .Values.dataNode.groups }}
---
{{ include "milvus.datanode.hpa" (merge (dict "group" $group) $) }}
{{- end }}
{{- else }}
{{ include "milvus.datanode.hpa" (merge (dict "group" (dict)) .) }}
{{- end }}
{{- end }}
6 changes: 6 additions & 0 deletions charts/milvus/templates/mixcoord-deployment.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,9 @@ metadata:
labels:
{{ include "milvus.labels" . | indent 4 }}
component: "mixcoord"
{{- with .Values.mixCoordinator.labels }}
{{ toYaml . | indent 4 }}
{{- end }}
{{ include "milvus.ud.labels" . | indent 4 }}
annotations:
{{ include "milvus.ud.annotations" . | indent 4 }}
Expand Down Expand Up @@ -34,6 +37,9 @@ spec:
{{ include "milvus.matchLabels" . | indent 8 }}
{{ include "milvus.ud.labels" . | indent 8 }}
component: "mixcoord"
{{- with .Values.mixCoordinator.labels }}
{{ toYaml . | indent 8 }}
{{- end }}
annotations:
{{- if .Values.mixCoordinator.profiling.enabled }}
pyroscope.io/scrape: "true"
Expand Down
Loading
Loading