[Bug]: DataNode panic and QueryNode SIGSEGV when processing Geometry field with RTREE index

[zhuwenxing]

Is there an existing issue for this?

• I have searched the existing issues

Environment

• Milvus version: master (commit: d221fce)
• Deployment mode: cluster
• MQ type: Pulsar v3
• SDK version: N/A (chaos test)
• OS: Linux (K8s, Rocky Linux nodes)
• CPU/Memory: Default chaos-testing resource limits

K8s Pod List

etcd-followers-pod-failure-24174-0                                1/1     Running             0                46m
etcd-followers-pod-failure-24174-1                                1/1     Running             2 (22m ago)      46m
etcd-followers-pod-failure-24174-2                                1/1     Running             0                46m
etcd-followers-pod-failure-24174-milvus-datanode-6b8dc8c49rtvsq   0/1     CrashLoopBackOff    11 (3m30s ago)   46m
etcd-followers-pod-failure-24174-milvus-datanode-6b8dc8c49wzndg   0/1     CrashLoopBackOff    11 (116s ago)    46m
etcd-followers-pod-failure-24174-milvus-mixcoord-577684777kdrfg   1/1     Running             2 (45m ago)      46m
etcd-followers-pod-failure-24174-milvus-proxy-54d694764c-xkx95    1/1     Running             2 (45m ago)      46m
etcd-followers-pod-failure-24174-milvus-querynode-d8ddb94528lcn   0/1     CrashLoopBackOff    10 (2m34s ago)   46m
etcd-followers-pod-failure-24174-milvus-querynode-d8ddb945b5ck7   0/1     CrashLoopBackOff    10 (3m13s ago)   46m
etcd-followers-pod-failure-24174-milvus-querynode-d8ddb945wzcpt   0/1     CrashLoopBackOff    10 (2m15s ago)   46m
etcd-followers-pod-failure-24174-milvus-streamingnode-6489f5dgv   1/1     Running             4 (27m ago)      46m
etcd-followers-pod-failure-24174-milvus-streamingnode-6489jk8rd   1/1     Running             2 (45m ago)      46m

Current Behavior

All DataNode (2 pods) and QueryNode (3 pods) enter CrashLoopBackOff. The crashes involve Geometry field data processing with RTREE index.

DataNode crashes with Go panic during index building:

panic: runtime error: index out of range [420] with length 5 [recovered]
panic: runtime error: index out of range [420] with length 5 [recovered]
panic: runtime error: index out of range [420] with length 5 [recovered]
panic: runtime error: index out of range [420] with length 5 [recovered]
panic: runtime error: index out of range [420] with length 5

Stack trace:

goroutine 662 [running]:
runtime.goPanicIndex(0x1a4, 0x5)
  /workspace/source/internal/datanode/index/task_index.go:326 +0x1ec5

Call chain:
  indexBuildLoop -> processTask -> indexBuildTask.Execute
  -> indexcgowrapper.CreateIndex -> _Cfunc_CreateIndex

QueryNode crashes with SIGSEGV during sealed segment loading:

SIGSEGV: segmentation violation
signal arrived during cgo execution

Stack trace:

Call chain:
  LoadSegments (gRPC handler)
  -> segmentLoader.Load -> ProcessFuncParallel
  -> loadSealedSegment (segment_loader.go:953)
  -> LoadSegment (segment_loader.go:1018)
  -> segment.Load(ctx) -> C.SegmentLoad

Both crashes are persistent — every restart triggers the same crash because the tasks/segments are replayed from metadata.

Expected Behavior

DataNode should build RTREE index for Geometry fields without panic, and QueryNode should load sealed segments containing Geometry data without SIGSEGV.

Steps To Reproduce

This was observed during automated chaos testing. Exact minimal reproduction steps are unclear.

Observed context:

• A collection with a Geometry field (data_type=Geometry, nullable=true) and FloatVector fields was created
• RTREE index was configured on the Geometry field, HNSW index on the vector field
• After data insertion (~90k rows), DataNode crashed when building RTREE index, and QueryNode crashed when loading sealed segments
• The crashes are persistent across pod restarts

Milvus Log

DataNode index build context (just before crash):

[INFO] [index/task_index.go:319] ["create index"] [buildID=464994407064120772]
  [collection=464994406713799211] [segmentID=464994406925095548]
  [currentIndexVersion=8]
  field_schema:{fieldID:106 name:"geometry_field" data_type:Geometry nullable:true}
  index_params:{key:"index_type" value:"RTREE"}
  num_rows:90016
  storage_version:3
  segment_insert_files: 11 field files (fieldIDs: 0,1,110,112,113,114,119,121,122,124,126)

QueryNode SIGSEGV timeline (all 3 pods, persistent across restarts):

2026-03-18T06:11:57Z  querynode-b5ck7  SIGSEGV: segmentation violation
2026-03-18T06:12:02Z  querynode-b5ck7  SIGSEGV: segmentation violation
2026-03-18T06:30:18Z  querynode-b5ck7  SIGSEGV: segmentation violation
2026-03-18T06:31:05Z  querynode-wzcpt  SIGSEGV: segmentation violation
2026-03-18T06:37:01Z  querynode-28lcn  SIGSEGV: segmentation violation (x8)
2026-03-18T06:37:44Z  querynode-wzcpt  SIGSEGV: segmentation violation

Anything else?

• Other components (mixcoord, proxy, streamingnode) are running normally
• Earlier RTREE index builds on smaller segments (9 fields, ~30-37k rows) succeeded at 05:59 UTC
• The crash occurs specifically on segments with 11 fields (additional fields 124, 126) and ~90k rows
• This was observed during chaos testing (etcd follower pod failure), but the etcd failure is unrelated to the crashes

[zhuwenxing]

/assign @xiaocai2333

[austingrant651-create]

Assign me

…

On Thu, Mar 19, 2026, 3:45 AM zhuwenxing ***@***.***> wrote: *zhuwenxing* left a comment (milvus-io/milvus#48375) <#48375 (comment)> /assign @xiaocai2333 <https://github.com/xiaocai2333> — Reply to this email directly, view it on GitHub <#48375 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/B74AFF6BCQS3H5QCBV4GJXD4ROXRHAVCNFSM6AAAAACWXUYJ2OVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHM2DAOBYGY2DENZTGM> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>