Feeding Milvus at Enterprise Scale: Introducing the OpenCrawling Output Connector (Seeking Feedback!)

[OpenPj]

Hi Milvus Community! 👋

I’m Piergiorgio, a Solution Architect and open source maintainer (ASF Member and PMC Chair of Apache ManifoldCF), and I’m excited to share a new tool that we hope will drive a massive amount of enterprise data directly into Milvus.

As organizations scale their RAG architectures, Python scripts and lightweight document loaders often hit a wall. Enterprises need to ingest data from complex sources—like ECM systems (Alfresco/CMIS), BPMN 2.0 Workflows, and data lakehouses (Apache Iceberg)—while strictly preserving Zero-Trust security (ACLs).

To solve this, we recently launched OpenCrawling, an open source cloud-native data federation engine built from scratch on Java 25, Spring Boot 4, and virtual threads to guarantee massive, non-blocking ingestion throughput.

Today, we are thrilled to announce our initial implementation of the Milvus Output Connector.

🎰 How it works:
OpenCrawling extracts documents, narrativizes workflows, and chunks data, bundling it all into the Open Ingestion Standard (OIS) payload. The Milvus Output Connector then takes these high volume streams of vector arrays, text payloads, and critical Zero-Trust ACL metadata, and natively pushes them into Milvus collections.

❓ Why we are posting here:
We want to make sure OpenCrawling is the absolute best and fastest way to pipe enterprise data into Milvus. Since this is our initial implementation using the Milvus Java SDK, we are looking for Milvus experts, developers, and power users to help us battle-test and optimize it.

❤️ We would love your contributions and feedback specifically on:

• High-Throughput Insertion: Are we utilizing the Milvus Java SDK and batch insertion APIs optimally alongside our Virtual Thread architecture to maximize write speeds?
• Metadata & ACL Filtering: We rely heavily on Milvus's scalar filtering capabilities to enforce Document-Level Security (ACL arrays) during RAG queries. We'd love advice on the most efficient schema designs for these metadata fields.
• Collection & Partition Strategy: Best practices for dynamically managing collections or partitions directly from the connector based on the incoming source systems.

🤝 Get Involved:
If you are passionate about vector db performance or Enterprise AI architectures, we’d love to have you test it out or review our implementation. Please consider that it's just a starting point for us 😶‍🌫️

Connector Code: https://github.com/opencrawling/opencrawling/tree/main/oc-milvus-output-connector

Architecture & Project: https://opencrawling.org/

Thanks for building such an incredible vector database. We look forward to collaborating to make the Milvus ecosystem even stronger!

Best regards,
Piergiorgio (PG) Lucidi
Founder & Lead Architect, OpenCrawling

[yhmo]

Nice job. I just browsed the Milvus-related source code of OpenCrawling. It looks good to me, only some small suggestions:

• Some field names are hardcode such as "lastModified"/"security_denied_read"/"security_allowed_read", it is better to define them as common strings
• For scarlar fields(Varchar/Int/Double/Float/Bool, etc), it is better use row.addProperty("text", message.text()) instead of row.add("text", gson.toJsonTree(message.text())) since the previous is clean and doesn't require a gson object, only the Array/Struct/Vector fields need gson.toJsonTree(): https://github.com/opencrawling/opencrawling/blob/8ac51eac100be9c9632f9110d65ef4921366d533/oc-milvus-output-connector/src/main/java/org/opencrawling/milvus/messaging/MilvusStoreWriterConsumer.java#L105

[OpenPj]

Hi @yhmo,
Thank you so much for your feedback, very appreciated and this is priceless for us.
I'm going to work on this and please consider that if you want to contribute, feel free to fork and create pull requests 😄

[OpenPj]

Everything is now updated including your suggestions. Thank you again @yhmo for your feedback!