Is there an existing issue for this?
Describe the bug
The synchronous flush APIs do not enforce timeout as one end-to-end budget.
MilvusClient.flush(collection_name, timeout=N) delegates to GrpcHandler.flush. The initial Flush RPC receives the full timeout. After that RPC returns, _wait_for_flushed starts a new timer and receives the same full timeout again. The low-level multi-collection path starts another full wait timer for each collection. GrpcHandler.flush_all has the same split-budget behavior between the initial FlushAll RPC and _wait_for_flush_all.
The polling sleeps can add further overshoot because timeout is checked before a fixed sleep of 0.5 seconds for flush, or 5 seconds for flush_all, instead of capping the sleep to the remaining budget.
On current master, a deterministic mocked test simulated 8 seconds spent in the initial Flush RPC and called flush(..., timeout=10). The method raised its timeout exception only after 18.5 seconds of simulated wall-clock time.
The async flush path is not affected in the same way because its outer retry wrapper applies asyncio.wait_for to the whole coroutine using the remaining timeout.
Expected Behavior
The timeout passed to a public synchronous flush API should bound the complete operation:
- compute one deadline when the API call starts;
- pass only the remaining budget to the initial RPC and every state-poll RPC;
- share the same deadline across all collection waits;
- cap polling sleeps to the remaining budget;
- apply the same rule to
flush_all.
Steps/Code To Reproduce behavior
- Mock
Flush.future().result() to advance a fake clock by 8 seconds and return a successful flush response.
- Mock
GetFlushState to keep returning flushed=False.
- Mock the polling sleep to advance the same fake clock.
- Call
GrpcHandler.flush(["coll"], timeout=10).
- Observe that the timeout is raised at 18.5 seconds rather than within the requested 10-second budget.
Environment details
- Hardware/Software conditions: macOS arm64
- Method of installation: source checkout
- Milvus version: not applicable; reproduced with mocked RPC responses
- PyMilvus version: current master at
c5e73a6e9deb
- Milvus configuration: not applicable
- Python version: 3.13.11
Anything else?
Discovered while reviewing #747. This timeout-budget bug is separate from that issue's request for periodic client-side progress logs.
Is there an existing issue for this?
Describe the bug
The synchronous flush APIs do not enforce
timeoutas one end-to-end budget.MilvusClient.flush(collection_name, timeout=N)delegates toGrpcHandler.flush. The initialFlushRPC receives the full timeout. After that RPC returns,_wait_for_flushedstarts a new timer and receives the same full timeout again. The low-level multi-collection path starts another full wait timer for each collection.GrpcHandler.flush_allhas the same split-budget behavior between the initialFlushAllRPC and_wait_for_flush_all.The polling sleeps can add further overshoot because timeout is checked before a fixed sleep of 0.5 seconds for
flush, or 5 seconds forflush_all, instead of capping the sleep to the remaining budget.On current master, a deterministic mocked test simulated 8 seconds spent in the initial
FlushRPC and calledflush(..., timeout=10). The method raised its timeout exception only after 18.5 seconds of simulated wall-clock time.The async
flushpath is not affected in the same way because its outer retry wrapper appliesasyncio.wait_forto the whole coroutine using the remaining timeout.Expected Behavior
The timeout passed to a public synchronous flush API should bound the complete operation:
flush_all.Steps/Code To Reproduce behavior
Flush.future().result()to advance a fake clock by 8 seconds and return a successful flush response.GetFlushStateto keep returningflushed=False.GrpcHandler.flush(["coll"], timeout=10).Environment details
c5e73a6e9debAnything else?
Discovered while reviewing #747. This timeout-budget bug is separate from that issue's request for periodic client-side progress logs.