fix: [poc_2.6.17] stop CDC acting on stale topology changes; answer topology queries on unconfigured clusters - #52729
Merged
Conversation
Contributor
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
czs007
force-pushed
the
cp-cdc-fixes-poc2617
branch
2 times, most recently
from
August 20, 2026 18:26
6389e1c to
c71ab0c
Compare
…er topology queries on unconfigured clusters Cherry-pick of two upstream fixes onto poc_2.6.17. milvus-io#52728 — a replicator resumes from the checkpoint the target reports, which after `restore secondary` is the position the backup was taken at. Replaying from there walks over topology changes that removed this edge before it was re-created, and those were acted on as if current: the replicator deleted its own replicate pchannel metadata and stopped moments after starting, and the same messages, forwarded to the secondary, turned it back into a standalone primary. The task's initialized checkpoint carries the time tick of the AlterReplicateConfig that created it, so anything at or before it predates the task and is now disregarded, both when deciding to remove the task and when deciding to forward the message. milvus-io#52687 — GetReplicateConfiguration never returned on a cluster that had never been given a replicate configuration, blocking until the caller's deadline. Such a cluster reports a nil configuration, and the watcher treated the resulting nil helper as "no assignment received yet". Wait only until an assignment arrives, return the possibly-nil helper, make the ConfigHelper read accessors nil-receiver safe, and reject replicate messages explicitly on a cluster with no configuration. Includes the unit tests from both upstream PRs; internal/cdc/util had no test file before. Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
…1144) issue: milvus-io#50905 pr: milvus-io#50919 - streamingcoord recovery: include pchannels recovered from RootCoord collection metadata through ConfigChannelProvider so missing WAL topics are added after stats initialization - dml channel compatibility: expand canonical DML-style pchannels up to the recovered index while preserving literal non-config and non-canonical pchannel names - channel stats: expose active pchannels from PChannelStatsManager for metadata-driven channel recovery --------- Signed-off-by: chyezh <chyezh@outlook.com> (cherry picked from commit 51749e6)
Cherry-pick from master (PR still open) pr: milvus-io#51049 issue: milvus-io#50633 issue: milvus-io#51048 Cherry-picked from master PR milvus-io#51049 (open - preemptive backport). Backports the CDC lag metric lifecycle fix: - Seed the lag gauge from InitializedCheckpoint before target access. - Overwrite the seed with the target-confirmed checkpoint. - Ignore nil and zero checkpoints. - Delete the lag series only on genuine ReplicateManager removal. - Preserve the live lag series during outdated revision cleanup. - Kept the 2.6 pkg/v2, go-api/v2, and log plus zap APIs. - Omitted the master-only documentation change as requested for 2.6. **Note**: Original PR milvus-io#51049 is still open. This CP PR should be updated if the original PR changes. - [x] Code and test file counts and line counts match the master PR after excluding documentation - [x] Per-file changed lines match after v3-to-v2 import normalization - [x] No conflict markers - [x] DCO sign-off - [ ] make static-check (skipped by cherry-pick workflow) Signed-off-by: bigsheeper <yihao.dai@zilliz.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com> (cherry picked from commit e6b5d93)
…configuration The watcher test still asserted the behaviour this branch changes: that a received-but-nil configuration keeps the caller waiting until its deadline. Replace it with the two cases that now hold — a nil configuration is returned promptly, and the call still blocks when no assignment has arrived at all. Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
czs007
force-pushed
the
cp-cdc-fixes-poc2617
branch
from
August 20, 2026 20:01
c71ab0c to
618a3a6
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Cherry-picks onto
poc_2.6.17for a customer running this branch, so a single image carrieseverything they need. Two fixes are open upstream against master (#52728, #52687); the other
two are already on 2.6 and are brought over because they bear on the same area.
#52728 — CDC stops itself on a topology change that predates the task
A replicator resumes from the checkpoint the target cluster reports, which after
milvus-backup restore secondaryis the position the backup was taken at. Replaying fromthere walks over every topology change made since — including ones that removed this edge
before it was re-created — and those were acted on as if they were current:
replicating-pchannel/key and stopped 0.2 s afterstarting;
its replicate checkpoint.
Nothing recovered from it: the topology query still reported the edge (only the per-pchannel
task keys were gone), and re-applying the same configuration was short-circuited as unchanged
and returned success.
ReplicatePChannelMeta.initialized_checkpointalready carries the time tick of theAlterReplicateConfigthat created the task, so a message at or before it predates the taskand cannot be an instruction for it. It is now disregarded both when deciding to remove the
task and when deciding to forward the message. A task with no initialized time tick keeps the
previous behaviour.
#52687 — topology query on a cluster with no replication blocks
GetReplicateConfigurationnever returned on a cluster that had never been given a replicateconfiguration — the state of every freshly installed cluster. Such a cluster reports a nil
configuration, and the watcher's wait condition treated the resulting nil helper as "no
assignment received yet", so the call ran to the caller's deadline and surfaced as
DEADLINE_EXCEEDED, indistinguishable from an unhealthy cluster.Wait only until an assignment has been received and return the possibly-nil helper; make the
ConfigHelperread accessors nil-receiver safe; reject replicate messages explicitly on acluster with no configuration, which previously could not be reached because the lookup
blocked first.
Tests
Both upstream PRs' unit tests are included.
internal/cdc/utilhad no test file before:the old behaviour;
Also picked from 2.6
51749e6f8d). streamingcoordrecovery adds WAL topics missing after stats initialization by reading them back from
RootCoord collection metadata. Directly relevant here: the customer reinstalled the
secondary, and recovery on a rebuilt cluster is exactly this path.
e6b5d9308d). The replication lagseries was wrong, so the one signal an operator would watch to notice replication had
stopped could not be trusted. Given the failure this branch fixes is otherwise silent, the
metric being correct matters.
Both apply cleanly; the only conflict was two tests appended to the same file, resolved by
keeping both.
#50849 — ignore stale replicated txn bodywas checked and is already present onpoc_2.6.17.#51454 — skip unreplicable replicated ddlis on master only and depends onmessage-property changes that are not on 2.6; it is left for a regular cherry-pick to 2.6
rather than being adapted here.