Skip to content

fix: [poc_2.6.17] stop CDC acting on stale topology changes; answer topology queries on unconfigured clusters - #52729

Merged
czs007 merged 4 commits into
milvus-io:poc_2.6.17from
czs007:cp-cdc-fixes-poc2617
Aug 20, 2026
Merged

czs007 merged 4 commits into
milvus-io:poc_2.6.17from
czs007:cp-cdc-fixes-poc2617

Conversation

@czs007

@czs007 czs007 commented Aug 20, 2026

Copy link
Copy Markdown
Collaborator

Cherry-picks onto poc_2.6.17 for a customer running this branch, so a single image carries
everything they need. Two fixes are open upstream against master (#52728, #52687); the other
two are already on 2.6 and are brought over because they bear on the same area.

#52728 — CDC stops itself on a topology change that predates the task

A replicator resumes from the checkpoint the target cluster reports, which after
milvus-backup restore secondary is the position the backup was taken at. Replaying from
there walks over every topology change made since — including ones that removed this edge
before it was re-created — and those were acted on as if they were current:

  • the replicator deleted its own replicating-pchannel/ key and stopped 0.2 s after
    starting;
  • the same message, forwarded to the secondary, made it a standalone primary again, dropping
    its replicate checkpoint.

Nothing recovered from it: the topology query still reported the edge (only the per-pchannel
task keys were gone), and re-applying the same configuration was short-circuited as unchanged
and returned success.

ReplicatePChannelMeta.initialized_checkpoint already carries the time tick of the
AlterReplicateConfig that created the task, so a message at or before it predates the task
and cannot be an instruction for it. It is now disregarded both when deciding to remove the
task and when deciding to forward the message. A task with no initialized time tick keeps the
previous behaviour.

#52687 — topology query on a cluster with no replication blocks

GetReplicateConfiguration never returned on a cluster that had never been given a replicate
configuration — the state of every freshly installed cluster. Such a cluster reports a nil
configuration, and the watcher's wait condition treated the resulting nil helper as "no
assignment received yet", so the call ran to the caller's deadline and surfaced as
DEADLINE_EXCEEDED, indistinguishable from an unhealthy cluster.

Wait only until an assignment has been received and return the possibly-nil helper; make the
ConfigHelper read accessors nil-receiver safe; reject replicate messages explicitly on a
cluster with no configuration, which previously could not be reached because the lookup
blocked first.

Tests

Both upstream PRs' unit tests are included. internal/cdc/util had no test file before:

  • ordering check in both directions, including the no-initialized-time-tick case that keeps
    the old behaviour;
  • a current detach still removes the replication, a detach that predates the task does not;
  • the consume loop neither forwards a stale topology change nor ends on it.

Also picked from 2.6

  • fix: [cp 2.6] recover pchannels from collection metadata #51144 — recover pchannels from collection metadata (51749e6f8d). streamingcoord
    recovery adds WAL topics missing after stats initialization by reading them back from
    RootCoord collection metadata. Directly relevant here: the customer reinstalled the
    secondary, and recovery on a rebuilt cluster is exactly this path.
  • fix: [cp2.6]correct the CDC replication lag metric #51365 — correct the CDC replication lag metric (e6b5d9308d). The replication lag
    series was wrong, so the one signal an operator would watch to notice replication had
    stopped could not be trusted. Given the failure this branch fixes is otherwise silent, the
    metric being correct matters.

Both apply cleanly; the only conflict was two tests appended to the same file, resolved by
keeping both.

#50849 — ignore stale replicated txn body was checked and is already present on
poc_2.6.17. #51454 — skip unreplicable replicated ddl is on master only and depends on
message-property changes that are not on 2.6; it is left for a regular cherry-pick to 2.6
rather than being adapted here.

@sre-ci-robot sre-ci-robot added the size/L Denotes a PR that changes 100-499 lines. label Aug 20, 2026
@sre-ci-robot

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
To complete the pull request process, please ask for approval from czs007 after the PR has been reviewed.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@sre-ci-robot sre-ci-robot added size/XL Denotes a PR that changes 500-999 lines. and removed size/L Denotes a PR that changes 100-499 lines. labels Aug 20, 2026
@czs007
czs007 force-pushed the cp-cdc-fixes-poc2617 branch 2 times, most recently from 6389e1c to c71ab0c Compare August 20, 2026 18:26
czs007 and others added 4 commits August 20, 2026 13:00
…er topology queries on unconfigured clusters

Cherry-pick of two upstream fixes onto poc_2.6.17.

milvus-io#52728 — a replicator resumes from the checkpoint the target
reports, which after `restore secondary` is the position the backup was taken
at. Replaying from there walks over topology changes that removed this edge
before it was re-created, and those were acted on as if current: the replicator
deleted its own replicate pchannel metadata and stopped moments after starting,
and the same messages, forwarded to the secondary, turned it back into a
standalone primary. The task's initialized checkpoint carries the time tick of
the AlterReplicateConfig that created it, so anything at or before it predates
the task and is now disregarded, both when deciding to remove the task and when
deciding to forward the message.

milvus-io#52687 — GetReplicateConfiguration never returned on a cluster
that had never been given a replicate configuration, blocking until the caller's
deadline. Such a cluster reports a nil configuration, and the watcher treated
the resulting nil helper as "no assignment received yet". Wait only until an
assignment arrives, return the possibly-nil helper, make the ConfigHelper read
accessors nil-receiver safe, and reject replicate messages explicitly on a
cluster with no configuration.

Includes the unit tests from both upstream PRs; internal/cdc/util had no test
file before.

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
…1144)

issue: milvus-io#50905
pr: milvus-io#50919

- streamingcoord recovery: include pchannels recovered from RootCoord
collection metadata through ConfigChannelProvider so missing WAL topics
are added after stats initialization
- dml channel compatibility: expand canonical DML-style pchannels up to
the recovered index while preserving literal non-config and
non-canonical pchannel names
- channel stats: expose active pchannels from PChannelStatsManager for
metadata-driven channel recovery

---------

Signed-off-by: chyezh <chyezh@outlook.com>
(cherry picked from commit 51749e6)
Cherry-pick from master (PR still open)

pr: milvus-io#51049
issue: milvus-io#50633
issue: milvus-io#51048

Cherry-picked from master PR milvus-io#51049 (open - preemptive backport).

Backports the CDC lag metric lifecycle fix:
- Seed the lag gauge from InitializedCheckpoint before target access.
- Overwrite the seed with the target-confirmed checkpoint.
- Ignore nil and zero checkpoints.
- Delete the lag series only on genuine ReplicateManager removal.
- Preserve the live lag series during outdated revision cleanup.

- Kept the 2.6 pkg/v2, go-api/v2, and log plus zap APIs.
- Omitted the master-only documentation change as requested for 2.6.

**Note**: Original PR milvus-io#51049 is still open. This CP PR should be updated
if the original PR changes.

- [x] Code and test file counts and line counts match the master PR
after excluding documentation
- [x] Per-file changed lines match after v3-to-v2 import normalization
- [x] No conflict markers
- [x] DCO sign-off
- [ ] make static-check (skipped by cherry-pick workflow)

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
(cherry picked from commit e6b5d93)
…configuration

The watcher test still asserted the behaviour this branch changes: that a
received-but-nil configuration keeps the caller waiting until its deadline.
Replace it with the two cases that now hold — a nil configuration is returned
promptly, and the call still blocks when no assignment has arrived at all.

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
@czs007
czs007 force-pushed the cp-cdc-fixes-poc2617 branch from c71ab0c to 618a3a6 Compare August 20, 2026 20:01
@czs007
czs007 merged commit 7e7c527 into milvus-io:poc_2.6.17 Aug 20, 2026
3 of 4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size/XL Denotes a PR that changes 500-999 lines.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants