RFC: ts-ct-migration: STM coexistence [PR 2] by sjust-redpanda · Pull Request #30887 · redpanda-data/redpanda

sjust-redpanda · 2026-06-23T20:33:02Z

Summary

In order to dynamically convert a TS partition to CT/TSv2, we need to be able to:

Dynamically switch over to ctp_stm on a partition which originally was TS
During the migration process, continue to use the archival_metadata_stm and log_eviction_stm, though the topic config is CT/TSv2.

The simplest way to achieve the above is for all TS and CT/TSv2 partitions to already have all three. This PR adjusts is_applicable_for on these stms to make the above happen and adjusts as needed trimming behavior to prevent empty stms from causing problems and to ensure that the archival_metadata_stm's constraints still apply during migration.

The PR also includes a minor ctp_stm change to allow specifying a reconciled log offset directly.

What's in this PR

cloud_topics/stm: don't pin truncation on an idle ctp_stm —
get_max_collectible_offset() returns offset::max() while no epoch has been
applied (_max_applied_epoch empty), so a ctp_stm that has seen no cloud data
never pins local-log truncation. Once advance_epoch() is called it falls back
to offset::min() to protect unreconciled data until the LRO advances.
cloud_topics/stm: pre-install an idle ctp_stm on tiered-storage
partitions — so a tiered→cloud migration needs no runtime STM install (STM
membership is fixed at construction). Also makes apply_raft_snapshot treat an
empty buffer as a no-op (the recovery fast-forward hands each STM an empty
snapshot to advance over).
cloud_topics/stm: explicit-log-offset advance_reconciled_offset
overload — lets a caller supply the reconciled log offset directly instead of
deriving it from the raft offset translator; the existing kafka-offset-only
entry point now delegates.
archival: create the archival STM on cloud-topic partitions — drop the
cloud_topic_enabled() == false guard so a migrated partition keeps its
archival STM after the storage-mode flip; an always-cloud topic carries an
inert, empty STM.
archival: clamp local-log trim while the manifest holds archived data — a
cloud-mode partition reports is_archival_enabled() == false, which would let
max_removable_local_log_offset collect-all and evict not-yet-uploaded tiered
data; keep constraining via cloud_recoverable_offset() while
holds_archived_data() (live manifest non-empty or spillover archive
present).
cluster: install the eviction STM on cloud-topic partitions — so a
migrating partition keeps supporting prefix truncation (DeleteRecords) and
retention while migrating; an inert passenger on a native cloud topic.

Testing

ctp_stm_state_test: an idle get_max_collectible_offset() returns max().
ctp_stm_test: applying an empty raft snapshot is a no-op (neither throws nor
resets applied state).
archival_metadata_stm_test: the local-trim clamp keeps constraining while the
manifest is non-empty.
full end-to-end coverage rides with the later migration-enable PR.

🤖 Generated with Claude Code

Design Doc

PR 0 · PR 1 · PR 2 · PR 3 · PR 4 · PR 5 · PR 6 · PR 7 · PR 8

sjust-redpanda · 2026-06-23T20:43:10Z

This seemed like the simpler approach. The other option would be to add infrastructure to dynamically add or remove stms. I have a branch for that I can resurrect if it there's interest.

Lazin · 2026-06-24T13:49:55Z

+    // no runtime STM install). Without this a passenger would pin the local log
+    // at offset::min() forever.
+    if (!_max_applied_epoch.has_value()) {
+        return model::offset::max();


This is not the only problem that ctp_stm creates. The ctp_stm also implements its own log eviction (same as log_eviction_stm).

Yeah, a subsequent PR goes over behaviors like that in both the TS and CT to prevent them from operating at the same time.

Lazin · 2026-06-24T13:58:23Z

+    // still holds archived data (a non-empty live manifest or a spillover
+    // archive) it is still served from tiered storage, so keep constraining via
+    // cloud_recoverable_offset() until the manifest is cleared.
+    collect_all = collect_all && !holds_archived_data();


is_archival_enabled() could be set to false by the user

Ah, that's a complication.

Lazin · 2026-06-24T14:02:03Z

+    // partition, and on restart/recovery a still-migrating partition is
+    // constructed cloud-mode -- in both cases the STM must be present. On a
+    // native cloud topic it is an inert passenger: retention and DeleteRecords
+    // go through the L1 path, so no local eviction requests are generated.


This is not correct, on a native cloud topic or TSv2 partition ctp_stm serves the same role as log_eviction_stm. It truncates the local log.

There is another big difference between the TSv1 and TSv2. The disk_log_impl instance is started with the background housekeeping loop disabled. TSv1 and local only partitions need this loop to funciton but ctp_stm does not rely on it.

The comment means that log_eviction_stm specifically won't do any local eviction.

Lazin

I think the approach with multiple STMs running all the time is not simpler compared to handoff (STM A stops, STM B starts). We will have to guarantee that both STMs are not doing something that they can't do together.

sjust-redpanda · 2026-06-24T15:08:55Z

I do have a branch with an approach to removing/adding stms, and I'd actually prefer that if there's an appetite for the additional mechanics. The problem of TS and CT background operations occurring potentially at the same time is still present, however. For the most part, those behaviors depend on the config rather than the presence or absence of an stm, and will still require adjustments to deal with mid-migration situation.

In order to dynamically switch a partition from TS to CT/TSv2, we need to be able to enable the ctp_stm on the fly. The simplest way to do that is to simply attach an empty ctp_stm to every partition. Unfortunately, with LRO unset, ctp_stm previously returned offset::min() for get_max_collectible_offset() preventing log trimming. This commit modifies that behavior to return offset::max() if _max_applied_epoch is empty making the presence of a ctp_stm harmless until advance_epoch() is called. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…tions is_applicable_for: pre-install the (idle) ctp_stm on tiered-storage partitions when cloud topics are available, so a tiered->cloud migration needs no runtime STM install -- the STM is already present and inert until the partition becomes a cloud topic. STM membership is fixed at construction, so it has to be there beforehand. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…verload Split advance_reconciled_offset so the reconciled log offset can be supplied explicitly instead of always deriving it from the raft offset translator. The existing kafka-offset-only entry point now delegates, deriving the log offset as before. This lets a caller seed the reconciliation baseline of a TS-migrated partition directly to the migration boundary -- passing the precise raft offset of the last uploaded tiered-storage segment -- so the already reconciled region can be trimmed without re-deriving the offset. ctp_stm carries no migration state of its own; this is expressed purely as "reconciliation starts at offset N". Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

During migration, we need the newly configured TSv2 or CT partition to continue to behave as an archival stm. Thus, we need the archival stm to continue to exist after the config switch. The simplest way to do this is to allow it to exist generally on CT/TSv2 topics. Drop the cloud_topic_enabled() == false guard from archival_metadata_stm_factory::is_applicable_for, so every cloud-topic partition carries an archival STM. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

A cloud-mode partition whose storage mode has been flipped reports is_archival_enabled() == false, which would make max_removable_local_log_offset collect_all and stop constraining local-log truncation -- evicting tiered-storage data not yet uploaded. Keep constraining via cloud_recoverable_offset() while the partition still holds archived data (a non-empty live manifest or a spillover archive), introduced here as holds_archived_data(); once the manifest is emptied, trimming resumes. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Drop the cloud_topic_enabled() guard from log_eviction_stm_factory so the eviction STM is installed on cloud-topic partitions too. A partition migrating from tiered storage is still served from its local log + archival manifest and must keep supporting prefix truncation (DeleteRecords) while migrating; the trigger flips storage_mode to cloud without reconstructing the partition, and a still-migrating partition is reconstructed cloud-mode on restart/recovery, so the STM must already be present. On a native cloud topic it is an inert passenger. Subsequent integration tests will provide coverage. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

github-actions Bot added area/build area/redpanda labels Jun 23, 2026

sjust-redpanda changed the title ~~ts-ct-migration: STM coexistence [PR 2]~~ RFC: ts-ct-migration: STM coexistence [PR 2] Jun 23, 2026

sjust-redpanda mentioned this pull request Jun 23, 2026

RFC ts-ct-migration: metastore and L1 reader [PR 1] #30800

Draft

1 task

sjust-redpanda requested review from Lazin, WillemKauf, andrwng and oleiman June 23, 2026 20:52

Lazin reviewed Jun 24, 2026

View reviewed changes

sjust-redpanda force-pushed the sjust/ts-ct-migration-pr2 branch from ef59066 to 58fdc72 Compare June 29, 2026 17:49

sjust-redpanda mentioned this pull request Jun 30, 2026

RFC ts-ct-migration: partition_mode substrate [PR 0] #30977

Draft

sjust-redpanda and others added 6 commits July 2, 2026 09:58

sjust-redpanda force-pushed the sjust/ts-ct-migration-pr2 branch from 58fdc72 to dfd6695 Compare July 2, 2026 20:57

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

RFC: ts-ct-migration: STM coexistence [PR 2]#30887

RFC: ts-ct-migration: STM coexistence [PR 2]#30887
sjust-redpanda wants to merge 6 commits into
redpanda-data:devfrom
sjust-redpanda:sjust/ts-ct-migration-pr2

sjust-redpanda commented Jun 23, 2026 •

edited

Loading

Uh oh!

sjust-redpanda commented Jun 23, 2026

Uh oh!

Lazin Jun 24, 2026

Uh oh!

sjust-redpanda Jun 24, 2026

Uh oh!

Lazin Jun 24, 2026

Uh oh!

sjust-redpanda Jun 24, 2026

Uh oh!

Lazin Jun 24, 2026

Uh oh!

sjust-redpanda Jun 24, 2026

Uh oh!

Lazin left a comment

Uh oh!

sjust-redpanda commented Jun 24, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

sjust-redpanda commented Jun 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

What's in this PR

Testing

Uh oh!

sjust-redpanda commented Jun 23, 2026

Uh oh!

Lazin Jun 24, 2026

Choose a reason for hiding this comment

Uh oh!

sjust-redpanda Jun 24, 2026

Choose a reason for hiding this comment

Uh oh!

Lazin Jun 24, 2026

Choose a reason for hiding this comment

Uh oh!

sjust-redpanda Jun 24, 2026

Choose a reason for hiding this comment

Uh oh!

Lazin Jun 24, 2026

Choose a reason for hiding this comment

Uh oh!

sjust-redpanda Jun 24, 2026

Choose a reason for hiding this comment

Uh oh!

Lazin left a comment

Choose a reason for hiding this comment

Uh oh!

sjust-redpanda commented Jun 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

sjust-redpanda commented Jun 23, 2026 •

edited

Loading

sjust-redpanda commented Jun 24, 2026 •

edited

Loading