compaction: Reduce record copying in compaction read paths

[WillemKauf]

Reduces the amount of record copying we perform in cloud topics compaction and local storage by adding record_batch::for_each_record_key(), which only materializes a record's key, offset delta and indication that the record is a tombstone via a new type record_key_metadata.

This is useful for a number of passes in local storage compaction and cloud topics compaction.

In an ad-hoc, end to end cloud topics L1 compaction reducer benchmark (not checked in):

  ┌──────────────┬──────────────────────────┬──────────────────────────┐
  │    codec     │        wall-clock        │       allocations        │
  ├──────────────┼──────────────────────────┼──────────────────────────┤
  │ zstd         │ 32.4 ms → 25.0 ms (−23%) │ 425,866 → 121,307 (−72%) │
  ├──────────────┼──────────────────────────┼──────────────────────────┤
  │ uncompressed │ 65.9 ms → 57.0 ms (−14%) │ 453,789 → 149,254 (−67%) │
  └──────────────┴──────────────────────────┴──────────────────────────┘

And from a benchmark (not checked in) of full record parsing (performed with
batch.for_each_record()) versus key-only (performed with
batch.for_each_record_key()):

  ┌────────────┬──────────┬──────────┬─────────┬──────────────────────────┐
  │ value size │   full   │ key-only │ speedup │ allocs (full → key-only) │
  ├────────────┼──────────┼──────────┼─────────┼──────────────────────────┤
  │ 64 B       │ 44.1 ns  │ 20.2 ns  │ 2.2×    │ 2.1 → 0                  │
  ├────────────┼──────────┼──────────┼─────────┼──────────────────────────┤
  │ 512 B      │ 45.2 ns  │ 19.9 ns  │ 2.3×    │ 2.1 → 0                  │
  ├────────────┼──────────┼──────────┼─────────┼──────────────────────────┤
  │ 4 KiB      │ 63.1 ns  │ 19.7 ns  │ 3.2×    │ 2.1 → 0                  │
  ├────────────┼──────────┼──────────┼─────────┼──────────────────────────┤
  │ 16 KiB     │ 140.7 ns │ 20.6 ns  │ 6.8×    │ 2.1 → 0                  │
  └────────────┴──────────┴──────────┴─────────┴──────────────────────────┘

Backports Required

• none - not a bug fix
• none - this is a backport
• none - issue does not exist in previous branches
• none - papercut/not impactful enough to backport
• v26.1.x
• v25.3.x
• v25.2.x

Release Notes

Improvements

• Optimize the cloud topics and local storage compaction implementations by reducing the amount of record copying performed

[Copilot]

Pull request overview

This PR reduces record copying in compaction paths (local storage + cloud topics) by introducing key-only record iteration (record_batch::for_each_record_key{,_async}) backed by a new lightweight model::record_key_metadata and parser.

Changes:

• Add record_key_metadata + parse_record_key_from_buffer() and expose key-only record iteration APIs on model::record_batch.
• Switch compaction/deduplication reducers and filters to use key-only iteration to avoid full record materialization.
• Add focused unit tests verifying correct key-only parsing/iteration behavior and update build deps accordingly.

Reviewed changes

Copilot reviewed 17 out of 17 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
src/v/storage/tests/segment_deduplication_test.cc	Updates test predicates to accept record_key_metadata instead of full records.
src/v/storage/segment_utils.h	Switches compaction decision helpers to use key-only metadata + adjusts logging.
src/v/storage/segment_utils.cc	Updates record callbacks to use record_key_metadata fields.
src/v/storage/segment_deduplication_utils.cc	Uses key-only metadata for deduplication key/offset checks.
src/v/storage/compaction_reducers.h	Changes reducer callback types to consume record_key_metadata.
src/v/storage/compaction_reducers.cc	Migrates reducers to for_each_record_key_async and moves keys into index/map.
src/v/model/tests/record_batch_test.cc	Adds deterministic tests for key-only parsing and async stop behavior.
src/v/model/tests/BUILD	Adds deps needed by new key-parse tests.
src/v/model/record.h	Adds for_each_record_key and for_each_record_key_async APIs.
src/v/model/record_utils.h	Introduces record_key_metadata and key-only parse function declaration.
src/v/model/record_utils.cc	Implements parse_record_key_from_buffer() with value/header skipping.
src/v/compaction/utils.h	Adds new overload of is_latest_record_for_key for pre-extracted key/offset.
src/v/compaction/utils.cc	Implements overload and refactors existing overload to call it.
src/v/cloud_topics/level_one/maintenance/compaction/compaction_source.cc	Migrates map-building reducer to key-only iteration.
src/v/cloud_topics/level_one/maintenance/compaction/compaction_filter.h	Refactors filter API to operate on key/offset/tombstone metadata.
src/v/cloud_topics/level_one/maintenance/compaction/compaction_filter.cc	Implements key-only filtering and offset-delta selection.
src/v/cloud_topics/level_one/maintenance/compaction/BUILD	Adds compaction key library dependency.

[vbotbuildovich]

CI test results

test results on build#86733

test_status	test_class	test_method	test_arguments	test_kind	job_url	passed	reason	test_history
FLAKY(PASS)	ListOffsetsLeaderEpochRedpandaTest	test_list_offsets_epoch	{"correct_epoch": true}	integration	https://buildkite.com/redpanda/redpanda/builds/86733#019f29c7-3d9f-4200-b55c-ce10fed46fad	19/21	Test PASSES after retries.No significant increase in flaky rate(baseline=0.0014, p0=0.0283, reject_threshold=0.0100. adj_baseline=0.1000, p1=0.3917, trust_threshold=0.5000)	https://redpanda.metabaseapp.com/dashboard/87-tests?tab=142-dt-individual-test-history&test_class=ListOffsetsLeaderEpochRedpandaTest&test_method=test_list_offsets_epoch
FLAKY(PASS)	NodeWiseRecoveryTest	test_recovery_local_data_missing	{"wait_for_final_manifest_uploads": true}	integration	https://buildkite.com/redpanda/redpanda/builds/86733#019f29c7-3d9e-4d61-b274-ec7024ea53df	9/11	Test PASSES after retries.No significant increase in flaky rate(baseline=0.0641, p0=0.4847, reject_threshold=0.0100. adj_baseline=0.1803, p1=0.4380, trust_threshold=0.5000)	https://redpanda.metabaseapp.com/dashboard/87-tests?tab=142-dt-individual-test-history&test_class=NodeWiseRecoveryTest&test_method=test_recovery_local_data_missing