Three new background tasks introduced by the
`enable_experimental_export_merge_tree_partition_feature`
forwardport call ZooKeeper without entering a component scope.
With `enforce_keeper_component_tracking = true` (set in
fast-test config via `zookeeper_enforce_component_name.yaml`),
this triggers a logical error in `Coordination::ZooKeeper::pushRequest`
the moment any `ReplicatedMergeTree` table activates the tasks
on startup, aborting the server. The 247 failing fast-test
replicated-table tests are all downstream effects of this abort
(they surface as KEEPER_EXCEPTION / TABLE_IS_READ_ONLY).
Wrap the entry of each background task method in
`Coordination::setCurrentComponent`, matching the convention used
by other replicated background work (e.g.
`ReplicatedMergeTreeRestartingThread`, `ReplicatedMergeTreeCleanupThread`).
Addresses 247 failing tests in the Fast test shard on
#1685. After this fix
the still-failing set shrank from 247 -> 0 (locally: 245 OK, 2
SKIPPED, 0 FAILED across the same input list).
zvonand
added
releasy
Changelog category (leave one):
Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):
Add cache for S3 list objects calls and support for exporting MergeTree parts and partitions. Fix Apache Iceberg queries not hitting the parquet metadata cache.
Add cache for S3 list objects calls and support for exporting MergeTree parts and partitions. Fix Apache Iceberg queries not hitting the parquet metadata cache (#1405 by @arthurpassos, #1388 by @arthurpassos, #1593 by @arthurpassos, #1517 by @arthurpassos, #1631 by @arthurpassos).
CI/CD Options
Exclude tests:
Regression jobs to run:
Combined port of 5 PR(s) (group
apassos-1). Cherry-picked from #1405, #1388, #1593, #1517, #1631.#1405: Antalya 26.1 - Forward port of list objects cache #1040
Documentation entry for user-facing changes
Cache for listobjects calls
#1388: Antalya 26.1 - Forward port of export part and partition
Documentation entry for user-facing changes
Export merge tree part and partition (we still need to rebase #1177 afterwards)
#1593: Export Partition - release the part lock when the query is cancelled
During export partition, parts are locked by replicas for exports. This PR introduces a change that releases these locks when an export task is cancelled. Previously, it would not release the lock. We did not catch this error before because the only cases an export task was cancelled we tested were
KILL EXPORT PARTITIONandDROP TABLE. In those cases, the entire task is cancelled, so it does not matter if a replica does not release its lock.But a query can also be cancelled with 'SYSTEM STOP MOVES', and in that case, it is a local operation. The lock must be released so other replicas can continue.
Documentation entry for user-facing changes
...
#1517: Fix IPartitionStrategy race condition
IPartitionStrategy::computePartitionKey might be called from different threads, and it writes to cached_result concurrently without any sort of protection. It would be easier to add a mutex around it, but we can actually make it lock-free by moving the cache write to the constructor.
Documentation entry for user-facing changes
...
#1631: Fix condition for using parquet metadata cache
Apache Iceberg queries were not htiting the parquet metadata cache because
object_info->getFileFormat()resolves toIcebergDataObjectInfo::getFileFormat, which gets its return value fromIcebergObjectSerializableInfo. This field is filled with the value from Apache Iceberg manifest file, and it is upper case by default, which then fails clickhouse check for parquet metadata cache usage.Documentation entry for user-facing changes
...