Skip to content

Fix Cloudberry restore point test race condition in WAL archiving #35

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
Copilot wants to merge 4 commits into
base: master
Choose a base branch
from
Draft

Fix Cloudberry restore point test race condition in WAL archiving #35

Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
712a77f
Initial plan
Copilot Feb 9, 2026
4837f47
Add retry logic and delays for WAL archiving in cloudberry restore po…
Copilot Feb 9, 2026
6926dcf
Address code review feedback - add comment and increase sleep interval
Copilot Feb 9, 2026
763a05f
Fix comment - update timeout calculation to reflect 5s interval
Copilot Feb 9, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
37 changes: 29 additions & 8 deletions docker/cloudberry_tests/scripts/tests/restore_point.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,24 +16,46 @@ setup_wal_archiving
wal-g --config=${TMP_CONFIG} delete everything FORCE --confirm

wal-g create-restore-point rp1 --config=${TMP_CONFIG}
# Wait a bit for the archiver to process the WAL switch
sleep 2

wal-g create-restore-point rp2 --config=${TMP_CONFIG}
# Wait for the archiver to process the second WAL switch
sleep 2

# Check whether the WAL log is correctly switched and uploaded to S3 after create-restore-point
# gpadmin@10f4a227f02b:/usr/local/gpdb_src$ wal-g st ls segments_005/seg0/wal_005/ --config=${TMP_CONFIG}
# type size last modified name
# obj 4624920 2025-05-21 07:17:16.052 +0000 UTC 000000010000000000000001.lz4
# obj 264275 2025-05-21 07:26:06.265 +0000 UTC 000000010000000000000002.lz4

#wait for wal-g to upload WALs
sleep 5

check_wal_upload() {
local path=$1

# Wait up to 5 minutes (60 attempts × 5 seconds) for WAL files to be uploaded
local max_attempts=60
local attempt=1

echo "Waiting for WAL files to be uploaded to $path..."

while [ $attempt -le $max_attempts ]; do
local count=$(wal-g st ls "$path" --config=${TMP_CONFIG} | awk '/^obj/ {count++} END {print count+0}')

if [ "$count" -ge 2 ]; then
echo "Found $count WAL files in $path (attempt $attempt/$max_attempts)"
wal-g st ls "$path" --config=${TMP_CONFIG}
return 0
fi

echo "Found $count WAL files in $path, waiting... (attempt $attempt/$max_attempts)"
# Use 5-second interval to reduce S3 API calls while still providing reasonable timeout
sleep 5
attempt=$((attempt + 1))
done

echo "Error: WAL files after create-restore-point were not correctly uploaded to S3 for $path"
echo "Final listing:"
wal-g st ls "$path" --config=${TMP_CONFIG}

wal-g st ls "$path" --config=${TMP_CONFIG} \
| awk '/^obj/ {count++} END {exit !(count >= 2)}'
return 1
}

# Check each segment
Expand All @@ -44,7 +66,6 @@ for seg_path in \
segments_005/seg2/wal_005/
do
if ! check_wal_upload "$seg_path"; then
echo "Error: WAL files after create-restore-point were not correctly uploaded to S3 for $seg_path"
exit 1
fi
done
Expand Down