export fails on swap partition

[anarcat]

it looks like ganeti is completely failing to export VMs that have a swap device. I created this instance with:

    gnt-instance add \
      -o debootstrap+bullseye \
      -t drbd --no-wait-for-sync \
      --net 0:ip=pool,network=gnt-fsn13-02 \
      --no-ip-check \
      --no-name-check \
      --disk 0:size=10G \
      --disk 1:size=2G,name=swap \
      --backend-parameters memory=1g,vcpus=1 \
      test-01.torproject.org

the swap is created through this hook, which basically does this:

    swapdev=$(eval "echo \$DISK_${i}_PATH")
    mkswap "$swapdev"

when i try to export this instance, ganeti fails with:

root@fsn-node-02:~# gnt-backup export -n fsn-node-02.torproject.org test-01.torproject.org
Mon Oct  3 20:35:00 2022 Shutting down instance test-01.torproject.org
Mon Oct  3 20:35:05 2022 Creating a snapshot of disk/0 on node fsn-node-01.torproject.org
Mon Oct  3 20:35:06 2022 Creating a snapshot of disk/1 on node fsn-node-01.torproject.org
Mon Oct  3 20:35:07 2022 Starting instance test-01.torproject.org
Mon Oct  3 20:35:08 2022 Exporting snapshot/0 from fsn-node-01.torproject.org to fsn-node-02.torproject.org
Mon Oct  3 20:35:08 2022 Exporting snapshot/1 from fsn-node-01.torproject.org to fsn-node-02.torproject.org
Mon Oct  3 20:35:11 2022 snapshot/0 is now listening, starting export
Mon Oct  3 20:35:11 2022 snapshot/1 is now listening, starting export
Mon Oct  3 20:35:14 2022 snapshot/0 is receiving data on fsn-node-02.torproject.org
Mon Oct  3 20:35:14 2022 snapshot/0 is sending data on fsn-node-01.torproject.org
Mon Oct  3 20:35:14 2022  - WARNING: export 'export-disk1-2022-10-03_20_35_14-g26kw0j_' on fsn-node-01.torproject.org failed: Exited with status 1
Mon Oct  3 20:35:14 2022 snapshot/1 failed to send data: Exited with status 1 (recent output: Cannot interpret kpartx output and get partition mapping\ndd: 0 bytes copied, 0.00339349 s, 0.0 kB/s)
Mon Oct  3 20:35:14 2022 Removing snapshot of disk/1 on node fsn-node-01.torproject.org
Mon Oct  3 20:35:15 2022  - WARNING: Aborting import 'import-disk1-2022-10-03_20_35_08-j2owrsjk' on 053e482a-c9f9-49a1-984d-50ae5b4563e6
Mon Oct  3 20:35:16 2022 snapshot/1 finished receiving data
Mon Oct  3 20:35:21 2022 snapshot/0 sent 513M, 89.6 MiB/s
Mon Oct  3 20:35:26 2022 snapshot/0 finished receiving data
Mon Oct  3 20:35:26 2022 snapshot/0 finished sending data
Mon Oct  3 20:35:26 2022 Removing snapshot of disk/0 on node fsn-node-01.torproject.org
Mon Oct  3 20:35:27 2022  - WARNING: Some disk exports have failed; there may be leftover data for instance test-01.torproject.org on node fsn-node-02.torproject.org
Failure: command execution error:
Export failed, errors in export finalization, disk export: disk(s) 1

it looks like it's failing on the dreaded map_disk0 code from common.sh:

instance-debootstrap/common.sh.in

Lines 110 to 126 in e0df6b1

	map_disk0() {
	blockdev="$1"
	filesystem_dev_base=`kpartx -l -p-part $blockdev | \
	grep -m 1 -- "-part1 : .*$blockdev" | \
	awk '{print $1}'`
	if [ -z "$filesystem_dev_base" ]; then
	log_error "Cannot interpret kpartx output and get partition mapping"
	exit 1
	fi
	kpartx -a -s -p-part $blockdev > /dev/null
	filesystem_dev="/dev/mapper/$filesystem_dev_base"
	if [ ! -b "$filesystem_dev" ]; then
	log_error "Can't find kpartx mapped partition: $filesystem_dev"
	exit 1
	fi
	echo "$filesystem_dev"
	}

specifically, the output of this command is empty:

kpartx -l -p-part $blockdev | \ 
                        grep -m 1 -- "-part1 : .*$blockdev" | \ 
                        awk '{print $1}'

if i kill the second disk, the backup works.

so i guess the question is: are backups just broken with multiple devices? or is this specific to swap?

later code in the export function certainly has me worried about backing up anything but a raw ext2+ partition here:

instance-debootstrap/export

Lines 46 to 49 in e0df6b1

	else
	echo "Can't dump partition of type ${vol_type}!" >&2
	exit 1
	fi

... why don't we just dd the heck out of this anyways, as a fallback?

[rbott]

I would actually opt to completly ditch the current import/export scripts and replace them with the versions from the noop OS provider, which essentially uses dd to make a bitwise copy of the disk.

This of course creates quite large exports and also exports potentially useless data (e.g. swap), but is guaranteed to work in all cases. The current logic is very prone to fail in various scenarios (differing disk layouts, xfs or other non-ext-filesystems etc.).

@saschalucas @apoikos @atta what do you think about this?

[anarcat]

it's effectively what we did here.

[atta]

@rbott yes i'm a 100% for you suggestion i can prepare a pull-request in order to get this common issue out of the way