diff --git a/harvester_robot_tests/keywords/addon.resource b/harvester_robot_tests/keywords/addon.resource index e396cb882..39f57964d 100644 --- a/harvester_robot_tests/keywords/addon.resource +++ b/harvester_robot_tests/keywords/addon.resource @@ -5,8 +5,18 @@ Library Process Resource variables.resource Resource common.resource +*** Variables *** +${LVM_ADDON_URL} https://raw.githubusercontent.com/harvester/experimental-addons/main/harvester-csi-driver-lvm/harvester-csi-driver-lvm.yaml + *** Keywords *** -Get Initial State +Install Addon From URL + [Arguments] ${url} + [Documentation] Install an addon by applying a YAML manifest from the given URL + Log Installing addon from ${url} + addon_keywords.install_addon_yaml ${url} + Log Addon YAML applied successfully + +Get Addon Initial State [Arguments] ${addon_name} [Documentation] Get initial state of addon (enabled/disabled) ... Args: @@ -18,7 +28,7 @@ Get Initial State Log Addon ${addon_name} initial state: ${initial_state} RETURN ${initial_state} -Enable +Enable Addon [Arguments] ${addon_name} [Documentation] Enable an addon ... Args: @@ -27,7 +37,7 @@ Enable addon_keywords.enable_addon ${addon_name} Log Addon ${addon_name} enable request sent -Disable +Disable Addon [Arguments] ${addon_name} [Documentation] Disable an addon ... Args: @@ -36,7 +46,7 @@ Disable addon_keywords.disable_addon ${addon_name} Log Addon ${addon_name} disable request sent -Wait For Enabled +Wait For Addon Enabled [Arguments] ${addon_name} ${timeout}=${WAIT_TIMEOUT} [Documentation] Wait until addon is enabled ... Args: @@ -46,7 +56,7 @@ Wait For Enabled addon_keywords.wait_for_addon_enabled ${addon_name} ${timeout} Log Addon ${addon_name} is now enabled -Wait For Disabled +Wait For Addon Disabled [Arguments] ${addon_name} ${timeout}=${WAIT_TIMEOUT} [Documentation] Wait until addon is disabled ... Args: @@ -56,7 +66,7 @@ Wait For Disabled addon_keywords.wait_for_addon_disabled ${addon_name} ${timeout} Log Addon ${addon_name} is now disabled -Is Enabled +Is Addon Enabled [Arguments] ${addon_name} [Documentation] Check if addon is enabled ... Args: @@ -76,7 +86,7 @@ Wait For Monitoring Pods Running addon_keywords.wait_for_monitoring_pods_running ${namespace} ${timeout} Log Monitoring pods are now running -Wait For Pods Running +Wait For Addon Pods Running [Arguments] ${namespace} ${label} ${timeout}=${WAIT_TIMEOUT} [Documentation] Wait for pods matching label to be running in namespace ... Args: @@ -87,7 +97,7 @@ Wait For Pods Running addon_keywords.wait_for_pods_running ${namespace} ${label} ${timeout} Log Pods with label ${label} are running in ${namespace} -Wait For Service Running +Wait For Addon Service Running [Arguments] ${namespace} ${service_name} ${timeout}=${WAIT_TIMEOUT} [Documentation] Wait for service to be running in namespace ... Args: @@ -98,7 +108,7 @@ Wait For Service Running addon_keywords.wait_for_service_running ${namespace} ${service_name} ${timeout} Log Service ${service_name} is running in ${namespace} -Configure Nvidia Toolkit +Configure Nvidia Toolkit Addon [Arguments] ${addon_name} ${image_repo} ${image_tag} ${driver_location} [Documentation] Configure the nvidia-driver-toolkit addon with image repo, tag, and driver location ... Args: @@ -110,7 +120,7 @@ Configure Nvidia Toolkit addon_keywords.configure_nvidia_toolkit_addon ${addon_name} ${image_repo} ${image_tag} ${driver_location} Log Nvidia-driver-toolkit addon configured -Verify Nvidia Toolkit Configured +Verify Nvidia Toolkit Addon Configured [Arguments] ${addon_name} ${image_repo} ${image_tag} ${driver_location} [Documentation] Verify nvidia-driver-toolkit addon configuration is applied ... Args: @@ -141,7 +151,7 @@ Port Forward To Prometheus addon_keywords.port_forward_to_prometheus ${namespace} ${pod_name} ${local_port} Log Port forward established -Stop Port Forward +Stop Prometheus Port Forward [Documentation] Stop port forwarding Log Stopping port forward addon_keywords.stop_port_forward @@ -173,7 +183,7 @@ Verify Prometheus Metric Exists Should Be True ${exists} Metric ${query} should exist Log Metric ${query} verified successfully -Restore State +Restore Addon State [Arguments] ${addon_name} ${initial_state} [Documentation] Restore addon to its initial state ... Args: @@ -187,16 +197,16 @@ Restore State Initial Addon State Is Captured [Arguments] ${addon_name} [Documentation] Capture and store the initial state of the addon - ${initial_state}= Get Initial State ${addon_name} + ${initial_state}= Get Addon Initial State ${addon_name} Set Suite Variable ${INITIAL_STATE_MONITORING} ${initial_state} Log Captured initial addon state: ${initial_state} Monitoring Addon Is Enabled [Arguments] ${addon_name} [Documentation] Enable the monitoring addon and wait for deployment - ${is_enabled}= Is Enabled ${addon_name} - Run Keyword If not ${is_enabled} Enable ${addon_name} - Wait For Enabled ${addon_name} timeout=900 + ${is_enabled}= Is Addon Enabled ${addon_name} + Run Keyword If not ${is_enabled} Enable Addon ${addon_name} + Wait For Addon Enabled ${addon_name} timeout=900 Log Monitoring addon ${addon_name} is enabled Monitoring Pods Should Be Running @@ -237,11 +247,11 @@ Suite Teardown For Addon Tests Log Running suite teardown for addon tests # Stop port forward if still running - Run Keyword And Ignore Error Stop Port Forward + Run Keyword And Ignore Error Stop Prometheus Port Forward # Restore addon to initial state Run Keyword If '${INITIAL_STATE_MONITORING}' != 'None' - ... Restore State ${ADDON_MONITORING} ${INITIAL_STATE_MONITORING} + ... Restore Addon State ${ADDON_MONITORING} ${INITIAL_STATE_MONITORING} # Standard cleanup Cleanup test resources @@ -265,28 +275,28 @@ Get Prometheus Pod Name Initial PCI Addon State Is Captured [Arguments] ${addon_name} [Documentation] Capture and store the initial state of the addon - ${initial_state}= Get Initial State ${addon_name} + ${initial_state}= Get Addon Initial State ${addon_name} Set Suite Variable ${INITIAL_STATE_PCIDEVICES} ${initial_state} Log Captured initial addon state: ${initial_state} PCI Devices Addon Is Enabled [Arguments] ${addon_name} [Documentation] Enable the pcidevices-controller addon and wait for deployment - ${is_enabled}= Is Enabled ${addon_name} - Run Keyword If not ${is_enabled} Enable ${addon_name} - Wait For Enabled ${addon_name} timeout=900 + ${is_enabled}= Is Addon Enabled ${addon_name} + Run Keyword If not ${is_enabled} Enable Addon ${addon_name} + Wait For Addon Enabled ${addon_name} timeout=900 Log PCI Devices addon ${addon_name} is enabled PCI Devices Controller Pods Should Be Running [Arguments] ${namespace} ${label} [Documentation] Verify pcidevices-controller pods are running - Wait For Pods Running ${namespace} ${label} timeout=900 + Wait For Addon Pods Running ${namespace} ${label} timeout=900 Log All pcidevices-controller pods are running in ${namespace} PCI Devices Webhook Service Should Be Running [Arguments] ${namespace} ${service_name} [Documentation] Verify pcidevices-webhook service is running - Wait For Service Running ${namespace} ${service_name} timeout=300 + Wait For Addon Service Running ${namespace} ${service_name} timeout=300 Log pcidevices-webhook service is running in ${namespace} Suite Setup For PCI Devices Addon Tests @@ -299,6 +309,6 @@ Suite Teardown For PCI Devices Addon Tests [Documentation] Cleanup and restore addon state after tests Log Running suite teardown for PCI Devices addon tests Run Keyword If '${INITIAL_STATE_PCIDEVICES}' != ${None} - ... Restore State ${ADDON_PCIDEVICES} ${INITIAL_STATE_PCIDEVICES} + ... Restore Addon State ${ADDON_PCIDEVICES} ${INITIAL_STATE_PCIDEVICES} Cleanup test resources Log Suite teardown completed diff --git a/harvester_robot_tests/keywords/common.resource b/harvester_robot_tests/keywords/common.resource index 3a587f20c..2a8340a89 100644 --- a/harvester_robot_tests/keywords/common.resource +++ b/harvester_robot_tests/keywords/common.resource @@ -1,6 +1,9 @@ *** Settings *** Documentation Common Keywords Library ../libs/keywords/common_keywords.py +Library ../libs/keywords/image_keywords.py +Library ../libs/keywords/network_keywords.py +Library ../libs/keywords/volume_keywords.py Resource variables.resource @@ -32,3 +35,50 @@ Count Pods By Label ${pods}= common_keywords.list_pods_by_label ${namespace} ${label_selector} ${status} ${count}= Get Length ${pods} RETURN ${count} + +Suite Setup For Shared Resources + [Documentation] Suite setup for shared resources + Log Setting up shared resources for the test suite + + # Initialize test variables + ${timestamp}= Evaluate __import__('datetime').datetime.now().strftime("%Y%m%d%H%M%S") + Set Suite Variable ${UNIQUE_NAME} ${timestamp} + Set Suite Variable ${VLAN_NETWORK_NAME} vlan-network-${VLAN_ID} + # Harvester enforces a 12-character max on ClusterNetwork and VlanConfig names. + ${ts_short}= Evaluate '${timestamp}'[-7:] + Set Suite Variable ${CLUSTER_NETWORK_NAME} cnet-${ts_short} + Set Suite Variable ${VLAN_CONFIG_NAME} vcfg-${ts_short} + Set Suite Variable ${IP_POOL_NAME} ippool-vlan${VLAN_ID} + Set Suite Variable ${IMAGE_NAME} opensuse-${timestamp} + + # Create shared resources + network_keywords.create_cluster_network ${CLUSTER_NETWORK_NAME} + network_keywords.create_vlan_config ${VLAN_CONFIG_NAME} ${CLUSTER_NETWORK_NAME} ${VLAN_NIC} + network_keywords.wait_for_cluster_network_ready ${CLUSTER_NETWORK_NAME} + network_keywords.create_vlan_network ${VLAN_NETWORK_NAME} ${VLAN_ID} ${CLUSTER_NETWORK_NAME} + network_keywords.create_ip_pool + ... ${IP_POOL_NAME} + ... ${IP_POOL_SUBNET} + ... ${IP_POOL_START} + ... ${IP_POOL_END} + ... ${DEFAULT_NAMESPACE}/${VLAN_NETWORK_NAME} + image_keywords.create image from url ${IMAGE_NAME} ${OPENSUSE_IMAGE_URL} + image_keywords.wait for image downloaded ${IMAGE_NAME} + + Log Test environment ready (image: ${IMAGE_NAME}, vlan: ${VLAN_NETWORK_NAME}, timestamp: ${timestamp}) console=yes + +Suite Teardown For Shared Resources + [Documentation] Cleanup all resources after tests. + Log Running shared resources teardown + + # Delete shared resources + Run Keyword And Ignore Error network_keywords.delete_ip_pool ${IP_POOL_NAME} + Run Keyword And Ignore Error network_keywords.delete_vlan_network ${VLAN_NETWORK_NAME} + Run Keyword And Ignore Error network_keywords.delete_vlan_config ${VLAN_CONFIG_NAME} + Run Keyword And Ignore Error network_keywords.wait_for_vlan_config_deleted ${VLAN_CONFIG_NAME} + Run Keyword And Ignore Error network_keywords.delete_cluster_network ${CLUSTER_NETWORK_NAME} + Run Keyword And Ignore Error volume_keywords.cleanup_volumes + Run Keyword And Ignore Error image_keywords.delete image ${IMAGE_NAME} + + Cleanup test resources + Log Suite teardown completed for shared resources diff --git a/harvester_robot_tests/keywords/image.resource b/harvester_robot_tests/keywords/image.resource index d4ba2e9fc..8a0c5207a 100644 --- a/harvester_robot_tests/keywords/image.resource +++ b/harvester_robot_tests/keywords/image.resource @@ -1,7 +1,6 @@ *** Settings *** Documentation Image Keywords Resource variables.resource -Resource common.resource Library ../libs/keywords/image_keywords.py *** Keywords *** diff --git a/harvester_robot_tests/keywords/network.resource b/harvester_robot_tests/keywords/network.resource index bef2b7a91..490a25348 100644 --- a/harvester_robot_tests/keywords/network.resource +++ b/harvester_robot_tests/keywords/network.resource @@ -45,6 +45,13 @@ Delete VLAN Config Log Deleting VLAN config: ${name} network_keywords.delete_vlan_config ${name} +Wait For VLAN Config Deleted + [Arguments] ${name} ${timeout}=${WAIT_TIMEOUT} + [Documentation] Wait for VLAN config to be fully removed before proceeding. + ... Required before deleting the parent ClusterNetwork. + Log Waiting for VLAN config ${name} to be deleted + network_keywords.wait_for_vlan_config_deleted ${name} ${timeout} + Wait For Cluster Network Ready [Arguments] ${name} ${timeout}=${WAIT_TIMEOUT} [Documentation] Wait for cluster network to become ready. diff --git a/harvester_robot_tests/keywords/setting.resource b/harvester_robot_tests/keywords/setting.resource index a2f2b9e29..63f860ecf 100644 --- a/harvester_robot_tests/keywords/setting.resource +++ b/harvester_robot_tests/keywords/setting.resource @@ -54,4 +54,22 @@ LHv2 Data Engine Is Enabled Wait Until LHv2 Data Engine Is Enabled Wait Until Keyword Succeeds ${WAIT_TIMEOUT} ${RETRY_INTERVAL} - ... LHv2 Data Engine Is Enabled \ No newline at end of file + ... LHv2 Data Engine Is Enabled + +# LVM Setting Keywords +Configure CSI Driver Setting For LVM + [Documentation] Add LVM provisioner entry to csi-driver-config setting + Log Configuring csi-driver-config for LVM snapshots + setting_keywords.configure_csi_driver + ... csi-driver-config + ... lvm.driver.harvesterhci.io + ... lvm-snapshot + Log CSI driver config updated for LVM + +Reset CSI Driver Setting For LVM + [Documentation] Remove LVM provisioner entry from csi-driver-config setting + Log Resetting csi-driver-config for LVM + setting_keywords.remove_csi_driver + ... csi-driver-config + ... lvm.driver.harvesterhci.io + Log CSI driver config reset for LVM diff --git a/harvester_robot_tests/keywords/storage.resource b/harvester_robot_tests/keywords/storage.resource index e3c6a2460..8ea893aec 100644 --- a/harvester_robot_tests/keywords/storage.resource +++ b/harvester_robot_tests/keywords/storage.resource @@ -1,6 +1,9 @@ *** Settings *** Documentation Storage Keywords Library ../libs/keywords/storage_keywords.py +Library ../libs/keywords/vm_keywords.py +Library ../libs/keywords/volume_keywords.py +Library Collections Resource variables.resource Resource common.resource Resource host.resource @@ -9,6 +12,10 @@ Resource host.resource *** Variables *** &{DATA_DISK_BY_NODE} &{EMPTY} ${LHv2_SC_REPLICAS} ${EMPTY} +${LVM_NODE_COUNT} ${0} +@{LVM_VOLUMES_CREATED} +&{LVM_DISK_BY_NODE} &{EMPTY} +&{LVM_VG_NODE_MAP} &{EMPTY} *** Keywords *** @@ -77,3 +84,95 @@ Wait Until LHv2 Storages Are Provisioned Wait Until Keyword Succeeds ${WAIT_TIMEOUT} ${RETRY_INTERVAL} ... Longhorn Node Disk Is Schedulable ${node} ${disk} END + +# LVM Suite Setup Keywords +Identify And Assign LVM Disks + [Documentation] Identify nodes with at least one additional disk >= 50 GiB for LVM. + ... Single-node cluster: needs 1 such node (creates vg-dm-thin only). + ... Multi-node cluster: randomly picks 2 nodes (vg-dm-thin + vg-dm-striped). + ... Skips the entire suite when no eligible nodes are found. + Log Identifying disks suitable for LVM (>= 50 GiB) + &{LVM_DISK_BY_NODE}= storage_keywords.identify_lvm_suitable_disks ${50} + ${node_count}= Get Length ${LVM_DISK_BY_NODE} + Skip If ${node_count} == 0 + ... No nodes found with a suitable disk >= 50 GiB — LVM tests require at least 1 eligible node + Set Suite Variable &{LVM_DISK_BY_NODE} + Set Suite Variable ${LVM_NODE_COUNT} ${node_count} + Log Found ${node_count} node(s) with LVM-suitable disks: ${LVM_DISK_BY_NODE} + +Create LVM Volume Groups + [Documentation] Create volume groups from identified disks. + ... Single-node: creates the VG matching ${LVM_VG_TYPE} (dm-thin or striped). + ... Multi-node: creates vg-dm-thin + vg-dm-striped on separate nodes. + Log Creating LVM volume groups (${LVM_NODE_COUNT} eligible node(s)) type=${LVM_VG_TYPE} + &{LVM_VG_NODE_MAP}= storage_keywords.create_lvm_volume_groups ${LVM_DISK_BY_NODE} ${LVM_VG_TYPE} + Set Suite Variable &{LVM_VG_NODE_MAP} + # Map the two accepted values to canonical VG/SC names and VM/VOL suffix + ${_vg_name}= Set Variable If '${LVM_VG_TYPE}' == 'dm-thin' vg-dm-thin vg-dm-striped + ${_sc_name}= Set Variable If '${LVM_VG_TYPE}' == 'dm-thin' lvm-sc-dm-thin lvm-sc-dm-striped + ${_suffix}= Set Variable If '${LVM_VG_TYPE}' == 'dm-thin' thin striped + Set Suite Variable ${LVM_VG_ACTIVE} ${_vg_name} + Set Suite Variable ${LVM_SC_ACTIVE} ${_sc_name} + Set Suite Variable ${LVM_VM_BLOCK} lvm-vm-block-${_suffix} + Set Suite Variable ${LVM_VM_FS} lvm-vm-fs-${_suffix} + Set Suite Variable ${LVM_VM_SC} lvm-vm-sc-${_suffix} + Set Suite Variable ${LVM_VOL_BLOCK} lvm-vol-block-${_suffix} + Set Suite Variable ${LVM_VOL_FS} lvm-vol-fs-${_suffix} + Log LVM volume groups created: ${LVM_VG_NODE_MAP} + Log Active config: VG=${_vg_name} SC=${_sc_name} VM/VOL suffix=${_suffix} + +# LVM Volume Keywords +LVM Block Volume Is Created + [Arguments] ${vol_name} ${size} ${sc_name} + [Documentation] Create a block volume using the LVM StorageClass + Log Creating LVM block volume ${vol_name} size=${size} sc=${sc_name} + volume_keywords.create_volume ${vol_name} ${size} storage_class=${sc_name} volume_mode=Block + Append To List ${LVM_VOLUMES_CREATED} ${vol_name} + Set Suite Variable @{LVM_VOLUMES_CREATED} + Log LVM block volume ${vol_name} created + +LVM Filesystem Volume Is Created + [Arguments] ${vol_name} ${size} ${sc_name} + [Documentation] Create a filesystem volume using the LVM StorageClass + Log Creating LVM filesystem volume ${vol_name} size=${size} sc=${sc_name} + volume_keywords.create_volume ${vol_name} ${size} storage_class=${sc_name} volume_mode=Filesystem + Append To List ${LVM_VOLUMES_CREATED} ${vol_name} + Set Suite Variable @{LVM_VOLUMES_CREATED} + Log LVM filesystem volume ${vol_name} created + +# LVM Volume Expand Keywords +LVM Volume Is Expanded Directly + [Arguments] ${vol_name} ${new_size} + [Documentation] Expand LVM volume by editing the volume directly + Log Expanding volume ${vol_name} to ${new_size} via volume edit + volume_keywords.expand_lvm_pvc ${vol_name} ${new_size} + Log Volume ${vol_name} expanded to ${new_size} + +# LVM Cleanup +Cleanup LVM Test Resources + [Documentation] Cleanup all LVM test resources + Log Cleaning up LVM test resources + # Delete VMs and root disk and wait for full termination before releasing volumes + FOR ${vm_name} IN @{VMS_CREATED} + Run Keyword And Ignore Error vm_keywords.delete_vm ${vm_name} + Run Keyword And Ignore Error vm_keywords.wait_for_vm_deleted ${vm_name} + Run Keyword And Ignore Error volume_keywords.delete_volume ${vm_name}-disk-0 + END + # Delete restored VMs (from snapshot restore to new VM tests) + FOR ${vm_name} IN @{VMS_RESTORED} + Run Keyword And Ignore Error vm_keywords.delete_vm ${vm_name} + Run Keyword And Ignore Error vm_keywords.wait_for_vm_deleted ${vm_name} + Run Keyword And Ignore Error volume_keywords.delete_volume ${vm_name}-disk-0 + END + + # Delete Volumes + FOR ${vol_name} IN @{LVM_VOLUMES_CREATED} + Run Keyword And Ignore Error volume_keywords.delete_volume ${vol_name} + END + # Delete StorageClasses + FOR ${sc_name} IN @{LVM_SCS_CREATED} + Run Keyword And Ignore Error storage_keywords.delete_lvm_storage_class ${sc_name} + END + # Remove volume groups + Run Keyword And Ignore Error storage_keywords.cleanup_lvm_volume_groups ${LVM_DISK_BY_NODE} + Log LVM test resources cleaned up diff --git a/harvester_robot_tests/keywords/storageclass.resource b/harvester_robot_tests/keywords/storageclass.resource new file mode 100644 index 000000000..183b71c9d --- /dev/null +++ b/harvester_robot_tests/keywords/storageclass.resource @@ -0,0 +1,25 @@ +*** Settings *** +Documentation StorageClass Keywords +Library ../libs/keywords/storage_keywords.py +Library Collections +Resource variables.resource + + +*** Variables *** +@{LVM_SCS_CREATED} + + +*** Keywords *** +LVM StorageClass Is Created + [Arguments] ${sc_name} ${vg_name} ${vg_type} + [Documentation] Create an LVM StorageClass with specified volume group + ... Args: + ... sc_name: Name of the StorageClass + ... vg_name: Volume group name + ... vg_type: Volume group type (dm-thin or striped) + Log Creating LVM StorageClass ${sc_name} for VG ${vg_name} type ${vg_type} + ${node}= storage_keywords.get_node_for_vg ${vg_name} ${LVM_VG_NODE_MAP} + storage_keywords.create_lvm_storage_class ${sc_name} ${vg_name} ${vg_type} ${node} + Append To List ${LVM_SCS_CREATED} ${sc_name} + Set Suite Variable @{LVM_SCS_CREATED} + Log LVM StorageClass ${sc_name} created successfully diff --git a/harvester_robot_tests/keywords/variables.resource b/harvester_robot_tests/keywords/variables.resource index a7d4dcea0..13adfba0b 100644 --- a/harvester_robot_tests/keywords/variables.resource +++ b/harvester_robot_tests/keywords/variables.resource @@ -33,6 +33,9 @@ ${DEFAULT_STORAGE_CLASS} harvester-longhorn ${DEFAULT_VOLUME_SIZE} 10Gi ${RWX_HOST_STORAGE_CLASS} %{RWX_HOST_STORAGE_CLASS=longhorn-rwx} +# LVM: dm-thin or striped +${LVM_VG_TYPE} %{LVM_VG_TYPE=striped} + # Test Execution ${LOOP_COUNT} 3 ${RETRY_COUNT} 100 diff --git a/harvester_robot_tests/keywords/virtualmachine.resource b/harvester_robot_tests/keywords/virtualmachine.resource index b79b009d5..717bdf6d1 100644 --- a/harvester_robot_tests/keywords/virtualmachine.resource +++ b/harvester_robot_tests/keywords/virtualmachine.resource @@ -4,6 +4,14 @@ Resource variables.resource Resource common.resource Resource image.resource Library ../libs/keywords/vm_keywords.py +Library ../libs/keywords/snapshot_keywords.py +Library Collections + +*** Variables *** +@{VMS_CREATED} +@{VMS_RESTORED} +&{VM_CHECKSUMS} &{EMPTY} +&{VM_SNAPSHOTS} &{EMPTY} *** Keywords *** List VMs by prefix @@ -69,3 +77,185 @@ Image is available for VM creation [Documentation] Create an image from URL (delegates to Image module). image.Create image from url with name ${image_name} ${image_url} image.Wait for image downloaded by name ${image_name} + +VM Is Created And Running + [Arguments] ${vm_name} ${image_name} ${sc_name}=${EMPTY} ${network_name}=${VLAN_NETWORK_NAME} + [Documentation] Create a VM from the given image and wait for it to be running. + Log Creating VM ${vm_name} with image ${image_name} + VM is created ${vm_name} image_id=${image_name} sc_name=${sc_name} network_name=${network_name} + VM should be running ${vm_name} + Append To List ${VMS_CREATED} ${vm_name} + Set Suite Variable @{VMS_CREATED} + Log VM ${vm_name} is running + +VM Is Created with Additional Volume using SC + [Arguments] ${vm_name} ${sc_name} ${image_name} ${network_name}=${VLAN_NETWORK_NAME} + [Documentation] Create a VM with the given image as root disk and an additional data volume from the StorageClass + Log Creating VM ${vm_name} with image ${image_name} and extra data volume from SC ${sc_name} + vm_keywords.create_vm_with_volume_using_sc ${vm_name} ${sc_name} ${image_name} network_name=${network_name} + Wait Until Keyword Succeeds ${WAIT_TIMEOUT} ${RETRY_INTERVAL} + ... VM State Is Running ${vm_name} + Append To List ${VMS_CREATED} ${vm_name} + Set Suite Variable @{VMS_CREATED} + Log VM ${vm_name} created with SC ${sc_name} and running + +Volume Is Attached To VM + [Arguments] ${vm_name} ${vol_name} + [Documentation] Attach an LVM volume to a running VM + Log Attaching volume ${vol_name} to VM ${vm_name} + vm_keywords.attach_volume_to_vm ${vm_name} ${vol_name} + Wait Until Keyword Succeeds ${WAIT_TIMEOUT} ${RETRY_INTERVAL} + ... VM State Is Running ${vm_name} + Log Volume ${vol_name} attached to VM ${vm_name} + +Get Block Device In VM + [Arguments] ${vm_name} ${expected_disk_size}=${None} + [Documentation] Discover the data block device inside the VM and return its path. + ... When expected_disk_size is provided (e.g. 5Gi) the check also + ... filters by disk size to avoid matching the wrong device. + ... Returns the device path (e.g. /dev/vdb) for use in subsequent steps. + Log Getting block device in VM ${vm_name} + ${device}= vm_keywords.check_block_device_in_vm ${vm_name} ${expected_disk_size} + Log Block device ${device} found in VM ${vm_name} + RETURN ${device} + +VM State Is Running + [Arguments] ${vm_name} + [Documentation] Assert that VM is in running state + ${is_running}= vm_keywords.is_vm_running ${vm_name} + Should Be True ${is_running} VM ${vm_name} is not running + +VM State Is Stopped + [Arguments] ${vm_name} + [Documentation] Assert that VM is in stopped state + ${is_stopped}= vm_keywords.is_vm_stopped ${vm_name} + Should Be True ${is_stopped} VM ${vm_name} is not stopped + +# LVM VM Power Operations +VM Is Powered Off + [Arguments] ${vm_name} + [Documentation] Stop the VM + Log Stopping LVM VM ${vm_name} + vm_keywords.stop_vm ${vm_name} + Wait Until Keyword Succeeds ${WAIT_TIMEOUT} ${RETRY_INTERVAL} + ... VM State Is Stopped ${vm_name} + Log LVM VM ${vm_name} stopped + +VM Is Powered On And Running + [Arguments] ${vm_name} + [Documentation] Start VM and wait for running + Log Starting LVM VM ${vm_name} + vm_keywords.start_vm ${vm_name} + Wait Until Keyword Succeeds ${WAIT_TIMEOUT} ${RETRY_INTERVAL} + ... VM State Is Running ${vm_name} + Log LVM VM ${vm_name} started and running + +# LVM Data Operations Keywords +Data Is Written To Disk And Checksum Recorded + [Arguments] ${vm_name} ${device} ${format_device}=${True} + [Documentation] Mount device, write data and record md5sum. + ... Set format_device=${False} for Filesystem volumes (pre-formatted by CSI). + ... Set format_device=${True} (default) for Block volumes (mkfs.ext4 first). + Log Writing data to ${device} on VM ${vm_name} + ${checksum}= vm_keywords.write_data_and_get_checksum_on_disk ${vm_name} ${device} ${format_device} + Set To Dictionary ${VM_CHECKSUMS} ${vm_name} ${checksum} + Set Suite Variable &{VM_CHECKSUMS} + Log Data written to VM ${vm_name}, checksum: ${checksum} + +Data Is Deleted From Test VMs + [Documentation] Delete test data from all running LVM VMs + Log Deleting data from LVM VMs + FOR ${vm_name} IN @{VMS_CREATED} + ${is_running}= vm_keywords.is_vm_running ${vm_name} + Run Keyword If ${is_running} vm_keywords.delete_data_from_vm ${vm_name} + END + Log Data deleted from all running LVM VMs + +# LVM Snapshot Keywords +VM Snapshots Are Taken + [Documentation] Take VM snapshots for all running LVM VMs + Log Taking snapshots of running LVM VMs + FOR ${vm_name} IN @{VMS_CREATED} + ${is_running}= vm_keywords.is_vm_running ${vm_name} + IF ${is_running} + ${snapshot_name}= snapshot_keywords.take_vm_snapshot ${vm_name} + Set To Dictionary ${VM_SNAPSHOTS} ${vm_name} ${snapshot_name} + END + END + Set Suite Variable &{VM_SNAPSHOTS} + Log Snapshots taken: ${VM_SNAPSHOTS} + +Snapshots Are Restored To New VMs + [Documentation] Restore snapshots to newly created VMs + Log Restoring LVM snapshots to new VMs + FOR ${vm_name} ${snapshot_name} IN &{VM_SNAPSHOTS} + ${new_vm_name}= Set Variable ${vm_name}-restored + snapshot_keywords.restore_snapshot_to_new_vm ${snapshot_name} ${new_vm_name} + Append To List ${VMS_RESTORED} ${new_vm_name} + Set Suite Variable @{VMS_RESTORED} + Wait Until Keyword Succeeds ${WAIT_TIMEOUT} ${RETRY_INTERVAL} + ... VM State Is Running ${new_vm_name} + END + Log All snapshots restored to new VMs + +Restored VMs Have Correct Data + [Documentation] Verify restored VMs contain original data + FOR ${vm_name} ${snapshot_name} IN &{VM_SNAPSHOTS} + ${new_vm_name}= Set Variable ${vm_name}-restored + ${expected_checksum}= Get From Dictionary ${VM_CHECKSUMS} ${vm_name} + vm_keywords.mount_data_disk_in_vm ${new_vm_name} + ${actual_checksum}= Wait Until Keyword Succeeds ${WAIT_TIMEOUT} ${RETRY_INTERVAL} + ... vm_keywords.get_data_checksum_from_vm ${new_vm_name} + Should Be Equal ${actual_checksum} ${expected_checksum} + ... Data mismatch on restored VM ${new_vm_name} + END + Log All restored VMs verified + +Snapshots Are Restored To Existing VMs + [Documentation] Restore snapshots back to existing VMs + ... All VMs are stopped first before any restore is initiated. + Log Stopping all VMs before restore + FOR ${vm_name} ${snapshot_name} IN &{VM_SNAPSHOTS} + vm_keywords.stop_vm ${vm_name} + END + FOR ${vm_name} ${snapshot_name} IN &{VM_SNAPSHOTS} + Wait Until Keyword Succeeds ${WAIT_TIMEOUT} ${RETRY_INTERVAL} + ... VM State Is Stopped ${vm_name} + END + Log All VMs stopped, restoring snapshots + FOR ${vm_name} ${snapshot_name} IN &{VM_SNAPSHOTS} + snapshot_keywords.restore_snapshot_to_existing_vm ${snapshot_name} ${vm_name} + END + FOR ${vm_name} ${snapshot_name} IN &{VM_SNAPSHOTS} + Wait Until Keyword Succeeds ${WAIT_TIMEOUT} ${RETRY_INTERVAL} + ... VM State Is Running ${vm_name} + END + Log All snapshots restored to existing VMs + +VMs Have Original Data After Restore + [Documentation] Verify VMs have original data after snapshot restore + FOR ${vm_name} ${snapshot_name} IN &{VM_SNAPSHOTS} + ${expected_checksum}= Get From Dictionary ${VM_CHECKSUMS} ${vm_name} + vm_keywords.mount_data_disk_in_vm ${vm_name} + ${actual_checksum}= vm_keywords.get_data_checksum_from_vm ${vm_name} + Should Be Equal ${actual_checksum} ${expected_checksum} + ... Data mismatch on VM ${vm_name} after restore + END + Log All VMs have correct data after restore + +# LVM Volume Expand Via VM Keywords +Volume Is Expanded Via VM Edit + [Arguments] ${vm_name} ${vol_name} ${new_size} + [Documentation] Expand LVM volume by editing the VM spec + Log Expanding volume ${vol_name} on VM ${vm_name} to ${new_size} via VM edit + vm_keywords.expand_volume_via_vm_edit ${vm_name} ${vol_name} ${new_size} + Log Volume ${vol_name} expanded to ${new_size} via VM edit + +Volume Size Inside VM Is Correct + [Arguments] ${vm_name} ${expected_size} + [Documentation] Verify the volume size inside the VM matches expected + Log Verifying volume size inside VM ${vm_name} is ${expected_size} + ${actual_size}= vm_keywords.get_volume_size_inside_vm ${vm_name} ${expected_size} + Should Be True ${actual_size} + ... Volume size inside VM ${vm_name} does not match expected ${expected_size} + Log Volume size inside VM ${vm_name} is correct: ${expected_size} diff --git a/harvester_robot_tests/keywords/volume.resource b/harvester_robot_tests/keywords/volume.resource new file mode 100644 index 000000000..92cba0e33 --- /dev/null +++ b/harvester_robot_tests/keywords/volume.resource @@ -0,0 +1,24 @@ +*** Settings *** +Documentation Volume Keywords +Library ../libs/keywords/volume_keywords.py +Resource variables.resource + +*** Keywords *** +Create Volume + [Arguments] ${volume_name} ${size}=2Gi ${replicas}=3 ${frontend}=blockdev + [Documentation] Create a volume + volume_keywords.create_volume ${volume_name} ${size} ${replicas} ${frontend} + +Delete Volume + [Arguments] ${volume_name} ${wait}=${True} + [Documentation] Delete a volume + volume_keywords.delete_volume ${volume_name} ${wait} + +Wait For Volume Deleted + [Arguments] ${volume_name} ${timeout}=${DEFAULT_TIMEOUT_SHORT} + [Documentation] Wait for a volume to be deleted + volume_keywords.wait_for_volume_deleted ${volume_name} ${timeout} + +Cleanup Volumes + [Documentation] Delete all test-labeled PVCs + volume_keywords.cleanup_volumes diff --git a/harvester_robot_tests/libs/addon/addon.py b/harvester_robot_tests/libs/addon/addon.py index 33e3c0637..27ac3f43e 100644 --- a/harvester_robot_tests/libs/addon/addon.py +++ b/harvester_robot_tests/libs/addon/addon.py @@ -110,3 +110,7 @@ def verify_nvidia_toolkit_configuration( def get_nvidia_toolkit_configuration(self, addon_name): """Get nvidia-driver-toolkit addon configuration - delegates to implementation""" return self.addon.get_nvidia_toolkit_configuration(addon_name) + + def install_addon_yaml(self, url): + """Install an addon by applying a YAML manifest from a URL - delegates to implementation""" + return self.addon.install_addon_yaml(url) diff --git a/harvester_robot_tests/libs/addon/base.py b/harvester_robot_tests/libs/addon/base.py index 86f1d1ff7..063268ff2 100644 --- a/harvester_robot_tests/libs/addon/base.py +++ b/harvester_robot_tests/libs/addon/base.py @@ -66,3 +66,8 @@ def query_prometheus(self, query, prometheus_url): def verify_prometheus_metric_exists(self, query, prometheus_url, retries=3, retry_interval=5): """Verify that a Prometheus metric exists with retry logic""" pass + + @abstractmethod + def install_addon_yaml(self, url): + """Install an addon by applying a YAML manifest from a URL""" + pass diff --git a/harvester_robot_tests/libs/addon/crd.py b/harvester_robot_tests/libs/addon/crd.py index ae3d0a695..dd04e43eb 100644 --- a/harvester_robot_tests/libs/addon/crd.py +++ b/harvester_robot_tests/libs/addon/crd.py @@ -664,3 +664,39 @@ def get_nvidia_toolkit_configuration(self, addon_name): 'image_tag': image_values.get('tag') or spec.get('image/tag'), 'driver_location': parsed_values.get('driverLocation') or spec.get('driverLocation') } + + def install_addon_yaml(self, url): + """Install an addon by applying a YAML manifest from a URL. + This must be done BEFORE enabling, as some addons are not bundled in the Harvester ISO. + """ + logging(f"Installing addon from URL: {url}") + result = subprocess.run( + ["kubectl", "apply", "-f", url], + capture_output=True, text=True, timeout=60 + ) + if result.returncode != 0: + raise Exception(f"Failed to install addon YAML from {url}: {result.stderr}") + logging(f"Addon YAML applied: {result.stdout.strip()}") + + def _wait_for_addon_cr_exists(self, addon_name, namespace="harvester-system", timeout=60): + """Wait for the addon custom resource to exist after kubectl apply""" + endtime = time.time() + timeout + while time.time() < endtime: + try: + self.custom_api.get_namespaced_custom_object( + group=HARVESTER_API_GROUP, + version=HARVESTER_API_VERSION, + namespace=namespace, + plural="addons", + name=addon_name + ) + logging(f"Addon CR {addon_name} exists") + return + except ApiException as e: + if e.status == 404: + time.sleep(2) + continue + raise + raise AssertionError( + f"Addon CR {addon_name} not found within {timeout}s after kubectl apply" + ) diff --git a/harvester_robot_tests/libs/addon/rest.py b/harvester_robot_tests/libs/addon/rest.py index b2e5da432..e51df5aec 100644 --- a/harvester_robot_tests/libs/addon/rest.py +++ b/harvester_robot_tests/libs/addon/rest.py @@ -685,3 +685,6 @@ def get_nvidia_toolkit_configuration(self, addon_name): 'image_tag': image_values.get('tag') or spec.get('image/tag'), 'driver_location': parsed_values.get('driverLocation') or spec.get('driverLocation') } + + def install_addon_yaml(self, url): + raise NotImplementedError("REST implementation not available for Addon.install_yaml") diff --git a/harvester_robot_tests/libs/blockdevice/base.py b/harvester_robot_tests/libs/blockdevice/base.py index d1c6722af..798d105bf 100644 --- a/harvester_robot_tests/libs/blockdevice/base.py +++ b/harvester_robot_tests/libs/blockdevice/base.py @@ -30,3 +30,34 @@ def provision_longhorn_storage(self, name, engine_version, namespace): Return nothing, but raise exception if operation fails """ pass + + @abstractmethod + def identify_lvm_suitable(self, min_size_gib): + """Identify blockdevices >= min_size_gib suitable for LVM. + + Returns dict {node_name: disk_name} with one disk per node. + """ + pass + + @abstractmethod + def create_lvm_volume_groups(self, disk_by_node, vg_type): + """Create LVM volume groups on selected nodes. + + Returns dict {vg_name: node_name}. + """ + pass + + @abstractmethod + def provision_lvm_disk(self, disk_name, node_name, vg_name): + """Provision a blockdevice for LVM with specified volume group""" + pass + + @abstractmethod + def wait_for_vgs_active(self, vg_node_map, timeout): + """Wait for all volume groups in vg_node_map to become active""" + pass + + @abstractmethod + def cleanup_lvm_volume_groups(self, disk_by_node): + """Remove LVM provisioning from disks""" + pass diff --git a/harvester_robot_tests/libs/blockdevice/blockdevice.py b/harvester_robot_tests/libs/blockdevice/blockdevice.py index 3ca44954e..8718696e3 100644 --- a/harvester_robot_tests/libs/blockdevice/blockdevice.py +++ b/harvester_robot_tests/libs/blockdevice/blockdevice.py @@ -35,3 +35,20 @@ def get(self, name, namespace): def provision_longhorn_storage(self, name, engine_version, namespace): self.blockdevice.provision_longhorn_storage(name, engine_version, namespace) + + def identify_lvm_suitable(self, min_size_gib): + return self.blockdevice.identify_lvm_suitable(min_size_gib) + + def create_lvm_volume_groups(self, disk_by_node, vg_type): + return self.blockdevice.create_lvm_volume_groups(disk_by_node, vg_type) + + def provision_lvm_disk(self, disk_name, node_name, vg_name): + return self.blockdevice.provision_lvm_disk(disk_name, node_name, vg_name) + + def wait_for_vgs_active(self, vg_node_map, timeout=None): + if timeout is not None: + return self.blockdevice.wait_for_vgs_active(vg_node_map, timeout) + return self.blockdevice.wait_for_vgs_active(vg_node_map) + + def cleanup_lvm_volume_groups(self, disk_by_node): + return self.blockdevice.cleanup_lvm_volume_groups(disk_by_node) diff --git a/harvester_robot_tests/libs/blockdevice/crd.py b/harvester_robot_tests/libs/blockdevice/crd.py index 47bed2bee..d4362764a 100644 --- a/harvester_robot_tests/libs/blockdevice/crd.py +++ b/harvester_robot_tests/libs/blockdevice/crd.py @@ -3,11 +3,17 @@ Layer 4: Component and its implementation """ +import random +import time from kubernetes import client from kubernetes.client.rest import ApiException from crd import get_cr, patch_cr -from constant import HARVESTER_API_GROUP, HARVESTER_API_VERSION -from utility.utility import logging +from constant import ( + HARVESTER_API_GROUP, HARVESTER_API_VERSION, + DEFAULT_TIMEOUT, GIBIBYTE, + LONGHORN_SYSTEM_NAMESPACE, LVM_VG_DM_THIN, LVM_VG_STRIPED, +) +from utility.utility import logging, get_retry_count_and_interval from .base import Base @@ -24,6 +30,7 @@ def __init__(self): "plural": "blockdevices" } self.port_forward_process = None + _, self.retry_interval = get_retry_count_and_interval() def list(self, namespace): try: @@ -74,3 +81,156 @@ def provision_longhorn_storage(self, name, engine_version, namespace): ) except ApiException as e: raise Exception(f"Failed to provision blockdevice {namespace}/{name}: {e}") + + def identify_lvm_suitable(self, min_size_gib): + """Identify blockdevices >= min_size_gib suitable for LVM. + Returns dict {node_name: disk_name} with one disk per node. + """ + min_size_bytes = int(min_size_gib) * GIBIBYTE + + blockdevices = self.custom_api.list_namespaced_custom_object( + group=HARVESTER_API_GROUP, + version=HARVESTER_API_VERSION, + namespace=LONGHORN_SYSTEM_NAMESPACE, + plural="blockdevices" + ).get("items", []) + + disk_by_node = {} + for bd in blockdevices: + node_name = bd.get("spec", {}).get("nodeName", "") + if not node_name: + continue + + if bd.get("status", {}).get("provisionPhase", "") == "Provisioned": + continue + + try: + size_bytes = int( + bd.get("status", {}) + .get("deviceStatus", {}) + .get("capacity", {}) + .get("sizeBytes", 0) + ) + except (ValueError, TypeError): + size_bytes = 0 + + if size_bytes < min_size_bytes: + continue + + if node_name not in disk_by_node: + disk_name = bd.get("metadata", {}).get("name", "") + if disk_name: + disk_by_node[node_name] = disk_name + + logging(f"Suitable disks for LVM: {disk_by_node}") + return disk_by_node + + def create_lvm_volume_groups(self, disk_by_node, vg_type): + """Create LVM volume groups on selected nodes. + Single-node: creates the VG matching vg_type (dm-thin or striped). + Multi-node: always creates vg-dm-thin + vg-dm-striped on separate nodes. + Returns dict {vg_name: node_name}. + """ + _VG_TYPE_MAP = { + "dm-thin": LVM_VG_DM_THIN, + "striped": LVM_VG_STRIPED, + } + nodes = list(disk_by_node.keys()) + if not nodes: + raise AssertionError("No nodes with suitable disks found for LVM") + + vg_node_map = {} + + if len(nodes) >= 2: + selected = random.sample(nodes, 2) + else: + selected = nodes[:1] + + node1 = selected[0] + single_vg = _VG_TYPE_MAP.get(vg_type, LVM_VG_DM_THIN) + self.provision_lvm_disk(disk_by_node[node1], node1, single_vg) + vg_node_map[single_vg] = node1 + logging(f"Assigned {single_vg} to node {node1}") + + if len(selected) >= 2: + node2 = selected[1] + self.provision_lvm_disk(disk_by_node[node2], node2, LVM_VG_STRIPED) + vg_node_map[LVM_VG_STRIPED] = node2 + logging(f"Assigned {LVM_VG_STRIPED} to node {node2}") + + self.wait_for_vgs_active(vg_node_map) + return vg_node_map + + def provision_lvm_disk(self, disk_name, node_name, vg_name): + """Provision a blockdevice for LVM with specified volume group""" + logging(f"Provisioning disk {disk_name} on {node_name} for VG {vg_name}") + patch_body = { + "spec": { + "provision": True, + "provisioner": { + "lvm": { + "vgName": vg_name + } + } + } + } + try: + patch_cr( + group=HARVESTER_API_GROUP, + version=HARVESTER_API_VERSION, + namespace=LONGHORN_SYSTEM_NAMESPACE, + plural="blockdevices", + name=disk_name, + body=patch_body + ) + except ApiException as e: + raise Exception(f"Failed to provision disk {disk_name} for VG {vg_name}: {e}") + + def wait_for_vgs_active(self, vg_node_map, timeout=DEFAULT_TIMEOUT): + """Wait for all volume groups in vg_node_map to become active""" + endtime = time.time() + timeout + for vg_name, node_name in vg_node_map.items(): + while time.time() < endtime: + try: + blockdevices = self.custom_api.list_namespaced_custom_object( + group=HARVESTER_API_GROUP, + version=HARVESTER_API_VERSION, + namespace=LONGHORN_SYSTEM_NAMESPACE, + plural="blockdevices" + ).get("items", []) + + for bd in blockdevices: + bd_node = bd.get("spec", {}).get("nodeName", "") + bd_vg = (bd.get("spec", {}).get("provisioner", {}) + .get("lvm", {}).get("vgName", "")) + if bd_node == node_name and bd_vg == vg_name: + phase = bd.get("status", {}).get("provisionPhase", "") + state = bd.get("status", {}).get("state", "") + if phase == "Provisioned" and state == "Active": + logging(f"VG {vg_name} on {node_name} is active") + break + else: + time.sleep(self.retry_interval) + continue + break + except ApiException: + time.sleep(self.retry_interval) + else: + raise AssertionError(f"VG {vg_name} on {node_name} not active within {timeout}s") + + def cleanup_lvm_volume_groups(self, disk_by_node): + """Remove LVM provisioning from disks""" + for node, disk_name in disk_by_node.items(): + try: + patch_body = {"spec": {"provision": False, "provisioner": {}}} + patch_cr( + group=HARVESTER_API_GROUP, + version=HARVESTER_API_VERSION, + namespace=LONGHORN_SYSTEM_NAMESPACE, + plural="blockdevices", + name=disk_name, + body=patch_body + ) + logging(f"Cleaned up LVM provisioning for disk {disk_name} on {node}") + except ApiException as e: + logging(f"Error cleaning up disk {disk_name}: {e}") diff --git a/harvester_robot_tests/libs/blockdevice/rest.py b/harvester_robot_tests/libs/blockdevice/rest.py index 9a51efae6..967c79393 100644 --- a/harvester_robot_tests/libs/blockdevice/rest.py +++ b/harvester_robot_tests/libs/blockdevice/rest.py @@ -13,3 +13,33 @@ class Rest(Base): def __init__(self): self.api_client = get_harvester_api_client() self.port_forward_process = None + + def list(self, namespace): + raise NotImplementedError("REST implementation not available for Blockdevice.list") + + def get(self, name, namespace): + raise NotImplementedError("REST implementation not available for Blockdevice.get") + + def provision_longhorn_storage(self, name, engine_version, namespace): + raise NotImplementedError("REST implementation not available" + "for Blockdevice.provision_longhorn_storage") + + def identify_lvm_suitable(self, min_size_gib): + raise NotImplementedError("REST implementation not available" + "for Blockdevice.identify_lvm_suitable") + + def create_lvm_volume_groups(self, disk_by_node, vg_type): + raise NotImplementedError("REST implementation not available" + "for Blockdevice.create_lvm_volume_groups") + + def provision_lvm_disk(self, disk_name, node_name, vg_name): + raise NotImplementedError("REST implementation not available" + "for Blockdevice.provision_lvm_disk") + + def wait_for_vgs_active(self, vg_node_map, timeout): + raise NotImplementedError("REST implementation not available" + "for Blockdevice.wait_for_vgs_active") + + def cleanup_lvm_volume_groups(self, disk_by_node): + raise NotImplementedError("REST implementation not available" + "for Blockdevice.cleanup_lvm_volume_groups") diff --git a/harvester_robot_tests/libs/constant.py b/harvester_robot_tests/libs/constant.py index 668efeb59..086ccffbd 100644 --- a/harvester_robot_tests/libs/constant.py +++ b/harvester_robot_tests/libs/constant.py @@ -17,6 +17,8 @@ class HarvesterOperationStrategy(Enum): HARVESTER_API_VERSION = "v1beta1" KUBEVIRT_API_GROUP = "kubevirt.io" KUBEVIRT_API_VERSION = "v1" +SNAPSHOT_API_GROUP = "snapshot.kubevirt.io" +SNAPSHOT_API_VERSION = "v1beta1" LONGHORN_API_GROUP = "longhorn.io" LONGHORN_API_VERSION = "v1beta2" @@ -43,13 +45,19 @@ class HarvesterOperationStrategy(Enum): TEBIBYTE = (GIBIBYTE * KIBIBYTE) LARGE_DISK_BYTE = TEBIBYTE -# VM States +# VM Related VM_STATE_STOPPED = "Stopped" VM_STATE_RUNNING = "Running" VM_STATE_STARTING = "Starting" VM_STATE_STOPPING = "Stopping" VM_STATE_MIGRATING = "Migrating" +DATA_PATH = "/mnt/test-data" +DATA_FILE = "testdata.bin" + +# Mount point used when formatting and mounting an LVM data disk inside a VM +BLOCK_DEVICE_MOUNT = "/mnt/vdb_drive" + # Volume States VOLUME_STATE_BOUND = "Bound" VOLUME_STATE_PENDING = "Pending" @@ -126,8 +134,8 @@ class HarvesterOperationStrategy(Enum): RANCHER_WAIT_TIMEOUT = 1800 # 30 minutes for cluster operations RANCHER_NAMESPACE = "fleet-default" -# Cloud-init user data for RKE2 nodes -DEFAULT_RKE2_USER_DATA = """#cloud-config +# Cloud-init user data +DEFAULT_USER_DATA = """#cloud-config password: password chpasswd: expire: false @@ -159,3 +167,9 @@ class HarvesterOperationStrategy(Enum): DEFAULT_RKE2_NODE_CPUS = 4 DEFAULT_RKE2_NODE_MEMORY = 8 # GB DEFAULT_RKE2_NODE_DISK = 80 # GB + +# LVM constants +LVM_PROVISIONER = "lvm.driver.harvesterhci.io" +LVM_VG_DM_THIN = "vg-dm-thin" +LVM_VG_STRIPED = "vg-dm-striped" +LONGHORN_SYSTEM_NAMESPACE = "longhorn-system" diff --git a/harvester_robot_tests/libs/keywords/addon_keywords.py b/harvester_robot_tests/libs/keywords/addon_keywords.py index 2f6febfe8..2275eefe5 100644 --- a/harvester_robot_tests/libs/keywords/addon_keywords.py +++ b/harvester_robot_tests/libs/keywords/addon_keywords.py @@ -294,3 +294,8 @@ def get_nvidia_toolkit_addon_config(self, addon_name): """ logging(f'Getting nvidia-driver-toolkit addon configuration for {addon_name}') return self.addon.get_nvidia_toolkit_configuration(addon_name) + + def install_addon_yaml(self, url): + """Install an addon by applying a YAML manifest from the given URL""" + logging(f'Installing addon from {url}') + return self.addon.install_addon_yaml(url) diff --git a/harvester_robot_tests/libs/keywords/common_keywords.py b/harvester_robot_tests/libs/keywords/common_keywords.py index f64d12303..1d8856d43 100644 --- a/harvester_robot_tests/libs/keywords/common_keywords.py +++ b/harvester_robot_tests/libs/keywords/common_keywords.py @@ -42,18 +42,6 @@ def cleanup_images(self): from image import Image Image().cleanup() - def cleanup_volumes(self): - """Cleanup volumes""" - logging('Cleanup volumes requested') - - def cleanup_networks(self): - """Cleanup networks""" - logging('Cleanup networks requested') - - def cleanup_backups(self): - """Cleanup backups""" - logging('Cleanup backups requested') - def list_pods_by_label(self, namespace, label_selector, status=None): """List pods by label""" pods = get_pods_by_label(namespace, label_selector) diff --git a/harvester_robot_tests/libs/keywords/network_keywords.py b/harvester_robot_tests/libs/keywords/network_keywords.py index 1ceb1e4d9..2dfa2c435 100644 --- a/harvester_robot_tests/libs/keywords/network_keywords.py +++ b/harvester_robot_tests/libs/keywords/network_keywords.py @@ -73,6 +73,17 @@ def delete_vlan_config(self, name): logging(f"Deleting VLAN config: {name}") self.network.delete_vlan_config(name) + def wait_for_vlan_config_deleted(self, name, timeout=120): + """ + Wait for VLAN config to be fully deleted. + + Args: + name: VLAN config name + timeout: Wait timeout in seconds + """ + logging(f"Waiting for VLAN config {name} to be deleted") + self.network.wait_for_vlan_config_deleted(name, int(timeout)) + def wait_for_cluster_network_ready(self, name, timeout=120): """ Wait for cluster network to become ready. diff --git a/harvester_robot_tests/libs/keywords/rancher_keywords.py b/harvester_robot_tests/libs/keywords/rancher_keywords.py index c865cce5a..0f2c012be 100644 --- a/harvester_robot_tests/libs/keywords/rancher_keywords.py +++ b/harvester_robot_tests/libs/keywords/rancher_keywords.py @@ -11,7 +11,7 @@ from utility.utility import logging, generate_name_with_suffix # noqa E402 from rancher import Rancher # noqa E402 -from constant import DEFAULT_TIMEOUT, DEFAULT_TIMEOUT_LONG, DEFAULT_RKE2_USER_DATA, DEFAULT_RKE2_NODE_CPUS, DEFAULT_RKE2_NODE_MEMORY, DEFAULT_RKE2_NODE_DISK # noqa E402 +from constant import DEFAULT_TIMEOUT, DEFAULT_TIMEOUT_LONG, DEFAULT_USER_DATA, DEFAULT_RKE2_NODE_CPUS, DEFAULT_RKE2_NODE_MEMORY, DEFAULT_RKE2_NODE_DISK # noqa E402 class rancher_keywords: @@ -511,7 +511,7 @@ def create_harvester_config(self, name, cpus=None, mems=None, disks=None, image_ if not disks: disks = DEFAULT_RKE2_NODE_DISK if not user_data: - user_data = DEFAULT_RKE2_USER_DATA + user_data = DEFAULT_USER_DATA logging(f"Creating Harvester config: {name}") return self.rancher.create_harvester_config( name, int(cpus), int(mems), int(disks), diff --git a/harvester_robot_tests/libs/keywords/setting_keywords.py b/harvester_robot_tests/libs/keywords/setting_keywords.py index d3ec95c0e..f76994321 100644 --- a/harvester_robot_tests/libs/keywords/setting_keywords.py +++ b/harvester_robot_tests/libs/keywords/setting_keywords.py @@ -42,3 +42,13 @@ def get_condition_message(self, setting_id: str, condition_type: str): def enable(self, setting_id): setting = self.setting.enable(setting_id) return setting + + def configure_csi_driver(self, setting_id, provisioner, snapshot_class): + """Configure a CSI driver entry in the csi-driver-config setting""" + logging(f'Configuring CSI driver for provisioner {provisioner}') + return self.setting.configure_csi_driver(setting_id, provisioner, snapshot_class) + + def remove_csi_driver(self, setting_id, provisioner): + """Remove a CSI driver entry from the csi-driver-config setting""" + logging(f'Removing CSI driver entry for provisioner {provisioner}') + return self.setting.remove_csi_driver(setting_id, provisioner) diff --git a/harvester_robot_tests/libs/keywords/snapshot_keywords.py b/harvester_robot_tests/libs/keywords/snapshot_keywords.py new file mode 100644 index 000000000..7ddd4ef4f --- /dev/null +++ b/harvester_robot_tests/libs/keywords/snapshot_keywords.py @@ -0,0 +1,54 @@ +""" +Snapshot Keywords - Layer 3 wrapper around the Snapshot component. +Exposes snapshot lifecycle operations (take, wait, delete) as Robot Framework keywords. +Restore operations remain in vm_keywords because they create/modify VMs. +""" +import os +import sys + +sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), '../'))) +from utility.utility import logging # noqa E402 +from snapshot import Snapshot # noqa E402 +from constant import DEFAULT_TIMEOUT, DEFAULT_NAMESPACE # noqa E402 + + +class snapshot_keywords: + """Snapshot keyword wrapper - creates Snapshot component and delegates""" + + def __init__(self): + self._snapshot = None + + @property + def snapshot(self): + """Lazy initialize Snapshot to allow API client setup first""" + if self._snapshot is None: + self._snapshot = Snapshot() + return self._snapshot + + def take_vm_snapshot(self, vm_name, namespace=DEFAULT_NAMESPACE): + """Take a snapshot of the given VM; returns the snapshot name""" + logging(f'Taking snapshot of VM {vm_name}') + return self.snapshot.take_snapshot(vm_name, namespace) + + def wait_for_vm_snapshot_ready(self, snapshot_name, namespace=DEFAULT_NAMESPACE, + timeout=DEFAULT_TIMEOUT): + """Wait until the snapshot is marked readyToUse""" + logging(f'Waiting for snapshot {snapshot_name} to be ready') + self.snapshot.wait_for_snapshot_ready(snapshot_name, namespace, timeout) + + def delete_vm_snapshot(self, snapshot_name, namespace=DEFAULT_NAMESPACE): + """Delete a VM snapshot""" + logging(f'Deleting snapshot {snapshot_name}') + self.snapshot.delete_snapshot(snapshot_name, namespace) + + def restore_snapshot_to_new_vm(self, snapshot_name, new_vm_name, + namespace=DEFAULT_NAMESPACE): + """Restore a snapshot to a newly created VM""" + logging(f'Restoring snapshot {snapshot_name} to new VM {new_vm_name}') + self.snapshot.restore_snapshot_to_new_vm(snapshot_name, new_vm_name, namespace) + + def restore_snapshot_to_existing_vm(self, snapshot_name, vm_name, + namespace=DEFAULT_NAMESPACE): + """Restore a snapshot to an existing (stopped) VM""" + logging(f'Restoring snapshot {snapshot_name} to existing VM {vm_name}') + self.snapshot.restore_snapshot_to_existing_vm(snapshot_name, vm_name, namespace) diff --git a/harvester_robot_tests/libs/keywords/storage_keywords.py b/harvester_robot_tests/libs/keywords/storage_keywords.py index 28e0d3a5a..e2b7d7dcc 100644 --- a/harvester_robot_tests/libs/keywords/storage_keywords.py +++ b/harvester_robot_tests/libs/keywords/storage_keywords.py @@ -11,6 +11,7 @@ from constant import LARGE_DISK_BYTE # noqa E402 from utility.utility import logging # noqa E402 from blockdevice import Blockdevice # noqa E402 +from storageclass import StorageClass # noqa E402 from host_keywords import host_keywords # noqa E402 @@ -20,6 +21,7 @@ class storage_keywords: def __init__(self): """Initialize storage keywords with lazy loading""" self._blockdevice = None + self._sc = None self._host = None @property @@ -36,6 +38,13 @@ def host(self): self._host = host_keywords() return self._host + @property + def sc(self): + """Lazy initialize StorageClass to allow API client setup first""" + if self._sc is None: + self._sc = StorageClass() + return self._sc + def list_blockdevices(self, namespace): return self.blockdevice.list(namespace) @@ -118,3 +127,35 @@ def get_lh_node_disk_status_condition(self, node_name, disk_name, condition_type for condition in disk_status.get("conditions", []): if condition.get("type") == condition_type: return condition.get("status") + + # LVM-specific methods + def identify_lvm_suitable_disks(self, min_size_gib): + """Identify disks suitable for LVM on all nodes (>= min_size_gib GiB)""" + logging(f'Identifying LVM-suitable disks >= {min_size_gib} GiB') + return self.blockdevice.identify_lvm_suitable(int(min_size_gib)) + + def create_lvm_volume_groups(self, disk_by_node, vg_type): + """Create volume groups from identified disks""" + logging(f'Creating LVM volume groups from {disk_by_node} type={vg_type}') + return self.blockdevice.create_lvm_volume_groups(dict(disk_by_node), str(vg_type)) + + def get_node_for_vg(self, vg_name, vg_node_map): + """Get the node associated with a volume group""" + node = dict(vg_node_map).get(vg_name, "") + logging(f'Node for VG {vg_name}: {node}') + return node + + def create_lvm_storage_class(self, sc_name, vg_name, vg_type, node): + """Create an LVM StorageClass""" + logging(f'Creating LVM StorageClass {sc_name} vg={vg_name} type={vg_type} node={node}') + return self.sc.create_lvm_sc(sc_name, vg_name, vg_type, node) + + def delete_lvm_storage_class(self, sc_name): + """Delete an LVM StorageClass""" + logging(f'Deleting LVM StorageClass {sc_name}') + return self.sc.delete(sc_name) + + def cleanup_lvm_volume_groups(self, disk_by_node): + """Remove LVM volume groups from disks""" + logging(f'Cleaning up LVM volume groups for {disk_by_node}') + return self.blockdevice.cleanup_lvm_volume_groups(dict(disk_by_node)) diff --git a/harvester_robot_tests/libs/keywords/vm_keywords.py b/harvester_robot_tests/libs/keywords/vm_keywords.py index 56d98e252..dec734fb9 100644 --- a/harvester_robot_tests/libs/keywords/vm_keywords.py +++ b/harvester_robot_tests/libs/keywords/vm_keywords.py @@ -9,6 +9,7 @@ sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), '../'))) # noqa E402 from utility.utility import logging # noqa E402 from vm import VM # noqa E402 +from snapshot import Snapshot # noqa E402 from constant import DEFAULT_TIMEOUT, DEFAULT_TIMEOUT_LONG, DEFAULT_NAMESPACE # noqa E402 @@ -17,6 +18,14 @@ class vm_keywords: def __init__(self): self.vm = VM() + self._snapshot = None + + @property + def snapshot(self): + """Lazy initialize Snapshot to allow API client setup first""" + if self._snapshot is None: + self._snapshot = Snapshot() + return self._snapshot def cleanup_vms(self): """Clean up all test VMs""" @@ -62,6 +71,14 @@ def migrate_vm(self, vm_name, target_node): logging(f'Migrating VM {vm_name} to {target_node}') self.vm.migrate(vm_name, target_node) + def is_vm_running(self, vm_name): + """Return True if VM is in running state""" + return self.vm.is_running(vm_name) + + def is_vm_stopped(self, vm_name): + """Return True if VM is in stopped state""" + return self.vm.is_stopped(vm_name) + def wait_for_vm_running(self, vm_name, timeout=DEFAULT_TIMEOUT): """Wait for VM to reach running state""" logging(f'Waiting for VM {vm_name} to be running') @@ -115,3 +132,55 @@ def wait_for_backup_completed(self, vm_name, backup_name, timeout=DEFAULT_TIMEOU """Wait for backup to complete""" logging(f'Waiting for backup {backup_name} to complete') self.vm.wait_for_backup_completed(vm_name, backup_name, timeout) + + def create_vm_with_volume_using_sc(self, vm_name, sc_name, image_name, network_name=None): + """Create a VM with Harvester image root disk and LVM StorageClass as additional volume""" + logging(f'Creating VM {vm_name} with LVM SC {sc_name} and image {image_name}') + return self.vm.create_vm_with_volume_using_sc(vm_name, sc_name, image_name, + network_name=network_name) + + def attach_volume_to_vm(self, vm_name, vol_name): + """Attach an volume to a VM""" + logging(f'Attaching volume {vol_name} to VM {vm_name}') + return self.vm.attach_volume(vm_name, vol_name) + + def check_block_device_in_vm(self, vm_name, expected_disk_size=None): + """Assert the data block device is visible in the VM and return its path""" + logging(f'Checking block device is available in VM {vm_name}') + return self.vm.check_block_device(vm_name, expected_disk_size=expected_disk_size) + + def write_data_and_get_checksum_on_disk(self, vm_name, device, format_device=True): + """Write data to the given device inside the VM and return md5sum checksum""" + logging(f'Writing data to {device} on VM {vm_name}') + return self.vm.write_data_and_get_checksum_on_disk(vm_name, device, format_device) + + def delete_data_from_vm(self, vm_name): + """Delete test data from VM""" + logging(f'Deleting data from VM {vm_name}') + return self.vm.delete_data(vm_name) + + def mount_data_disk_in_vm(self, vm_name): + """Mount the data disk in VM after a restore""" + logging(f'Mounting data disk in VM {vm_name}') + return self.vm.mount_data_disk(vm_name) + + def get_data_checksum_from_vm(self, vm_name): + """Get md5sum checksum of data in VM. Raises if VM not ready or result is invalid.""" + import re + logging(f'Getting data checksum from VM {vm_name}') + result = self.vm.get_data_checksum(vm_name) + if not result or not re.match(r'^[0-9a-f]{32}$', result): + raise AssertionError( + f"No valid md5sum from VM {vm_name!r} (got: {result!r:.120})" + ) + return result + + def expand_volume_via_vm_edit(self, vm_name, vol_name, new_size): + """Expand a volume by patching the VM spec""" + logging(f'Expanding volume {vol_name} on VM {vm_name} to {new_size}') + return self.vm.expand_volume_via_vm_edit(vm_name, vol_name, new_size) + + def get_volume_size_inside_vm(self, vm_name, expected_size): + """Verify volume size inside VM matches expected; returns True/False""" + logging(f'Verifying volume size inside VM {vm_name} is {expected_size}') + return self.vm.verify_volume_size(vm_name, expected_size) diff --git a/harvester_robot_tests/libs/keywords/volume_keywords.py b/harvester_robot_tests/libs/keywords/volume_keywords.py index 79f683f83..f27cbdad1 100644 --- a/harvester_robot_tests/libs/keywords/volume_keywords.py +++ b/harvester_robot_tests/libs/keywords/volume_keywords.py @@ -22,7 +22,7 @@ def cleanup_volumes(self): """Clean up all test volumes""" self.volume.cleanup() - def create_volume(self, volume_name, size="2Gi", numberOfReplicas=3, frontend="blockdev", **kwargs): # NOQA + def create_volume(self, volume_name, size="2Gi", numberOfReplicas=None, frontend=None, **kwargs): # NOQA """Create a volume""" logging(f'Creating volume {volume_name}') self.volume.create(volume_name, size, numberOfReplicas, frontend, **kwargs) @@ -86,3 +86,9 @@ def restore_volume_from_snapshot(self, volume_name, snapshot_name, new_volume_na """Restore volume from snapshot""" logging(f'Restoring volume from snapshot {snapshot_name}') self.volume.restore_from_snapshot(volume_name, snapshot_name, new_volume_name) + + # LVM PVC methods + def expand_lvm_pvc(self, vol_name, new_size): + """Expand an LVM PVC to a new size""" + logging(f'Expanding LVM PVC {vol_name} to {new_size}') + return self.volume.expand_pvc(vol_name, new_size) diff --git a/harvester_robot_tests/libs/network/base.py b/harvester_robot_tests/libs/network/base.py index fcf62dd66..0358aec79 100644 --- a/harvester_robot_tests/libs/network/base.py +++ b/harvester_robot_tests/libs/network/base.py @@ -53,6 +53,16 @@ def delete_vlan_config(self, name): """ pass + @abstractmethod + def wait_for_vlan_config_deleted(self, name, timeout): + """Wait for VLAN config to be fully deleted. + + Args: + name: VLAN config name + timeout: Wait timeout in seconds + """ + pass + @abstractmethod def wait_for_cluster_network_ready(self, name, timeout): """Wait for cluster network to become ready. diff --git a/harvester_robot_tests/libs/network/crd.py b/harvester_robot_tests/libs/network/crd.py index 09cb4f31d..e0d1b82bd 100644 --- a/harvester_robot_tests/libs/network/crd.py +++ b/harvester_robot_tests/libs/network/crd.py @@ -132,6 +132,25 @@ def delete_vlan_config(self, name): f"Failed to delete VLAN config: {e}" ) + def wait_for_vlan_config_deleted(self, name, timeout=120): + """Wait for a VlanConfig CR to be fully removed""" + logging(f"Waiting for VLAN config {name} to be deleted") + deadline = time.time() + int(timeout) + while time.time() < deadline: + try: + self.custom_api.get_cluster_custom_object( + group="network.harvesterhci.io", + version="v1beta1", + plural="vlanconfigs", + name=name + ) + except ApiException as e: + if e.status == 404: + logging(f"VLAN config {name} deleted") + return + time.sleep(5) + raise Exception(f"VLAN config {name} not deleted within {timeout}s") + def wait_for_cluster_network_ready(self, name, timeout=120): """Wait for cluster network to become ready""" logging(f"Waiting for cluster network {name} to be ready") diff --git a/harvester_robot_tests/libs/network/network.py b/harvester_robot_tests/libs/network/network.py index 59e363acf..5ae190dec 100644 --- a/harvester_robot_tests/libs/network/network.py +++ b/harvester_robot_tests/libs/network/network.py @@ -56,6 +56,10 @@ def delete_vlan_config(self, name): """Delete VLAN config""" return self.network.delete_vlan_config(name) + def wait_for_vlan_config_deleted(self, name, timeout=120): + """Wait for VLAN config to be fully deleted""" + return self.network.wait_for_vlan_config_deleted(name, timeout) + def wait_for_cluster_network_ready(self, name, timeout=120): """Wait for cluster network to become ready""" return self.network.wait_for_cluster_network_ready(name, timeout) diff --git a/harvester_robot_tests/libs/network/rest.py b/harvester_robot_tests/libs/network/rest.py index 7f23ee53c..b4788e71d 100644 --- a/harvester_robot_tests/libs/network/rest.py +++ b/harvester_robot_tests/libs/network/rest.py @@ -116,7 +116,19 @@ def delete_vlan_config(self, name): logging(f"Deleted VLAN config: {name}") - def wait_for_cluster_network_ready(self, name, timeout=120): + def wait_for_vlan_config_deleted(self, name, timeout=120): + """Wait for a VlanConfig to be fully removed""" + logging(f"Waiting for VLAN config {name} to be deleted") + deadline = time.time() + int(timeout) + while time.time() < deadline: + code, _ = self.harvester_api.get( + f"v1/harvester/network.harvesterhci.io.vlanconfigs/{name}" + ) + if code == 404: + logging(f"VLAN config {name} deleted") + return + time.sleep(5) + raise Exception(f"VLAN config {name} not deleted within {timeout}s") """Wait for cluster network to become ready""" logging(f"Waiting for cluster network {name} to be ready") deadline = time.time() + int(timeout) diff --git a/harvester_robot_tests/libs/setting/base.py b/harvester_robot_tests/libs/setting/base.py index 6a37e7676..305f07b7a 100644 --- a/harvester_robot_tests/libs/setting/base.py +++ b/harvester_robot_tests/libs/setting/base.py @@ -18,3 +18,13 @@ def get(self, setting_id): def enable(self, setting_id): """Enable a setting""" pass + + @abstractmethod + def configure_csi_driver(self, setting_id, provisioner, snapshot_class): + """Configure csi-driver-config to add a provisioner entry""" + pass + + @abstractmethod + def remove_csi_driver(self, setting_id, provisioner): + """Remove a provisioner entry from csi-driver-config""" + pass diff --git a/harvester_robot_tests/libs/setting/crd.py b/harvester_robot_tests/libs/setting/crd.py index 26ff23f7a..07cbcca00 100644 --- a/harvester_robot_tests/libs/setting/crd.py +++ b/harvester_robot_tests/libs/setting/crd.py @@ -3,12 +3,14 @@ Layer 4: Component and its implementation """ +import json from kubernetes import client from kubernetes.client.rest import ApiException from crd import get_cr, patch_cr from constant import ( HARVESTER_API_GROUP, HARVESTER_API_VERSION, ) +from utility.utility import logging from .base import Base @@ -54,3 +56,89 @@ def enable(self, setting_id): return setting except ApiException as e: raise Exception(f"Failed to enable setting {setting_id}: {e}") + + def configure_csi_driver(self, setting_id, provisioner, snapshot_class): + """Configure csi-driver-config to add a provisioner entry. + Preserves existing entries by reading value or default fallback. + """ + logging(f"Configuring csi-driver-config for provisioner={provisioner}") + try: + setting = get_cr( + group=HARVESTER_API_GROUP, + version=HARVESTER_API_VERSION, + namespace="", + plural="settings", + name=setting_id + ) + except ApiException: + setting = self.custom_api.get_cluster_custom_object( + group=HARVESTER_API_GROUP, + version=HARVESTER_API_VERSION, + plural="settings", + name=setting_id + ) + + current_value = setting.get("value") or setting.get("default", "{}") + try: + config = json.loads(current_value) if current_value else {} + except (json.JSONDecodeError, TypeError): + config = {} + + if provisioner not in config: + config[provisioner] = {} + config[provisioner]["volumeSnapshotClassName"] = snapshot_class + + patch_body = {"value": json.dumps(config)} + try: + self.custom_api.patch_cluster_custom_object( + group=HARVESTER_API_GROUP, + version=HARVESTER_API_VERSION, + plural="settings", + name=setting_id, + body=patch_body + ) + logging(f"csi-driver-config updated for {provisioner}") + except ApiException as e: + raise Exception(f"Failed to update csi-driver-config: {e}") + + def remove_csi_driver(self, setting_id, provisioner): + """Remove a provisioner entry from csi-driver-config setting.""" + logging(f"Removing csi-driver-config entry for provisioner={provisioner}") + try: + setting = get_cr( + group=HARVESTER_API_GROUP, + version=HARVESTER_API_VERSION, + namespace="", + plural="settings", + name=setting_id + ) + except ApiException: + setting = self.custom_api.get_cluster_custom_object( + group=HARVESTER_API_GROUP, + version=HARVESTER_API_VERSION, + plural="settings", + name=setting_id + ) + + current_value = setting.get("value") or setting.get("default", "{}") + try: + config = json.loads(current_value) if current_value else {} + except (json.JSONDecodeError, TypeError): + config = {} + + if provisioner in config: + del config[provisioner] + patch_body = {"value": json.dumps(config)} + try: + self.custom_api.patch_cluster_custom_object( + group=HARVESTER_API_GROUP, + version=HARVESTER_API_VERSION, + plural="settings", + name=setting_id, + body=patch_body + ) + logging(f"csi-driver-config entry removed for {provisioner}") + except ApiException as e: + raise Exception(f"Failed to update csi-driver-config: {e}") + else: + logging(f"No csi-driver-config entry found for {provisioner}, skipping") diff --git a/harvester_robot_tests/libs/setting/rest.py b/harvester_robot_tests/libs/setting/rest.py index dcf3b551e..210cba397 100644 --- a/harvester_robot_tests/libs/setting/rest.py +++ b/harvester_robot_tests/libs/setting/rest.py @@ -13,3 +13,17 @@ class Rest(Base): def __init__(self): self.api_client = get_harvester_api_client() self.port_forward_process = None + + def get(self, setting_id): + raise NotImplementedError("REST implementation not available for Setting.get") + + def enable(self, setting_id): + raise NotImplementedError("REST implementation not available for Setting.enable") + + def configure_csi_driver(self, setting_id, provisioner, snapshot_class): + raise NotImplementedError("REST implementation not available" + "for Setting.configure_csi_driver") + + def remove_csi_driver(self, setting_id, provisioner): + raise NotImplementedError("REST implementation not available" + "for Setting.remove_csi_driver") diff --git a/harvester_robot_tests/libs/setting/setting.py b/harvester_robot_tests/libs/setting/setting.py index ef3e8da2a..f09dee731 100644 --- a/harvester_robot_tests/libs/setting/setting.py +++ b/harvester_robot_tests/libs/setting/setting.py @@ -33,3 +33,11 @@ def enable(self, setting_id): """Enable a setting - delegates to implementation""" setting = self.setting.enable(setting_id) return setting + + def configure_csi_driver(self, setting_id, provisioner, snapshot_class): + """Configure csi-driver-config setting - delegates to implementation""" + return self.setting.configure_csi_driver(setting_id, provisioner, snapshot_class) + + def remove_csi_driver(self, setting_id, provisioner): + """Remove a provisioner entry from csi-driver-config - delegates to implementation""" + return self.setting.remove_csi_driver(setting_id, provisioner) diff --git a/harvester_robot_tests/libs/snapshot/__init__.py b/harvester_robot_tests/libs/snapshot/__init__.py new file mode 100644 index 000000000..c29e977ad --- /dev/null +++ b/harvester_robot_tests/libs/snapshot/__init__.py @@ -0,0 +1,4 @@ +"""Snapshot module - Layer 4 component""" +from .snapshot import Snapshot + +__all__ = ["Snapshot"] diff --git a/harvester_robot_tests/libs/snapshot/base.py b/harvester_robot_tests/libs/snapshot/base.py new file mode 100644 index 000000000..e04662a01 --- /dev/null +++ b/harvester_robot_tests/libs/snapshot/base.py @@ -0,0 +1,35 @@ +"""Snapshot Component: Base Class + +Layer 4: Component and its implementation +""" + +from abc import ABC, abstractmethod + + +class Base(ABC): + """Base class for Snapshot implementations""" + + @abstractmethod + def take_snapshot(self, vm_name, namespace): + """Take a VM snapshot; returns snapshot_name""" + pass + + @abstractmethod + def wait_for_snapshot_ready(self, snapshot_name, namespace, timeout): + """Wait for a snapshot to be ready""" + pass + + @abstractmethod + def delete_snapshot(self, snapshot_name, namespace): + """Delete a snapshot""" + pass + + @abstractmethod + def restore_snapshot_to_new_vm(self, snapshot_name, new_vm_name, namespace): + """Restore a snapshot to a new VM""" + pass + + @abstractmethod + def restore_snapshot_to_existing_vm(self, snapshot_name, vm_name, namespace): + """Restore a snapshot to an existing VM (must be stopped)""" + pass diff --git a/harvester_robot_tests/libs/snapshot/crd.py b/harvester_robot_tests/libs/snapshot/crd.py new file mode 100644 index 000000000..0012ac696 --- /dev/null +++ b/harvester_robot_tests/libs/snapshot/crd.py @@ -0,0 +1,194 @@ +"""Snapshot Component: CRD Implementation +Layer 4: Component and its implementation +""" + +import time +from kubernetes import client +from kubernetes.client.rest import ApiException +from crd import create_cr, delete_cr +from constant import ( + HARVESTER_API_GROUP, HARVESTER_API_VERSION, + KUBEVIRT_API_GROUP, KUBEVIRT_API_VERSION, + DEFAULT_NAMESPACE, DEFAULT_TIMEOUT, + LABEL_TEST, LABEL_TEST_VALUE, +) +from utility.utility import logging, get_retry_count_and_interval, generate_name_with_suffix +from .base import Base + + +class CRD(Base): + """CRD implementation for VM Snapshot operations via Harvester VirtualMachineBackup""" + + def __init__(self): + self.custom_api = client.CustomObjectsApi() + _, self.retry_interval = get_retry_count_and_interval() + + def take_snapshot(self, vm_name, namespace=DEFAULT_NAMESPACE): + """Take a VM snapshot via Harvester VirtualMachineBackup; returns snapshot_name""" + snapshot_name = generate_name_with_suffix(vm_name, "snap") + logging(f"Taking snapshot {snapshot_name} of VM {vm_name}") + + # Fetch VM UID for ownerReference (best-effort) + try: + vm = self.custom_api.get_namespaced_custom_object( + group=KUBEVIRT_API_GROUP, version=KUBEVIRT_API_VERSION, + namespace=namespace, plural="virtualmachines", name=vm_name + ) + vm_uid = vm.get("metadata", {}).get("uid", "") + except ApiException: + vm_uid = "" + + body = { + "apiVersion": f"{HARVESTER_API_GROUP}/{HARVESTER_API_VERSION}", + "kind": "VirtualMachineBackup", + "metadata": { + "name": snapshot_name, + "namespace": namespace, + "labels": {LABEL_TEST: LABEL_TEST_VALUE}, + "ownerReferences": [{ + "apiVersion": f"{KUBEVIRT_API_GROUP}/v1", + "kind": "VirtualMachine", + "name": vm_name, + "uid": vm_uid + }] + }, + "spec": { + "type": "snapshot", + "source": { + "apiGroup": KUBEVIRT_API_GROUP, + "kind": "VirtualMachine", + "name": vm_name + } + } + } + + try: + create_cr( + group=HARVESTER_API_GROUP, + version=HARVESTER_API_VERSION, + namespace=namespace, + plural="virtualmachinebackups", + body=body + ) + except ApiException as e: + raise Exception(f"Failed to take snapshot of VM {vm_name}: {e}") + + self.wait_for_snapshot_ready(snapshot_name, namespace) + return snapshot_name + + def wait_for_snapshot_ready(self, snapshot_name, namespace=DEFAULT_NAMESPACE, + timeout=DEFAULT_TIMEOUT): + """Wait for a VirtualMachineBackup snapshot to be ready""" + endtime = time.time() + timeout + while time.time() < endtime: + try: + snap = self.custom_api.get_namespaced_custom_object( + group=HARVESTER_API_GROUP, + version=HARVESTER_API_VERSION, + namespace=namespace, + plural="virtualmachinebackups", + name=snapshot_name + ) + if snap.get("status", {}).get("readyToUse", False): + logging(f"Snapshot {snapshot_name} is ready") + return + except ApiException: + pass + time.sleep(self.retry_interval) + + raise AssertionError(f"Snapshot {snapshot_name} not ready within {timeout}s") + + def delete_snapshot(self, snapshot_name, namespace=DEFAULT_NAMESPACE): + """Delete a VirtualMachineBackup snapshot""" + try: + delete_cr( + group=HARVESTER_API_GROUP, + version=HARVESTER_API_VERSION, + namespace=namespace, + plural="virtualmachinebackups", + name=snapshot_name + ) + logging(f"Snapshot {snapshot_name} deleted") + except ApiException as e: + if e.status != 404: + logging(f"Error deleting snapshot {snapshot_name}: {e}") + + def restore_snapshot_to_new_vm(self, snapshot_name, new_vm_name, + namespace=DEFAULT_NAMESPACE): + """Restore a Harvester VirtualMachineBackup snapshot to a new VM""" + restore_name = f"{new_vm_name}-restore" + logging(f"Restoring snapshot {snapshot_name} to new VM {new_vm_name}") + body = { + "apiVersion": f"{HARVESTER_API_GROUP}/{HARVESTER_API_VERSION}", + "kind": "VirtualMachineRestore", + "metadata": { + "name": restore_name, + "namespace": namespace, + "labels": {LABEL_TEST: LABEL_TEST_VALUE} + }, + "spec": { + "target": { + "apiGroup": KUBEVIRT_API_GROUP, + "kind": "VirtualMachine", + "name": new_vm_name + }, + "virtualMachineBackupName": snapshot_name, + "virtualMachineBackupNamespace": namespace, + "newVM": True + } + } + try: + create_cr(group=HARVESTER_API_GROUP, version=HARVESTER_API_VERSION, + namespace=namespace, plural="virtualmachinerestores", body=body) + except ApiException as e: + raise Exception(f"Failed to restore snapshot to new VM: {e}") + self._wait_for_restore_complete(restore_name, namespace) + + def restore_snapshot_to_existing_vm(self, snapshot_name, vm_name, + namespace=DEFAULT_NAMESPACE): + """Restore a Harvester VirtualMachineBackup snapshot to an existing (stopped) VM""" + restore_name = f"{vm_name}-restore-{int(time.time())}" + logging(f"Restoring snapshot {snapshot_name} to existing VM {vm_name}") + body = { + "apiVersion": f"{HARVESTER_API_GROUP}/{HARVESTER_API_VERSION}", + "kind": "VirtualMachineRestore", + "metadata": { + "name": restore_name, + "namespace": namespace, + "labels": {LABEL_TEST: LABEL_TEST_VALUE} + }, + "spec": { + "target": { + "apiGroup": KUBEVIRT_API_GROUP, + "kind": "VirtualMachine", + "name": vm_name + }, + "virtualMachineBackupName": snapshot_name, + "virtualMachineBackupNamespace": namespace, + "deletionPolicy": "retain" + } + } + try: + create_cr(group=HARVESTER_API_GROUP, version=HARVESTER_API_VERSION, + namespace=namespace, plural="virtualmachinerestores", body=body) + except ApiException as e: + raise Exception(f"Failed to restore snapshot to existing VM: {e}") + self._wait_for_restore_complete(restore_name, namespace) + + def _wait_for_restore_complete(self, restore_name, namespace=DEFAULT_NAMESPACE, + timeout=DEFAULT_TIMEOUT): + """Poll until the Harvester VirtualMachineRestore reports complete""" + endtime = time.time() + timeout + while time.time() < endtime: + try: + restore = self.custom_api.get_namespaced_custom_object( + group=HARVESTER_API_GROUP, version=HARVESTER_API_VERSION, + namespace=namespace, plural="virtualmachinerestores", name=restore_name + ) + if restore.get("status", {}).get("complete", False): + logging(f"Restore {restore_name} completed") + return + except ApiException: + pass + time.sleep(self.retry_interval) + raise AssertionError(f"Restore {restore_name} not complete within {timeout}s") diff --git a/harvester_robot_tests/libs/snapshot/rest.py b/harvester_robot_tests/libs/snapshot/rest.py new file mode 100644 index 000000000..c36ba1e2a --- /dev/null +++ b/harvester_robot_tests/libs/snapshot/rest.py @@ -0,0 +1,33 @@ +"""Snapshot Component: REST Implementation (stub) + +Layer 4: Component and its implementation +""" + +from utility.utility import get_harvester_api_client +from .base import Base + + +class Rest(Base): + """REST implementation stub for Snapshot operations""" + + def __init__(self): + self.api_client = get_harvester_api_client() + + def take_snapshot(self, vm_name, namespace): + raise NotImplementedError("REST implementation not available for Snapshot.take") + + def wait_for_snapshot_ready(self, snapshot_name, namespace, timeout): + raise NotImplementedError("REST implementation not available" + "for Snapshot.wait_for_snapshot_ready") + + def delete_snapshot(self, snapshot_name, namespace): + raise NotImplementedError("REST implementation not available" + "for Snapshot.delete_snapshot") + + def restore_snapshot_to_new_vm(self, snapshot_name, new_vm_name, namespace): + raise NotImplementedError("REST implementation not available" + "for Snapshot.restore_to_new_vm") + + def restore_snapshot_to_existing_vm(self, snapshot_name, vm_name, namespace): + raise NotImplementedError("REST implementation not available" + "for Snapshot.restore_to_existing_vm") diff --git a/harvester_robot_tests/libs/snapshot/snapshot.py b/harvester_robot_tests/libs/snapshot/snapshot.py new file mode 100644 index 000000000..533e4c342 --- /dev/null +++ b/harvester_robot_tests/libs/snapshot/snapshot.py @@ -0,0 +1,40 @@ +"""Snapshot Component + +Layer 4: Component and its implementation +""" + +import os +from constant import HarvesterOperationStrategy +from .base import Base +from .crd import CRD +from .rest import Rest + + +class Snapshot(Base): + def __init__(self): + try: + strategy_str = os.getenv("HARVESTER_OPERATION_STRATEGY", "crd").lower() + self._strategy = HarvesterOperationStrategy(strategy_str) + except ValueError: + self._strategy = HarvesterOperationStrategy.CRD + + match self._strategy: + case HarvesterOperationStrategy.CRD: + self.snapshot = CRD() + case HarvesterOperationStrategy.REST: + self.snapshot = Rest() + + def take_snapshot(self, vm_name, namespace): + return self.snapshot.take_snapshot(vm_name, namespace) + + def wait_for_snapshot_ready(self, snapshot_name, namespace, timeout): + return self.snapshot.wait_for_snapshot_ready(snapshot_name, namespace, timeout) + + def delete_snapshot(self, snapshot_name, namespace): + return self.snapshot.delete_snapshot(snapshot_name, namespace) + + def restore_snapshot_to_new_vm(self, snapshot_name, new_vm_name, namespace): + return self.snapshot.restore_snapshot_to_new_vm(snapshot_name, new_vm_name, namespace) + + def restore_snapshot_to_existing_vm(self, snapshot_name, vm_name, namespace): + return self.snapshot.restore_snapshot_to_existing_vm(snapshot_name, vm_name, namespace) diff --git a/harvester_robot_tests/libs/storageclass/__init__.py b/harvester_robot_tests/libs/storageclass/__init__.py new file mode 100644 index 000000000..19989c2f7 --- /dev/null +++ b/harvester_robot_tests/libs/storageclass/__init__.py @@ -0,0 +1,4 @@ +"""StorageClass module - Layer 4 component""" +from .storageclass import StorageClass + +__all__ = ["StorageClass"] diff --git a/harvester_robot_tests/libs/storageclass/base.py b/harvester_robot_tests/libs/storageclass/base.py new file mode 100644 index 000000000..f1d5d3bdd --- /dev/null +++ b/harvester_robot_tests/libs/storageclass/base.py @@ -0,0 +1,25 @@ +"""StorageClass Component: Base Class + +Layer 4: Component and its implementation +""" + +from abc import ABC, abstractmethod + + +class Base(ABC): + """Base class for StorageClass implementations""" + + @abstractmethod + def create_lvm_sc(self, sc_name, vg_name, vg_type, node): + """Create an LVM StorageClass""" + pass + + @abstractmethod + def delete(self, sc_name): + """Delete a StorageClass""" + pass + + @abstractmethod + def get_node(self, sc_name): + """Get the node parameter from a StorageClass""" + pass diff --git a/harvester_robot_tests/libs/storageclass/crd.py b/harvester_robot_tests/libs/storageclass/crd.py new file mode 100644 index 000000000..1a335a44b --- /dev/null +++ b/harvester_robot_tests/libs/storageclass/crd.py @@ -0,0 +1,82 @@ +"""StorageClass Component: CRD Implementation + +Layer 4: Makes actual Kubernetes API calls +""" + +from kubernetes import client +from kubernetes.client.rest import ApiException +from constant import LABEL_TEST, LABEL_TEST_VALUE, LVM_PROVISIONER +from utility.utility import logging +from .base import Base + + +class CRD(Base): + """CRD implementation for StorageClass operations using Kubernetes API""" + + def __init__(self): + self.storage_api = client.StorageV1Api() + + def create_lvm_sc(self, sc_name, vg_name, vg_type, node): + """Create an LVM StorageClass""" + logging(f"Creating LVM StorageClass: {sc_name}") + + # Map internal VG-type names to the values the CSI driver accepts + _SC_TYPE_MAP = { + "dm-thin": "dm-thin", + "striped": "striped", + } + sc_type = _SC_TYPE_MAP.get(vg_type, vg_type) + + body = client.V1StorageClass( + api_version="storage.k8s.io/v1", + kind="StorageClass", + metadata=client.V1ObjectMeta( + name=sc_name, + labels={LABEL_TEST: LABEL_TEST_VALUE} + ), + provisioner=LVM_PROVISIONER, + parameters={ + "node": node, + "vgName": vg_name, + "type": sc_type + }, + reclaim_policy="Delete", + volume_binding_mode="WaitForFirstConsumer", + allow_volume_expansion=True, + allowed_topologies=[ + client.V1TopologySelectorTerm( + match_label_expressions=[ + client.V1TopologySelectorLabelRequirement( + key="topology.lvm.csi/node", + values=[node] + ) + ] + ) + ] + ) + + try: + self.storage_api.create_storage_class(body=body) + logging(f"StorageClass {sc_name} created") + except ApiException as e: + if e.status == 409: + logging(f"StorageClass {sc_name} already exists") + else: + raise Exception(f"Failed to create StorageClass {sc_name}: {e}") + + def delete(self, sc_name): + """Delete a StorageClass""" + try: + self.storage_api.delete_storage_class(name=sc_name) + logging(f"StorageClass {sc_name} deleted") + except ApiException as e: + if e.status != 404: + logging(f"Error deleting StorageClass {sc_name}: {e}") + + def get_node(self, sc_name): + """Get the node parameter from a StorageClass""" + try: + sc = self.storage_api.read_storage_class(name=sc_name) + return sc.parameters.get("node", "") + except ApiException: + return "" diff --git a/harvester_robot_tests/libs/storageclass/rest.py b/harvester_robot_tests/libs/storageclass/rest.py new file mode 100644 index 000000000..c4ff2ec4e --- /dev/null +++ b/harvester_robot_tests/libs/storageclass/rest.py @@ -0,0 +1,24 @@ +"""StorageClass Component: REST Implementation (stub) + +Layer 4: Component and its implementation +""" + +from utility.utility import get_harvester_api_client +from .base import Base + + +class Rest(Base): + """REST implementation stub for StorageClass operations""" + + def __init__(self): + self.api_client = get_harvester_api_client() + + def create_lvm_sc(self, sc_name, vg_name, vg_type, node): + raise NotImplementedError( + "REST implementation not available for StorageClass.create_lvm_sc") + + def delete(self, sc_name): + raise NotImplementedError("REST implementation not available for StorageClass.delete") + + def get_node(self, sc_name): + raise NotImplementedError("REST implementation not available for StorageClass.get_node") diff --git a/harvester_robot_tests/libs/storageclass/storageclass.py b/harvester_robot_tests/libs/storageclass/storageclass.py new file mode 100644 index 000000000..afdf08f5a --- /dev/null +++ b/harvester_robot_tests/libs/storageclass/storageclass.py @@ -0,0 +1,34 @@ +"""StorageClass Component + +Layer 4: Component and its implementation +""" + +import os +from constant import HarvesterOperationStrategy +from .base import Base +from .crd import CRD +from .rest import Rest + + +class StorageClass(Base): + def __init__(self): + try: + strategy_str = os.getenv("HARVESTER_OPERATION_STRATEGY", "crd").lower() + self._strategy = HarvesterOperationStrategy(strategy_str) + except ValueError: + self._strategy = HarvesterOperationStrategy.CRD + + match self._strategy: + case HarvesterOperationStrategy.CRD: + self.sc = CRD() + case HarvesterOperationStrategy.REST: + self.sc = Rest() + + def create_lvm_sc(self, sc_name, vg_name, vg_type, node): + return self.sc.create_lvm_sc(sc_name, vg_name, vg_type, node) + + def delete(self, sc_name): + return self.sc.delete(sc_name) + + def get_node(self, sc_name): + return self.sc.get_node(sc_name) diff --git a/harvester_robot_tests/libs/vm/base.py b/harvester_robot_tests/libs/vm/base.py index 6592f7630..9156b5e07 100644 --- a/harvester_robot_tests/libs/vm/base.py +++ b/harvester_robot_tests/libs/vm/base.py @@ -67,3 +67,50 @@ def get_status(self, vm_name): def cleanup(self): """Clean up test resources""" pass + + @abstractmethod + def attach_volume(self, vm_name, vol_name, namespace=DEFAULT_NAMESPACE): + """Attach an existing PVC to a VM (stop, patch, start)""" + pass + + @abstractmethod + def is_running(self, vm_name, namespace=DEFAULT_NAMESPACE): + """Return True if VM is in Running state""" + pass + + @abstractmethod + def is_stopped(self, vm_name, namespace=DEFAULT_NAMESPACE): + """Return True if VM is stopped (no VMI)""" + pass + + @abstractmethod + def write_data_and_get_checksum_on_disk( + self, vm_name, device, format_device=True, + namespace=DEFAULT_NAMESPACE): + """Write data to the given device inside VM and return md5sum""" + pass + + @abstractmethod + def mount_data_disk(self, vm_name, namespace=DEFAULT_NAMESPACE): + """Mount the data disk after a restore""" + pass + + @abstractmethod + def delete_data(self, vm_name, namespace=DEFAULT_NAMESPACE): + """Delete test data from VM""" + pass + + @abstractmethod + def get_data_checksum(self, vm_name, namespace=DEFAULT_NAMESPACE): + """Return md5sum checksum of test data in VM""" + pass + + @abstractmethod + def expand_volume_via_vm_edit(self, vm_name, vol_name, new_size, namespace=DEFAULT_NAMESPACE): + """Expand volume by patching the VM volumeClaimTemplates annotation""" + pass + + @abstractmethod + def verify_volume_size(self, vm_name, expected_size, namespace=DEFAULT_NAMESPACE): + """Verify volume size inside VM matches expected (uses lsblk)""" + pass diff --git a/harvester_robot_tests/libs/vm/crd.py b/harvester_robot_tests/libs/vm/crd.py index d7a130faf..5edc08269 100644 --- a/harvester_robot_tests/libs/vm/crd.py +++ b/harvester_robot_tests/libs/vm/crd.py @@ -1,21 +1,56 @@ """ VM CRD Implementation """ +import ast import json +import re import time from kubernetes import client from kubernetes.client.rest import ApiException -from crd import get_cr, create_cr, delete_cr, list_cr, wait_for_cr_deleted +from kubernetes.stream import stream +from crd import get_cr, create_cr, delete_cr, list_cr, wait_for_cr_deleted, patch_cr from constant import ( KUBEVIRT_API_GROUP, KUBEVIRT_API_VERSION, VIRTUALMACHINE_PLURAL, VIRTUALMACHINEINSTANCE_PLURAL, DEFAULT_NAMESPACE, LABEL_TEST, LABEL_TEST_VALUE, - DEFAULT_TIMEOUT_SHORT + DEFAULT_TIMEOUT, DEFAULT_TIMEOUT_SHORT, + VM_STATE_RUNNING, VM_STATE_STOPPED, + RUN_STRATEGY_RERUN_ON_FAILURE, + DATA_FILE, + DEFAULT_USER_DATA, ) from utility.utility import logging, get_retry_count_and_interval from vm.base import Base +def _parse_virsh_output(raw): + """Parse virsh qemu-agent-command output into a Python dict. + """ + # First pass: find the first valid JSON object in the raw string. + decoder = json.JSONDecoder() + idx = 0 + while idx != -1: + idx = raw.find("{", idx) + if idx == -1: + break + try: + obj, _ = decoder.raw_decode(raw, idx) + return obj + except json.JSONDecodeError: + idx += 1 + + # Fallback: some kubernetes client versions return str(dict) which uses + # single-quoted Python syntax. ast.literal_eval handles this safely. + try: + result = ast.literal_eval(raw.strip()) + if isinstance(result, dict): + return result + except (ValueError, SyntaxError): + pass + + raise ValueError(f"No JSON object found in stream output: {raw!r}") + + class CRD(Base): """ VM CRD implementation - uses Kubernetes Custom Resources @@ -25,6 +60,7 @@ def __init__(self): """Initialize CRD client.""" self.obj_api = client.CustomObjectsApi() self.core_api = client.CoreV1Api() + self.storage_api = client.StorageV1Api() self.retry_count, self.retry_interval = ( get_retry_count_and_interval() ) @@ -54,10 +90,24 @@ def _create_virtual_machine( self, vm_name, cpu, memory, image_id, namespace, **kwargs): """Create VM matching Harvester's exact structure.""" + # Optional node affinity — either explicit node_name or derived from sc_name + node_name = kwargs.get('node_name') or '' + sc_name = kwargs.get('sc_name') or '' + if sc_name and not node_name: + node_name = self._get_sc_node(sc_name) + + # Optional secondary VLAN NIC + network_name = kwargs.get('network_name') or '' + interfaces = [{"masquerade": {}, "model": "virtio", "name": "default"}] + networks = [{"name": "default", "pod": {}}] + if network_name: + interfaces.append({"bridge": {}, "model": "virtio", "name": "vlan-nic"}) + networks.append({ + "name": "vlan-nic", + "multus": {"networkName": f"default/{network_name}"} + }) + # Look up the image's actual storage class from Harvester. - # Since v1.8.0 (harvester#5165), storage classes use lh- - # instead of longhorn-. We read status.storageClassName - # which works for both old and new Harvester versions. try: img_obj = self.obj_api.get_namespaced_custom_object( group="harvesterhci.io", @@ -139,6 +189,10 @@ def _create_virtual_machine( "threads": 1 }, "devices": { + "disks": [ + {"bootOrder": 1, "disk": {"bus": "virtio"}, "name": "disk-0"}, + {"disk": {"bus": "virtio"}, "name": "cloudinit"} + ], "inputs": [ { "bus": "usb", @@ -146,13 +200,7 @@ def _create_virtual_machine( "type": "tablet" } ], - "interfaces": [ - { - "masquerade": {}, - "model": "virtio", - "name": "default" - } - ] + "interfaces": interfaces }, "features": { "acpi": {"enabled": True} @@ -174,8 +222,12 @@ def _create_virtual_machine( }, "evictionStrategy": "LiveMigrateIfPossible", "hostname": vm_name, - "networks": [ - {"name": "default", "pod": {}} + "networks": networks, + "volumes": [ + {"name": "disk-0", + "persistentVolumeClaim": {"claimName": f"{vm_name}-disk-0"}}, + {"name": "cloudinit", + "cloudInitNoCloud": {"userData": DEFAULT_USER_DATA}} ], "terminationGracePeriodSeconds": 120 } @@ -183,6 +235,21 @@ def _create_virtual_machine( } } + if node_name: + body["spec"]["template"]["spec"]["affinity"] = { + "nodeAffinity": { + "requiredDuringSchedulingIgnoredDuringExecution": { + "nodeSelectorTerms": [{ + "matchExpressions": [{ + "key": "kubernetes.io/hostname", + "operator": "In", + "values": [node_name] + }] + }] + } + } + } + try: create_cr( group=KUBEVIRT_API_GROUP, @@ -556,3 +623,570 @@ def cleanup(self): logging(f'Error deleting VM {vm_name}: {e}', 'WARNING') except Exception as e: logging(f'Error during VM cleanup: {e}', 'WARNING') + + def create_for_lvm(self, vm_name, sc_name, image_id, network_name=None): + """Create a VM with a Harvester image root disk and node affinity from the StorageClass""" + node_name = self._get_sc_node(sc_name) + self._create_vm_with_image(vm_name, image_id, node_name=node_name, + network_name=network_name) + logging(f"VM {vm_name} created") + + def create_vm_with_volume_using_sc(self, vm_name, sc_name, image_id, network_name=None): + """Create a VM with a Harvester image root disk and + an additional LVM StorageClass volume""" + node_name = self._get_sc_node(sc_name) + self._create_vm_with_image(vm_name, image_id, node_name=node_name, + sc=sc_name, network_name=network_name) + logging(f"VM {vm_name} with LVM volume created") + + def _get_sc_node(self, sc_name): + """Get the node parameter from a StorageClass""" + try: + sc = self.storage_api.read_storage_class(name=sc_name) + return sc.parameters.get("node", "") + except ApiException: + return "" + + def _get_image_storage_class(self, image_id, namespace=DEFAULT_NAMESPACE): + """Look up the Harvester image's storageClassName from its status""" + try: + img_obj = self.obj_api.get_namespaced_custom_object( + group="harvesterhci.io", + version="v1beta1", + namespace=namespace, + plural="virtualmachineimages", + name=image_id + ) + storage_class = img_obj.get("status", {}).get("storageClassName", "") + if not storage_class: + raise Exception(f"Image {image_id} has no storageClassName in status") + logging(f"Resolved storage class for image {image_id}: {storage_class}") + return storage_class + except ApiException as e: + raise Exception(f"Failed to look up image {image_id}: {e}") + + def _create_vm_with_image( + self, vm_name, image_id, namespace=DEFAULT_NAMESPACE, + node_name=None, sc=None, network_name=None): + """Create a VM using a Harvester image root disk, + with optional node affinity and additional volume""" + storage_class = self._get_image_storage_class(image_id, namespace) + + volume_claim_templates = [ + { + "metadata": { + "name": f"{vm_name}-disk-0", + "annotations": { + "harvesterhci.io/imageId": f"{namespace}/{image_id}" + } + }, + "spec": { + "accessModes": ["ReadWriteMany"], + "resources": {"requests": {"storage": "10Gi"}}, + "volumeMode": "Block", + "storageClassName": storage_class + } + } + ] + disks = [ + {"name": "disk-0", "disk": {"bus": "virtio"}, "bootOrder": 1}, + {"name": "cloudinitdisk", "disk": {"bus": "virtio"}} + ] + volumes = [ + {"name": "disk-0", "persistentVolumeClaim": {"claimName": f"{vm_name}-disk-0"}}, + { + "name": "cloudinitdisk", + "cloudInitNoCloud": { + "userData": ( + "#cloud-config\n" + "password: password\n" + "chpasswd:\n" + " expire: false\n" + "ssh_pwauth: true\n" + ) + } + } + ] + + if sc: + disks.append({"name": "lvm-disk", "disk": {"bus": "virtio"}}) + volumes.append({ + "name": "lvm-disk", + "persistentVolumeClaim": {"claimName": f"{vm_name}-lvm-disk"} + }) + volume_claim_templates.append({ + "metadata": {"name": f"{vm_name}-lvm-disk", "namespace": namespace}, + "spec": { + "accessModes": ["ReadWriteOnce"], + "storageClassName": sc, + "resources": {"requests": {"storage": "5Gi"}}, + "volumeMode": "Filesystem" + } + }) + + body = { + "apiVersion": f"{KUBEVIRT_API_GROUP}/{KUBEVIRT_API_VERSION}", + "kind": "VirtualMachine", + "metadata": { + "name": vm_name, + "namespace": namespace, + "annotations": { + "harvesterhci.io/vmRunStrategy": RUN_STRATEGY_RERUN_ON_FAILURE, + "harvesterhci.io/volumeClaimTemplates": json.dumps(volume_claim_templates), + "harvesterhci.io/sshNames": "[]" + }, + "labels": { + LABEL_TEST: LABEL_TEST_VALUE, + "harvesterhci.io/creator": "robot-framework" + } + }, + "spec": { + "runStrategy": RUN_STRATEGY_RERUN_ON_FAILURE, + "template": { + "metadata": { + "labels": {"harvesterhci.io/vmName": vm_name}, + "annotations": {"harvesterhci.io/sshNames": "[]"} + }, + "spec": { + "domain": { + "cpu": {"cores": 2, "sockets": 1, "threads": 1}, + "devices": { + "disks": disks, + "interfaces": [{"masquerade": {}, + "model": "virtio", "name": "default"}] + + ([{"bridge": {}, "model": "virtio", + "name": "vlan-nic"}] if network_name else []) + }, + "machine": {"type": "q35"}, + "memory": {"guest": "4Gi"}, + "resources": { + "limits": {"cpu": "2", "memory": "4Gi"}, + "requests": {"cpu": "125m", "memory": "2730Mi"} + } + }, + "networks": [{"name": "default", "pod": {}}] + + ([{"name": "vlan-nic", + "multus": {"networkName": f"default/{network_name}"}}] + if network_name else []), + "volumes": volumes, + "hostname": vm_name + } + } + } + } + + if node_name: + body["spec"]["template"]["spec"]["affinity"] = { + "nodeAffinity": { + "requiredDuringSchedulingIgnoredDuringExecution": { + "nodeSelectorTerms": [{ + "matchExpressions": [{ + "key": "kubernetes.io/hostname", + "operator": "In", + "values": [node_name] + }] + }] + } + } + } + + try: + create_cr( + group=KUBEVIRT_API_GROUP, + version=KUBEVIRT_API_VERSION, + namespace=namespace, + plural="virtualmachines", + body=body + ) + except ApiException as e: + raise Exception(f"Failed to create VM {vm_name}: {e}") + + def attach_volume(self, vm_name, vol_name, namespace=DEFAULT_NAMESPACE): + """Attach an existing PVC to a VM (stop, patch volumes, start)""" + logging(f"Attaching volume {vol_name} to VM {vm_name}") + self.stop(vm_name) + self._wait_for_lvm_vm_state(vm_name, VM_STATE_STOPPED, namespace) + + vm = get_cr( + group=KUBEVIRT_API_GROUP, + version=KUBEVIRT_API_VERSION, + namespace=namespace, + plural="virtualmachines", + name=vm_name + ) + spec = vm.get("spec", {}).get("template", {}).get("spec", {}) + disks = spec.get("domain", {}).get("devices", {}).get("disks", []) + volumes_list = spec.get("volumes", []) + + disks.append({"name": vol_name, "disk": {"bus": "virtio"}}) + volumes_list.append({"name": vol_name, "persistentVolumeClaim": {"claimName": vol_name}}) + + patch_body = { + "spec": { + "template": { + "spec": { + "domain": {"devices": {"disks": disks}}, + "volumes": volumes_list + } + } + } + } + try: + patch_cr( + group=KUBEVIRT_API_GROUP, + version=KUBEVIRT_API_VERSION, + namespace=namespace, + plural="virtualmachines", + name=vm_name, + body=patch_body + ) + except ApiException as e: + raise Exception(f"Failed to attach volume {vol_name} to VM {vm_name}: {e}") + + self.start(vm_name) + self._wait_for_lvm_vm_state(vm_name, VM_STATE_RUNNING, namespace) + + def is_running(self, vm_name, namespace=DEFAULT_NAMESPACE): + """Return True if VM is in Running state""" + try: + vmi = self.obj_api.get_namespaced_custom_object( + group=KUBEVIRT_API_GROUP, + version=KUBEVIRT_API_VERSION, + namespace=namespace, + plural="virtualmachineinstances", + name=vm_name + ) + return vmi.get("status", {}).get("phase", "") == "Running" + except ApiException: + return False + + def is_stopped(self, vm_name, namespace=DEFAULT_NAMESPACE): + """Return True if VM is stopped (no VMI exists)""" + try: + self.obj_api.get_namespaced_custom_object( + group=KUBEVIRT_API_GROUP, + version=KUBEVIRT_API_VERSION, + namespace=namespace, + plural="virtualmachineinstances", + name=vm_name + ) + return False + except ApiException as e: + return e.status == 404 + + def _wait_for_lvm_vm_state(self, vm_name, expected_state, namespace=DEFAULT_NAMESPACE, + timeout=DEFAULT_TIMEOUT): + """Wait for VM to reach expected state""" + endtime = time.time() + timeout + while time.time() < endtime: + if expected_state == VM_STATE_RUNNING and self.is_running(vm_name, namespace): + return + if expected_state == VM_STATE_STOPPED and self.is_stopped(vm_name, namespace): + return + time.sleep(self.retry_interval) + raise AssertionError(f"VM {vm_name} did not reach state " + f"{expected_state} within {timeout}s") + + def _get_vmi_ip(self, vm_name, namespace=DEFAULT_NAMESPACE): + """Get the IP address of a running VMI""" + try: + vmi = self.obj_api.get_namespaced_custom_object( + group=KUBEVIRT_API_GROUP, + version=KUBEVIRT_API_VERSION, + namespace=namespace, + plural="virtualmachineinstances", + name=vm_name + ) + for iface in vmi.get("status", {}).get("interfaces", []): + ip = iface.get("ipAddress", "") + if ip: + return ip + except ApiException as e: + raise Exception(f"Failed to get VMI IP for {vm_name}: {e}") + raise Exception(f"No IP address found for VM {vm_name}") + + def _get_virt_launcher_pod(self, vm_name, namespace=DEFAULT_NAMESPACE): + """Get the virt-launcher pod name for a VM""" + try: + pods = self.core_api.list_namespaced_pod( + namespace=namespace, + label_selector=f"harvesterhci.io/vmName={vm_name}" + ) + for pod in pods.items: + if pod.metadata.name.startswith("virt-launcher") and pod.status.phase == "Running": + return pod.metadata.name + except ApiException: + pass + return None + + def _wait_for_guest_agent(self, vm_name, namespace=DEFAULT_NAMESPACE, + timeout=DEFAULT_TIMEOUT_SHORT): + """Wait until the QEMU guest agent reports as connected""" + logging(f"Waiting for guest agent on VM {vm_name}") + deadline = time.time() + timeout + while time.time() < deadline: + try: + vmi = self.obj_api.get_namespaced_custom_object( + group=KUBEVIRT_API_GROUP, + version=KUBEVIRT_API_VERSION, + namespace=namespace, + plural=VIRTUALMACHINEINSTANCE_PLURAL, + name=vm_name + ) + for cond in vmi.get("status", {}).get("conditions", []): + if (cond.get("type") == "AgentConnected" + and cond.get("status") == "True"): + logging(f"Guest agent connected on VM {vm_name}") + return + except ApiException: + pass + time.sleep(self.retry_interval) + raise AssertionError( + f"Guest agent not connected on VM {vm_name} within {timeout}s" + ) + + def _exec_in_vm(self, vm_name, namespace, commands): + """Execute commands inside a VM via the qemu guest agent in the virt-launcher pod. + + Uses qemu:///session (the correct URI inside virt-launcher) and the + domain name format '{namespace}_{vm_name}' that KubeVirt assigns. + Polls guest-exec-status until the command exits and returns stdout. + """ + pod_name = self._get_virt_launcher_pod(vm_name, namespace) + if not pod_name: + raise Exception(f"No virt-launcher pod found for VM {vm_name}") + + # KubeVirt names the libvirt domain _ + domain = f"{namespace}_{vm_name}" + full_command = " && ".join(commands) + logging(f"Executing in VM {vm_name} via pod {pod_name}: {full_command}") + + # Wait for the guest agent to be ready before sending any commands. + # The agent may not be connected immediately after a VM restart. + self._wait_for_guest_agent(vm_name, namespace) + + # Step 1: start the command via guest-exec and obtain its PID + exec_payload = json.dumps({ + "execute": "guest-exec", + "arguments": { + "path": "/bin/sh", + "arg": ["-c", full_command], + "capture-output": True + } + }) + start_result = stream( + self.core_api.connect_get_namespaced_pod_exec, + pod_name, namespace, + command=["virsh", "qemu-agent-command", domain, exec_payload], + stderr=True, stdin=False, stdout=True, tty=False + ) + pid = _parse_virsh_output(start_result)["return"]["pid"] + logging(f"Guest exec PID {pid} started in VM {vm_name}") + + # Step 2: poll guest-exec-status until the command exits (up to 120 s) + status_payload = json.dumps({ + "execute": "guest-exec-status", + "arguments": {"pid": pid} + }) + status_cmd = ["virsh", "qemu-agent-command", domain, status_payload] + for _ in range(120): + time.sleep(1) + status_result = stream( + self.core_api.connect_get_namespaced_pod_exec, + pod_name, namespace, + command=status_cmd, + stderr=True, stdin=False, stdout=True, tty=False + ) + status = _parse_virsh_output(status_result)["return"] + if status.get("exited"): + import base64 + stdout = "" + if status.get("out-data"): + stdout = base64.b64decode(status["out-data"]).decode() + exit_code = status.get("exitcode", 0) + if exit_code != 0: + stderr = "" + if status.get("err-data"): + stderr = base64.b64decode(status["err-data"]).decode() + raise Exception( + f"Command failed (exit {exit_code}) in VM {vm_name}.\n" + f"CMD : {full_command}\n" + f"STDOUT: {stdout}\nSTDERR: {stderr}" + ) + return stdout + + raise Exception( + f"Command timed out after 120s in VM {vm_name}: {full_command}" + ) + + def _find_data_disk(self, vm_name, namespace=DEFAULT_NAMESPACE, expected_size=None): + """Dynamically find the data disk device inside the VM. + + Runs lsblk to list raw disk devices, excludes vda (the root/image disk), + and returns the first remaining virtio disk whose size matches expected_size + (e.g. '5Gi'). When expected_size is None the size filter is skipped. + """ + if expected_size: + # Convert Kubernetes size notation to lsblk SIZE column format: + # '5Gi' -> '5G', '512Mi' -> '512M', '1Ti' -> '1T' + # Due to filesystem overhead Filesystem volume size is smaller than the + # requested PVC capacity, so callers should omit expected_size. + lsblk_size = expected_size.rstrip('i') + cmd = ( + f"lsblk -d -n -o NAME,SIZE " + f"| awk '/^vd/ && !/^vda/ && $2==\"{lsblk_size}\" {{print $1}}' " + f"| head -1" + ) + else: + # Skip vda (root) and disks in the K/M range (e.g. cloud-init ISO ~1M). + # Only return a disk whose size is reported in G/T range. + cmd = ( + "lsblk -d -n -o NAME,SIZE " + "| awk '/^vd/ && !/^vda/ && $2 ~ /[GT]$/ {print $1}' " + "| head -1" + ) + + output = self._exec_in_vm(vm_name, namespace, [cmd]) + device_name = output.strip() + if not device_name: + size_hint = f" with size {expected_size}" if expected_size else "" + raise AssertionError( + f"No data disk found in VM {vm_name}: " + f"no non-root virtio disk{size_hint} visible in lsblk" + ) + device = f"/dev/{device_name}" + logging(f"Discovered data disk {device} in VM {vm_name}") + return device + + def check_block_device(self, vm_name, namespace=DEFAULT_NAMESPACE, expected_disk_size=None): + """Assert a data disk (non-root virtio block device) is visible in the VM. + + When expected_disk_size is provided (e.g. '5Gi') the check also verifies + the disk matches that size, avoiding confusion with other attached disks. + + Returns the discovered device path (e.g. '/dev/vdb') for use by callers. + """ + logging(f"Checking data block device is visible in VM {vm_name}") + device = self._find_data_disk(vm_name, namespace, expected_disk_size) + logging(f"Data block device {device} confirmed in VM {vm_name}") + return device + + def write_data_and_get_checksum_on_disk( + self, vm_name, device, format_device=True, namespace=DEFAULT_NAMESPACE): + """Write data to the given device inside the VM and return its md5sum. + + Formats the device to ext4 when format_device=True (Block volumes). + Filesystem volumes are pre-formatted by the CSI driver; pass + format_device=False to skip mkfs and mount directly. + + """ + logging(f"Writing data to {device} on VM {vm_name}") + mount_point = f"/mnt/{vm_name}" + commands = [] + if format_device: + commands.append(f"mkfs.ext4 {device}") + commands += [ + f"mkdir -p {mount_point}", + f"mount {device} {mount_point}", + f"dd if=/dev/urandom of={mount_point}/{DATA_FILE} bs=1M count=100", + f"md5sum {mount_point}/{DATA_FILE} | awk '{{print $1}}'" + ] + output = self._exec_in_vm(vm_name, namespace, commands).strip() + # mkfs prints multi-line noise; extract just the 32-char md5sum hash + match = re.search(r'\b([0-9a-f]{32})\b', output) + return match.group(1) if match else output + + def mount_data_disk(self, vm_name, namespace=DEFAULT_NAMESPACE): + """Mount the data disk inside the VM after a restore. + + After a snapshot restore the data disk exists but is not mounted. + This re-mounts it at BLOCK_DEVICE_MOUNT so subsequent md5sum checks + can read the data. + """ + logging(f"Mounting data disk in VM {vm_name}") + mount_point = f"/mnt/{vm_name}" + device = self._find_data_disk(vm_name, namespace) + commands = [ + f"mkdir -p {mount_point}", + f"mount {device} {mount_point} || true", + ] + self._exec_in_vm(vm_name, namespace, commands) + logging(f"Data disk {device} mounted at {mount_point} in VM {vm_name}") + + def delete_data(self, vm_name, namespace=DEFAULT_NAMESPACE): + """Delete test data from VM""" + mount_point = f"/mnt/{vm_name}" + commands = [f"rm -f {mount_point}/{DATA_FILE}"] + self._exec_in_vm(vm_name, namespace, commands) + + def get_data_checksum(self, vm_name, namespace=DEFAULT_NAMESPACE): + """Return md5sum checksum of test data in VM""" + mount_point = f"/mnt/{vm_name}" + commands = [f"md5sum {mount_point}/{DATA_FILE} | awk '{{print $1}}'"] + output = self._exec_in_vm(vm_name, namespace, commands).strip() + match = re.search(r'\b([0-9a-f]{32})\b', output) + return match.group(1) if match else output + + def expand_volume_via_vm_edit(self, vm_name, vol_name, new_size, namespace=DEFAULT_NAMESPACE): + """Expand volume by patching the VM volumeClaimTemplates annotation and PVC""" + logging(f"Expanding volume {vol_name} on VM {vm_name} to {new_size}") + vm = get_cr(group=KUBEVIRT_API_GROUP, version=KUBEVIRT_API_VERSION, + namespace=namespace, plural="virtualmachines", name=vm_name) + annotations = vm.get("metadata", {}).get("annotations", {}) + vct_json = annotations.get("harvesterhci.io/volumeClaimTemplates", "[]") + try: + vcts = json.loads(vct_json) + except (json.JSONDecodeError, TypeError): + vcts = [] + for vct in vcts: + if vct.get("metadata", {}).get("name") == vol_name: + vct["spec"]["resources"]["requests"]["storage"] = new_size + break + patch_body = { + "metadata": { + "annotations": { + "harvesterhci.io/volumeClaimTemplates": json.dumps(vcts) + } + } + } + try: + patch_cr(group=KUBEVIRT_API_GROUP, version=KUBEVIRT_API_VERSION, + namespace=namespace, plural="virtualmachines", + name=vm_name, body=patch_body) + except ApiException as e: + raise Exception(f"Failed to expand volume via VM edit: {e}") + # Also patch the PVC directly + pvc_patch = {"spec": {"resources": {"requests": {"storage": new_size}}}} + try: + self.core_api.patch_namespaced_persistent_volume_claim( + name=vol_name, namespace=namespace, body=pvc_patch) + logging(f"PVC {vol_name} expanded to {new_size}") + except ApiException as e: + raise Exception(f"Failed to expand PVC {vol_name}: {e}") + + def verify_volume_size(self, vm_name, expected_size, namespace=DEFAULT_NAMESPACE): + """Verify volume size inside VM matches expected (uses lsblk)""" + from constant import GIBIBYTE + logging(f"Verifying volume size in VM {vm_name}, expected={expected_size}") + commands = ["lsblk -b -o NAME,SIZE --json"] + try: + result = self._exec_in_vm(vm_name, namespace, commands) + size_value = int(expected_size.replace("Gi", "")) + expected_bytes = size_value * GIBIBYTE + import json as _json + try: + block_info = _json.loads(result) + for device in block_info.get("blockdevices", []): + size = int(device.get("size", 0)) + if abs(size - expected_bytes) < (expected_bytes * 0.05): + return True + for child in device.get("children", []): + size = int(child.get("size", 0)) + if abs(size - expected_bytes) < (expected_bytes * 0.05): + return True + except (_json.JSONDecodeError, TypeError): + logging("Failed to parse lsblk output") + return False + except Exception as e: + logging(f"Error verifying volume size in VM: {e}") + return False diff --git a/harvester_robot_tests/libs/vm/rest.py b/harvester_robot_tests/libs/vm/rest.py index 54aca1b96..92041d0dd 100644 --- a/harvester_robot_tests/libs/vm/rest.py +++ b/harvester_robot_tests/libs/vm/rest.py @@ -7,7 +7,7 @@ from utility.utility import get_retry_count_and_interval from utility.utility import logging from vm.base import Base -from constant import DEFAULT_NAMESPACE +from constant import DEFAULT_NAMESPACE, DEFAULT_USER_DATA class Rest(Base): @@ -37,6 +37,8 @@ def create(self, vm_name, cpu, memory, image_id, **kwargs): level="WARNING") vm_spec.add_image("disk-0", image_id, image_uid=image_uid) + vm_spec.user_data = DEFAULT_USER_DATA + code, data = api.vms.create(vm_name, vm_spec) assert code == 201, f"Failed to create VM: {code}, {data}" return data @@ -181,12 +183,6 @@ def get_data_checksum(self, vm_name): """Get checksum of data in VM""" return self.checksums.get(vm_name, "") - def create_snapshot(self, vm_name, snapshot_name): - """Create a snapshot of the VM""" - api = get_harvester_api_client() - code, data = api.vms.create_snapshot(vm_name, snapshot_name) - assert code == 201, f"Failed to create snapshot: {code}, {data}" - def create_backup(self, vm_name, backup_name): """Create a backup of the VM""" api = get_harvester_api_client() @@ -214,3 +210,51 @@ def cleanup(self): """Clean up all VMs""" logging('Cleaning up test VMs') self.checksums.clear() + + def create_vm_with_volume_using_sc(self, vm_name, sc_name, image_id): + raise NotImplementedError("REST implementation not available for VM.create_with_lvm_sc") + + def create_for_lvm(self, vm_name, sc_name, image_id): + raise NotImplementedError("REST implementation not available for VM.create_for_lvm") + + def attach_volume(self, vm_name, vol_name, namespace=None): + raise NotImplementedError("REST implementation not available for VM.attach_volume") + + def is_running(self, vm_name, namespace=None): + raise NotImplementedError("REST implementation not available for VM.is_running") + + def is_stopped(self, vm_name, namespace=None): + raise NotImplementedError("REST implementation not available for VM.is_stopped") + + def write_data_and_get_checksum_on_disk( + self, vm_name, device, format_device=True, namespace=None): + raise NotImplementedError( + "REST implementation not available " + "for VM.write_data_and_get_checksum_on_disk") + + def mount_data_disk(self, vm_name, namespace=None): + raise NotImplementedError( + "REST implementation not available for VM.mount_data_disk") + + def delete_data(self, vm_name, namespace=None): + raise NotImplementedError("REST implementation not available for VM.delete_data") + + def restore_snapshot_to_new_vm(self, snapshot_name, new_vm_name, namespace=None): + raise NotImplementedError("REST implementation not available" + "for VM.restore_snapshot_to_new_vm") + + def restore_snapshot_to_existing_vm(self, snapshot_name, vm_name, namespace=None): + raise NotImplementedError("REST implementation not available" + "for VM.restore_snapshot_to_existing_vm") + + def wait_for_restore_complete(self, restore_name, namespace, timeout): + raise NotImplementedError("REST implementation not available" + "for VM.wait_for_restore_complete") + + def expand_volume_via_vm_edit(self, vm_name, vol_name, new_size, namespace=None): + raise NotImplementedError("REST implementation not available" + "for VM.expand_volume_via_vm_edit") + + def verify_volume_size(self, vm_name, expected_size, namespace=None): + raise NotImplementedError("REST implementation not available" + "for VM.verify_volume_size") diff --git a/harvester_robot_tests/libs/vm/vm.py b/harvester_robot_tests/libs/vm/vm.py index 097fead48..9bfdb6bea 100644 --- a/harvester_robot_tests/libs/vm/vm.py +++ b/harvester_robot_tests/libs/vm/vm.py @@ -63,11 +63,13 @@ def verify_on_node(self, vm_name, expected_node): def write_data(self, vm_name, data_size_mb): return self.vm.write_data(vm_name, data_size_mb) - def get_data_checksum(self, vm_name): - return self.vm.get_data_checksum(vm_name) + def get_data_checksum(self, vm_name, namespace=None): + ns = namespace or DEFAULT_NAMESPACE + return self.vm.get_data_checksum(vm_name, ns) - def create_snapshot(self, vm_name, snapshot_name): - return self.vm.create_snapshot(vm_name, snapshot_name) + def mount_data_disk(self, vm_name, namespace=None): + ns = namespace or DEFAULT_NAMESPACE + return self.vm.mount_data_disk(vm_name, ns) def create_backup(self, vm_name, backup_name): return self.vm.create_backup(vm_name, backup_name) @@ -77,3 +79,42 @@ def wait_for_backup_completed(self, vm_name, backup_name, timeout): def cleanup(self): return self.vm.cleanup() + + def attach_volume(self, vm_name, vol_name, namespace=None): + ns = namespace or DEFAULT_NAMESPACE + return self.vm.attach_volume(vm_name, vol_name, ns) + + def is_running(self, vm_name, namespace=None): + ns = namespace or DEFAULT_NAMESPACE + return self.vm.is_running(vm_name, ns) + + def is_stopped(self, vm_name, namespace=None): + ns = namespace or DEFAULT_NAMESPACE + return self.vm.is_stopped(vm_name, ns) + + def check_block_device(self, vm_name, namespace=None, expected_disk_size=None): + ns = namespace or DEFAULT_NAMESPACE + return self.vm.check_block_device(vm_name, ns, expected_disk_size) + + def write_data_and_get_checksum_on_disk( + self, vm_name, device, format_device=True, namespace=None): + ns = namespace or DEFAULT_NAMESPACE + return self.vm.write_data_and_get_checksum_on_disk( + vm_name, device, format_device, ns) + + def create_vm_with_volume_using_sc(self, vm_name, sc_name, image_id, + namespace=None, network_name=None): + return self.vm.create_vm_with_volume_using_sc( + vm_name, sc_name, image_id, network_name=network_name) + + def delete_data(self, vm_name, namespace=None): + ns = namespace or DEFAULT_NAMESPACE + return self.vm.delete_data(vm_name, ns) + + def expand_volume_via_vm_edit(self, vm_name, vol_name, new_size, namespace=None): + ns = namespace or DEFAULT_NAMESPACE + return self.vm.expand_volume_via_vm_edit(vm_name, vol_name, new_size, ns) + + def verify_volume_size(self, vm_name, expected_size, namespace=None): + ns = namespace or DEFAULT_NAMESPACE + return self.vm.verify_volume_size(vm_name, expected_size, ns) diff --git a/harvester_robot_tests/libs/volume/base.py b/harvester_robot_tests/libs/volume/base.py index b39a7bad8..173d31c8c 100644 --- a/harvester_robot_tests/libs/volume/base.py +++ b/harvester_robot_tests/libs/volume/base.py @@ -76,3 +76,8 @@ def wait_for_deleted(self, volume_name, timeout): def cleanup(self): """Clean up all test volumes""" pass + + @abstractmethod + def expand_pvc(self, vol_name, new_size, namespace): + """Expand a PVC to new_size""" + pass diff --git a/harvester_robot_tests/libs/volume/crd.py b/harvester_robot_tests/libs/volume/crd.py index 9dfbfd97a..fd2b6f356 100644 --- a/harvester_robot_tests/libs/volume/crd.py +++ b/harvester_robot_tests/libs/volume/crd.py @@ -10,7 +10,7 @@ VOLUME_STATE_BOUND, ACCESS_MODE_RWO, LABEL_TEST, LABEL_TEST_VALUE, - DEFAULT_TIMEOUT_SHORT + DEFAULT_TIMEOUT_SHORT, DEFAULT_RETRY_INTERVAL ) from utility.utility import logging, get_retry_count_and_interval from volume.base import Base @@ -34,8 +34,21 @@ def create(self, volume_name, size, numberOfReplicas, frontend, **kwargs): namespace = kwargs.get('namespace', DEFAULT_NAMESPACE) storage_class = kwargs.get('storage_class', DEFAULT_STORAGE_CLASS) access_mode = kwargs.get('access_mode', ACCESS_MODE_RWO) + volume_mode = kwargs.get('volume_mode', None) # Build PVC manifest + spec = { + "accessModes": [access_mode], + "storageClassName": storage_class, + "resources": { + "requests": { + "storage": size + } + } + } + if volume_mode: + spec["volumeMode"] = volume_mode + body = { "apiVersion": "v1", "kind": "PersistentVolumeClaim", @@ -49,15 +62,7 @@ def create(self, volume_name, size, numberOfReplicas, frontend, **kwargs): "volume.kubernetes.io/storage-provisioner": "driver.longhorn.io" # Noqa } }, - "spec": { - "accessModes": [access_mode], - "storageClassName": storage_class, - "resources": { - "requests": { - "storage": size - } - } - } + "spec": spec } # Add Longhorn-specific parameters @@ -92,22 +97,34 @@ def create(self, volume_name, size, numberOfReplicas, frontend, **kwargs): logging(f"Failed to create PVC {volume_name}: {e}") raise Exception(f"Failed to create volume: {e.status}, {e.reason}") - def delete(self, volume_name, wait=True): - """Delete a PersistentVolumeClaim""" - namespace = DEFAULT_NAMESPACE + def delete(self, volume_name, wait=True, namespace=DEFAULT_NAMESPACE, + timeout=120, interval=DEFAULT_RETRY_INTERVAL): + """Delete a PersistentVolumeClaim. - try: - logging(f"Deleting PersistentVolumeClaim {namespace}/{volume_name}") - self.core_api.delete_namespaced_persistent_volume_claim( - name=volume_name, - namespace=namespace - ) - - if wait: - self.wait_for_deleted(volume_name) - - except ApiException as e: - if e.status != 404: # Ignore not found errors + Retries on 409 conflict (volume still attached to a VM whose + virt-launcher pod has not fully terminated yet). + """ + deadline = time.time() + timeout + while True: + try: + logging(f"Deleting PersistentVolumeClaim {namespace}/{volume_name}") + self.core_api.delete_namespaced_persistent_volume_claim( + name=volume_name, + namespace=namespace + ) + if wait: + self.wait_for_deleted(volume_name, namespace=namespace) + return + except ApiException as e: + if e.status == 404: + return + if e.status == 409 and time.time() < deadline: + logging( + f"PVC {volume_name} still attached (409), " + f"retrying in {interval}s …" + ) + time.sleep(interval) + continue logging(f"Error deleting PVC {volume_name}: {e}") raise @@ -147,7 +164,8 @@ def get(self, volume_name, namespace=DEFAULT_NAMESPACE): } } except ApiException as e: - logging(f"Failed to get PVC {volume_name}: {e}") + if e.status != 404: + logging(f"Failed to get PVC {volume_name}: {e}") raise def list(self, namespace=DEFAULT_NAMESPACE, label_selector=None): @@ -233,15 +251,15 @@ def wait_for_attached(self, volume_name, timeout=DEFAULT_TIMEOUT_SHORT): raise AssertionError(f"PVC {namespace}/{volume_name} did not bind within {timeout}s") - def wait_for_deleted(self, volume_name, timeout=DEFAULT_TIMEOUT_SHORT): + def wait_for_deleted( + self, volume_name, timeout=DEFAULT_TIMEOUT_SHORT, + namespace=DEFAULT_NAMESPACE): """Wait for PVC to be deleted""" - namespace = DEFAULT_NAMESPACE endtime = time.time() + timeout while time.time() < endtime: try: self.get(volume_name, namespace) - logging("Waiting for PVC to be deleted...") except ApiException as e: if e.status == 404: logging(f"PVC {namespace}/{volume_name} deleted") @@ -414,3 +432,17 @@ def cleanup(self): logging(f'Error deleting PVC {pvc_name}: {e}', "WARNING") except Exception as e: logging(f'Error during volume cleanup: {e}', 'WARNING') + + + def expand_pvc(self, vol_name, new_size, namespace=None): + """Expand a PVC to new_size""" + from constant import DEFAULT_NAMESPACE + ns = namespace or DEFAULT_NAMESPACE + body = {"spec": {"resources": {"requests": {"storage": new_size}}}} + try: + self.core_api.patch_namespaced_persistent_volume_claim( + name=vol_name, namespace=ns, body=body + ) + logging(f"PVC {vol_name} expanded to {new_size}") + except Exception as e: + raise Exception(f"Failed to expand PVC {vol_name}: {e}") diff --git a/harvester_robot_tests/libs/volume/rest.py b/harvester_robot_tests/libs/volume/rest.py index f9a24cbcc..19aa9a4cc 100644 --- a/harvester_robot_tests/libs/volume/rest.py +++ b/harvester_robot_tests/libs/volume/rest.py @@ -121,3 +121,6 @@ def restore_from_snapshot(self, volume_name, snapshot_name, new_volume_name): def cleanup(self): """Clean up all test volumes""" logging('Cleaning up test volumes') + + def expand_pvc(self, vol_name, new_size, namespace): + raise NotImplementedError("REST implementation not available for Volume.expand_pvc") diff --git a/harvester_robot_tests/libs/volume/volume.py b/harvester_robot_tests/libs/volume/volume.py index 0573cfbb0..89c61f2d2 100644 --- a/harvester_robot_tests/libs/volume/volume.py +++ b/harvester_robot_tests/libs/volume/volume.py @@ -59,3 +59,8 @@ def restore_from_snapshot(self, volume_name, snapshot_name, new_volume_name): def cleanup(self): return self.volume.cleanup() + + def expand_pvc(self, vol_name, new_size, namespace=None): + from constant import DEFAULT_NAMESPACE + ns = namespace or DEFAULT_NAMESPACE + return self.volume.expand_pvc(vol_name, new_size, ns) diff --git a/harvester_robot_tests/tests/regression/test_addon.robot b/harvester_robot_tests/tests/regression/test_addon.robot index 1583df783..a3a71dd16 100644 --- a/harvester_robot_tests/tests/regression/test_addon.robot +++ b/harvester_robot_tests/tests/regression/test_addon.robot @@ -42,7 +42,7 @@ Test Rancher Monitoring Addon Deploys And Accessibility And Essential Harvester Metrics Should Exist # Cleanup - [Teardown] addon.Stop Port Forward + [Teardown] addon.Stop Prometheus Port Forward Test PCI Devices Controller Addon Deploys And Accessibility [Tags] p0 coretest pcidevices-controller diff --git a/harvester_robot_tests/tests/regression/test_lvm.robot b/harvester_robot_tests/tests/regression/test_lvm.robot new file mode 100644 index 000000000..e33f2f65f --- /dev/null +++ b/harvester_robot_tests/tests/regression/test_lvm.robot @@ -0,0 +1,187 @@ +*** Settings *** +Documentation LVM Local Storage Test Cases +... This suite tests LVM local storage feature in Harvester. +... Ref: https://docs.harvesterhci.io/v1.8/advanced/addons/lvm-local-storage +... Ref: https://github.com/harvester/harvester/issues/5724 +Test Tags regression lvm addon + +Resource ../../keywords/variables.resource +Resource ../../keywords/common.resource +Resource ../../keywords/storage.resource +Resource ../../keywords/storageclass.resource +Resource ../../keywords/addon.resource +Resource ../../keywords/setting.resource +Resource ../../keywords/virtualmachine.resource + +Suite Setup LVM Suite Setup +Suite Teardown LVM Suite Teardown + +*** Variables *** +${LVM_ADDON_NAME} harvester-csi-driver-lvm +${LVM_VG_ACTIVE} vg-dm-thin +${VG_STRIPED} vg-dm-striped +${LVM_SC_ACTIVE} lvm-sc-dm-thin +${LVM_SC_STRIPED} lvm-sc-striped +${LVM_VM_BLOCK} lvm-vm-block-thin +${LVM_VM_FS} lvm-vm-fs-thin +${LVM_VM_SC} lvm-vm-sc-thin +${LVM_VOL_BLOCK} lvm-vol-block-thin +${LVM_VOL_FS} lvm-vol-fs-thin +${LVM_VOLUME_SIZE} 5Gi +${LVM_EXPANDED_SIZE} 10Gi + +*** Test Cases *** +Test LVM Block Volume Attach And Data Integrity + [Tags] p0 smoke + [Documentation] Create LVM block volume, attach to VM, write data and verify md5sum + ... Steps: + ... 1. Create StorageClass for dm-thin volume group + ... 2. Create a block volume using the LVM StorageClass + ... 3. Create a VM + ... 4. Add the LVM volume to the VM + ... 5. Mount the disk inside VM, write data and compute md5sum + ... Expected Result: + ... - Volume is created and attached to VM + ... - Data is written and md5sum is recorded + + Given LVM StorageClass Is Created ${LVM_SC_ACTIVE} ${LVM_VG_ACTIVE} ${LVM_VG_TYPE} + And LVM Block Volume Is Created ${LVM_VOL_BLOCK} ${LVM_VOLUME_SIZE} ${LVM_SC_ACTIVE} + And VM Is Created And Running ${LVM_VM_BLOCK} ${IMAGE_NAME} sc_name=${LVM_SC_ACTIVE} + When Volume Is Attached To VM ${LVM_VM_BLOCK} ${LVM_VOL_BLOCK} + + ${DATA_DISK}= Get Block Device In VM ${LVM_VM_BLOCK} ${LVM_VOLUME_SIZE} + + Then Data Is Written To Disk And Checksum Recorded ${LVM_VM_BLOCK} ${DATA_DISK} + +Test LVM Filesystem Volume Attach And Data Integrity + [Tags] p0 + [Documentation] Create LVM filesystem volume, attach to VM, write data and verify md5sum + ... Steps: + ... 1. Create StorageClass for dm-thin volume group + ... 2. Create a filesystem volume using the LVM StorageClass + ... 3. Create a VM + ... 4. Add the LVM volume to the VM + ... 5. Write data and compute md5sum + ... Expected Result: + ... - Volume is created and attached to VM + ... - Data is written and md5sum is recorded + + Given LVM StorageClass Is Created ${LVM_SC_ACTIVE} ${LVM_VG_ACTIVE} ${LVM_VG_TYPE} + And LVM Filesystem Volume Is Created ${LVM_VOL_FS} ${LVM_VOLUME_SIZE} ${LVM_SC_ACTIVE} + And VM Is Created And Running ${LVM_VM_FS} ${IMAGE_NAME} sc_name=${LVM_SC_ACTIVE} + When Volume Is Attached To VM ${LVM_VM_FS} ${LVM_VOL_FS} + + ${DATA_DISK}= Get Block Device In VM ${LVM_VM_FS} + + Then Data Is Written To Disk And Checksum Recorded ${LVM_VM_FS} ${DATA_DISK} format_device=${False} + +Test LVM Volume Direct VM Creation With StorageClass + [Tags] p0 + [Documentation] Create VM directly using LVM StorageClass (vg-dm-thin) and verify data + ... Steps: + ... 1. Create StorageClass for dm-thin volume group + ... 2. Create VM directly using the LVM StorageClass + ... 3. Mount the disk inside VM, write data and compute md5sum + ... Expected Result: + ... - VM is created with LVM volume + ... - Data is written and md5sum is recorded + + Given LVM StorageClass Is Created ${LVM_SC_ACTIVE} ${LVM_VG_ACTIVE} ${LVM_VG_TYPE} + When VM Is Created with Additional Volume using SC ${LVM_VM_SC} ${LVM_SC_ACTIVE} ${IMAGE_NAME} + + ${DATA_DISK}= Get Block Device In VM ${LVM_VM_SC} + + Then Data Is Written To Disk And Checksum Recorded ${LVM_VM_SC} ${DATA_DISK} format_device=${False} + +Test LVM Volume Snapshot And Restore To New VM + [Tags] p1 smoke + [Documentation] Take snapshots of running VMs, delete data, restore and verify + ... Steps: + ... 1. Take snapshots from above three VMs if running + ... 2. Restore snapshots to new VMs + ... 3. Check the data integrity on new VMs + ... Expected Result: + ... - Snapshots are taken successfully + ... - Restored VMs contain original data + + Given VM Snapshots Are Taken + When Snapshots Are Restored To New VMs + Then Restored VMs Have Correct Data + +Test LVM Volume Snapshot Restore To Existing VM + [Tags] p1 + [Documentation] Restore snapshots to existing VMs and verify data + ... Steps: + ... 1. Take snapshots from above three VMs if running + ... 2. Delete data from original VMs + ... 3. Restore to the snapshots to existing VMs + ... 4. Assert the data of VMs + ... Expected Result: + ... - Existing VMs are restored with original data + ... - Data integrity is verified + + Given VM Snapshots Are Taken + AND Data Is Deleted From Test VMs + WHEN Snapshots Are Restored To Existing VMs + Then VMs Have Original Data After Restore + +Test LVM Volume Expand Via VM Edit + [Tags] p1 + [Documentation] Shut down VM, expand volume via VM edit, verify size + ... Steps: + ... 1. Shut down the VM + ... 2. Expand volume using VM edit + ... 3. Power on the VM + ... 4. Assert the volume size inside VM + ... Expected Result: + ... - Volume is expanded successfully + ... - VM shows correct expanded size + + Given VM Is Powered Off ${LVM_VM_BLOCK} + When Volume Is Expanded Via VM Edit ${LVM_VM_BLOCK} ${LVM_VOL_BLOCK} ${LVM_EXPANDED_SIZE} + And VM Is Powered On And Running ${LVM_VM_BLOCK} + Then Volume Size Inside VM Is Correct ${LVM_VM_BLOCK} ${LVM_EXPANDED_SIZE} + +Test LVM Volume Expand Via Volume Edit + [Tags] p1 + [Documentation] Shut down VM, expand volume via volume edit, verify size + ... Steps: + ... 1. Shut down the VM + ... 2. Expand volume using volume edit + ... 3. Power on the VM + ... 4. Assert the volume size inside VM + ... Expected Result: + ... - Volume is expanded successfully + ... - VM shows correct expanded size + + Given VM Is Powered Off ${LVM_VM_FS} + When LVM Volume Is Expanded Directly ${LVM_VOL_FS} ${LVM_EXPANDED_SIZE} + And VM Is Powered On And Running ${LVM_VM_FS} + Then Volume Size Inside VM Is Correct ${LVM_VM_FS} ${LVM_EXPANDED_SIZE} + + +*** Keywords *** +LVM Suite Setup + [Documentation] Setup LVM test environment + Set up test environment + Suite Setup For Shared Resources + # LVM Addon Setup + addon.Install Addon From URL ${LVM_ADDON_URL} + addon.Enable Addon ${LVM_ADDON_NAME} + addon.Wait For Addon Enabled ${LVM_ADDON_NAME} + setting.Configure CSI Driver Setting For LVM + storage.Identify And Assign LVM Disks + storage.Create LVM Volume Groups + +LVM Suite Teardown + [Documentation] Cleanup LVM test environment + # Clean up LVM resources (VMs waited on, volumes, SCs, VGs) while addon is still running + Run Keyword And Ignore Error storage.Cleanup LVM Test Resources + # Disable LVM addon + Run Keyword And Ignore Error addon.Disable Addon ${LVM_ADDON_NAME} + Run Keyword And Ignore Error addon.Wait For Addon Disabled ${LVM_ADDON_NAME} + # Reset CSI driver setting now that addon is disabled + Run Keyword And Ignore Error setting.Reset CSI Driver Setting For LVM + + # Clean up shared resources (image, network) and general resources + Suite Teardown For Shared Resources diff --git a/harvester_robot_tests/tests/regression/test_nvidia_toolkit_addon.robot b/harvester_robot_tests/tests/regression/test_nvidia_toolkit_addon.robot index 917248863..aa29b7aef 100644 --- a/harvester_robot_tests/tests/regression/test_nvidia_toolkit_addon.robot +++ b/harvester_robot_tests/tests/regression/test_nvidia_toolkit_addon.robot @@ -74,28 +74,28 @@ Test Nvidia Driver Toolkit Addon End-to-End Initial Nvidia Addon State Is Captured [Arguments] ${addon_name} [Documentation] Capture and store the initial state of the nvidia-driver-toolkit addon - ${initial_state}= addon.Get Initial State ${addon_name} + ${initial_state}= addon.Get Addon Initial State ${addon_name} Set Suite Variable ${INITIAL_STATE_NVIDIA_TOOLKIT} ${initial_state} Log Captured initial nvidia-driver-toolkit addon state: ${initial_state} Nvidia Addon Is Enabled [Arguments] ${addon_name} [Documentation] Enable the nvidia-driver-toolkit addon and wait for deployment - ${is_enabled}= addon.Is Enabled ${addon_name} - Run Keyword If not ${is_enabled} addon.Enable ${addon_name} - addon.Wait For Enabled ${addon_name} timeout=900 + ${is_enabled}= addon.Is Addon Enabled ${addon_name} + Run Keyword If not ${is_enabled} addon.Enable Addon ${addon_name} + addon.Wait For Addon Enabled ${addon_name} timeout=900 Log Nvidia-driver-toolkit addon ${addon_name} is enabled Nvidia Toolkit Addon Is Configured [Arguments] ${addon_name} ${image_repo} ${image_tag} ${driver_location} [Documentation] Set image repo, tag, and driver location for the nvidia-driver-toolkit addon - addon.Configure Nvidia Toolkit ${addon_name} ${image_repo} ${image_tag} ${driver_location} + addon.Configure Nvidia Toolkit Addon ${addon_name} ${image_repo} ${image_tag} ${driver_location} Log Nvidia-driver-toolkit addon configured with repo: ${image_repo}, tag: ${image_tag}, driver: ${driver_location} Nvidia Toolkit Configuration Should Be Applied [Arguments] ${addon_name} ${image_repo} ${image_tag} ${driver_location} [Documentation] Verify the nvidia-driver-toolkit addon configuration has been applied - addon.Verify Nvidia Toolkit Configured ${addon_name} ${image_repo} ${image_tag} ${driver_location} + addon.Verify Nvidia Toolkit Addon Configured ${addon_name} ${image_repo} ${image_tag} ${driver_location} Log Nvidia-driver-toolkit addon configuration verified successfully Suite Setup For Nvidia Addon Tests @@ -109,10 +109,10 @@ Suite Teardown For Nvidia Addon Tests Log Running suite teardown for nvidia-driver-toolkit addon tests # Restore nvidia-driver-toolkit addon to initial state Run Keyword If '${INITIAL_STATE_NVIDIA_TOOLKIT}' != ${None} - ... addon.Restore State ${ADDON_NVIDIA_TOOLKIT} ${INITIAL_STATE_NVIDIA_TOOLKIT} + ... addon.Restore Addon State ${ADDON_NVIDIA_TOOLKIT} ${INITIAL_STATE_NVIDIA_TOOLKIT} # Restore pcidevices-controller addon to initial state Run Keyword If '${INITIAL_STATE_PCIDEVICES}' != 'None' - ... addon.Restore State ${ADDON_PCIDEVICES} ${INITIAL_STATE_PCIDEVICES} + ... addon.Restore Addon State ${ADDON_PCIDEVICES} ${INITIAL_STATE_PCIDEVICES} # Standard cleanup Cleanup test resources Log Suite teardown completed @@ -120,28 +120,28 @@ Suite Teardown For Nvidia Addon Tests Initial PCI Addon State Is Captured [Arguments] ${addon_name} [Documentation] Capture and store the initial state of the addon - ${initial_state}= addon.Get Initial State ${addon_name} + ${initial_state}= addon.Get Addon Initial State ${addon_name} Set Suite Variable ${INITIAL_STATE_PCIDEVICES} ${initial_state} Log Captured initial addon state: ${initial_state} PCI Devices Addon Is Enabled [Arguments] ${addon_name} [Documentation] Enable the pcidevices-controller addon and wait for deployment - ${is_enabled}= addon.Is Enabled ${addon_name} - Run Keyword If not ${is_enabled} addon.Enable ${addon_name} - addon.Wait For Enabled ${addon_name} timeout=900 + ${is_enabled}= addon.Is Addon Enabled ${addon_name} + Run Keyword If not ${is_enabled} addon.Enable Addon ${addon_name} + addon.Wait For Addon Enabled ${addon_name} timeout=900 Log PCI Devices addon ${addon_name} is enabled PCI Devices Controller Pods Should Be Running [Arguments] ${namespace} ${label} [Documentation] Verify pcidevices-controller pods are running - addon.Wait For Pods Running ${namespace} ${label} timeout=900 + addon.Wait For Addon Pods Running ${namespace} ${label} timeout=900 Log All pcidevices-controller pods are running in ${namespace} PCI Devices Webhook Service Should Be Running [Arguments] ${namespace} ${service_name} [Documentation] Verify pcidevices-webhook service is running - addon.Wait For Service Running ${namespace} ${service_name} timeout=300 + addon.Wait For Addon Service Running ${namespace} ${service_name} timeout=300 Log pcidevices-webhook service is running in ${namespace} Suite Setup For PCI Devices Addon Tests @@ -154,6 +154,6 @@ Suite Teardown For PCI Devices Addon Tests [Documentation] Cleanup and restore addon state after tests Log Running suite teardown for PCI Devices addon tests Run Keyword If '${INITIAL_STATE_PCIDEVICES}' != 'None' - ... addon.Restore State ${ADDON_PCIDEVICES} ${INITIAL_STATE_PCIDEVICES} + ... addon.Restore Addon State ${ADDON_PCIDEVICES} ${INITIAL_STATE_PCIDEVICES} Cleanup test resources Log Suite teardown completed