Skip to content

fix(e2e): skip GPU tests on additional allocation failure error codes #8269

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
ganeshkumarashok wants to merge 1 commit into
base: main
Choose a base branch
from
+15 −3

fix(e2e): skip GPU tests on additional allocation failure error codes

d46529d
Select commit
Loading
Failed to load commit list.
Open

fix(e2e): skip GPU tests on additional allocation failure error codes #8269

fix(e2e): skip GPU tests on additional allocation failure error codes
d46529d
Select commit
Loading
Failed to load commit list.
Azure Pipelines / Agentbaker E2E failed Apr 10, 2026 in 15m 46s

Build #20260409.49 had test failures

1 errors / 3 warnings

Details

Tests

  • Failed: 2 (1.85%)
  • Passed: 106 (98.15%)
  • Other: 0 (0.00%)
  • Total: 108

Annotations

Check failure on line 775 in Build log

See this annotation in the file changed.

@azure-pipelines azure-pipelines / Agentbaker E2E

Build log #L775 

Script failed with exit code: 1

Check failure on line 1 in Test_Ubuntu2204_MessageOfTheDay

See this annotation in the file changed.

@azure-pipelines azure-pipelines / Agentbaker E2E

Test_Ubuntu2204_MessageOfTheDay 

Failed
Raw output 
=== RUN   Test_Ubuntu2204_MessageOfTheDay
=== PAUSE Test_Ubuntu2204_MessageOfTheDay
=== CONT  Test_Ubuntu2204_MessageOfTheDay
    test_helpers.go:361: [14.218s] TAGS {Name:Test_Ubuntu2204_MessageOfTheDay ImageName:2204gen2containerd OS:ubuntu Arch:amd64 NetworkIsolated:false NonAnonymousACR:false GPU:false WASM:false BootstrapTokenFallback:false KubeletCustomConfig:false Scriptless:false VHDCaching:false MockAzureChinaCloud:false VMSeriesCoverageTest:false}
    test_helpers.go:200: [21.986s] → running scenario...
    test_helpers.go:232: [25.773s] → preparing AKS node...
    vmss.go:324: [33.557s] → creating VMSS 84zn-2026-04-10-ubuntu2204messageoftheday...
    vmss.go:232: [52.179s] VMSS portal link: https://ms.portal.azure.com/#@microsoft.onmicrosoft.com/resource/subscriptions/8ecadfc9-d1a3-4ea4-b844-0d9f87e4d7c8/resourceGroups/MC_abe2e-westus3_abe2e-kubenet-v4-e1f58_westus3/providers/Microsoft.Compute/virtualMachineScaleSets/84zn-2026-04-10-ubuntu2204messageoftheday/overview
    vmss.go:238: [52.267s] Managed cluster portal link: https://ms.portal.azure.com/#@microsoft.onmicrosoft.com/resource/subscriptions/8ecadfc9-d1a3-4ea4-b844-0d9f87e4d7c8/resourceGroups/MC_abe2e-westus3_abe2e-kubenet-v4-e1f58_westus3/providers/Microsoft.ContainerService/managedClusters/abe2e-kubenet-v4-e1f58/overview
    vmss.go:357: [69.690s] VM will be automatically deleted after the test finishes, to preserve it for debugging purposes set KEEP_VMSS=true or pause the test with a breakpoint before the test finishes or failed
    vmss.go:361: [69.690s] SSH Instructions: (may take a few minutes for the VM to be ready for SSH)
        ========================
        az network bastion ssh --target-resource-id "/subscriptions/8ecadfc9-d1a3-4ea4-b844-0d9f87e4d7c8/resourceGroups/MC_abe2e-westus3_abe2e-kubenet-v4-e1f58_westus3/providers/Microsoft.Compute/virtualMachineScaleSets/84zn-2026-04-10-ubuntu2204messageoftheday/virtualMachines/0" --name "abe2e-kubenet-v4-e1f58-bastion" --resource-group MC_abe2e-westus3_abe2e-kubenet-v4-e1f58_westus3 --auth-type ssh-key --username azureuser --ssh-key /tmp/private-key-3953472458
        
    bastionssh.go:304: [172.322s] Attempt 1/5 establishing SSH over bastion to 10.224.0.151
    vmss.go:411: [176.436s] VM reached running state
    vmss.go:381: [176.436s] ✓ creating VMSS 84zn-2026-04-10-ubuntu2204messageoftheday done (142.9s)
    kube.go:152: [176.436s] → waiting for node 84zn-2026-04-10-ubuntu2204messageoftheday to be ready...
    kube.go:184: [176.478s] node 84zn-2026-04-10-ubuntu2204messageoftheday000000 is ready. Taints: [{"key":"node.kubernetes.io/network-unavailable","effect":"NoSchedule","timeAdded":"2026-04-10T00:03:39Z"}] Conditions: [{"type":"NetworkUnavailable","status":"True","lastHeartbeatTime":"2026-04-10T00:03:39Z","lastTransitionTime":"2026-04-10T00:03:39Z","reason":"NodeInitialization","message":"Waiting for cloud routes"},{"type":"MemoryPressure","status":"False","lastHeartbeatTime":"2026-04-10T00:03:23Z","lastTransitionTime":"2026-04-10T00:03:23Z","reason":"KubeletHasSufficientMemory","message":"kubelet has sufficient memory available"},{"type":"DiskPressure","status":"False","lastHeartbeatTime":"2026-04-10T00:03:23Z","lastTransitionTime":"2026-04-10T00:03:23Z","reason":"KubeletHasNoDiskPressure","message":"kubelet has no disk pressure"},{"type":"PIDPressure","status":"False","lastHeartbeatTime":"2026-04-10T00:03:23Z","lastTransitionTime":"2026-04-10T00:03:23Z","reason":"KubeletHasSufficientPID","message":"kubelet has sufficient PID available"},{"type":"Ready","status":"True","lastHeartbeatTime":"2026-04-10T00:03:23Z","lastTransitionTime":"2026-04-10T00:03:23Z","reason":"KubeletReady","message":"kubelet is posting ready status"}]
    kube.go:185: [176.479s] ✓ waiting for node 84zn-2026-04-10-ubuntu2204messageoftheday to be ready done (0.0s)
    test_helpers.go:314: [176.479s] Node 84zn-2026-04-10-ubuntu2204messageoftheday took 2m22.972580321s to be created and 42.592841ms to be ready
    test_help

Check failure on line 1 in Test_Ubuntu2404_NPD_Basic

See this annotation in the file changed.

@azure-pipelines azure-pipelines / Agentbaker E2E

Test_Ubuntu2404_NPD_Basic 

Failed
Raw output 
=== RUN   Test_Ubuntu2404_NPD_Basic
=== PAUSE Test_Ubuntu2404_NPD_Basic
=== CONT  Test_Ubuntu2404_NPD_Basic
    azure.go:472: [0.000s] Looking up images in https://ms.portal.azure.com/#@microsoft.onmicrosoft.com/resource/subscriptions/c4c3550e-a965-4993-a50c-628fd38cd3e1/resourceGroups/aksvhdtestbuildrg/providers/Microsoft.Compute/galleries/PackerSigGalleryEastUS/images/aks-ubuntu-containerd-24.04-gen2/overview
    azure.go:561: [30.915s] Image version /subscriptions/c4c3550e-a965-4993-a50c-628fd38cd3e1/resourceGroups/aksvhdtestbuildrg/providers/Microsoft.Compute/galleries/PackerSigGalleryEastUS/images/2404gen2containerd/versions/1.1775722270.17245 is already in region westus3
    vhd.go:320: [30.915s] got version by tag branch=refs/heads/main: https://ms.portal.azure.com/#@microsoft.onmicrosoft.com/resource/subscriptions/c4c3550e-a965-4993-a50c-628fd38cd3e1/resourceGroups/aksvhdtestbuildrg/providers/Microsoft.Compute/galleries/PackerSigGalleryEastUS/images/aks-ubuntu-containerd-24.04-gen2/versions/1.1775722270.17245/overview
    test_helpers.go:361: [30.915s] TAGS {Name:Test_Ubuntu2404_NPD_Basic ImageName:2404gen2containerd OS:ubuntu Arch:amd64 NetworkIsolated:false NonAnonymousACR:false GPU:false WASM:false BootstrapTokenFallback:false KubeletCustomConfig:false Scriptless:false VHDCaching:false MockAzureChinaCloud:false VMSeriesCoverageTest:false}
    test_helpers.go:200: [30.915s] → running scenario...
    test_helpers.go:232: [30.915s] → preparing AKS node...
    vmss.go:324: [33.594s] → creating VMSS 67i9-2026-04-10-ubuntu2404npdbasic...
    vmss.go:232: [44.339s] VMSS portal link: https://ms.portal.azure.com/#@microsoft.onmicrosoft.com/resource/subscriptions/8ecadfc9-d1a3-4ea4-b844-0d9f87e4d7c8/resourceGroups/MC_abe2e-westus3_abe2e-kubenet-v4-e1f58_westus3/providers/Microsoft.Compute/virtualMachineScaleSets/67i9-2026-04-10-ubuntu2404npdbasic/overview
    vmss.go:238: [44.339s] Managed cluster portal link: https://ms.portal.azure.com/#@microsoft.onmicrosoft.com/resource/subscriptions/8ecadfc9-d1a3-4ea4-b844-0d9f87e4d7c8/resourceGroups/MC_abe2e-westus3_abe2e-kubenet-v4-e1f58_westus3/providers/Microsoft.ContainerService/managedClusters/abe2e-kubenet-v4-e1f58/overview
2026/04/10 00:01:37 Using VM extension version 1.426 for extension type Compute.AKS.Linux.AKSNode in region westus3
    vmss.go:357: [63.868s] VM will be automatically deleted after the test finishes, to preserve it for debugging purposes set KEEP_VMSS=true or pause the test with a breakpoint before the test finishes or failed
    vmss.go:361: [63.871s] SSH Instructions: (may take a few minutes for the VM to be ready for SSH)
        ========================
        az network bastion ssh --target-resource-id "/subscriptions/8ecadfc9-d1a3-4ea4-b844-0d9f87e4d7c8/resourceGroups/MC_abe2e-westus3_abe2e-kubenet-v4-e1f58_westus3/providers/Microsoft.Compute/virtualMachineScaleSets/67i9-2026-04-10-ubuntu2404npdbasic/virtualMachines/0" --name "abe2e-kubenet-v4-e1f58-bastion" --resource-group MC_abe2e-westus3_abe2e-kubenet-v4-e1f58_westus3 --auth-type ssh-key --username azureuser --ssh-key /tmp/private-key-3953472458
        
    bastionssh.go:304: [225.931s] Attempt 1/5 establishing SSH over bastion to 10.224.0.128
    vmss.go:411: [228.737s] VM reached running state
    vmss.go:381: [228.737s] ✓ creating VMSS 67i9-2026-04-10-ubuntu2404npdbasic done (194.8s)
    kube.go:152: [228.737s] → waiting for node 67i9-2026-04-10-ubuntu2404npdbasic to be ready...
    kube.go:184: [228.850s] node 67i9-2026-04-10-ubuntu2404npdbasic000000 is ready. Taints: [{"key":"node.kubernetes.io/network-unavailable","effect":"NoSchedule","timeAdded":"2026-04-10T00:04:04Z"}] Conditions: [{"type":"KubeletProblem","status":"False","lastHeartbeatTime":"2026-04-10T00:04:06Z","lastTransitionTime":"2026-04-10T00:04:05Z","reason":"KubeletIsUp","message":"kubelet service is up"},{"type":"ContainerRuntimeProblem","status":"False","lastHeartbeatTime":"2026-04-10T00:04:06Z","lastTransitionTime":"2026-04-10
View more details on Azure Pipelines