[CI Monitor] Daily Report - 2026-07-12

[amd-bot]

Daily Cross-Workflow Summary — 2026-07-12

Snapshot: 2026-07-12 23:24 UTC · Only completed runs/jobs counted in trends · Auto-updated every 30 min

TL;DR

🔴 RED · 8 tracked clusters · 2 🆕 today (R262 + R261's FP8 leg) · 6 carrying over · 3 in-flight fixes (R260 #30942 🆕draft, R195 #29965, R258 #30506)
👉 Today's ask: (1) Review/land R260 fix #30942 ("Pin cmake==4.3.4") — a fix PR now exists (opened today, still draft) for the MORI/cmake-4.4 break that has left all AMD nightly docker publishing RED for a 2nd day (both rocm7.0 + rocm7.2). (2) Triage R262 (🆕) — a fresh sglang code regression (CLIPImageProcessor has no attribute 'tokenizer') crashes the InternVL server and aborts the nightly VLM-MMMU job; passed Jul-11, broke Jul-12, ❌ no fix. (3) R257 GPU-Hang wave is now 4th day and still dominates the nightly + PR MI35x fleet — chase infra ⚠️ #30431 and A/B the pre/post base-image at a fixed SHA.

Important

The Jul-12 nightly + PR runs are still IN-FLIGHT at snapshot (status=queued on runs 29202683625, 29202600186, 29192376210, 29202664092) — their failures are [IN-FLIGHT, partial data], excluded from trend numbers. The completed Jul-12 runs are both release-docker (❌ R260) and mi355x-disagg (❌). The latest completed nightly + PR runs are the Jul-11 ones (nightly runs concluded cancelled; PR runs failure).

Workflow status

Workflow	Runs (latest completed)	✅	❌	Trend (completed)	Δ vs yesterday
nightly-test-amd	Jul-11 29162254653 (cancelled)	7	23	R257 wave persists	Jul-11 now completed; Jul-12 IN-FLIGHT
nightly-test-amd-rocm720	Jul-11 29162179641 (cancelled)	8	25	R257 wave persists	+🆕 R262 (InternVL) in Jul-12 IN-FLIGHT
release-docker-amd-nightly	Jul-12 29192571824	0	2	green×10d → RED×2	R260 continues (now has fix #30942)
release-docker-amd-rocm720-nightly	Jul-12 29192413224	0	4	green → RED×2	R260 continues
nightly-amd-mi355x-disagg	Jul-12 29179694296	most	2	R241 + R261	R261 leg = DSV4-Flash FP8 today (was FP4)
amd-aiter-scout	no run this cycle	—	—	—	no run digested
pr-test-amd	Jul-11 29152390308	—	11	diffusion/mi35x mix	stable; Jul-12 IN-FLIGHT
pr-test-amd-rocm720	Jul-11 29162225443	—	10	diffusion/mi35x mix	stable; Jul-12 IN-FLIGHT

aiter byte-identical across pass/fail (dev image 9127c94a1 v0.1.17.dev110) — exonerated for R257/R262. Both release-docker clusters are build failures (aiter N/A).

🆕 NEW clusters today

R262 · 🆕 · InternVL server crash — CLIPImageProcessor has no attribute 'tokenizer' — nightly-rocm720 · ❌ none

• Status: NEW today; first crash 2026-07-12, InternVL server passed Jul-11/Jul-10/Jul-05. A genuine code regression (not accuracy). The crash fires at popen_launch_server before the per-model try/except guard, so it aborts the whole test_mmmu_vlm_models run early (Ran 1 test in 1526s vs ~2800s), and the 7 VLMs after InternVL never ran.
• Top hypothesis: [MEDIUM] A commit in the Jul-11→Jul-12 window changed VLM processor/tokenizer resolution so get_tokenizer_from_processor calls .tokenizer on a CLIPImageProcessor (image-only object). Supporting: symptom is a clean AttributeError at server launch, model-specific to InternVL2_5-2B, appeared exactly at the Jul-11→Jul-12 boundary. Disconfirming: run is IN-FLIGHT (nightly-rocm720 still queued); not yet confirmed on the rocm7.0 sister; no code-level commit pinned yet.
• In-flight fix: ❌ none found (searched get_tokenizer_from_processor, CLIPImageProcessor+tokenizer — 0 relevant open PRs). Suggested triage: bisect 9b4bb415d..96a04cb13 for VLM processor/get_tokenizer_from_processor changes; re-run InternVL2_5-2B alone at HEAD 96a04cb13.

Workflow	Job (shard)	Test File	Test Function	Error	Log
nightly-test-amd-rocm720	nightly-accuracy-2-gpu-vlm-rocm720 [IN-FLIGHT]	test/registered/amd/accuracy/mi30x/test_vlms_mmmu_eval_amd.py	test_mmmu_vlm_models (InternVL2_5-2B)	AttributeError: 'CLIPImageProcessor' has no attribute 'tokenizer' → server exit 1	Log

Top active carry-over clusters

R260 · MORI gtest_discover_tests cmake-4.4 JSON parse break — all AMD release-docker builds RED — release-docker ×2 · 🆕 #30942 (draft)

• Status: 2nd day (first Jul-11 after 10 green days Jul-01→Jul-10). Fails the [stage-4 23/28] MORI build in every publish job across rocm7.0 + rocm7.2; push_local_registry jobs then cascade (missing image artifact). Blocks all AMD nightly image publishing since Jul-10.
• Top hypothesis: [HIGH] Unpinned pip install … cmake (docker/rocm.Dockerfile:361) floated 4.3.4 → 4.4.0 between Jul-10 (pass) and Jul-11 (fail); cmake 4.4.0's ParseTestList.cmake:96 rejects the empty/invalid JSON the mori gtest binaries emit during GPU-less discovery. Supporting: version diff verified from log (4.3.4 pass → 4.4.0 fail), erroring module is literally cmake-4.4/Modules/GoogleTest/ParseTestList.cmake, MORI_COMMIT pinned & unchanged. Disconfirming: (none found).
• In-flight fix: 🆕 #30942 "[AMD] Pin cmake==4.3.4 in ROCm Dockerfile" — opened today 18:13 UTC, still draft. This directly matches the [HIGH] hypothesis. Suggested triage: review + un-draft/land #30942, rebuild both arch images with ENABLE_MORI=1.

Workflow	Job (shard)	Test File	Test Function	Error	Log
release-docker-amd-nightly	publish (gfx942, all)	docker build [stage-4 23/28] MORI	N/A (docker build)	ParseTestList.cmake:96 JSON Syntax error → ninja exit 1 (cmake 4.4.0)	Log
release-docker-amd-nightly	push_local_registry (gfx942)	Download image tag artifact	N/A	cascade (no image from failed publish)	Log
release-docker-amd-rocm720-nightly	publish (gfx942-rocm720, all)	docker build [stage-4] MORI	N/A	same cmake 4.4 JSON parse error	Log
release-docker-amd-rocm720-nightly	publish (gfx950-rocm720, all)	docker build [stage-4] MORI	N/A	same	Log
release-docker-amd-rocm720-nightly	push_local_registry (gfx950/gfx942-rocm720)	Download image tag artifact	N/A	cascade ×2	Log

R257 · Fleet-wide MI35x (gfx950) "GPU Hang" (HW Exception) during weight load / model init — all AMD GPU workflows · ⚠️ #30431 (infra, WIP)

• Status: 4th day (first Jul-09/10). Still dominant: ~23/30 completed nightly-amd + ~25/33 nightly-rocm720 Jul-11 jobs, plus MI35x shards on both PR workflows; continues into the Jul-12 IN-FLIGHT runs. Signature: all weight shards load → HW Exception … reason :GPU Hang → Fatal Python error: Aborted → exit 134; faulthandler points into fused_moe_triton/layer.py weight-loader (model-independent).
• Top hypothesis: [MEDIUM] MI35x node/driver-level GPU hang (correlates with the Jul-10 base-image bump). Disconfirming: model- and workflow-independent, aiter byte-identical across pass/fail, alternates pass/fail across differing head SHAs → environment over a bisectable code regression.
• In-flight fix: ⚠️ #30431 (WIP, self-heal VRAM leaks — not a confirmed hang fix). Suggested triage: A/B one 8-GPU MI35x job pre- vs post-bump image at a fixed SHA; if pre-bump passes, image is implicated — else escalate node health + reboot MI35x pool.

Workflow	Job (shard)	Test File	Test Function	Error	Log
nightly-test-amd	nightly-2-gpu-mi35x-deepseek-r1-mxfp4-tp2	test/registered/amd/accuracy/mi35x/test_deepseek_r1_mxfp4_tp2_mi35x.py	setUpClass	GPU Hang → exit 134	Log
nightly-test-amd	nightly-8-gpu-mi35x-grok2	test/registered/amd/test_grok2_eval_mi35x.py	setUpClass	GPU Hang → exit 134	Log
nightly-test-amd-rocm720	nightly-2-gpu-mi35x-deepseek-r1-mxfp4-tp2-rocm720	test_deepseek_r1_mxfp4_tp2_mi35x.py	setUpClass	GPU Hang → exit 134	Log
nightly-test-amd-rocm720	nightly-8-gpu-mi35x-glm51-rocm720	test_glm51_eval_mi35x.py	setUpClass	GPU Hang → watchdog SIGABRT	Log
pr-test-amd	stage-c-test-large-8-gpu-amd-mi35x	test_deepseek_r1_mxfp4_8gpu.py	setUpClass	GPU Hang → exit 134	Log

(+~40 more MI35x jobs across the four AMD workflows carry the identical GPU-Hang signature — omitted for length; see per-workflow drill-down.)

R241 · Kimi-K2.6 FP8 non-MTP 1P1D disagg GSM8K ~0.882 < 0.92 — mi355x-disagg · ✅ #30433 (diagnostic)

• Status: never-passed since Jul-05 (0.887/0.886/0.892/0.886/0.882 today). [MEDIUM] genuine degradation in the non-MTP disagg decode path (MTP leg scores ~0.95, single-node non-MTP 0.944). #30433 is a DO-NOT-MERGE diagnostic; no merge-ready fix.

Workflow	Job (shard)	Test File	Test Function	Error	Log
nightly-amd-mi355x-disagg	nightly-mi355x-benchmark (kimik26-fp8-1k1k-1p1d)	scripts/ci/slurm/recipes/mi355x-fp8/kimik26/1k1k/1p1d.yaml	GSM8K 8-shot gate	accuracy=0.882 threshold=0.92 → exit 1	Log

R261 · DeepSeek-V4-Flash disagg marginal GSM8K miss near tight gate — mi355x-disagg · ❌ none

• Status: recurring family; today's failing leg is DSV4-Flash FP8 (0.905 < 0.91) — different from yesterday's FP4 (0.908). FP8 leg passed the two prior runs (0.919 → 0.912 → 0.905, a narrow downward drift straddling the gate). [LOW] run-to-run variance near a tight gate; weak secondary: an FP8/graph-numerics commit in bbcfcaee..592c0438. Disconfirming: only one below-gate point; sibling FP4/Pro legs passed today. Suggested triage: re-run the leg 2-3× at HEAD 592c04381; if it clears 0.91, variance — else bisect the window.

Workflow	Job (shard)	Test File	Test Function	Error	Log
nightly-amd-mi355x-disagg	nightly-mi355x-benchmark (dsv4flash-fp8-1k1k-1p1d-dp8ep8)	scripts/ci/slurm/recipes/mi355x-fp8/dsv4flash/.../1p1d-dp8ep8.yaml	GSM8K 8-shot gate	accuracy=0.905 threshold=0.91 → exit 1	Log

R195 · Mamba extra_buffer needs CUDA/MUSA/NPU (FLA) on ROCm — nightly-amd + rocm720 · ✅ #29965 (ready)

• Status: recurring/never-passed for test_qwen35_eval_mi35x.py. [HIGH] mamba auto radix-cache picks extra_buffer; ROCm not in the CUDA/MUSA/NPU path. Land-ready fix falls back to no_buffer.

Workflow	Job (shard)	Test File	Test Function	Error	Log
nightly-test-amd	nightly-8-gpu-mi35x-qwen35	test_qwen35_eval_mi35x.py	test_lm_eval	extra_buffer needs CUDA/MUSA/NPU (FLA) → exit 1	Log
nightly-test-amd-rocm720	nightly-8-gpu-mi35x-qwen35-rocm720	test_qwen35_eval_mi35x.py	test_lm_eval	same	Log

R248 · HF / Xet weight-download stall → server-launch / watchdog timeout — nightly + PR · ❌ none

• Status: recurring (Jul-06…Jul-12). Cold weight-cache miss / xet_get stall on distinct models ⇒ host/network, not model-specific. In-flight ❌ (retry / pre-seed runner cache).

Workflow	Job (shard)	Test File	Test Function	Error	Log
nightly-test-amd-rocm720	nightly-8-gpu-mi35x-qwen3-235b-mxfp4-rocm720	test_qwen3_instruct_mxfp4.py	setUpClass	HF Xet download hang → 300s watchdog → exit 134	Log

R258 · Native sgl_kernel SIGSEGV on gfx950 (fast_topk_v2 / rotary_embedding) — pr-test-amd + rocm720 · ✅ #30506 (candidate)

• Status: carry-over on stage-b-1gpu-small-mi35x; two ops segfault (SIGSEGV -11), distinct from R257 hang. In-flight ✅ #30506 "Disable DSA fused top-k v2 on ROCm" directly addresses the topk side.

Workflow	Job (shard)	Test File	Test Function	Error	Log
pr-test-amd-rocm720	stage-b-test-1-gpu-small-amd-mi35x-rocm720	test_dsa_skip_logits_clean.py	test_skip_logits_clean_topk_equivalence	SIGSEGV (-11) fast_topk_v2	Log

Known stable / PR-side recurring clusters (no new action today) · click to expand

ID	Cluster	Workflow · Rep. job	Test / fn	Status	In-flight fix
—	VLM MMMU accuracy ~random-chance on AMD MI30x (Qwen2/2.5/3-VL all below thr)	nightly-amd + rocm720 · nightly-accuracy-2-gpu-vlm	test_mmmu_vlm_models	recurring/never-passed ≥7d	❌ none
—	GSM8K accuracy below threshold — Mixtral/Mistral answer-extraction	nightly-amd + rocm720 · nightly-accuracy-2-gpu	test_gsm8k_eval_amd.py	recurring	❌ none
—	Diffusion update_weights_from_disk HTTP 500 (InferenceMode inplace) + FLUX.2 setup assert	pr-test-amd + rocm720 · multimodal-gen	test_update_weights_from_disk.py	recurring since Jul-04	❌ none
—	Diffusion server startup / HF-download timeout → port-5555 --strict-ports cascade + flux2 FP8 _scaled_mm HIPBLAS_NOT_SUPPORTED	pr-test-amd + rocm720 · multimodal-gen	test_server_1_gpu.py	recurring	❌ none
—	GLM-Image AR invalid HF repo id (.../vision_language_encoder/) → server launch fail / 150-min timeout	pr-test-amd + rocm720 · multimodal-gen-2gpu	test_ar_models.py::TestGLMImage	never-passed	❌ none
—	Disagg ZImage encoder numpy ambiguous-truth ValueError → warmup 500	pr-test-amd + rocm720 · multimodal-gen-2gpu	test_disagg_server.py	recurring since Jul-08	❌ none
—	KV-canary verify_position violation crashing PD decode scheduler (baseline canary)	pr-test-amd-rocm720 · stage-b	PD decode	recurring	❌ none
—	AMD remote-instance TransferEngine leaves dst weights zeroed	pr-test-amd · stage-b-2gpu-large	test_load_weights_from_remote_instance.py	flaky	✅ #30035
—	ModuleNotFoundError: msgpack in pi05 vla protocol	pr-test-amd-rocm720 · multimodal-gen-unit	unit test	minor	❌ none
—	Self-hosted MI325 runner lost communication / container-start abort; MORI/spdk submodule clone GnuTLS network flake	pr-test-amd + rocm720 · extra-a / jit-kernel	N/A (infra)	intermittent	❌ none
—	Triton-backend serving throughput below AMD perf gate	pr-test-amd-rocm720 · stage-b-1gpu-large	test_bench_serving_1gpu_part1	recurring	❌ none

Yesterday's clusters — status today · click to expand

• R257 (fleet-wide MI35x GPU Hang): still dominant, 4th day — carries over.
• R260 (MORI cmake-4.4 docker break): carries over, 2nd day — now has 🆕 fix #30942 (draft).
• R261 (DSV4-Flash marginal GSM8K): carries over as a family; today's failing leg is FP8 (0.905), not yesterday's FP4.
• R241 / R195 / R248 / R258: carry over (above).
• 🆕 R262 (InternVL CLIPImageProcessor.tokenizer crash): new today in the Jul-12 nightly-rocm720 IN-FLIGHT run.

Workflow drill-down · click to expand

nightly-test-amd · latest completed 29162254653 (Jul-11, cancelled, head 9b4bb415d) · 23 ❌ / 7 ✅. Jul-12 29202683625 IN-FLIGHT (head 96a04cb13).

Job (shard)	Test File	Cluster	Error
nightly-2-gpu-mi35x-deepseek-r1-mxfp4-tp2	test_deepseek_r1_mxfp4_tp2_mi35x.py	R257	GPU Hang → 134
nightly-8-gpu-mi35x-qwen35	test_qwen35_eval_mi35x.py	R195	extra_buffer FLA
nightly-accuracy-2-gpu-vlm	test_vlms_mmmu_eval_amd.py	VLM MMMU	accuracy < thr
+~19 more MI35x GPU-Hang + accuracy-2-gpu	N/A	R257 / GSM8K	GPU Hang / accuracy

nightly-test-amd-rocm720 · latest completed 29162179641 (Jul-11, cancelled) · 25 ❌ / 8 ✅. Jul-12 29202600186 IN-FLIGHT (carries 🆕 R262 InternVL crash).

Job (shard)	Test File	Cluster	Error
nightly-2-gpu-mi35x-deepseek-r1-mxfp4-tp2-rocm720	test_deepseek_r1_mxfp4_tp2_mi35x.py	R257	GPU Hang → 134
nightly-8-gpu-mi35x-qwen35-rocm720	test_qwen35_eval_mi35x.py	R195	extra_buffer FLA
nightly-8-gpu-mi35x-qwen3-235b-mxfp4-rocm720	test_qwen3_instruct_mxfp4.py	R248	HF Xet stall → watchdog
nightly-accuracy-2-gpu-vlm-rocm720 [IN-FLIGHT Jul-12]	test_vlms_mmmu_eval_amd.py	🆕 R262	CLIPImageProcessor.tokenizer crash
+~22 more MI35x GPU-Hang jobs	N/A	R257	GPU Hang → 134

release-docker-amd-nightly · 29192571824 (Jul-12, completed ❌) · 2 failed — all R260 (publish gfx942 + push cascade). gfx950 leg also hit the same MORI/cmake break.
release-docker-amd-rocm720-nightly · 29192413224 (Jul-12, completed ❌) · 4 failed — R260 (publish gfx942 + gfx950 + 2 push cascades).
nightly-amd-mi355x-disagg · 29179694296 (Jul-12, completed ❌) · 2 failed — R241 (kimik26 non-MTP 0.882) + R261 (dsv4flash-fp8 0.905). DSV4 FP4/Pro + MTP legs passed.

pr-test-amd · latest completed 29152390308 (Jul-11, ❌) · 11 failed — MI35x stage-c = R257; multimodal-gen = diffusion/GLM-Image/port-5555 (known); jit-kernel = MORI submodule clone (infra). Jul-12 29192376210 IN-FLIGHT.
pr-test-amd-rocm720 · latest completed 29162225443 (Jul-11, ❌) · 10 failed — MI35x stage-c = R257; stage-b-1gpu-small = R258; multimodal-gen-unit = msgpack; KV-canary + diffusion 2-gpu known. Jul-12 29202664092 IN-FLIGHT.

How this report is generated

• Only status == "completed" runs/jobs counted in trends. Jul-12 nightly + PR runs are IN-FLIGHT and labelled; their failures are partial. Completed Jul-12 runs: both release-docker (❌ R260) + mi355x-disagg (❌ R241/R261). Latest completed nightly/PR runs are Jul-11 (nightly cancelled, PR failure).
• 🆕 today: R262 (InternVL CLIPImageProcessor.tokenizer code regression, [MEDIUM]); R261's failing leg switched FP4→FP8 ([LOW]).
• Fixes: R260 🆕 #30942 (draft), R195 ✅ #29965, R258 ✅ #30506. R241 diagnostic #30433; R257 infra #30431 (WIP).
• aiter byte-identical across pass/fail (v0.1.17.dev110); no amd-aiter-scout run this cycle.
• Confidence: FACT/HIGH/MEDIUM/LOW/SPECULATION. Bot does NOT assign Priority — engineers decide from cluster size + persistence + fix availability.

---

Generated by amd-bot · last updated 2026-07-12 23:24 UTC

---

Generated by amd-bot using Claude Code CLI (last updated: 2026-07-12 23:24 UTC)

---

CI Monitor — 2026-07-12

Repo: sgl-project/sglang

Monitored Workflows:

• nightly-test-amd.yml
• nightly-test-amd-rocm720.yml
• release-docker-amd-nightly.yml
• release-docker-amd-rocm720-nightly.yml
• nightly-amd-mi355x-disagg.yml
• amd-aiter-scout.yml
• pr-test-amd.yml
• pr-test-amd-rocm720.yml

Per-workflow failure reports are appended as comments below; the Daily Cross-Workflow Summary is rendered above this section.

[amd-bot]

release-docker-amd-rocm720-nightly.yml — 6 failure(s)

Run started (UTC): 2026-07-11 12:19 UTC
Last scanned (UTC): 2026-07-12 14:48 UTC
Commits: sglang 80856ab

Summary

Counts: 6 failures · 2 clusters · 0 carrying over (≥3 days) · 6 all part of one MORI-build regression first seen 2026-07-11

The 6 failures are one root cause: the MORI docker build breaks in cmake's gtest_discover_tests (3 publish jobs), which starves the 3 downstream push_local_registry jobs of their image-tag artifact (cascade). No test failures — all are docker-build / CI-plumbing failures.

#	Cluster	Job	Test File	Test Function	Status	Hypothesis (confidence)
1	MORI cmake gtest-discovery build failure	publish (gfx942-rocm720, all)	N/A (docker build)	N/A	Failing since Jul 11 (2 runs)	[HIGH] unpinned pip cmake 4.3.4→4.4.0 breaks MORI gtest_discover_tests
2	MORI cmake gtest-discovery build failure	publish (gfx942-rocm720, all)	N/A (docker build)	N/A	Failing since Jul 11 (2 runs)	[HIGH] cmake 4.4.0 empty-JSON parse (same cluster)
3	MORI cmake gtest-discovery build failure	publish (gfx950-rocm720, all)	N/A (docker build)	N/A	New Jul 12 (gfx950 leg)	[MEDIUM] cmake 4.4.0 × non-deterministic empty test-list output
4	push_local_registry artifact cascade	push_local_registry (gfx942-rocm720)	N/A (CI plumbing)	N/A	Failing since Jul 11 (2 runs)	[FACT] cascade — no artifact from failed publish
5	push_local_registry artifact cascade	push_local_registry (gfx942-rocm720)	N/A (CI plumbing)	N/A	Failing since Jul 11 (2 runs)	[FACT] cascade — no artifact from failed publish
6	push_local_registry artifact cascade	push_local_registry (gfx950-rocm720)	N/A (CI plumbing)	N/A	New Jul 12 (gfx950 leg)	[FACT] cascade — no artifact from failed publish

Cluster: MORI cmake gtest-discovery build failure

• Affected: (publish gfx942-rocm720, N/A, N/A) Jul 11 + Jul 12; (publish gfx950-rocm720, N/A, N/A) Jul 12.
• Shared facts: identical stack — MORI build stage stage-4 23/28 (docker/rocm.Dockerfile), cmake-4.4/Modules/GoogleTest/ParseTestList.cmake:96 → string(JSON) fails on Line 1, Column 1 … value, object or array expected → ninja: build stopped → setup.py develop exit 1. MORI pin e31d426a, base-image digest, and docker/rocm.Dockerfile all FACT-verified unchanged across the pass→fail boundary; the only changed input is the unpinned pip cmake.
• Top hypothesis: [HIGH] unpinned cmake auto-upgraded 4.3.4→4.4.0 (log-verified) at the exact Jul-10→Jul-11 pass→fail boundary; cmake 4.4's stricter GoogleTest JSON parser rejects the empty output MORI's umbp test binaries emit at build time. Disconfirming: gfx950 leg passed Jul 11 under the same cmake 4.4.0 and only failed Jul 12 — cmake 4.4 makes discovery fragile/intermittent, not 100% deterministic (does not overturn the version-drift trigger).
• In-flight fix: ❌ none found (searched open PRs for mori+gtest, mori+cmake, cmake+pin, gtest_discover, rocm+docker+mori — all hits unrelated).
• Suggested triage: pin cmake==4.3.4 (or <4.4) at docker/rocm.Dockerfile:361 and re-run both publish legs — cheapest confirm+likely fix; alternatively set MORI DISCOVERY_MODE PRE_TEST / disable build-time gtest discovery so GPU-linked test binaries aren't run in the builder.

Cluster: push_local_registry artifact cascade

• Affected: (push_local_registry gfx942-rocm720, N/A, N/A) Jul 11 + Jul 12; (push_local_registry gfx950-rocm720, N/A, N/A) Jul 12.
• Shared facts: each fails at its first step, actions/download-artifact@v4: Artifact not found for name: image-tag-<arch>-rocm720. Per release-docker-amd-rocm720-nightly.yml:106-116,136-150, the artifact is uploaded only after a successful publish build+push; push_local_registry runs needs: publish under if: !cancelled(), so it always attempts and fails when publish produced no image.
• Top hypothesis: [FACT] pure downstream cascade of the MORI cluster above — not an independent failure. Resolving the MORI build resolves all three.
• In-flight fix: ❌ none (same root cause as cluster A).
• Suggested triage: none independently — fix the publish MORI build; these clear automatically. push_local_registry red is a symptom, not the bug.

Suggested triage order

Rank	Triage step	Targets	Why this first
1	Pin cmake==4.3.4/<4.4 at docker/rocm.Dockerfile:361, re-run both publish legs	Cluster A (rows 1-3)	[HIGH] version-drift is the one changed input at the pass→fail boundary; single cheap change confirms + likely fixes
2	Re-verify the 3 push_local_registry legs go green after step 1	Cluster B (rows 4-6)	Pure cascade — should self-resolve once publish produces an artifact
3	If pinning is undesirable, disable MORI build-time gtest discovery (DISCOVERY_MODE PRE_TEST / tests off) + digest-pin base image & build toolchain	Cluster A (durable)	Removes the "unpinned dependency drift breaks a green build with no sglang change" recurrence class

---

Job	Failed Steps	Started
push_local_registry (gfx942-rocm720)	Download image tag artifact	2026-07-11T15:00
publish (gfx942-rocm720, all)	Build and Push to rocm/sgl-dev	2026-07-11T12:19
publish (gfx942-rocm720, all)	Build and Push to rocm/sgl-dev	2026-07-12T12:20
publish (gfx950-rocm720, all)	Build and Push to rocm/sgl-dev	2026-07-12T12:19
push_local_registry (gfx950-rocm720)	Download image tag artifact	2026-07-12T14:39
push_local_registry (gfx942-rocm720)	Download image tag artifact	2026-07-12T14:39

Per-Job Analysis

push_local_registry (gfx942-rocm720) — failed step(s): Download image tag artifact

Commit Info

• sglang: 32cb89d41
• aiter: N/A (docker-build workflow; no [CI-AITER-CHECK] markers in either log)
• MORI: e31d426a (pinned in docker/rocm.Dockerfile:113, verified in build log: "HEAD is now at e31d426a feat(ep): add AccumNum=9 vec8 intranode combine for shared-expert fusion (#396)")

Failed Tests

This is not a test failure — it is a CI pipeline/build failure. Two levels:

Stage	What failed	Error	Log
push_local_registry (gfx942-rocm720) (job 86555992047)	actions/download-artifact@v4 step	Unable to download artifact(s): Artifact not found for name: image-tag-gfx942-rocm720	Log
publish (gfx942-rocm720, all) (job 86543981482, root cause)	docker build [stage-4 23/28] MORI build	CMake Error … ParseTestList.cmake:96 … string sub-command JSON failed parsing json string: Line 1, Column 1 Syntax error: value, object or array expected → ninja: build stopped → exit code: 1	Log

Failure Summary

The reported job push_local_registry (gfx942-rocm720) did not do any real work: it fails immediately at the artifact-download step because the artifact image-tag-gfx942-rocm720 does not exist. That artifact is only uploaded by the upstream publish (gfx942-rocm720, all) job after a successful rocm/sgl-dev push. The publish job failed during docker build at the MORI build step ([stage-4 23/28]), so no artifact was produced. This cascade is by design — the workflow comment (release-docker-amd-rocm720-nightly.yml:129-135) notes that publish legs without an artifact fail at download and "be the only ones marked red." The gfx950 legs of both publish and push_local_registry succeeded in the same run.

Regression Status

New failure today (first failure in the queryable window). The MORI build step of publish (gfx942-rocm720) — and therefore push_local_registry (gfx942-rocm720) — passed on every prior scheduled run.

Recent history of release-docker-amd-rocm720-nightly.yml (branch main, event=schedule, completed runs only). gfx942 build/push tracked:

Date	Run	gfx942 publish+push	gfx950 publish+push
Jul 11	29152436775	❌ Failed (MORI gtest-discover)	✅ Passed
Jul 10	29093667269	✅ Passed	✅ Passed
Jul 09	29019594442	✅ Passed	✅ Passed
Jul 08	28942988408	✅ Passed	✅ Passed
Jul 07	28867437602	✅ Passed	✅ Passed

First observed failure: 2026-07-11 (run 29152436775). Last known passing: 2026-07-10 (run 29093667269).

Failure Cluster

"MORI build failure — gtest_discover_tests JSON parse error during ROCm docker image build (gfx942)"

Facts (verified observations, no interpretation)

• Reported job 86555992047 failed at step 2 (actions/download-artifact@v4): Artifact not found for name: image-tag-gfx942-rocm720 (log line 28). No build/test work ran in this job.
• The artifact is uploaded only by the publish job's "Upload image tag artifact" step, which is gated behind a successful rocm/sgl-dev push (release-docker-amd-rocm720-nightly.yml:106-116).
• Upstream publish (gfx942-rocm720, all) 86543981482 failed in docker build … -f docker/rocm.Dockerfile at [stage-4 23/28] (the ENABLE_MORI=1 step).
• Exact error (publish log ~line 23771): linking src/umbp/tests/test_client_registry_external_kv succeeded, then cmake's gtest_discover_tests post-build step ran the binary and ParseTestList.cmake:96 failed: string sub-command JSON failed parsing json string … Line 1, Column 1 Syntax error: value, object or array expected → ninja: build stopped: subcommand failed → subprocess.CalledProcessError: Command '['cmake', '--build', '.', '-j', '64']' returned non-zero exit status 1.
• MORI commit is a hardcoded pin e31d426a (docker/rocm.Dockerfile:113), last modified 2026-07-09 by e703f9e56; runs on Jul 09 and Jul 10 passed with this same pin.
• --no-cache is used, so MORI is re-cloned and rebuilt from identical source each run.
• The gfx950-rocm720 leg passed in the same run: publish (gfx950-rocm720, all) 86543981508 ✅ and push_local_registry (gfx950-rocm720) 86555992039 ✅.
• No [CI-AITER-CHECK] markers in either log; aiter is not involved.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [MEDIUM] Transient/flaky failure in MORI's gtest_discover_tests step (the just-built test_client_registry_external_kv binary produced empty/non-JSON output when run at build time, which is exactly what yields the Line 1, Column 1 JSON parse error).
  • Supporting: MORI source is byte-identical to Jul 09/10 (pinned commit unchanged, --no-cache), yet those runs passed; the gfx950 leg passed in this very run with the same Dockerfile and MORI commit; gtest_discover_tests executes the test binary during docker build, and a crash/segfault/empty stdout there produces precisely this cmake JSON error; only 1 of 146 ninja targets failed (a single test's discovery), the rest linked fine.
  • Disconfirming: no direct log evidence of why the binary produced empty output (no segfault/stack trace captured — cmake swallows the binary's own stderr); cannot rule out a host-specific toolchain/cmake state difference on the gfx942 build container.
  • How to verify: re-run the failed publish (gfx942-rocm720) job (or the whole workflow) — if the next run is green with no code change, flakiness is confirmed. For a deeper repro: docker build . -f docker/rocm.Dockerfile --build-arg GPU_ARCH=gfx942-rocm720 --build-arg ENABLE_MORI=1 --no-cache on an amd-docker-scale host and inspect whether ./src/umbp/tests/test_client_registry_external_kv --gtest_list_tests emits valid output.
• [LOW] Environment/toolchain issue in MORI's CMake config (cmake 4.4 GoogleTest module strictness with the ROCm 7.2 clang toolchain) intermittently mis-handling the discovery output.
  • Supporting: the failure is inside cmake-4.4/Modules/GoogleTest/ParseTestList.cmake, a third-party module invoked by MORI's build.
  • Disconfirming: same cmake/toolchain passed on gfx950 in the same run and on gfx942 on prior days — a deterministic toolchain bug would fail consistently.
  • How to verify: check whether repeated gfx942 builds fail deterministically at this same target; if so, escalate upstream to ROCm/mori.
• [SPECULATION] sglang code change in 32cb89d41 or the Jul 10→11 delta caused this.
  • Supporting: none — this is the head SHA that failed.
  • Disconfirming: the failing step builds the external MORI repo at a fixed pin and does not compile any sglang source; the failure is in src/umbp/tests/ of MORI, unreachable from sglang code. This is almost certainly not an sglang regression.

In-flight Fix Check

Searched open PRs with keywords mori+gtest and mori+docker:

• In-flight fix: ❌ none found — the one mori+gtest hit (#22037 "Multi-stage build for ROCm image") and the mori+docker hits (#30651, #30052) are unrelated to this gtest-discovery build break. (API: search/issues?q=repo:sgl-project/sglang+is:pr+is:open+mori+gtest)

Suggested Triage Steps

• Re-run first. Re-run publish (gfx942-rocm720, all) (or dispatch the workflow with job_select=publish). Given the identical-source-passed-yesterday + gfx950-passed-same-run evidence, a clean re-run is the cheapest disconfirmation of the flakiness hypothesis.
• If the re-run is green: mark this as a transient MORI-build flake; no sglang action needed.
• If it fails deterministically at the same test_client_registry_external_kv discovery: reproduce locally with docker build … --build-arg GPU_ARCH=gfx942-rocm720 --build-arg ENABLE_MORI=1 --no-cache and run test_client_registry_external_kv --gtest_list_tests inside the container to capture the binary's actual (missing) output; escalate to ROCm/mori as a build-time gtest-discovery issue.
• Consider (maintainer's call) making MORI's test discovery non-fatal to the image build (e.g., building MORI with tests disabled), so a test-discovery hiccup cannot block the nightly image publish. Bumping the MORI_COMMIT pin (docker/rocm.Dockerfile:113) is only warranted if upstream MORI has a fix.

Status

• Persistence: New today — first failure at 2026-07-11 12:19 UTC (run 29152436775); previous 7 scheduled runs (Jul 04–10) were green.
• Scope: gfx942-rocm720 leg only (2 jobs: publish root cause + push_local_registry cascade). gfx950-rocm720 unaffected. Single workflow (release-docker-amd-rocm720-nightly.yml).
• Blocked work: nightly ROCm 7.2 rocm/sgl-dev / lmsysorg/sglang-rocm mi30x (gfx942) image was not published today, and its local-registry mirror did not run. mi35x (gfx950) image published normally.
• In-flight fix: ❌ none found — needs a re-run to confirm flakiness before any new PR.

publish (gfx942-rocm720, all) — failed step(s): Build and Push to rocm/sgl-dev

Commit Info

• sglang: 32cb89d41
• aiter: N/A (docker-build workflow — no [CI-AITER-CHECK] markers in log)

Failed Tests

Test File	Test Function	Error	Log
N/A (Docker image build)	N/A	Docker build failed at [stage-4 23/28] (MORI build); gtest_discover_tests for test_client_registry_external_kv → CMake Error … string sub-command JSON failed parsing json string … Syntax error: value, object or array expected → ninja: build stopped → setup.py … CalledProcessError → exit code 1	Job log

This is not a pytest/unittest failure — it is a container image build failure in the publish (gfx942-rocm720, all) job.

Failure Summary

The nightly ROCm 7.2 image publish job failed while building the optional MORI component (ENABLE_MORI=1, NIC_BACKEND=none) inside docker/rocm.Dockerfile at build stage stage-4 23/28. MORI's CMake build linked its gtest executables successfully, then CMake's gtest_discover_tests POST_BUILD step for test_client_registry_external_kv ran the freshly-linked binary to enumerate its tests and tried to parse the (empty/invalid) --gtest_list_tests JSON output; CMake 4.4's GoogleTest/ParseTestList.cmake:96 raised a hard JSON parse error, stopping ninja and failing setup.py develop, which failed the Docker build.

Regression Status

New regression (first failure 2026-07-11). The previous 7 consecutive scheduled runs (Jul 4–Jul 10) all succeeded.

Recent history of the publish (gfx942-rocm720, all) job / MORI build stage (release-docker-amd-rocm720-nightly.yml, branch=main, event=schedule, completed runs only):

Date	Run	Job	MORI Build Status	Detail
Jul 11	29152436775	publish (gfx942-rocm720, all)	❌ Failed	gtest_discover_tests JSON parse error (cmake 4.4)
Jul 10	29093667269	publish (gfx942-rocm720, all)	✅ Passed	[MORI] Done. (cmake 4.3)
Jul 9	29019594442	publish (gfx942-rocm720, all)	✅ Passed	—
Jul 8	28942988408	publish (gfx942-rocm720, all)	✅ Passed	—
Jul 7	28867437602	publish (gfx942-rocm720, all)	✅ Passed	—

First observed failure: 2026-07-11. Last known passing: 2026-07-10 (run 29093667269).

Failure Cluster

MORI Docker build failure — CMake gtest_discover_tests JSON parse error (empty gtest list output)

Facts (verified observations)

• Docker build invocation: docker build . -f docker/rocm.Dockerfile … --build-arg ENABLE_MORI=1 … --no-cache (log line 1492). --no-cache ⇒ every stage, including MORI, rebuilds from scratch each night.
• Failing stage: #28 [stage-4 23/28] MORI build; env [MORI] Enabling MORI (NIC_BACKEND=none), MORI_GPU_ARCHS=gfx942 (log lines 22006, 22430).
• Verbatim error (log line ~23769–23778):

  FAILED: … src/umbp/tests/test_client_registry_external_kv …
  CMake Error at /opt/venv/lib/python3.10/site-packages/cmake/data/share/cmake-4.4/Modules/GoogleTest/ParseTestList.cmake:96 (string):
    string sub-command JSON failed parsing json string:
      * Line 1, Column 1
      Syntax error: value, object or array expected.
  Call Stack: … GoogleTestAddTests.cmake:245 (parse_tests_from_json)
              test_client_registry_external_kv_e3b0c442_discovery.cmake:2 (gtest_discover_tests_impl)
  ninja: build stopped: subcommand failed.
  subprocess.CalledProcessError: Command '['cmake', '--build', '.', '-j', '64']' returned non-zero exit status 1.
  

• The gtest executable linked OK immediately before the failure ([129/146] Linking CXX executable src/umbp/tests/test_client_registry_external_kv) — this is a test-discovery failure, not a compile/link failure.
• CMake version delta (the key fact): last passing run (Jul 10) used pip cmake 4.3; this failing run (Jul 11) used pip cmake 4.4 (verified from cmake-4.x module paths in both logs).
• MORI_COMMIT is pinned to e31d426a and TILELANG_COMMIT to a fixed SHA (docker/rocm.Dockerfile:113, :103) — unchanged across both runs.
• cmake is unpinned in the Dockerfile: docker/rocm.Dockerfile:361 runs pip install --upgrade … cmake ninja …, so each --no-cache nightly build pulls whatever cmake is newest on PyPI.
• No sglang commit in the regression window 94de28764..32cb89d41 (30 commits) touched docker/rocm.Dockerfile, and none mention mori/cmake/gtest/tilelang/docker.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [HIGH] The unpinned pip cmake upgrade 4.3 → 4.4 broke MORI's gtest_discover_tests. CMake 4.4's GoogleTest module parses --gtest_list_tests output as JSON via string(JSON …) and hard-fails when that output is empty/invalid; the test_client_registry_external_kv binary evidently emits no valid test-list JSON when run at Docker build time.
  • Supporting: the error is thrown from cmake 4.4's own module (cmake-4.4/…/GoogleTest/ParseTestList.cmake:96); Jul 10 built the identical pinned MORI commit successfully with cmake 4.3; cmake is the only environmental input that changed (verified 4.3→4.4); MORI/TileLang commits pinned; no sglang Dockerfile change in window.
  • Disconfirming: MORI submodules float (git submodule update --init --recursive with no pin), so a MORI 3rdparty submodule change could in principle have altered the test binary so it now emits empty list output — but even so, cmake 4.3 tolerated whatever the binary produced and 4.4 does not, so the cmake bump is the proximate trigger.
  • How to verify: rebuild the image pinning cmake to the last-good version, e.g. add "cmake==4.3.*" (or <4.4) to docker/rocm.Dockerfile:361, and re-run the MORI stage; or git bisect-style A/B the two cmake wheels against the same pinned MORI commit in container rocm/sgl-dev.
• [LOW] A floating MORI 3rdparty submodule regressed the test_client_registry_external_kv gtest so it now produces empty --gtest_list_tests output.
  • Supporting: MORI submodules are not pinned.
  • Disconfirming: does not explain why cmake 4.3 succeeded on Jul 10; the discovery-step JSON error is cmake-version-specific.
  • How to verify: diff MORI submodule SHAs between a Jul 10 and Jul 11 build; run the linked test binary manually with --gtest_list_tests --gtest_output=json to inspect its output.

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs for mori+cmake, cmake+pin, gtest_discover, rocm+docker+mori; matches returned (#30651, #22646, #20588, etc.) are unrelated MORI/CMake features, none address the cmake-4.4 gtest-discovery breakage.

Suggested Triage Steps

• Pin cmake below 4.4 in docker/rocm.Dockerfile:361 (e.g. pip install --upgrade "cmake<4.4" ninja …) and re-run the publish (gfx942-rocm720, all) job to confirm the MORI stage builds again — this both verifies the hypothesis and is the likely fix direction.
• If pinning cmake resolves it, consider disabling MORI's gtest discovery at build time (e.g. -DMORI_BUILD_TESTS=OFF if MORI exposes such a flag, or gtest_discover_tests(... DISCOVERY_MODE PRE_TEST) upstream) so image builds don't depend on running test binaries in a GPU-less builder.
• As a durable safeguard, pin MORI submodules and all unpinned pip build tools (cmake, ninja, scikit-build-core) so --no-cache nightly builds are reproducible.
• To confirm cmake as the sole cause, A/B build the pinned MORI commit e31d426a with cmake 4.3 vs 4.4 in the rocm/sgl-dev builder.

Status

• Persistence: First failure 2026-07-11 (1 completed run); previous 7 scheduled runs (Jul 4–10) all passed.
• Scope: publish (gfx942-rocm720, all) job of release-docker-amd-rocm720-nightly.yml; blocks nightly ROCm 7.2 image publication. Other ROCm/docker nightly images using the same unpinned-cmake + MORI pattern in docker/rocm.Dockerfile are likely at risk of the same break.
• Blocked work: Nightly rocm/sgl-dev:*-rocm720 image is not published for 2026-07-11.
• In-flight fix: ❌ none found in open PRs — needs a new PR (pin cmake <4.4) or investigation.

publish (gfx942-rocm720, all) — failed step(s): Build and Push to rocm/sgl-dev

Commit Info

• sglang: 80856aba (Make the mxfp8 MoE runner backend list extensible (#30828))
• aiter: N/A (docker-build workflow — no [CI-AITER-CHECK] markers in log)
• MORI: e31d426a (pinned in docker/rocm.Dockerfile:113, unchanged across pass/fail runs)

Failed Tests

Test File	Test Function	Error	Log
N/A — Docker image build failure	Dockerfile step #28 (stage-4 23/28, MORI build)	CMake Error at .../GoogleTest/ParseTestList.cmake:96: string sub-command JSON failed parsing json string: Syntax error → ninja: build stopped → CalledProcessError: 'cmake --build .' returned non-zero exit status 1	Log

This is not an sglang unit test. It is a build failure of the ROCm/mori dependency, which is git-cloned and compiled inside the ROCm image (python3 setup.py develop). Two MORI gtest binaries (test_ssd_read_lease_gating, test_ssd_reliability under src/umbp/tests) fail CMake's gtest_discover_tests POST_BUILD step.

Failure Summary

The publish (gfx942-rocm720, all) job builds the nightly ROCm 7.2 Docker image. Dockerfile step #28 clones and builds MORI; the build reaches [129/146] Linking … test_ssd_read_lease_gating, then CMake runs that freshly-linked binary to enumerate its gtest cases. The binary produces empty output, so ParseTestList.cmake:96 fails to parse the (empty) JSON test list ("Line 1, Column 1 … value, object or array expected"). Ninja stops, setup.py develop returns non-zero, and the Docker build exits 1.

Regression Status

Known recurring failure — started 2026-07-11, now 2 consecutive completed runs.

Recent history of the publish (gfx942-rocm720, all) job in release-docker-amd-rocm720-nightly.yml (branch main, event schedule, completed runs only):

Date	Run	Job Status	Failure
Jul 12	29192413224	❌ Failed	MORI ParseTestList.cmake:96 JSON parse (MORI e31d426a) — (overall run still in_progress, but this job completed as failed)
Jul 11	29152436775	❌ Failed	Identical MORI ParseTestList.cmake:96 JSON parse, same MORI e31d426a
Jul 10	29093667269	✅ Passed	—
Jul 09	29019594442	✅ Passed	—
Jul 08	28942988408	✅ Passed	—

First observed failure: 2026-07-11. Last known pass: 2026-07-10.

Failure Cluster

MORI gtest discovery JSON parse failure during ROCm image build (ParseTestList.cmake empty-JSON).

Facts (verified observations, no interpretation)

• Docker step #28 [stage-4 23/28] failed; the whole job failed with ##[error]Process completed with exit code 1 (log tail).
• Verbatim error (log line ~23753): CMake Error at /opt/venv/lib/python3.10/site-packages/cmake/data/share/cmake-4.4/Modules/GoogleTest/ParseTestList.cmake:96 (string): string sub-command JSON failed parsing json string: … Syntax error: value, object or array expected. Triggered from test_ssd_read_lease_gating_e3b0c442_discovery.cmake:2 (gtest_discover_tests_impl) and again for test_ssd_reliability.
• Terminal errors: ninja: build stopped: subcommand failed → subprocess.CalledProcessError: Command '['cmake', '--build', '.', '-j', '64']' returned non-zero exit status 1 (from /sgl-workspace/mori/setup.py:609).
• CMake in use: 4.4 (from /opt/venv/.../cmake/data/share/cmake-4.4). Toolchain: clang++ from /opt/rocm-7.2.0/lib/llvm.
• MORI commit is identical in the Jul 11 fail log (switching to 'e31d426a…') and pinned unchanged at docker/rocm.Dockerfile:113.
• git log 94de28764c..32cb89d412 -- docker/rocm.Dockerfile → empty: no change to the ROCm Dockerfile (nor the MORI block) between the last passing (Jul 10) and first failing (Jul 11) sglang head SHAs. The window is 30 sglang commits; none touch the MORI build path.
• MORI ARG block was added by 0d7e8cfb (2026-07-09), which predates the last green run (Jul 10) — so MORI built fine with this exact pin on Jul 10.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [MEDIUM] Environmental regression in the ROCm 7.2 base image / build toolchain (not an sglang code change). The MORI test binary emits empty output at gtest_discover_tests time, and the only variable that changed between the Jul 10 pass and Jul 11 fail is the freshly-pulled base image / apt+pip toolchain (MORI source pin and sglang Dockerfile are both unchanged).
  • Supporting: MORI pin identical across runs; zero docker/MORI-related sglang commits in the pass→fail window; failure is deterministic across two consecutive nightlies; cmake-4.4 is a very recent CMake whose gtest_discover_tests treats empty binary output as a hard error.
  • Disconfirming: I did not diff the actual base-image digests between Jul 10 and Jul 11, so the environmental delta is inferred, not directly observed. The empty output could alternatively stem from the SSD-oriented test binaries needing a device/mount absent in the build container — but that would presumably have failed on Jul 10 too.
  • How to verify: rerun the Jul 10 image build (SHA 94de28764c) today; if it now also fails with the same MORI error at the same pin, the regression is environmental (base image / cmake), confirming sglang is exonerated. Compare docker pull digests of the ROCm base image between Jul 10 and Jul 11.
• [LOW] Analyzed sglang commit 80856aba caused it.
  • Supporting: it is the head SHA of the failing run.
  • Disconfirming: it only touches the mxfp8 MoE runner backend list (Python) — nothing in the Docker/MORI build path; the identical failure already occurred on Jul 11 at a different head SHA (32cb89d4). Effectively ruled out.
• [LOW] A MORI-upstream bug at pin e31d426a where test_ssd_* binaries don't list tests.
  • Supporting: the failing binaries are MORI's own SSD tests.
  • Disconfirming: same pin built successfully on Jul 10; a source-only bug at a fixed commit cannot regress on its own.

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs for mori (48 open, all feature/PD work — e.g. #30706, #30651, #27093; none address the gtest-discovery build break) and rocm docker (58 open; #27054 "Fix rocm.Dockerfile" and #27209 gfx950 support are unrelated to this MORI/gtest failure). API: search/issues?q=repo:sgl-project/sglang+is:pr+is:open+mori and …+rocm+docker.

Suggested Triage Steps

• Re-run the ROCm image build at the last green sglang SHA 94de2876 today. If it now fails identically, the regression is environmental (base image / cmake 4.4), confirming no sglang-side fix is needed.
• Diff the ROCm base-image digest and installed cmake version between the Jul 10 image and today's; a bumped cmake that hardened gtest_discover_tests empty-output handling is the leading suspect.
• As a candidate MORI-side mitigation to investigate (not a directive): set DISCOVERY_MODE PRE_TEST or TEST_LIST / disable gtest_discover_tests for the test_ssd_* targets so image build no longer executes the test binary at build time — this belongs upstream in ROCm/mori, or via bumping MORI_COMMIT to a commit that fixes it.
• If the base-image delta is confirmed as cause, consider pinning the ROCm base image digest in docker/rocm.Dockerfile to restore reproducibility.

Status (factual, no priority assignment)

• Persistence: Failing since 2026-07-11 (2 consecutive completed nightly runs: Jul 11 + Jul 12).
• Scope: publish (gfx942-rocm720, all) in release-docker-amd-rocm720-nightly.yml; the sister push_local_registry (gfx942-rocm720) job also failed on Jul 11 (downstream of the failed publish/build).
• Blocked work: No nightly ROCm 7.2 (gfx942) Docker image has been published since 2026-07-10.
• In-flight fix: ❌ none found in open PRs — needs new investigation (environmental / MORI-side).

publish (gfx950-rocm720, all) — failed step(s): Build and Push to rocm/sgl-dev

Commit Info

• sglang: 80856aba85c6
• aiter: N/A — docker-build workflow, no [CI-AITER-CHECK] markers in log. (For reference, the Dockerfile clones aiter at 9127c94a1; not relevant to this failure.)

Failed Tests

This is a docker image build failure, not a pytest run. The build breaks while compiling the MORI dependency (ROCm/mori, pinned at e31d426a in docker/rocm.Dockerfile:113) inside the image.

Build Step	Failing Component	Error	Log
stage-4 23/28 MORI build (cmake --build)	mori gtest discovery: test_external_kv_hit_index, test_client_registry	CMake Error … string sub-command JSON failed parsing json string: Syntax error: value, object or array expected → ninja: build stopped → CalledProcessError: cmake --build … exit 1	Log

Failure Summary

The docker build step of the nightly ROCm 7.2 image publish fails at rocm.Dockerfile:421 while building the MORI dependency. Two of MORI's umbp test targets fail during gtest_discover_tests (CMake's post-build test enumeration): the discovery step runs the freshly-built test binary to emit its test list as JSON, but the binary produces empty output, so cmake-4.4.0's ParseTestList.cmake errors on the empty string and ninja stops. The image is never built or pushed.

Regression Status

Flaky / non-deterministic MORI build failure (workflow-level regression first appeared 2026-07-11).

Recent history of job publish (gfx950-rocm720, all) (workflow release-docker-amd-rocm720-nightly.yml, branch main, event schedule, completed runs only):

Date	Run	gfx950 leg	gfx942 leg	MORI build result
Jul 12	29192413224	❌ Failed	❌ Failed	JSON-discovery error on test_external_kv_hit_index, test_client_registry
Jul 11	29152436775	✅ Passed	❌ Failed	gfx942 failed on test_client_registry_external_kv; gfx950 built cleanly
Jul 10	29093667269	✅ Passed	✅ Passed	—
Jul 09	29019594442	✅ Passed	✅ Passed	—
Jul 08	28942988408	✅ Passed	✅ Passed	—

For this specific gfx950 leg, today is the first failure; the underlying MORI-build cluster first appeared 2026-07-11 on the gfx942 leg. The set of failing mori test targets differs between runs (Jul 11 gfx942: test_client_registry_external_kv; Jul 12 gfx950: test_external_kv_hit_index + test_client_registry), which is characteristic of a non-deterministic failure rather than a fixed code break.

Failure Cluster

MORI build failure — empty-JSON in gtest_discover_tests under cmake-4.4.0

Facts (verified observations)

• The failing Dockerfile step is stage-4 23/28 at rocm.Dockerfile:421 (the MORI build block). Verbatim error: subprocess.CalledProcessError: Command '['cmake', '--build', '.', '-j', '64']' returned non-zero exit status 1.
• Root cmake error (verbatim): CMake Error at …/cmake-4.4/Modules/GoogleTest/ParseTestList.cmake:96 (string): string sub-command JSON failed parsing json string: … Line 1, Column 1 Syntax error: value, object or array expected. Call stack → GoogleTestAddTests.cmake:245 (parse_tests_from_json) → <target>_e3b0c442_discovery.cmake:2 (gtest_discover_tests_impl).
• docker/rocm.Dockerfile was not modified in the regression window: git log 94de28764c88..80856aba85c6 -- docker/rocm.Dockerfile returns empty. MORI_COMMIT is pinned at e31d426a and unchanged.
• The build is --no-cache and the TileLang step (stage-4 20/28) runs pip install --upgrade … cmake …, so cmake is pulled fresh from PyPI on every run. Both Jul 11 and Jul 12 runs downloaded cmake-4.4.0 (cmake-4.4.0-py3-none-manylinux…whl).
• Same cmake version, opposite outcomes: in the Jul 11 run, the gfx950 leg (86543981508) built MORI cleanly with cmake-4.4.0 (0 JSON-parse errors, [MORI] Done), while the gfx942 leg (86543981482) failed with the same JSON error under the same cmake-4.4.0.
• 43 sglang commits are in the window 94de28764c88..80856aba85c6; none touch the docker recipe or the MORI build.

Hypothesised Causes (with confidence)

• [MEDIUM] Interaction between cmake-4.4.0's stricter GoogleTest JSON parser and MORI's gtest_discover_tests running host/GPU test binaries at docker build time (which intermittently emit empty output).
  • Supporting: the error originates entirely inside cmake-4.4.0's ParseTestList.cmake/GoogleTestAddTests.cmake; it is triggered by an empty string being passed to string(JSON …). cmake is freshly upgraded each --no-cache build, so a recent cmake bump aligns with the Jul-11 onset.
  • Disconfirming: cmake-4.4.0 was also used in the Jul 11 gfx950 run that passed — so cmake 4.4.0 alone is not sufficient to cause the failure; an additional non-deterministic factor (empty test-binary output) is required.
  • How to verify: rebuild the image at this SHA with cmake pinned to the last-known-good version (e.g. pip install "cmake<4.4" in the TileLang/MORI steps) and re-run; separately, run one of the failing binaries (./test_client_registry --gtest_list_tests) manually inside the image to see whether it emits valid JSON.
• [LOW] Non-deterministic failure of the MORI test binaries themselves at build time (e.g. a binary that crashes / writes nothing when executed on a GPU-less build host), independent of cmake version.
  • Supporting: the failing target set varies run-to-run (test_client_registry_external_kv vs test_external_kv_hit_index+test_client_registry); gfx950 passed and gfx942 failed within the same run.
  • Disconfirming: builds were consistently green through Jul 10, so whatever changed (most plausibly the cmake/toolchain floating version) is external and recent.
  • How to verify: add --gtest_list_tests diagnostics or set DISCOVERY_MODE=PRE_TEST for mori's gtest_discover_tests, or disable mori test-target discovery in the build.

This failure is not attributable to any sglang commit — the docker recipe and MORI pin are unchanged across the passing→failing window. Root cause lies in the build environment (floating cmake version) and/or upstream MORI test-discovery behavior.

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs for mori (48 results, all feature/CI-integration PRs, none addressing this build break), cmake+gtest (0), gtest_discover (0), rocm+docker+mori (12, none a fix). No open PR pins cmake or fixes MORI gtest discovery. API: search/issues?q=repo:sgl-project/sglang+is:pr+is:open+<kw>.

Suggested Triage Steps

• Confirm the cmake floating-version hypothesis: rebuild docker/rocm.Dockerfile at 80856aba85c6 with cmake pinned below 4.4 (the MORI/TileLang steps currently do pip install --upgrade … cmake …) and re-run the publish (gfx950-rocm720, all) leg.
• If cmake is implicated, consider pinning cmake in the Dockerfile MORI/TileLang steps for reproducible builds (currently unpinned + --no-cache pulls latest PyPI each night).
• Inside the built image, run the failing discovery manually (build/src/umbp/tests/test_client_registry --gtest_list_tests --gtest_output=json) to see whether the binary emits empty JSON on the GPU-less build host; if so, the fix belongs upstream in MORI (e.g. DISCOVERY_MODE=PRE_TEST) or by disabling mori test-target discovery in the docker build.
• Track upstream at ROCm/mori e31d426a; a MORI-pin bump to a commit that hardens gtest_discover_tests would also resolve this.

Status (factual)

• Persistence: MORI-build cluster failing since 2026-07-11 (first on gfx942 leg); both matrix legs failing as of 2026-07-12. Intermittent — one leg passed as recently as Jul 11 under the same cmake-4.4.0.
• Scope: workflow release-docker-amd-rocm720-nightly.yml — both publish legs (gfx942 + gfx950) failed today; downstream push_local_registry legs also failed (no image artifact was produced to mirror). Blocks nightly publish of rocm/sgl-dev / lmsysorg/sglang-rocm ROCm 7.2 images.
• In-flight fix: ❌ none found in open PRs — needs a new PR (pin cmake or fix/skip MORI gtest discovery) or upstream MORI fix.

push_local_registry (gfx950-rocm720) — failed step(s): Download image tag artifact

Commit Info

• sglang: 80856aba85c60eb8a5ffe294132cc76acf7af67f
• aiter: N/A — this is a docker-build workflow; no [CI-AITER-CHECK] markers are emitted.
• MORI: e31d426a13e96e1cbff96a1c904d291aefe8c46a (pinned in docker/rocm.Dockerfile:113, unchanged since #29394 on 2026-07-06)

Failed Tests

Test File	Test Function	Error	Log
N/A (build/infra)	N/A	Unable to download artifact(s): Artifact not found for name: image-tag-gfx950-rocm720	Log

This job runs no tests. Its failure is a downstream cascade of the publish (gfx950-rocm720, all) docker build, whose root failure is:

Component	Failing step	Error	Log
MORI 3rd-party build	gtest_discover_tests (cmake --build)	ParseTestList.cmake:96 string(JSON) — Syntax error: value, object or array expected → ninja: build stopped → CalledProcessError: cmake --build returned exit 1	Log

Failure Summary

push_local_registry (gfx950-rocm720) failed at its first step because the image-tag-gfx950-rocm720 artifact does not exist. The workflow uploads that artifact only after publish (gfx950-rocm720, all) successfully builds and pushes the docker image. The publish job failed during the MORI (ROCm/mori) build: cmake's build-time gtest_discover_tests ran the freshly-linked umbp test executables (test_external_kv_hit_index, test_client_registry), which produced empty/invalid output, so ParseTestList.cmake failed to parse the test-list JSON and ninja aborted. All 4 jobs in this run failed (both publish legs, both push_local_registry legs).

Regression Status

New regression — began 2026-07-11 (gfx942 leg), spread to both arches 2026-07-12.

Recent history of the gfx950 docker-build chain (publish (gfx950-rocm720) → push_local_registry (gfx950-rocm720)), event=schedule, main, completed runs only:

Date	Run	publish (gfx950)	push_local_registry (gfx950)	Note
Jul 12	29192413224	❌ MORI build fail	❌ cascade (no artifact)	both arches fail
Jul 11	29152436775	✅ Passed	✅ Passed	gfx942 leg failed same way
Jul 10	29093667269	✅ Passed	✅ Passed	all green
Jul 09	29019594442	✅ Passed	✅ Passed	all green
Jul 08	28942988408	✅ Passed	✅ Passed	all green

• First failure of the gfx950 chain: 2026-07-12. First failure of the gfx942 chain: 2026-07-11 (job, same ParseTestList.cmake signature). Last fully-green run: 2026-07-10.
• Non-deterministic: on 2026-07-11 the gfx950 leg passed while gfx942 failed with the identical error — same Dockerfile, same MORI pin, same base image. This variance across arch/day is the key signal.

Failure Cluster

MORI docker build fails in cmake gtest_discover_tests (empty test-list JSON) → publish image never built → downstream push_local_registry artifact-not-found cascade.

Facts (verified observations)

• push_local_registry (gfx950-rocm720) error (verbatim): ##[error]Unable to download artifact(s): Artifact not found for name: image-tag-gfx950-rocm720 (job log line 28).
• Workflow topology (release-docker-amd-rocm720-nightly.yml:106-116, 136-150): publish uploads image-tag-${gpu_arch} only after "Build and Push to rocm/sgl-dev" succeeds; push_local_registry needs: publish and runs under !cancelled(), so a leg whose publish build failed will always fail at artifact download.
• Root failure (publish gfx950, job): FAILED: [code=1] src/umbp/tests/test_external_kv_hit_index …_tests.cmake and test_client_registry; CMake Error at .../cmake-4.4/Modules/GoogleTest/ParseTestList.cmake:96 (string): string sub-command JSON failed parsing json string … Line 1, Column 1 Syntax error; then ninja: build stopped: subcommand failed → subprocess.CalledProcessError: Command '['cmake', '--build', '.', '-j', '64']' returned non-zero exit status 1 (MORI setup.py develop at stage-4 23/28).
• The failing executables link against /opt/rocm/lib/libhsa-runtime64.so and libamdhip64.so.7.2.70200 (from the FAILED link+discovery command line) and are executed at build time by cmake to enumerate gtests.
• Same signature in the 07-11 gfx942 failure: FAILED … test_client_registry_external_kv … + ParseTestList.cmake:96 (job).
• MORI pin e31d426a unchanged since 2026-07-06; docker/rocm.Dockerfile last modified 2026-07-06 — no sglang Dockerfile change between the last green (07-10) and first red (07-11) runs.
• Floating dependencies in the build: base image rocm/pytorch:rocm7.2_ubuntu22.04_py3.10_pytorch_release_2.9.1 is a mutable tag (this run resolved it to digest sha256:376bfab5…, not pinned in the Dockerfile); cmake is pip-installed unpinned — docker/rocm.Dockerfile:361 "$VENV_PIP" install --upgrade "setuptools…" wheel cmake ninja scikit-build-core, and the failing cmake is cmake-4.4.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [MEDIUM] MORI's build-time gtest_discover_tests runs the umbp test binaries (which link the ROCm HSA/HIP runtime) during docker build; with no usable GPU/HSA device at build time the binaries abort and emit empty stdout, so cmake's JSON parse of the (empty) test list fails.
  • Supporting: the FAILED command chains the link with cmake -P …_discovery.cmake; the error is Line 1, Column 1 … value, object or array expected (i.e. empty input); the executables link libhsa-runtime64.so/libamdhip64.so.
  • Disconfirming: this discovery step has apparently worked on prior nightly runs with the same MORI pin (07-06 → 07-10 green), so a build-time GPU requirement alone doesn't explain why it started failing on 07-11.
  • How to verify: on the amd-docker-scale build runner, rebuild docker/rocm.Dockerfile for gfx950-rocm720 and inspect whether test_external_kv_hit_index --gtest_list_tests (the discovery invocation) exits non-zero / prints nothing.
• [MEDIUM] A floating dependency changed between 07-10 and 07-11 — most likely the unpinned pip cmake (now 4.4) and/or a re-push of the mutable rocm/pytorch:rocm7.2… base tag — altering gtest_discover_tests behavior (newer CMake defaults to build-time discovery + JSON parsing).
  • Supporting: no sglang source/Dockerfile change in the window (MORI pin and Dockerfile static since 07-06); both cmake and the base image are unpinned; failure is CMake 4.4's ParseTestList.cmake.
  • Disconfirming: cannot confirm from the log alone that cmake's version or the base digest actually differs from the 07-10 build (that build's log/digest not compared here); the 07-11 gfx950 pass shows the change, if any, is not 100% deterministic.
  • How to verify: diff pip show cmake and the resolved base-image digest (sha256:…) between the 07-10 (green) and 07-12 (red) build logs; if cmake bumped, pin cmake in docker/rocm.Dockerfile:361 and rerun.
• [LOW] Transient build-runner/environment flake (resource pressure, GPU device disappearing under -j 64).
  • Supporting: non-deterministic across arch/day (07-11 gfx950 passed, gfx942 failed) — classic flake signature.
  • Disconfirming: 07-12 failed on both arches, and the error is a specific, repeatable cmake JSON-parse failure rather than a random timeout/OOM.
  • How to verify: re-run the 07-12 workflow; if it passes intermittently with no code change, flakiness is confirmed.

No hypothesis reaches HIGH; the sglang commit delta 94de28764c..32cb89d412 is unlikely to be causal because the Dockerfile/MORI pin did not change in it — the floating cmake/base-image dependencies are the more probable trigger.

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs for gtest_discover_tests (0), ParseTestList cmake (0), mori docker rocm build (4 results, all unrelated: #22037, #27209, #30433, #19777), rocm docker build failure cmake (3 results, all unrelated). Needs a new PR or investigation.

Suggested Triage Steps

• Compare the resolved cmake version (pip show cmake / the cmake-<ver> path in the log) and the base-image digest between the last green build (run 29093667269, Jul 10) and this red build — this cheaply confirms/refutes the floating-dependency hypothesis.
• If cmake bumped to 4.4: pin cmake to the last-known-good version at docker/rocm.Dockerfile:361 (change cmake → cmake==<good_ver>) on a branch and re-run the workflow.
• Consider disabling MORI's unit-test build/discovery in the image (e.g. a MORI CMake option analogous to -DBUILD_UNIT_TESTS=OFF, already used for Mooncake at docker/rocm.Dockerfile:267) so GPU-linked test binaries are never run at docker build time — file upstream at ROCm/mori if no such flag exists.
• Digest-pin the rocm/pytorch:rocm7.2… base tag (BASE_IMAGE_*_ROCM720, docker/rocm.Dockerfile:25,27) to make nightly builds reproducible and eliminate mutable-tag drift.
• Re-run the failed workflow once to measure flakiness; if it passes with no change, weight the [LOW] transient-environment hypothesis higher.

Status

• Persistence: gfx950 chain failing since 2026-07-12; gfx942 chain since 2026-07-11. Last fully-green nightly: 2026-07-10.
• Scope: all 4 jobs in run 29192413224 — both publish legs (root: MORI build) and both push_local_registry legs (cascade). Blocks nightly ROCm 7.2 image publish to rocm/sgl-dev / lmsysorg/sglang-rocm and the in-network registry mirror for both gfx942 and gfx950.
• In-flight fix: ❌ none found in open PRs.

---

Generated by amd-bot using Claude Code CLI (last updated: 2026-07-12 14:48 UTC)

[amd-bot]

nightly-amd-mi355x-disagg.yml — 4 failure(s)

Run started (UTC): 2026-07-11 05:39 UTC
Last scanned (UTC): 2026-07-12 08:17 UTC
Commits: sglang 592c043

Summary

Counts: 4 failures · 2 clusters · 2 carrying over (≥3 days) · 2 new today

#	Cluster	Job	Test File	Test Function	Status	Hypothesis (confidence)
1	Kimi-K2.6 non-MTP disagg gsm8k below gate (~0.88 vs 0.92)	kimik26-fp8-1k1k-1p1d (Jul 12)	scripts/ci/slurm/recipes/mi355x-fp8/kimik26/1k1k/1p1d.yaml	N/A (accuracy gate)	Never-passed since Jul 5 (0.882)	[MEDIUM] genuine accuracy drop on non-MTP disagg path — see #30433
2	Kimi-K2.6 non-MTP disagg gsm8k below gate (~0.88 vs 0.92)	kimik26-fp8-1k1k-1p1d (Jul 11)	scripts/ci/slurm/recipes/mi355x-fp8/kimik26/1k1k/1p1d.yaml	N/A (accuracy gate)	Never-passed since Jul 5 (0.886)	[MEDIUM] genuine accuracy drop on non-MTP disagg path — see #30433
3	DSV4-Flash disagg gsm8k marginally below gate (borderline)	dsv4flash-fp8-1k1k-1p1d-dp8ep8 (Jul 12)	dsv4flash-fp8-1k1k-1p1d-dp8ep8 gate (scripts/ci/slurm/launch_mi355x.sh)	N/A (accuracy gate)	New today; passed Jul 9/11 (0.905 vs 0.91)	[LOW] benchmark variance at tight gate; trend 0.919→0.912→0.905
4	DSV4-Flash disagg gsm8k marginally below gate (borderline)	dsv4flash-fp4-1k1k-1p1d-dp8ep8 (Jul 11)	dsv4flash-fp4-1k1k-1p1d-dp8ep8 gate	N/A (accuracy gate)	New (Jul 11); passed Jul 8/9 (0.908 vs 0.91)	[MEDIUM] marginal accuracy variance at gate boundary

Cluster: Kimi-K2.6 non-MTP disagg gsm8k below gate (~0.88 vs 0.92)

• Affected: kimik26-fp8-1k1k-1p1d (non-MTP), rows 1–2 — Jul 12 job, Jul 11 job. Both accuracy-gate misses (not crashes).
• Shared facts: All 1319 gsm8k questions served HTTP 200; stable sub-gate accuracy across 5 completed runs (0.882 / 0.886 / 0.892 / 0.886 / 0.887); the MTP sister config passes (~0.95) and single-node non-MTP passes (>0.92) on the same image — deficit is specific to the non-MTP disaggregated path. Never passed since config added ~Jul 5, so no pass_sha..fail_sha window to bisect.
• Top hypothesis: [MEDIUM] genuine accuracy degradation on the non-MTP 2-node 1P1D KV-transfer/decode path. Disconfirming: no green baseline exists; drop is only ~0.04 and could partly reflect a gate copied verbatim from the single-node test — though MTP disagg hitting ~0.95 argues 0.92 is attainable.
• In-flight fix: ✅ #30433 ([DO NOT MERGE] diagnostic for this exact non-MTP GSM8K drop, open since 2026-07-07). Related WIP: #30543, #30336, #29125. Do NOT open a duplicate.
• Suggested triage: diff per-question gsm8k outputs between the passing MTP and failing non-MTP variants on run 29139507423 to localize decode divergence; A/B the decode-meta (#30543) and MORI single-MR clamp (#30336) fixes.

Cluster: DSV4-Flash disagg gsm8k marginally below gate (borderline)

• Affected: dsv4flash-fp8-1k1k-1p1d-dp8ep8 (row 3, Jul 12) and dsv4flash-fp4-1k1k-1p1d-dp8ep8 (row 4, Jul 11). Two precisions of the same DSV4-Flash 1P1D dp8ep8 disagg family.
• Shared facts: Both served all 1319 questions HTTP 200 (no crash/OOM); both missed the 0.91 gate by a hair (FP8 0.905, FP4 0.908); both passed their two immediately-prior completed runs. FP8 shows a slight downward trend 0.919→0.912→0.905.
• Top hypothesis: [MEDIUM] marginal accuracy variance at a tight gate boundary rather than a code regression (row 4); row 3 has no candidate ≥ MEDIUM. Disconfirming: FP8's monotonic decline could indicate a small real regression; no A/B rerun on the same SHA was done.
• In-flight fix: ❌ none directly targets these gsm8k gate misses. Related DSV4-Flash disagg WIP: #30248, #29125, #27197.
• Suggested triage: re-run each leg 2–5× on its head SHA to separate variance from regression; if FP8 stays ≤0.905, bisect bbcfcaeefe36..592c04381d21 prioritizing 9b4bb415 (#30834) and 348e6fd2 (#30847).

Suggested triage order

Rank	Triage step	Targets	Why this first
1	Chase diagnostic PR #30433; diff MTP vs non-MTP gsm8k outputs to localize decode divergence	Cluster Kimi (rows 1–2)	Largest cluster, longest persistence (never-passed since Jul 5), dedicated in-flight thread already open
2	Re-run both DSV4-Flash legs 2–5× on head SHA to distinguish variance from regression	Cluster DSV4-Flash (rows 3–4)	Cheap evidence; both are marginal boundary misses after prior passes
3	If DSV4-Flash FP8 stays ≤0.905, bisect bbcfcaeefe36..592c04381d21	Row 3	Only pursue if rerun confirms a real regression; downward trend warrants watching

---

Job	Failed Steps	Started
nightly-mi355x-benchmark (dsv4flash-fp4-1k1k-1p1d-dp8ep8, dsv4flash-fp4-mi355x-dp8ep8-sglang, dee...	Launch MI355X 2N 1P1D benchmark	2026-07-11T05:39
nightly-mi355x-benchmark (kimik26-fp8-1k1k-1p1d, kimik26-fp8-mi355x-sglang, moonshotai/Kimi-K2.6,...	Launch MI355X 2N 1P1D benchmark	2026-07-11T11:26
nightly-mi355x-benchmark (dsv4flash-fp8-1k1k-1p1d-dp8ep8, dsv4flash-fp8-mi355x-dp8ep8-sglang, sgl...	Launch MI355X 2N 1P1D benchmark	2026-07-12T05:18
nightly-mi355x-benchmark (kimik26-fp8-1k1k-1p1d, kimik26-fp8-mi355x-sglang, moonshotai/Kimi-K2.6,...	Launch MI355X 2N 1P1D benchmark	2026-07-12T07:47

Per-Job Analysis

nightly-mi355x-benchmark (dsv4flash-fp4-1k1k-1p1d-dp8ep8, dsv4flash-fp4-mi355x-dp8ep8-sglang, dee... — failed step(s): Launch MI355X 2N 1P1D benchmark

Commit Info

• sglang: bbcfcaeefe36 — "fix(pd): do not abort when req.disagg_prefill_dp_rank is used (#27546)"
• aiter: N/A (no [CI-AITER-CHECK] markers in this Slurm-based MI355X benchmark log)

Failed Tests

Test File	Test Function	Error	Log
gsm8k benchmark (config dsv4flash-fp4-1k1k-1p1d-dp8ep8)	N/A (accuracy gate, not a pytest test)	accuracy=0.908 threshold=0.91 — below threshold	Log

Note: this MI355X 2N 1P1D disaggregated benchmark is not a pytest suite. It runs a gsm8k accuracy gate before a throughput sweep; the gate failed, so the sweep never ran ("failing before sweep").

Failure Summary

The dsv4flash-fp4-1k1k-1p1d-dp8ep8 benchmark (DeepSeek-V4-Flash FP4, 1P1D disaggregated, DP8/EP8 on MI355X 2-node) reached gsm8k accuracy 0.908, just under the 0.91 gate — a miss of 0.002 (~2-3 of 1319 questions). Because the accuracy gate fails fast before the throughput sweep, the whole job exited rc=1 (step 5 "Launch MI355X 2N 1P1D benchmark"). One unrelated config in the same matrix run also failed (kimik26-fp8-1k1k-1p1d, job 86509975073) — separate issue, not analyzed here.

Regression Status

Flaky / marginal accuracy miss at the threshold boundary.

Recent history of gsm8k accuracy gate for config dsv4flash-fp4-1k1k-1p1d-dp8ep8 (branch main, event schedule, completed runs only):

Date	Run	Job	Config Status	Result	Error
Jul 11	run	job	❌ Failed	acc 0.908	below 0.91 gate
Jul 10	run	—	⏭️ Run cancelled	—	not comparable
Jul 9	run	job	✅ Passed	—	—
Jul 8	run	job	✅ Passed	—	—
Jul 5	run	job	❌ Failed (different mode)	server crash	Traceback, Accuracy: <not found in bench.log> — bench crashed before completing, NOT an accuracy miss

First observed as an accuracy miss: Jul 11. Last known passing: Jul 9. The Jul 5 failure is a distinct symptom (server crash / bench never produced an accuracy number), so it does not represent the same failure. Lookback window: Jul 1–11 (event=schedule, branch=main).

Failure Cluster

gsm8k accuracy just below gate on DSV4-Flash FP4 1P1D disagg (MI355X)

Facts (verified observations)

• Accuracy: 0.908, Invalid: 0.001, Latency: 45.646 s, Output throughput: 2827.711 token/s over 1319 gsm8k questions (log lines 4148-4151).
• Gate logic printed [gsm8k] accuracy=0.908 threshold=0.91 then [gsm8k] accuracy below threshold -- failing before sweep (lines 4152-4153).
• The run completed the full 1319-question sweep (100%|██████████| 1319/1319) and the server returned HTTP 200 for all /generate requests — i.e. no crash, no numerical NaN; the model simply scored 0.002 under gate.
• Config passed on Jul 8 and Jul 9 (same branch, same schedule event) — see history table.
• Miss magnitude: 0.002 = ~2-3 questions out of 1319, within typical gsm8k run-to-run sampling variance.
• Job step that failed: step 5 "Launch MI355X 2N 1P1D benchmark"; downstream Slurm teardown produced the exit code 137 / docker kill lines (cleanup, not root cause).

Hypothesised Causes (with confidence, with disconfirming evidence)

• [MEDIUM] Marginal accuracy variance at the gate boundary (flaky threshold), not a code regression.
  • Supporting: benchmark ran to completion with all requests HTTP 200; miss is only 0.002 (~2-3 questions); the same config passed the two immediately prior completed runs (Jul 8, Jul 9); gsm8k with sampling has inherent per-run variance of this order.
  • Disconfirming: cannot fully rule out a small real quality regression — no A/B rerun of the same head SHA was performed; the accuracy delta vs the passing runs' margin is unknown (passing runs' exact accuracy not captured here).
  • How to verify: re-run the dsv4flash-fp4-1k1k-1p1d-dp8ep8 benchmark 3-5× on the same head SHA; if it clears 0.91 in most reruns, this is flaky boundary noise.
• [LOW] A quality regression from a commit in the Jul 9→Jul 11 window (69ddbf9..bbcfcaee), possibly PD-disagg related (head commit touches disagg_prefill_dp_rank).
  • Supporting: head commit bbcfcaeefe36 modifies PD-disaggregation abort logic, and this is a 1P1D disagg config; AMD has open PRs actively debugging DSV4-Flash PD state transfer on MI355X (see In-flight Fix Check).
  • Disconfirming: the miss is only 0.002 and the config passed Jul 9; a real PD-transfer correctness bug would more likely cause a large accuracy drop or a crash (as seen Jul 5), not a 0.002 boundary miss. The abort-logic fix in #27546 is unlikely to change token-level outputs.
  • How to verify: bisect 69ddbf9..bbcfcaee with reruns of this one config; compare accuracy before/after bbcfcaeefe36.

In-flight Fix Check

• In-flight fix: 🟡 partial — AMD is actively investigating the DSV4-Flash disaggregated PD path on MI355X:

  • #30248 "[WIP][AMD] A/B disable DSV4 Flash C128 state PD transfer"
  • #30271 "[Draft][CI] Debug MI355X DSV4 Flash C128 PD state transfer"
  • #30336 "[AMD] WIP - Clamp disagg max_total_num_tokens to MORI's single-MR size limit"

  These target the same model family + disagg path, but none is a specific fix for this 0.002 gsm8k gate miss. No open PR found that adjusts the gsm8k threshold for this config. (Searched: dsv4flash mi355x, mi355x threshold.)

Suggested Triage Steps

• Re-run dsv4flash-fp4-1k1k-1p1d-dp8ep8 3-5× on head bbcfcaeefe36 to establish the accuracy distribution and confirm whether 0.908 is within normal variance or a persistent sub-gate result.
• If persistently sub-0.91, bisect 69ddbf9..bbcfcaee, focusing on PD-disagg commits, and cross-check against the DSV4-Flash PD investigations in #30248 / #30271.
• If confirmed flaky boundary noise, consider a small threshold margin or averaging over reruns for this config's accuracy gate (maintainer decision).

Status (factual)

• Persistence: 1 completed run as an accuracy miss (Jul 11). Config passed the two prior completed runs (Jul 8, Jul 9). The Jul 5 failure was a distinct crash symptom.
• Scope: 1 config (dsv4flash-fp4-1k1k-1p1d-dp8ep8) in the nightly-amd-mi355x-disagg.yml matrix. A separate config (kimik26-fp8-1k1k-1p1d) also failed in the same run but is unrelated.
• Blocked work: throughput sweep for this config did not run (fails fast on accuracy gate); nightly MI355X DSV4-Flash-FP4 disagg perf signal missing for Jul 11.
• In-flight fix: 🟡 related DSV4-Flash PD debugging PRs open (#30248, #30271, #30336); none directly addresses this gate miss.

nightly-mi355x-benchmark (kimik26-fp8-1k1k-1p1d, kimik26-fp8-mi355x-sglang, moonshotai/Kimi-K2.6,... — failed step(s): Launch MI355X 2N 1P1D benchmark

Commit Info

• sglang: bbcfcaee
• aiter: N/A — no [CI-AITER-CHECK] markers in this disagg-benchmark log; aiter version is not tracked by this workflow.

Failed Tests

Test File	Test Function	Error	Log
scripts/ci/slurm/recipes/mi355x-fp8/kimik26/1k1k/1p1d.yaml (GSM8K accuracy gate, config kimik26-fp8-1k1k-1p1d)	N/A (benchmark correctness gate, not a pytest)	[gsm8k] accuracy=0.886 threshold=0.92 — failing before sweep → srun ... Exited with exit code 1	Log

Failure Summary

The kimik26-fp8-1k1k-1p1d matrix entry of the AMD MI355X 2-node 1P1D disaggregated nightly ran the full 1319-question, 8-shot GSM8K correctness gate through the PD path and scored 0.886, below the 0.92 gate defined in the recipe, so the job aborted before the perf sweep (rc=1). The subsequent exit code 137 on mia1-p02-g09 is the teardown docker kill mi355x_prefill after the gate failed — a consequence of the abort, not an independent crash.

Regression Status

Known recurring failure / Never-passed — this config has failed the GSM8K gate on every completed scheduled run since it entered the MI355X matrix (~Jul 5); no green run exists in the queryable window. The sister kimik26-fp8-1k1k-1p1d-**mtp** entry passes on the same runs, so the failure is specific to the non-MTP disagg path.

Recent history of the GSM8K accuracy gate for config kimik26-fp8-1k1k-1p1d (same branch main, event=schedule, completed runs only):

Date	Run	Job	Gate Status	Accuracy	Error
Jul 11	93	kimik26-fp8-1k1k-1p1d	❌ Failed	0.886	below 0.92 gate
Jul 10	90	—	⚪ Cancelled	—	run cancelled (excluded)
Jul 9	87	kimik26-fp8-1k1k-1p1d	❌ Failed	0.892	below 0.92 gate
Jul 8	79	kimik26-fp8-1k1k-1p1d	❌ Failed	0.886	below 0.92 gate
Jul 5	36	kimik26-fp8-1k1k-1p1d	❌ Failed	0.887	below 0.92 gate
Jul 1–2	21	—	➖ N/A	—	matrix ran DeepSeek-V4 only; Kimi config not present

First observed failure: Jul 5 (earliest completed run carrying this config). No passing run found for this config in the lookback window.

Failure Cluster

Kimi-K2.6 non-MTP disagg GSM8K accuracy below gate (~0.88 vs 0.92)

Facts (verified observations)

• Gate output (verbatim): Accuracy: 0.886 / Invalid: 0.002 / [gsm8k] accuracy=0.886 threshold=0.92 / [gsm8k] accuracy below threshold -- failing before sweep.
• The gate is configured in scripts/ci/slurm/recipes/mi355x-fp8/kimik26/1k1k/1p1d.yaml line 75: threshold: 0.92, num_questions: 1319, 8-shot.
• The server started and served all 1319 requests successfully (200s, 100%|██████████| 1319/1319); this is an accuracy shortfall, not a crash or infra failure.
• The -mtp sister config passes on the same runs (verified: run 87 job 86228240190 = success, run 79 job 85787096165 = success, run 36 job 85193219590 = success). Both recipes use the same threshold: 0.92.
• Accuracy is stable across runs (0.886 / 0.892 / 0.886 / 0.887) — a persistent ~3–4 point deficit, not run-to-run noise.
• Per in-flight PR #30433: non-MTP ~0.88, MTP ~0.95, single-node non-MTP passes at 0.944 — i.e. the deficit is specific to the non-MTP disaggregated path.

Hypothesised Causes (with confidence)

• [MEDIUM] The non-MTP disaggregated-PD path produces a genuine accuracy drop for Kimi-K2.6 fp8 (a correctness bug in the 1P1D KV-transfer / decode path), not a mis-set threshold.
  • Supporting: single-node non-MTP scores 0.944 and MTP disagg scores ~0.95 on the same model/threshold, while non-MTP disagg is stuck at ~0.88 across 4 runs; PR #30433 confirms page_size and all disagg runtime args were already ruled out.
  • Disconfirming: root cause not yet isolated to a specific commit or code path; the AMD team's diagnostic PR is still open (DO NOT MERGE), so this remains an active investigation rather than a confirmed defect.
  • How to verify: continue the A/B matrix in PR #30433 (single-node vs disagg, MTP vs non-MTP) and bisect the disagg decode/KV-transfer path.
• [LOW] The 0.92 gate is simply too tight for the non-MTP disagg config (a test-config issue rather than a model regression).
  • Supporting: accuracy is very stable at ~0.886–0.892, only ~3 points under gate.
  • Disconfirming: single-node non-MTP hits 0.944 with the same model/precision, so 0.92 is achievable — arguing the deficit is real, not an over-tight gate.
  • How to verify: compare non-MTP disagg vs single-node token-level outputs on a fixed GSM8K subset.

No commit-level regression can be pinned: this config has never passed since it was added, so there is no pass_sha..fail_sha window to bisect within the disagg workflow.

In-flight Fix Check

• In-flight fix: ✅ #30433 [DO NOT MERGE][AMD][DI][CI] Kimi-K2.6 disagg diagnostic recipes for non-MTP GSM8K drop (open since 2026-07-07, updated 2026-07-10) — directly tracks this exact symptom; it is a diagnostic/investigation PR, so no fix has merged yet. Chase this thread; do NOT open a duplicate.
• Related open PRs: #30543 (kimik26 disagg decodemeta, WIP), #30336 (clamp disagg max_total_num_tokens to MORI single-MR limit — targets dsv4flash disagg but same workflow).
• API used: search/issues?q=repo:sgl-project/sglang+is:pr+is:open+kimik26.

Suggested Triage Steps

• Track the diagnostic matrix in #30433; the key open question is why non-MTP disagg loses ~6 points vs non-MTP single-node (0.944) on the same model/precision — focus on the 1P1D KV-transfer and decode path.
• Diff the effective server args / sampling config between the passing -mtp recipe and the failing non-MTP recipe under scripts/ci/slurm/recipes/mi355x-fp8/kimik26/1k1k/ to confirm MTP is the only intended difference.
• If the investigation concludes the non-MTP disagg accuracy is a known model limitation rather than a regression, consider gating this config behind a diagnostic/allow-fail flag so it stops blocking the nightly signal until a fix lands (a maintainer decision).

Status

• Persistence: Failing every completed scheduled run since 2026-07-05 (Jul 5, 8, 9, 11); never passed in the queryable window. Accuracy stable at 0.886–0.892 vs 0.92 gate.
• Scope: One matrix entry (kimik26-fp8-1k1k-1p1d, non-MTP) in workflow nightly-amd-mi355x-disagg.yml. The -mtp sister config passes; other models in the matrix are unaffected by this cluster.
• In-flight fix: ✅ #30433 open (diagnostic, no merged fix yet).

nightly-mi355x-benchmark (dsv4flash-fp8-1k1k-1p1d-dp8ep8, dsv4flash-fp8-mi355x-dp8ep8-sglang, sgl... — failed step(s): Launch MI355X 2N 1P1D benchmark

Commit Info

• sglang: 592c04381d21
• aiter: N/A (Slurm MI355X benchmark; no [CI-AITER-CHECK] markers in log)

Failed Tests

Test File	Test Function	Error	Log
N/A — gsm8k accuracy gate (scripts/ci/slurm/launch_mi355x.sh, step "Launch MI355X 2N 1P1D benchmark")	gsm8k accuracy gate, config dsv4flash-fp8-1k1k-1p1d-dp8ep8	accuracy=0.905 < threshold 0.91 → [gsm8k] accuracy below threshold -- failing before sweep	Log

This is not a pytest/unit-test failure. It is a benchmark accuracy pre-gate: the harness runs gsm8k against the served DeepSeek-V4-flash-FP8 model over the disaggregated 1P1D (dp8ep8) path, and aborts before the throughput sweep if accuracy is below 0.91.

Failure Summary

The dsv4flash-fp8-1k1k-1p1d-dp8ep8 benchmark leg served the model successfully (all 1319 gsm8k requests returned HTTP 200) but scored Accuracy: 0.905 against the gate threshold > 0.91. The gate failed the leg with exit code 1 before running the perf sweep, and teardown killed the prefill container (rc=137, expected on kill). No crash, OOM, or serving error occurred — this is purely the accuracy gate tripping by 0.005.

Regression Status

Borderline / likely benchmark variance around a tight threshold (not a confirmed code regression).

Recent history of the gsm8k accuracy gate for leg dsv4flash-fp8-1k1k-1p1d-dp8ep8 (workflow nightly-amd-mi355x-disagg.yml, event=schedule, branch=main, completed runs only):

Date	Run	Job	Gate Status	gsm8k Accuracy	Note
Jul 12	29179694296	leg	❌ Failed	0.905 (< 0.91)	this run
Jul 11	29139507423	leg	✅ Passed	0.912	—
Jul 09	28994366909	leg	✅ Passed	0.919	—
Jul 08	28917370988	leg	✅ Passed	—	accuracy gate passed
Jul 05	28729727896	leg	❌ Failed	n/a	different cluster: PD path not serving (prefill server failed), aborted before accuracy
Jul 03	28638558731	leg	❌ Failed	n/a	different cluster: PD path not serving, aborted before accuracy

First observed accuracy-gate failure for this leg in the queryable window: Jul 12 (today). Last accuracy-gate pass: Jul 11 (0.912). The Jul 3/5 failures are a distinct serving-path failure, not this accuracy gate. Note the last three accuracy-producing runs trend downward within a narrow band: 0.919 → 0.912 → 0.905, straddling the 0.91 gate.

Failure Cluster

gsm8k accuracy below threshold on DeepSeek-V4-flash-FP8 disagg (borderline gate)

Facts (verified observations, no interpretation)

• Log: [gsm8k] accuracy=0.905 threshold=0.91 then [gsm8k] accuracy below threshold -- failing before sweep (job.log lines 4122–4123).
• Accuracy: 0.905, Invalid: 0.000, Latency: 39.712 s, Output throughput: 3311.143 token/s; all 1319/1319 gsm8k requests returned status_code=200.
• Gate exits before the perf sweep: bench_exit RC=1 → teardown docker kill mi355x_prefill → prefill rc=137 (kill signal, expected). No serving error, OOM, or stack trace in this run.
• gsm8k accuracy in the two prior passing runs: 0.912 (Jul 11), 0.919 (Jul 09) — verified from each job's log. Threshold 0.91 unchanged across all runs.
• 1319-sample gsm8k: 0.905 ≈ 1194 correct, 0.91 ≈ 1200 correct — a ~6-question gap.
• 18 commits merged between Jul 11 pass (bbcfcaeefe36) and Jul 12 fail (592c04381d21).
• No [CI-AITER-CHECK] markers in the log; aiter is not a variable in this benchmark.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [LOW] Benchmark variance against a too-tight threshold. gsm8k sampling + kernel non-determinism on FP8 easily moves ~6 questions run-to-run; three recent values (0.919 / 0.912 / 0.905) all sit within ~1.4% and the gate 0.91 lands in the middle of that band.
  • Supporting: leg passed the last two runs; miss is only 0.005; no serving error or crash.
  • Disconfirming: the three points form a monotonic decline, which could equally indicate a slow real accuracy regression rather than symmetric noise.
  • How to verify: re-run this leg on the same SHA 2–3× (nightly-amd-mi355x-disagg.yml, config dsv4flash-fp8-1k1k-1p1d-dp8ep8); if it oscillates above/below 0.91, it's variance and the threshold should be reviewed.
• [LOW] A commit in bbcfcaeefe36..592c04381d21 affecting FP8/MoE/cuda-graph numerics. Most plausible candidates by code area: 9b4bb415 (#30834, cuda-graph shared-buffer sizing / row-count slicing) and 348e6fd2 (#30847, guard kernel OOB / harden runtime edge cases).
  • Supporting: both touch kernel/graph paths exercised by the served forward pass; land exactly in the pass→fail window.
  • Disconfirming: no stack trace or code-level evidence ties either to gsm8k scoring; a 0.007 accuracy drop is within observed run-to-run variance, so temporal correlation alone is weak.
  • How to verify: bisect the 18-commit window by re-running this leg with each candidate reverted on a branch.
• [LOW] 7bac9c8c (#30853, draft-extend cuda graph for DeepSeek-V4 attention backend).
  • Supporting: touches DeepSeek-V4 attention.
  • Disconfirming: this leg is dp8ep8 with no MTP/speculative (the -mtp legs are separate jobs), so the draft/spec path is not exercised here — likely irrelevant.
  • How to verify: confirm no spec decoding is enabled in this config's server args.

No hypothesis reaches MEDIUM confidence; needs a re-run (variance check) and/or git bisect between bbcfcaeefe36..592c04381d21.

In-flight Fix Check

No PR directly targets the gsm8k accuracy threshold for this leg, but several open WIP efforts touch exactly this MI355X DSV4-flash disagg path:

• In-flight (related): #30248 — "[WIP][AMD] A/B disable DSV4 Flash C128 state PD transfer" (PD KV-transfer correctness on this exact model — could bear on accuracy)
• In-flight (related): #29125 — "[WIP][AMD][CI] Debug MI355X 1P1D disaggregation nightly"
• In-flight (related): #27197 — "Fix DeepSeek V4 Flash on MI355X"
• Direct gsm8k-threshold fix: ❌ none found (searched open PRs for mi355x gsm8k, dsv4flash accuracy)

Suggested Triage Steps

• Re-run the dsv4flash-fp8-1k1k-1p1d-dp8ep8 leg 2–3× on SHA 592c04381d21 to distinguish variance from a real regression. If accuracy oscillates around 0.91, treat as a gate-tightness issue and consider widening the margin or averaging multiple gsm8k runs.
• If accuracy stays ≤0.905 across re-runs, bisect bbcfcaeefe36..592c04381d21, prioritizing 9b4bb415 (#30834) and 348e6fd2 (#30847); re-run this leg with each reverted on a branch.
• Check whether in-flight #30248 (DSV4 Flash C128 state PD transfer) changes accuracy on this config — the disagg KV-transfer path is a plausible accuracy lever.
• Track the accuracy trend (0.919 → 0.912 → 0.905) over the next few nightlies; a continued decline confirms a real regression rather than noise.

Status

• Persistence: First accuracy-gate failure for this leg in the queryable window (Jul 12); the two immediately prior runs passed (0.912, 0.919). Earlier Jul 3/5 failures were a different cluster (PD path not serving).
• Scope: One benchmark leg (dsv4flash-fp8-1k1k-1p1d-dp8ep8) in nightly-amd-mi355x-disagg.yml. Sister legs (dsv4pro-fp8-...-dp8ep8, dsv4flash-fp4-...-dp8ep8) passed this run; the overall workflow run 29179694296 was still queued at snapshot (other legs in flight) — this leg alone concluded failure.
• Blocked work: Blocks the perf sweep for this one config only; benchmark throughput numbers for dsv4flash-fp8-1k1k-1p1d-dp8ep8 were not emitted this run.
• In-flight fix: No direct fix; related WIP PRs #30248, #29125, #27197.

nightly-mi355x-benchmark (kimik26-fp8-1k1k-1p1d, kimik26-fp8-mi355x-sglang, moonshotai/Kimi-K2.6,... — failed step(s): Launch MI355X 2N 1P1D benchmark

Commit Info

• sglang: 592c04381d210a062d6dc193864e5db9396aabc5
• aiter: N/A (no [CI-AITER-CHECK] markers — this is a Slurm benchmark workflow using pinned docker image lmsysorg/sglang-rocm:v0.5.13.post1-rocm720-mi35x-20260623)

Failed Tests

Test File	Test Function	Error	Log
scripts/ci/slurm/recipes/mi355x-fp8/kimik26/1k1k/1p1d.yaml (GSM8K accuracy gate)	accuracy correctness gate (mirrors test/registered/amd/accuracy/mi35x/test_kimi_k26_eval_mi35x.py)	[gsm8k] accuracy=0.882 threshold=0.92 — failing before sweep	Log

Failure Summary

The Kimi-K2.6 FP8 2-node 1P1D disaggregation, non-MTP benchmark ran the full GSM8K set (1319/1319 questions, all HTTP 200) and scored accuracy 0.882, below the 0.92 correctness gate. The gate fails before the perf sweep, so srun exits 1 and the job aborts (the subsequent exit code 137 is just the docker teardown docker kill mi355x_decode, not the root cause).

Regression Status

Never-passed (no green run found) — for this specific kimik26-fp8-1k1k-1p1d (non-MTP) config. The config was added to the matrix ~2026-07-05 and has failed the GSM8K gate in every completed scheduled run since, with a stable accuracy band of 0.88–0.89. Lookback window queried: 2026-07-05 → 2026-07-12 (event=schedule, branch=main, completed runs only).

Recent history of kimik26-fp8-1k1k-1p1d GSM8K accuracy gate (same branch, event=schedule, completed runs only):

Date	Run	Job	Test Status	Accuracy	Error
Jul 12	29179694296	kimik26-fp8-1k1k-1p1d	❌ Failed	0.882	below 0.92 gate
Jul 11	29139507423	kimik26-fp8-1k1k-1p1d	❌ Failed	0.886	below 0.92 gate
Jul 10	29069463215	—	⏭️ Cancelled	—	run cancelled
Jul 09	28994366909	kimik26-fp8-1k1k-1p1d	❌ Failed	0.892	below 0.92 gate
Jul 08	28917370988	kimik26-fp8-1k1k-1p1d	❌ Failed	0.886	below 0.92 gate
Jul 05	28729727896	kimik26-fp8-1k1k-1p1d	❌ Failed	0.887	below 0.92 gate

Failure Cluster

Kimi-K2.6 non-MTP 1P1D-disagg GSM8K accuracy below gate (~0.88 vs 0.92)

Facts (verified observations, no interpretation)

• Benchmark completed all 1319 GSM8K questions; every /generate request returned status_code=200. This is not a crash, OOM, or hang (log line 100%|██████████| 1319/1319).
• Verbatim gate line: [gsm8k] accuracy=0.882 threshold=0.92 → [gsm8k] accuracy below threshold -- failing before sweep (log line 4723).
• Reported metrics: Accuracy: 0.882 · Invalid: 0.003 · Latency: 261.328 s · Output throughput: 595.600 token/s.
• exit code 137 (prefill@mia1-p02-g09 rc=137) comes from the docker kill mi355x_prefill/mi355x_decode teardown after the gate already failed — it is a consequence, not the cause.
• Accuracy is stable and consistently sub-gate across 5 completed runs: 0.882 / 0.886 / 0.892 / 0.886 / 0.887 (all < 0.92).
• Sister-config comparison (same run family): on Jul 11 run 29139507423, the MTP variant kimik26-fp8-1k1k-1p1d-mtp passed while the non-MTP kimik26-fp8-1k1k-1p1d failed — the accuracy drop is specific to the non-MTP disagg path.
• Config gate threshold 0.92 is copied verbatim from the single-node registered test (recipe comment: "Mirrors the single-node registered test ... GSM8K > 0.92").

Hypothesised Causes (with confidence, with disconfirming evidence)

• [MEDIUM] Genuine accuracy degradation specific to the Kimi-K2.6 non-MTP 2-node 1P1D disaggregation path (MORI cross-node KV transfer + aiter prefill / triton decode split backends).
  • Supporting: MTP variant passes (~0.95) and single-node non-MTP passes (>0.92) on the same hardware/image, while non-MTP disagg is stuck at ~0.88; the gap is systematic (not noisy) across 5 runs. This is exactly the symptom documented in open PR #30433 ("Kimi-K2.6 non-MTP GSM8K only reaches ~0.88, gate is 0.92, while MTP passes ~0.95 and single-node non-MTP passes").
  • Disconfirming: no green baseline exists for this config, so we cannot bisect to a culprit commit — the accuracy drop may be a config/backend-setup issue rather than a code regression. The drop is small (~0.04) and could partly reflect an over-tight gate inherited from the single-node test.
  • How to verify: follow the diagnostic recipes in #30433; A/B the decode-meta path via #30543 and the MORI single-MR clamp via #30336; compare per-question outputs between MTP vs non-MTP disagg to localize where decode diverges.
• [LOW] Gate threshold (0.92) too strict for the disagg path rather than a real regression.
  • Supporting: gap is only ~0.04 and the threshold was copied from the single-node test without disagg-specific calibration.
  • Disconfirming: MTP disagg reaches ~0.95 on the same path, so 0.92 is clearly attainable in disagg — arguing the non-MTP shortfall is real, not a mis-set gate.
  • How to verify: root-cause the non-MTP/MTP gap first; only re-baseline the threshold if the outputs are confirmed correct.

In-flight Fix Check

• In-flight fix: ✅ #30433 ([DO NOT MERGE][AMD][DI][CI] Kimi-K2.6 disagg diagnostic recipes for non-MTP GSM8K drop, open since 2026-07-07, updated 2026-07-10) — dedicated diagnostic branch for this exact failure. Do NOT open a duplicate; chase this thread.
• Related active AMD investigation branches: #30543 (kimik26 disagg decodemeta, WIP), #30336 (clamp disagg max_total_num_tokens to MORI single-MR size, WIP), #29125 (debug MI355X 1P1D disagg nightly, WIP).
• API call used: GET /search/issues?q=repo:sgl-project/sglang+is:pr+is:open+{kimik26,mi355x+accuracy,1p1d+accuracy+threshold}.

Suggested Triage Steps

• Track diagnostic PR #30433 — it already scopes the non-MTP GSM8K drop; land its diagnostic recipes to localize the divergence.
• Diff per-question GSM8K outputs between the passing MTP variant and the failing non-MTP variant on run 29139507423 to find where decode diverges (isolates whether MTP masks a decode-path bug).
• A/B the two candidate fixes on a branch and re-run kimik26-fp8-1k1k-1p1d: decode-meta path (#30543) and MORI single-MR max_total_num_tokens clamp (#30336).
• If outputs are later confirmed correct, revisit whether the 0.92 gate (copied from the single-node test) is appropriate for the non-MTP disagg path — but only after the MTP-vs-non-MTP gap is root-caused.

Status (factual, no priority assignment)

• Persistence: Never-passed since the config was added (~2026-07-05); failed every completed scheduled run through 2026-07-12, accuracy stable at 0.88–0.89.
• Scope: Single config — kimik26-fp8-1k1k-1p1d (non-MTP) in nightly-amd-mi355x-disagg.yml. Sister MTP config (...-1p1d-mtp) passes; single-node Kimi-K2.6 eval passes.
• Blocked work: Blocks green signal for the Kimi-K2.6 non-MTP 1P1D disagg nightly benchmark on MI355X.
• In-flight fix: ✅ #30433 (diagnostic, open since 2026-07-07) + WIP branches #30543, #30336.

---

⏳ 9 job(s) still running (run) — will update when complete

---

Generated by amd-bot using Claude Code CLI (last updated: 2026-07-12 08:17 UTC)

[amd-bot]

release-docker-amd-nightly.yml — 6 failure(s)

Run started (UTC): 2026-07-11 12:24 UTC
Last scanned (UTC): 2026-07-12 15:17 UTC
Commits: sglang 80856ab

Summary

Counts: 6 failures · 2 clusters · 0 carrying over (≥3 days) · all 6 new since 2026-07-11 (MORI build root cause + cascade)

#	Cluster	Job	Test File	Test Function	Status	Hypothesis (confidence)
1	MORI build cmake gtest-discovery JSON parse error	publish (gfx942, all)	docker/rocm.Dockerfile (stage-4 23/28, MORI)	N/A (build)	2 days (Jul 11–12), gfx942	[HIGH] unpinned cmake upgraded 4.3.4→4.4.0 (docker/rocm.Dockerfile:361) — 4.4.0's ParseTestList.cmake hard-errors on empty gtest-discovery JSON
2	MORI build cmake gtest-discovery JSON parse error	publish (gfx942, all)	docker/rocm.Dockerfile (stage-4 23/28, MORI)	N/A (build)	Jul 11	[HIGH] same cmake 4.4.0 regression (Dockerfile + MORI pin e31d426a unchanged)
3	MORI build cmake gtest-discovery JSON parse error	publish (gfx950, all)	docker/rocm.Dockerfile (stage-4 23/28, MORI)	N/A (build)	Jul 11	[HIGH] same cmake 4.4.0 regression
4	Artifact-not-found cascade (mirror leg)	push_local_registry (gfx942)	N/A (download-artifact)	N/A	2 days (Jul 11–12), gfx942	[HIGH] downstream cascade — publish (gfx942) produced no image-tag-gfx942 artifact
5	Artifact-not-found cascade (mirror leg)	push_local_registry (gfx942)	N/A (download-artifact)	N/A	Jul 11	[HIGH] cascade of publish (gfx942) MORI build failure
6	Artifact-not-found cascade (mirror leg)	push_local_registry (gfx950)	N/A (download-artifact)	N/A	Jul 11	[HIGH] cascade of publish (gfx950) MORI build failure

Cluster: MORI build cmake gtest-discovery JSON parse error

• Affected: publish (gfx942, all) (Jul 12), publish (gfx942, all) (Jul 11), publish (gfx950, all) (Jul 11) — all docker/rocm.Dockerfile MORI stage, no sglang test involved.
• Shared facts: All fail at /opt/venv/.../cmake-4.4/Modules/GoogleTest/ParseTestList.cmake:96 (string) ← GoogleTestAddTests.cmake:245 ← test_<name>_e3b0c442_discovery.cmake, on empty/corrupt JSON from freshly-linked umbp test binaries (test_external_kv_block_index, test_client_registry_external_kv, etc.) → ninja: build stopped → exit 1. MORI_COMMIT=e31d426a (docker/rocm.Dockerfile:113) and the Dockerfile are byte-identical across the pass→fail boundary (git log 94de28764c..80856aba -- docker/rocm.Dockerfile empty). Build is --no-cache with unpinned cmake (docker/rocm.Dockerfile:361).
• Top hypothesis: [HIGH] cmake pip dep bumped 4.3.4 (Jul 10 pass) → 4.4.0 (Jul 11/12 fail) — verified from log (Successfully installed cmake-4.4.0), timing aligns with cmake 4.4.0's 2026-07-10 PyPI release, and the erroring module path is cmake-4.4/.... Disconfirming: publish (gfx950, all) passed on Jul 12 with the same cmake 4.4, so 4.4 is not a clean deterministic break — it hard-fails an intermittent empty-output condition (gtest discovery runs the binary on a GPU-less build host) rather than being the sole culprit.
• In-flight fix: ❌ none found (searched cmake+4.4, MORI+build, gtest_discover, ParseTestList, pin+cmake — all unrelated).
• Suggested triage: (1) A/B pin cmake==4.3.4 at docker/rocm.Dockerfile:361, rebuild gfx942 with ENABLE_MORI=1 --no-cache; if the MORI stage recovers, cmake 4.4.0 confirmed. (2) If empty-output persists, pursue MORI-side change (gtest_discover_tests(... DISCOVERY_MODE PRE_TEST) or disable in-image test builds) upstream at ROCm/mori. Do not treat a single green re-run as a fix (gfx950 passed Jul 12).

Cluster: Artifact-not-found cascade (mirror leg)

• Affected: push_local_registry (gfx942) (Jul 12), push_local_registry (gfx942) (Jul 11), push_local_registry (gfx950) (Jul 11).
• Shared facts: Each fails only at the actions/download-artifact@v4 step (Artifact not found for name: image-tag-gfx{942,950}). The image-tag-* artifact is uploaded only after the matching publish leg's docker build+push succeeds (release-docker-amd-nightly.yml:97-107); push_local_registry runs under !cancelled(), so a failed publish leg deterministically fails its mirror leg. No independent defect here.
• Top hypothesis: [HIGH] pure downstream cascade of the MORI build cluster. Disconfirming: (none found).
• In-flight fix: ❌ none — no separate fix needed; these legs go green automatically once publish produces artifacts.
• Suggested triage: fix the MORI build cluster; optionally surface "upstream publish leg failed" in the mirror job so the artifact-not-found message stops obscuring the real cause.

Suggested triage order

Rank	Triage step	Targets	Why this first
1	A/B pin cmake==4.3.4 in docker/rocm.Dockerfile:361, rebuild gfx942/gfx950 with ENABLE_MORI=1 --no-cache	Cluster 1 (rows 1-3)	[HIGH] cmake 4.3.4→4.4.0 is the only changed build input across the pass→fail boundary; single rebuild confirms or refutes
2	If empty-JSON persists under 4.3.4, raise MORI-side discovery-mode change (PRE_TEST / disable in-image gtest builds) upstream	Cluster 1 (rows 1-3)	Addresses the intermittent empty-output that gfx950's Jul 12 pass shows cmake 4.4 merely unmasks
3	Re-run cluster 1 fix; verify mirror legs recover with no separate change	Cluster 2 (rows 4-6)	Cascade resolves automatically once publish produces image-tag-* artifacts

---

Job	Failed Steps	Started
publish (gfx950, all)	Build and Push to rocm/sgl-dev	2026-07-11T12:25
publish (gfx942, all)	Build and Push to rocm/sgl-dev	2026-07-11T12:24
push_local_registry (gfx950)	Download image tag artifact	2026-07-11T14:58
push_local_registry (gfx942)	Download image tag artifact	2026-07-11T14:58
publish (gfx942, all)	Build and Push to rocm/sgl-dev	2026-07-12T12:24
push_local_registry (gfx942)	Download image tag artifact	2026-07-12T15:05

Per-Job Analysis

publish (gfx950, all) — failed step(s): Build and Push to rocm/sgl-dev

Commit Info

• sglang: 32cb89d4
• aiter: N/A (docker-build workflow — no [CI-AITER-CHECK] markers in log)

Failed Tests

Test File	Test Function	Error	Log
docker/rocm.Dockerfile (stage-4 23/28, MORI build)	N/A — build step, not a test	CMake Error … string sub-command JSON failed parsing json string … Syntax error: value, object or array expected → ninja: build stopped → exit code 1	Log

This is not an sglang test failure. It is a Docker image build failure inside the external ROCm/mori sub-build that rocm.Dockerfile compiles when ENABLE_MORI=1.

Failure Summary

The publish (gfx950, all) job builds rocm/sgl-dev:... via docker build --no-cache. The build failed at [stage-4 23/28] during MORI's ninja build: while linking src/umbp/tests/test_client_registry_external_kv, CMake's GoogleTest test-discovery step (gtest_discover_tests) tried to parse the test-list JSON emitted by the test binary and hit a JSON syntax error, causing ninja: build stopped: subcommand failed and job exit code 1. The MORI source is a pinned external commit; the failure surfaced in the pip-resolved CMake tooling, not in sglang code.

Regression Status

New regression (first failing run 2026-07-11; the prior 7 consecutive scheduled runs all succeeded).

Recent history of release-docker-amd-nightly.yml / publish (gfx950, all) (branch main, event schedule, completed runs only):

Date	Run	Job Status	Notes
Jul 11	29152576569	❌ Failed	MORI build — gtest-discovery JSON parse error
Jul 10	29093958349	✅ Passed	—
Jul 09	29020296834	✅ Passed	—
Jul 08	28943370684	✅ Passed	—
Jul 07	28867829963	✅ Passed	—

First observed failure: 2026-07-11 12:24 UTC. Last known pass: 2026-07-10 12:51 UTC.

Failure Cluster

MORI docker-build failure: CMake GoogleTest test-discovery JSON parse error (ParseTestList.cmake / cmake 4.4)

Facts (verified observations, no interpretation)

• Build failed at #28 [stage-4 23/28] RUN … [MORI] …; ENABLE_MORI=1, NIC_BACKEND=none, MORI_GPU_ARCHS=gfx950 (log ~lines 22390–22391).
• FAILED target: [130/146] Linking CXX executable src/umbp/tests/test_client_registry_external_kv → FAILED: [code=1] (log line 24117).
• The error is raised by .../cmake-4.4/Modules/GoogleTest/ParseTestList.cmake:96 (string) via GoogleTestAddTests.cmake:245 (parse_tests_from_json) (log ~24232). CMake in use is the pip-installed cmake 4.4 at /opt/venv/lib/python3.10/site-packages/cmake/data/share/cmake-4.4/.
• The JSON handed to string(JSON …) is garbled/interleaved, e.g. "line": e": 53, stray fragments ses", and i\/src\/umbp\/tests\/test_client_registry.cpp", and * Line 12, Column 19 … Syntax error: value, object or array expected (log ~24173–24232).
• ninja: build stopped: subcommand failed. (log line 24289) → ##[error]Process completed with exit code 1. (log line 24429).
• The Dockerfile is byte-identical between the last passing (Jul 10, 94de2876) and first failing (Jul 11, 32cb89d4) runs: git log 94de28764c..32cb89d412 -- docker/rocm.Dockerfile returns no commits (verified).
• MORI_COMMIT is pinned to e31d426a13e96e1cbff96a1c904d291aefe8c46a (rocm.Dockerfile:113), last modified 2026-07-09 — i.e. unchanged across the pass→fail window, so the MORI source built was identical both days.
• cmake is installed unpinned: "$VENV_PIP" install --upgrade … cmake … (rocm.Dockerfile:361). Build uses --no-cache, so pip re-resolves cmake on every run.
• gfx950 base image is date-pinned rocm/sgl-dev:rocm7-vllm-20250904 (rocm.Dockerfile:26).

Hypothesised Causes (with confidence, with disconfirming evidence)

• [MEDIUM] Unpinned pip install --upgrade cmake (rocm.Dockerfile:361) resolved to cmake 4.4 between the Jul 10 pass and Jul 11 fail; cmake 4.4's gtest_discover_tests JSON handling is the exact code in the failing stack trace.
  • Supporting: the Dockerfile and MORI pin are identical across the pass→fail window, so a floating external dependency must have changed; cmake is the only such dependency that appears directly in the failing stack trace (cmake-4.4/.../ParseTestList.cmake, GoogleTestAddTests.cmake).
  • Disconfirming: I could not read the Jul 10 passing build log to confirm it used an older cmake (e.g. 4.3), so the version bump is inferred, not observed. The JSON itself is corrupted (interleaved fragments) rather than merely a schema cmake rejects — that points partly at the producing binary, not only the parser.
  • How to verify: pin cmake in rocm.Dockerfile:361 to the version used on Jul 10 (e.g. cmake==4.3.* or whatever the passing build resolved), rebuild docker build . -f docker/rocm.Dockerfile --build-arg GPU_ARCH=gfx950 --build-arg ENABLE_MORI=1 --no-cache; if it passes, cmake 4.4 is implicated.
• [LOW] Flaky/racy gtest test-discovery producing garbled JSON output (the interleaved fragments suggest a buffer/output race in the discovery run of test_client_registry_external_kv).
  • Supporting: the JSON is physically corrupt (out-of-order fragments), which a pure schema/version mismatch would not produce; discovery runs the freshly-built binary.
  • Disconfirming: this is the first failure after 7+ consecutive green runs of an identical Dockerfile, which is more consistent with a changed input than with pure nondeterminism.
  • How to verify: re-run the workflow (or rebuild with --no-cache); if it passes without any change, treat as flake and consider --gtest_output serialization / DISCOVERY_MODE=PRE_TEST in MORI.

In-flight Fix Check

• In-flight fix: ❌ none found — searched open sglang PRs for cmake, mori, gtest discover; no open PR addresses a cmake pin or the MORI gtest-discovery build break (matches were unrelated FlashMLA/Mooncake/MoRI-feature PRs). Cited: search/issues?q=repo:sgl-project/sglang+is:pr+is:open+cmake (and +mori, +gtest+discover).

Suggested Triage Steps

• Confirm the cmake version delta: diff the pip cmake version between the Jul 10 passing build log and this failing log (search each for cmake-4. / Installed … cmake). If Jul 10 used < 4.4, that confirms the bump.
• If confirmed, pin cmake in docker/rocm.Dockerfile:361 (e.g. cmake==<last-good>), rebuild gfx950 with --no-cache, and re-run release-docker-amd-nightly.yml.
• Independent of the version question, consider making the MORI sub-build resilient to test-discovery flakiness (e.g. build MORI with tests disabled, or set gtest_discover_tests(DISCOVERY_MODE PRE_TEST)) since these gtest binaries are not executed by the docker publish job anyway. This is an upstream ROCm/mori change.
• Re-run once to rule out pure flakiness before pursuing a code fix.

Status

• Persistence: 1 failing run so far (2026-07-11 12:24 UTC); prior 7 scheduled runs green (last pass 2026-07-10 12:51 UTC).
• Scope: publish (gfx950, all) in release-docker-amd-nightly.yml. Other matrix legs (gfx942 / rocm720) may share the same unpinned-cmake + MORI build path — check whether they failed in the same run.
• Blocked work: blocks publication of the nightly rocm/sgl-dev gfx950 (MI35x) image.
• In-flight fix: ❌ none found in open PRs — needs a new PR (cmake pin) or a re-run to confirm flakiness.

publish (gfx942, all) — failed step(s): Build and Push to rocm/sgl-dev

Commit Info

• sglang: 32cb89d4
• aiter: N/A (docker-build workflow — no [CI-AITER-CHECK] markers in log)
• MORI: e31d426a (Dockerfile default pin, docker/rocm.Dockerfile:113; not overridden by the workflow)

Failed Tests

Test File	Test Function	Error	Log
docker/rocm.Dockerfile (stage-4 23/28, rocm.Dockerfile:421 — MORI build)	N/A (image build step, not an sglang test)	CMake Error … string sub-command JSON failed parsing json string → ninja: build stopped → CalledProcessError: 'cmake --build . -j 64' … exit status 1 → build exit code 1	Log

This is a Docker image build failure, not a Python test failure. The publish step failed while building the MORI dependency inside the ROCm image.

Failure Summary

The nightly ROCm docker image build failed at stage-4 23/28 (docker/rocm.Dockerfile:421, the optional-MORI build with ENABLE_MORI=1). MORI compiles its own C++ unit tests, and CMake's GoogleTest gtest_discover_tests post-link step for the target test_client_registry_external_kv fed a corrupted JSON test list into string(JSON …) in .../cmake-4.4/Modules/GoogleTest/ParseTestList.cmake:96, which aborted the ninja/cmake --build . -j 64 build with exit 1. All 4 jobs in the run (publish (gfx942, all), publish (gfx950, all), and the two push_local_registry jobs) failed at the same step.

Regression Status

New regression (as of the 2026-07-11 scheduled run). This workflow was green every day from 2026-07-02 through 2026-07-10 and failed for the first time today.

Recent history of the MORI build step in workflow release-docker-amd-nightly.yml, job publish (branch main, event schedule, completed runs only):

Date	Run	Job	Step Status	Detail
Jul 11	29152576569	publish (gfx942, all)	❌ Failed	MORI gtest_discover_tests JSON parse error
Jul 10	29093958349	publish	✅ Passed	—
Jul 09	29020296834	publish	✅ Passed	—
Jul 08	28943370684	publish	✅ Passed	—
Jul 07	28867829963	publish	✅ Passed	—

First observed failure: 2026-07-11 12:24 UTC. Last known passing: 2026-07-10 (29093958349). (The 2026-06-30 run also failed, but the ~11 green runs since exonerate that as unrelated.)

Failure Cluster

MORI gtest_discover_tests JSON parse failure during ROCm docker build

Facts (verified observations)

• Exact CMake error (verbatim): CMake Error at /opt/venv/lib/python3.10/site-packages/cmake/data/share/cmake-4.4/Modules/GoogleTest/ParseTestList.cmake:96 (string): string sub-command JSON failed parsing json string: followed by Syntax error: value, object or array expected. at Line 12, Column 19.
• The JSON fed to the parser is visibly corrupted / interleaved: e.g. "line": e": 53 and a stray ses", fragment appearing mid-array — i.e. two output streams were interleaved into one JSON buffer/file.
• The failing target's discovery script is test_client_registry_external_kv_e3b0c442_discovery.cmake; the hash segment e3b0c442 is the well-known SHA-256 prefix of the empty string, i.e. an empty/degenerate discriminator.
• Build ran under cmake --build . -j 64 (high parallelism); ninja printed build stopped: subcommand failed and setup.py … subprocess.CalledProcessError.
• MORI commit is the Dockerfile default e31d426a (docker/rocm.Dockerfile:113); the workflow (release-docker-amd-nightly.yml:90) passes only ENABLE_MORI=1 — no MORI_COMMIT override.
• docker/rocm.Dockerfile was not modified in the sglang commit window 94de28764c..32cb89d4 (30 commits, git log --oneline … -- docker/rocm.Dockerfile empty). The MORI pin was last touched 2026-07-09 by e703f9e5 — before the Jul 10 green run.
• Build uses --no-cache (workflow line 90) → every nightly rebuild pulls fresh unpinned apt/pip toolchain, including cmake (pip install … cmake …); the discovery ran under cmake 4.4.
• All 4 jobs in the run failed at this same step.

Hypothesised Causes (with confidence)

• [MEDIUM] Not an sglang-code regression. The ROCm Dockerfile and the pinned MORI commit e31d426a were byte-for-byte identical between the Jul 10 green run and the Jul 11 red run, yet the same MORI source built fine on Jul 10.
  • Supporting: no Dockerfile change in the commit window; MORI pin unchanged since Jul 9; sglang HEAD commit 32cb89d4 only prunes uv cache (irrelevant to the MORI/CMake step).
  • Disconfirming: (none found) — the changed sglang code has no path into MORI's C++/CMake build.
  • How to verify: rerun the workflow with no sglang change; if it still fails, sglang is exonerated.
• [LOW] Flaky parallel race in MORI's gtest discovery corrupting the shared JSON test list under -j 64. The interleaved "line": e": 53 / ses", fragments and the empty-string discriminator hash e3b0c442 are consistent with two discovery processes writing the same JSON output file concurrently.
  • Supporting: corrupted-JSON symptom is characteristic of a write race, not a syntactically-broken generator; deterministic bad JSON would fail every run, but the step passed 9 days straight.
  • Disconfirming: cannot confirm concurrency without a second (passing) run diff; the corruption could equally come from item below.
  • How to verify: rerun the failing job 2–3× — intermittent pass ⇒ flake; retry MORI build with -j 1 on a branch.
• [LOW] Unpinned toolchain drift in the --no-cache daily rebuild (most likely a cmake/GoogleTest-module bump to 4.4 that changed gtest_discover_tests JSON emission or string(JSON) strictness).
  • Supporting: build pins nothing for cmake; only external variable that changed between Jul 10 pass and Jul 11 fail is whatever the fresh pip/apt resolve pulled; error originates inside cmake-4.4/Modules/GoogleTest.
  • Disconfirming: could not diff the Jul 10 vs Jul 11 cmake version (Jul 10 log not fetched); the error is a parse failure on corrupt input, which a version bump alone would not obviously produce.
  • How to verify: compare cmake --version / installed cmake in the Jul 10 29093958349 publish log vs today's; if cmake bumped, pin it.

In-flight Fix Check

In-flight fix: ❌ none found — searched open sglang PRs for MORI, gtest_discover_tests, and release-docker-amd (GET /search/issues?q=repo:sgl-project/sglang+is:pr+is:open+<kw>). Open MORI PRs (#30706, #30651, #30052) are feature/CI additions, none address this build break.

Suggested Triage Steps

• Rerun the failing job 2–3× (or with the build's MORI step forced to -j 1) to distinguish a flaky JSON-write race from a deterministic toolchain break.
• Diff the toolchain between the Jul 10 green 29093958349 and Jul 11 red 29152576569 publish logs — grep both for the installed cmake version; if cmake bumped to 4.4, try pinning cmake<4.4 in the MORI build block of docker/rocm.Dockerfile.
• If reproducible, report upstream to ROCm/mori — the gtest_discover_tests discriminator hash e3b0c442 (empty string) suggests MORI's test CMake is not giving each target a unique discovery output path, causing JSON collisions under parallel discovery. A workaround is set(CMAKE_GTEST_DISCOVER_TESTS_DISCOVERY_MODE PRE_TEST) or disabling test discovery in the image build.
• As a fast unblock for the nightly image, MORI tests are not needed in the published image — consider building MORI with its test target disabled (e.g. a -DMORI_BUILD_TESTS=OFF-style flag if MORI exposes one) in docker/rocm.Dockerfile.

Status

• Persistence: First failure 2026-07-11 12:24 UTC (1 run); green the preceding 9 consecutive daily runs (Jul 2–10).
• Scope: All 4 jobs of the run — publish (gfx942, all), publish (gfx950, all), push_local_registry (gfx942), push_local_registry (gfx950); the push_local_registry jobs fail downstream because no image is published.
• Blocked work: No nightly ROCm rocm/sgl-dev images published for 2026-07-11 (both gfx942 and gfx950).
• In-flight fix: ❌ none found in open PRs — needs a new PR or a rerun to confirm flakiness.

push_local_registry (gfx950) — failed step(s): Download image tag artifact

Commit Info

• sglang: 32cb89d4 (ci: prune uv cache in job teardown to bound its growth #30813)
• aiter: N/A — this is a docker-build/release workflow; no [CI-AITER-CHECK] markers in the log.

Failed Tests

This is not a pytest/unittest failure. It is a two-level build/artifact failure:

Stage	What	Error	Log
push_local_registry (gfx950) (this job)	actions/download-artifact@v4 step	Unable to download artifact(s): Artifact not found for name: image-tag-gfx950	Log
publish (gfx950, all) (upstream root cause)	docker build docker/rocm.Dockerfile → stage-4 23/28 MORI build	CMake gtest_discover_tests → ParseTestList.cmake:96 string(JSON ...) Syntax error: value, object or array expected on test_client_registry_external_kv; ninja: build stopped: subcommand failed	Log

Failure Summary

This job (push_local_registry (gfx950)) failed at its very first step because the artifact image-tag-gfx950 did not exist. That artifact is only uploaded by the upstream publish (gfx950, all) job after its docker build+docker push succeed (release-docker-amd-nightly.yml:93-107). The publish (gfx950, all) job 86544356094 failed during the ROCm docker build while building the third-party MORI library (ROCm/mori), so the artifact was never produced and this job's download failed as a cascade. The publish (gfx942, all) leg 86544356085 failed with the identical MORI error, taking down push_local_registry (gfx942) 86555828361 too — all 4 jobs in the run are red for this one root cause.

Regression Status

Flaky / nondeterministic build failure — first observed 2026-07-11.

Recent history of workflow release-docker-amd-nightly.yml (event=schedule, MORI build stage of publish):

Date	Run	Build Status	Note
Jul 11	29152576569	❌ Failed	MORI gtest_discover_tests JSON parse error (both arches)
Jul 10	29093958349	✅ Passed	same MORI pin e31d426a, same test binary built, no parse error
Jul 09	29020296834	✅ Passed	—
Jul 08	28943370684	✅ Passed	—
Jul 04–07	4 runs	✅ Passed	—

Lookback window: last 8 scheduled runs (2026-07-04 → 2026-07-11). Today is the first and only failure.

Failure Cluster

MORI docker build failure — CMake gtest_discover_tests JSON parse error on test_client_registry_external_kv (cascades to artifact-not-found in push_local_registry).

Facts (verified observations)

• This job's error (verbatim): ##[error]Unable to download artifact(s): Artifact not found for name: image-tag-gfx950 — the actions/download-artifact@v4 step, the only substantive step that ran.
• Upstream root cause (publish log): CMake Error at .../GoogleTest/ParseTestList.cmake:96 (string): string sub-command JSON failed parsing json string: ... * Line 12, Column 19 Syntax error: value, object or array expected. while running test_client_registry_external_kv_e3b0c442_discovery.cmake (i.e. gtest_discover_tests executing the freshly-built src/umbp/tests/test_client_registry_external_kv binary to enumerate its tests). Followed by ninja: build stopped: subcommand failed and Docker ERROR: failed to build: failed to solve ... exit code: 1 at Dockerfile stage-4 23/28 (the MORI clone+build RUN).
• The captured JSON is corrupted/interleaved, not merely malformed: fragments appear out of order and overwrite each other, e.g. "line": e": 53, stray ses",, i\/src\/umbp\/tests\/test_client_registry.cpp", and two distinct testsuite documents mixed together. This is the signature of extra/racing stdout output from the test binary at discovery time, not a static syntax error.
• MORI is pinned: docker/rocm.Dockerfile:113 ARG MORI_COMMIT="e31d426a13e96e1cbff96a1c904d291aefe8c46a" (log confirms Note: switching to 'e31d426a13e96...'). Pin unchanged since 2026-07-09 (verified via git log -S).
• Same-input control: yesterday's passing run 29093958349 checked out the same MORI commit e31d426a and built test_client_registry_external_kv (4×), with zero Syntax error: value / GoogleTestAddTests occurrences.
• Both arches affected identically: gfx942 86544356085 and gfx950 86544356094 both fail at the same MORI test_client_registry_external_kv discovery step.
• The build uses --no-cache (release-docker-amd-nightly.yml:90), so MORI is re-cloned and rebuilt from scratch every run.

Hypothesised Causes (with confidence)

• [HIGH] This job's own failure is a downstream cascade, not an independent bug: image-tag-gfx950 is uploaded only after publish (gfx950)'s docker push succeeds, and push_local_registry runs under if: !cancelled(), so a failed publish leg deterministically fails the matching push_local_registry leg at the download step.
  • Supporting: workflow definition lines 97-107 gate the upload on build+push success; publish (gfx950) is red; this job's only error is artifact-not-found.
  • Disconfirming: (none found).
  • How to verify: re-run the workflow; if publish goes green, this job will find the artifact.
• [MEDIUM] The upstream publish failure is a nondeterministic/flaky MORI build failure in gtest_discover_tests, where the test_client_registry_external_kv binary emits racing/extra stdout (plausibly spdlog/mori_log output or the client-registry "Reaper" background thread — this suite has Reaper* and lease/heartbeat timing tests) that corrupts the JSON gtest writes for CMake to parse.
  • Supporting: byte-identical MORI pin + same test binary passed yesterday and failed today (rules out a deterministic code change in the pin); the parse error is on interleaved/out-of-order JSON ("line": e": 53, two mixed documents), which is a runtime/output-race artifact, not a fixed malformed string; the affected suites involve background reaper threads and timing.
  • Disconfirming: I did not capture the raw pre-parse stdout to prove the interleaving comes from the test binary vs. from ninja -j64 interleaving two concurrent discovery runs' output into the same file — either would be a MORI/CMake-side race, but the precise source is unconfirmed.
  • How to verify: rebuild docker/rocm.Dockerfile with the same MORI_COMMIT=e31d426a a few times; if it passes on retry, flakiness is confirmed. To pin the source, run MORI's build with ninja -j1 or -DCMAKE_GTEST_DISCOVER_TESTS_DISCOVERY_MODE=PRE_TEST and inspect the captured discovery stdout.
• [SPECULATION] sglang HEAD commit 32cb89d4 contributed.
  • Supporting: it is today's head SHA.
  • Disconfirming: it only prunes the uv cache in CI teardown (#30813) and touches nothing in docker/rocm.Dockerfile, the MORI build, or artifact handling. Not a plausible cause.

In-flight Fix Check

• In-flight fix: ❌ none found — searched open sglang PRs for mori+gtest_discover_tests (0 hits), gtest_discover_tests+json (0 hits), and MORI+build+docker (6 hits, all unrelated Dockerfile/feature PRs). No open PR addresses this MORI discovery-time JSON corruption.

Suggested Triage Steps

• Re-run the failed workflow first — given yesterday's identical-input pass, a plain re-run of 29152576569 will likely go green and unblock the nightly image + local-registry mirror. Confirms the flakiness hypothesis cheaply.
• If it recurs, reproduce the MORI build locally with MORI_COMMIT=e31d426a (docker build . -f docker/rocm.Dockerfile --build-arg ENABLE_MORI=1 --no-cache) a few times to establish the flake rate.
• To harden against the discovery-time race (MORI-side / ROCm/mori), consider building MORI's C++ tests with gtest_discover_tests(... DISCOVERY_MODE PRE_TEST) or -j1 for the discovery step, or disabling test-target builds in the image entirely (the runtime image does not need MORI's gtest binaries). File upstream at ROCm/mori if confirmed.
• Independently, consider decoupling push_local_registry from silent cascade: it already isolates failed legs, but the artifact-not-found message obscures the real (upstream build) cause — a note/step that surfaces "upstream publish leg failed" would speed future triage.

Status

• Persistence: First failure on 2026-07-11 (run started 12:24 UTC); the prior 7 consecutive scheduled runs (2026-07-04 → 2026-07-10) passed.
• Scope: All 4 jobs in this run red from one root cause — both publish legs (gfx942, gfx950) fail in the MORI build; both push_local_registry legs cascade to artifact-not-found. Blocks today's nightly rocm/sgl-dev / lmsysorg/sglang-rocm image publish and the in-network local-registry mirror.
• In-flight fix: ❌ none found in open PRs — needs a re-run to confirm flakiness, then a MORI-side hardening PR if it recurs.

push_local_registry (gfx942) — failed step(s): Download image tag artifact

Commit Info

• sglang: 32cb89d4
• aiter: N/A (docker-build workflow — no [CI-AITER-CHECK] markers in the log)

Failed Tests

This is not a test failure — it is a cascade from an upstream docker-build failure. No test suite ran in this job.

What failed	Where	Error	Log
Download image tag artifact step	push_local_registry (gfx942) (job 86555828361)	Unable to download artifact(s): Artifact not found for name: image-tag-gfx942	Log
Root cause — Build and Push to rocm/sgl-dev step (MORI build)	publish (gfx942, all) (job 86544356085)	cmake --build . -j 64 exit 1 → setup.py develop failed at rocm.Dockerfile:421 ([stage-4 23/28])	Log

Failure Summary

push_local_registry (gfx942) failed at its very first step because it could not download the image-tag-gfx942 artifact. That artifact was never produced: its needs: dependency publish (gfx942, all) failed at the Build and Push to rocm/sgl-dev step, so the Save published image tag and Upload image tag artifact steps were skipped (verified in the publish job's step list — steps 8 & 9 skipped). The underlying docker build aborted during the MORI build stage. This is a cascade failure by design: the workflow comment notes push_local_registry runs under !cancelled(), so legs whose publish leg produced no artifact "fail at download and be the only ones marked red."

Regression Status

New regression today (workflow-level), but not caused by an sglang code change (see Hypothesised Causes).

Recent history of the ROCm nightly docker build (release-docker-amd-nightly.yml, branch=main, event=schedule, completed runs only):

Date	Run	push_local_registry (gfx942)	publish (gfx942, all)
Jul 11	29152576569	❌ Failed (artifact missing)	❌ Failed (MORI build)
Jul 10	29093958349	✅ Passed	✅ Passed
Jul 09	29020296834	✅ Passed	✅ Passed
Jul 08	28943370684	✅ Passed	✅ Passed
Jul 07	28867829963	✅ Passed	✅ Passed

Green for 10 consecutive scheduled runs (Jul 1–10); first red today. (Prior isolated failure Jun 30 28445643935.) The gfx950 legs failed identically in the same run.

Failure Cluster

MORI build failure during ROCm docker image build — GoogleTest gtest_discover_tests JSON parse error.

Facts (verified observations)

• This job failed only at Download image tag artifact: ##[error]Unable to download artifact(s): Artifact not found for name: image-tag-gfx942 (job log line 28). No other steps ran.
• Publish job 86544356085 step sequence: step 7 Build and Push to rocm/sgl-dev = failure; steps 8 Save published image tag & 9 Upload image tag artifact = skipped (they only run on prior success, so no artifact was uploaded).
• The docker build aborted at rocm.Dockerfile:421 (MORI stage [stage-4 23/28]): subprocess.CalledProcessError: Command '['cmake', '--build', '.', '-j', '64']' returned non-zero exit status 1.
• Underlying failures are 4 GoogleTest test-discovery steps for umbp targets: test_external_kv_block_index, test_client_registry, test_tier_priority_route_get, test_ssd_read_lease_gating. Each: string sub-command JSON failed parsing json string: Syntax error: value, object or array expected. gtest_discover_tests runs each compiled test binary at build time to enumerate tests as JSON; the binaries produced non-JSON/empty output → ninja: build stopped: subcommand failed. No C++ compile error: lines precede these — only a deprecation warning (get_temporary_buffer).
• MORI is pinned to ROCm/mori commit e31d426a (docker/rocm.Dockerfile:113), unchanged since Jul 9 (verified via git log -S). The build runs --no-cache, so MORI is rebuilt fresh each night.
• Build emitted repeated DNS warnings: Failed to fetch http://compute-artifactory.amd.com/... Something wicked happened resolving 'compute-artifactory.amd.com:http' (-5 - No address associated with hostname) — these are apt warnings (ignored), not the fatal error.
• Window Jul10→Jul11 = 30 sglang commits (94de28764c..32cb89d4). Only one touches docker/: 3f1694f5 "Update sgl-deep-gemm to 0.1.4.post1" — and it edits docker/Dockerfile (CUDA), not docker/rocm.Dockerfile or the MORI stage.

Hypothesised Causes (with confidence)

• [MEDIUM] Environmental / transient failure in the MORI gtest_discover_tests step, not an sglang regression.
  • Supporting: identical MORI pin (e31d426a) and identical rocm.Dockerfile MORI recipe built successfully on Jul 10; --no-cache means fresh apt/base packages and a fresh MORI compile every run, so a nondeterministic test-binary crash during discovery, or a newly-pulled upstream package/base-image change, can flip the result without any sglang change. No sglang commit in the window touches the ROCm docker path.
  • Disconfirming: the exact reason the compiled umbp test binaries emitted non-JSON output is not visible in the log (no runtime stack trace captured for the discovery run), so "transient" is inferred, not proven; a persistent upstream MORI/base-image break would produce the same symptom and only re-running will disambiguate.
  • How to verify: re-run the workflow (workflow_dispatch); if it goes green, transient confirmed. If it stays red, build docker/rocm.Dockerfile with ENABLE_MORI=1 GPU_ARCH=gfx942 locally and inspect the gtest_discover_tests invocation output for the umbp targets.
• [LOW] DNS/network flakes to compute-artifactory.amd.com degraded the build environment.
  • Supporting: repeated No address associated with hostname warnings during the MORI apt phase.
  • Disconfirming: those lines are apt warnings that were explicitly ignored ("old ones used instead"); the fatal error is the gtest JSON parse, which is downstream of a successful compile, not a fetch failure.
  • How to verify: check runner network health at 14:28–14:31 UTC; not actionable until the transient hypothesis is ruled out.
• [SPECULATION] sglang commit in the window caused the MORI break — strongly disfavored: no sglang commit in 94de28764c..32cb89d4 touches rocm.Dockerfile, the MORI stage, or the MORI pin.

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs for mori+rocm (20 hits, all unrelated features: e.g. #30706 fp4 combine, #30651 MORI disagg cookbook) and rocm+docker (57 hits, none targeting the MORI gtest-discovery build). No open PR addresses this MORI build break.

Suggested Triage Steps

• Re-run the nightly via workflow_dispatch first — cheapest disambiguation between transient vs persistent. If green, close as a flake.
• If it stays red, reproduce locally: docker build . -f docker/rocm.Dockerfile --build-arg ENABLE_MORI=1 --build-arg GPU_ARCH=gfx942 --no-cache and capture the output of the failing gtest_discover_tests runs for src/umbp/tests/test_{external_kv_block_index,client_registry,tier_priority_route_get,ssd_read_lease_gating}.
• If persistent, check ROCm/mori upstream and consider whether the umbp GoogleTest discovery should be disabled at build time (gtest_discover_tests → PRE_TEST / gtest_add_tests, or -DBUILD_TESTING=OFF) since these tests are not needed in the shipped image — raise with the MORI-integration owner.
• Note this job (push_local_registry) needs no separate fix — it will go green automatically once publish (gfx942, all) produces the artifact again.

Status (factual)

• Persistence: First failure today (Jul 11, run started 12:24 UTC) after 10 consecutive green scheduled runs (Jul 1–10).
• Scope: All 4 jobs of the run red — both publish legs (gfx942, gfx950) failed the MORI build, cascading to both push_local_registry legs. Confined to release-docker-amd-nightly.yml (ROCm 7.0 nightly image).
• Blocked work: No fresh rocm/sgl-dev / lmsysorg/sglang-rocm nightly image published today, and the in-network registry mirror was not updated — AMD CI runners will continue using the Jul 10 image.
• In-flight fix: ❌ none found in open PRs.

publish (gfx942, all) — failed step(s): Build and Push to rocm/sgl-dev

Commit Info

• sglang: 80856aba
• aiter: N/A (docker-build workflow — no [CI-AITER-CHECK] markers in log)

Failed Tests

Test File	Test Function	Error	Log
N/A (Docker build failure)	N/A	ninja: build stopped: subcommand failed → cmake --build . exit 1 in MORI stage	Log

This is a Docker image build failure, not a test failure. Step 7 "Build and Push to rocm/sgl-dev" failed at Dockerfile stage [stage-4 23/28] (the MORI build, docker/rocm.Dockerfile:420-504, gated behind ENABLE_MORI=1).

Failure Summary

The nightly ROCm image build for gfx942 failed while building ROCm/mori (pinned to MORI_COMMIT=e31d426a). The MORI CMake build aborts during GoogleTest test-discovery: cmake -P .../ParseTestList.cmake fails to parse the JSON test list emitted by three freshly-linked mori test binaries (test_external_kv_block_index, test_external_kv_hit_index, test_peer_dram_allocator), stopping ninja and returning exit 1.

Regression Status

New regression — first failed 2026-07-11, recurring since (2 consecutive nightly runs).

Recent history of MORI build stage in job publish (gfx942, all) (branch main, event schedule, completed runs only):

Date	Run	Job	Stage Status	Failure	cmake version
Jul 12	29192571824	publish (gfx942, all)	❌ Failed	ParseTestList.cmake JSON parse	4.4.0
Jul 11	29152576569	publish (gfx942, all)	❌ Failed	same ParseTestList.cmake JSON parse	4.4.0
Jul 10	29093958349	publish (gfx942, all)	✅ Passed	—	4.3.4
Jul 09	29020296834	publish (gfx942, all)	✅ Passed	—	(pre-4.4)

Note: today's run 29192571824 is overall in_progress (other matrix legs still running), but this job (publish (gfx942, all), job 86649709878) is completed with conclusion failure.

Failure Cluster

CMake GoogleTest discovery JSON parse error during MORI build (ParseTestList.cmake — empty/invalid JSON from mori gtest binaries)

Facts (verified observations)

• Verbatim error (all 3 binaries): CMake Error at /opt/venv/lib/python3.10/site-packages/cmake/data/share/cmake-4.4/Modules/GoogleTest/ParseTestList.cmake:96 (string): string sub-command JSON failed parsing json string … Syntax error: value, object or array expected. (log lines 24079, 24097, 24115).
• Call stack: GoogleTestAddTests.cmake:245 (parse_tests_from_json) ← test_<name>_e3b0c442_discovery.cmake:2 (gtest_discover_tests_impl) — i.e. CMake ran the just-built test executable to enumerate tests and got empty/non-JSON output.
• cmake version diff (FACT, from log): last passing run (Jul 10) installed cmake 4.3.4 with 0 ParseTestList errors; both failing runs (Jul 11, Jul 12) installed cmake 4.4.0 (uninstalling 3.31.6). Today's log line 21122: Successfully installed cmake-4.4.0; Jul 10 log line 21100: Successfully installed cmake-4.3.4.
• cmake is unpinned in docker/rocm.Dockerfile:361 (TileLang stage): "$VENV_PIP" install --upgrade … wheel cmake ninja scikit-build-core → always pulls latest; the venv cmake is then reused by the MORI build stage.
• MORI_COMMIT (docker/rocm.Dockerfile:113) and the mori source are unchanged across passing and failing runs (pinned commit e31d426a); no recent sglang commit modified the MORI/cmake Dockerfile logic (git log -- docker/rocm.Dockerfile HEAD is unrelated PR #30654).
• Build is --no-cache (workflow line 90), so every run re-resolves the latest cmake wheel.

Hypothesised Causes (with confidence)

• [HIGH] The unpinned cmake pip dependency upgraded from 4.3.4 → 4.4.0 between Jul 10 and Jul 11; cmake 4.4.0's GoogleTestAddTests.cmake/ParseTestList.cmake discovery no longer tolerates the empty/invalid JSON produced when the mori gtest binaries are executed for discovery inside the GPU-less docker build.
  • Supporting: three independent evidence chains converge — (a) exact version diff verified from log (4.3.4 pass → 4.0.0… 4.4.0 fail), (b) timing aligns precisely (failure begins the first run 4.4.0 appears), (c) the erroring module path literally is cmake-4.4/Modules/GoogleTest/ParseTestList.cmake, and mori source + MORI_COMMIT are provably constant.
  • Disconfirming: (none found) — no sglang or mori source change coincides with the regression window; the only changed build input is the cmake version.
  • How to verify: pin cmake==4.3.4 in docker/rocm.Dockerfile:361 (and any other pip install … cmake sites) on a branch, rebuild the gfx942 image with ENABLE_MORI=1; if the MORI stage passes, cmake 4.4.0 is confirmed.
• [LOW] The mori test binaries themselves cannot execute during discovery (no GPU / missing runtime lib), yielding empty JSON that only 4.4.0 rejects.
  • Supporting: discovery requires running the binary; docker build host has no GPU.
  • Disconfirming: this was true under 4.3.4 as well yet the build passed — so it is at most a co-factor, not the trigger.
  • How to verify: run one mori test binary with --gtest_list_tests --gtest_output=json: inside the build container and inspect the output.

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs for cmake+4.4, gtest_discover, ParseTestList (0 hits each); cmake+pin and mori+build hits are unrelated (CUDA-13 CCCL, Mooncake pin, disaggregation legs). API: GET /search/issues?q=repo:sgl-project/sglang+is:pr+is:open+<kw>.

Suggested Triage Steps

• Pin the cmake version in docker/rocm.Dockerfile:361 to the last-known-good (cmake==4.3.4) and rebuild the gfx942/gfx950 images with ENABLE_MORI=1 to confirm the MORI stage recovers.
• If pinning is undesirable, check upstream ROCm/mori for a newer commit that disables in-build gtest discovery (e.g. gtest_discover_tests(... DISCOVERY_MODE PRE_TEST) or BUILD_UNIT_TESTS=OFF), then bump MORI_COMMIT.
• File/track an upstream mori issue: gtest discovery emits empty JSON in a GPU-less build container and breaks under cmake 4.4.0.
• Verify the sibling publish (gfx950, all) leg to confirm it hit the identical MORI/cmake failure (same unpinned cmake path), which would confirm this is environment-wide, not arch-specific.

Status

• Persistence: MORI build stage failing since 2026-07-11 (Jul 11 + Jul 12 nightly runs); Jul 10 and earlier were green.
• Scope: publish (gfx942, all) in release-docker-amd-nightly.yml; the gfx950 leg and the Jul 11 run's gfx950 leg also failed, indicating the whole nightly ROCm image publish (both archs) is blocked. Downstream push_local_registry legs depend on the publish artifact and also failed Jul 11.
• Blocked work: No nightly rocm/sgl-dev / lmsysorg/sglang-rocm ROCm 7.0 image published since 2026-07-10; local-registry mirror for AMD CI runners not refreshed.
• In-flight fix: ❌ none found in open PRs — needs a new PR (pin cmake, or bump MORI_COMMIT).

---

Generated by amd-bot using Claude Code CLI (last updated: 2026-07-12 15:17 UTC)

[amd-bot]

pr-test-amd-rocm720.yml — 26 failure(s)

Run started (UTC): 2026-07-11 17:46 UTC
Last scanned (UTC): 2026-07-12 23:21 UTC
Commits: sglang 96a04cb

Summary

Counts: 26 failures · 10 clusters · 9 clusters carrying over (≥3 days) · 1 recent (msgpack, since Jul 11) — across 2 completed scheduled runs 29162225443 (Jul 11) and 29202664092 (Jul 12). Not a scout workflow — no Origin column.

#	Cluster	Job	Test File	Test Function	Status	Hypothesis (confidence)
1	GPU Hang / KFD leak (MI35x)	stage-c-8gpu-mi35x (0)	test/registered/amd/test_kimi_k25_mxfp4_bcg_mi35x.py	test_bcg_gsm8k	last pass Jul 6; red Jul 11–12	[MEDIUM] MI35x KFD/VRAM-leak infra
2	GPU Hang / KFD leak (MI35x)	stage-c-8gpu-mi35x (0)	test_kimi_k25_mxfp4_bcg_mi35x.py	test_bcg_gsm8k	2 consecutive runs	[MEDIUM] #30431
3	GPU Hang / KFD leak (MI35x)	stage-c-8gpu-mi35x (1)	test/registered/amd/test_deepseek_r1_mxfp4_8gpu.py	setUpClass	last pass Jul 8	[MEDIUM] #30431
4	GPU Hang / KFD leak (MI35x)	stage-c-8gpu-mi35x (1)	test_deepseek_r1_mxfp4_8gpu.py	N/A (load hang @93%)	Jul 11–12	[LOW] runner GPU hang
5	GPU Hang / KFD leak (MI35x)	stage-b-1gpu-small-mi35x	N/A (suite crashed pre-test)	N/A	since Jul 9	[MEDIUM] flaky MI35x node
6	GPU Hang / KFD leak (MI35x)	stage-b-1gpu-small-mi35x	test/registered/quant/test_quark_mxfp4.py	TestOnlineQuantizationMemoryLoadDense.setUpClass	new Jul 11–12	[MEDIUM] #30431
7	1-GPU diffusion suite fail (MI325)	mm-gen-1gpu (0)	.../test/server/test_server_1_gpu.py	unpinnable (60-min timeout)	recurring	[MEDIUM] suite > 60-min budget
8	1-GPU diffusion suite fail (MI325)	mm-gen-1gpu (0)	undetermined (log 404)	—	pass Jul 9/10, red Jul 11–12	[LOW-MED] step timeout
9	1-GPU diffusion suite fail (MI325)	mm-gen-1gpu (1)	N/A (runner comm-loss)	N/A	5/6 runs	[MEDIUM] resource/time exhaustion
10	1-GPU diffusion suite fail (MI325)	mm-gen-1gpu (1)	run_suite.py --suite 1-gpu -p1	N/A (HF-download timeout)	never-passed in window	[MEDIUM] HF re-download > 60 min
11	1-GPU diffusion suite fail (MI325)	mm-gen-1gpu (3)	.../test_update_weights_from_disk.py	TestUpdateWeightsFromDisk::*[Qwen-Image]	≥8 days	[LOW→MED] InferenceMode copy (distinct root; log 404 today)
12	1-GPU diffusion suite fail (MI325)	mm-gen-1gpu (3)	N/A (runner comm-loss)	N/A	4 runs Jul 9–12	[MEDIUM] resource exhaustion
13	MI35x disagg GPU instability	stage-b-8gpu-mi35x-disagg	.../disaggregation/test_disaggregation_basic.py (+_pp,+_nixl)	setUpClass	intermittent (green Jul 7/8)	[MEDIUM] runner GPU state
14	MI35x disagg GPU instability	stage-b-8gpu-mi35x-disagg	test_disaggregation_basic.py	whole file	red since Jul 9	[MEDIUM] HW GPU hang
15	DSV4-Flash GPU Hang (bring-up)	dsv4-flash-fp4-fp8	test/registered/amd/test_deepseek_v4_flash_fp4.py	setUpClass	Never-passed	[LOW] DSV4 bring-up unstable
16	DSV4-Flash GPU Hang (bring-up)	dsv4-flash-fp4-fp8	test_deepseek_v4_flash_fp4.py	TestDeepseekV4Fp4.setUpClass	Never-passed	[LOW] graph-capture hang
17	Disagg _is_default array crash	mm-gen-2gpu (1)	.../single_test_file/test_disagg_server.py	TestDisaggZImage* (×3)	Never-passed since Jul 9	[HIGH] #30492 _is_default
18	Disagg _is_default array crash	mm-gen-2gpu (1)	test_disagg_server.py	TestDisaggZImage* (×3)	≥4 days	[FACT] array truthiness in extract_transfer_fields
19	flux2 FP8 HIPBLAS + port cascade	mm-gen-2gpu (0)	.../test/server/test_server_2_gpu.py	[flux2_modelopt_fp8_tp2_t2i] (+joy_echo +16 cascade)	Never-passed	[HIGH] #28889 hipBLAS FP8
20	flux2 FP8 HIPBLAS + port cascade	mm-gen-2gpu (0)	test_server_2_gpu.py	[flux2_modelopt_fp8_tp2_t2i] (+lora port leak, 18 cascade)	≥12 days	[HIGH] #28889
21	Triton throughput <3500 gate	stage-b-1gpu-large (0)	test/registered/perf/test_bench_serving_1gpu_part1.py	test_offline_throughput_with_triton_attention_backend	Never-passed ≥6 days	[MEDIUM] AMD gate too high
22	Triton throughput <3500 gate	stage-b-1gpu-large (0)	test_bench_serving_1gpu_part1.py	test_offline_throughput_with_triton_attention_backend	persistent (3150–3212 tok/s)	[MEDIUM] gate mis-calibrated
23	msgpack missing (ROCm image)	mm-gen-unit-test	.../test/unit/test_pi05_action_api.py	test_msgpack_roundtrip_preserves_string_keys_and_numpy_payloads	new Jul 11	[MEDIUM] #30633 dep not installed
24	msgpack missing (ROCm image)	mm-gen-unit-test	test_pi05_action_api.py	test_msgpack_roundtrip_...	Jul 11–12	[HIGH] #30633
25	GLM-Image invalid repo id	mm-gen-2gpu (2)	.../single_test_file/test_ar_models.py	TestGLMImage::test_generates_image	Never-passed since Jul 9	[HIGH] #25381 concatenated repo id
26	Remote TransferEngine zeroed params	stage-b-2gpu-large (1)	test/registered/model_loading/test_load_weights_from_remote_instance.py	test_load_weights_from_remote_instance	flaky (2 fails / 5 clean runs)	[HIGH] #30035 Mooncake flake

Cluster: GPU Hang / KFD leak (MI35x) — rows 1–6

• Affected: stage-c-8gpu-mi35x (0)+(1) and stage-b-1gpu-small-mi35x, both runs. All crash with HW Exception … reason: GPU Hang → SIGABRT (exit 134); both attempts (incl. HF_HUB_OFFLINE=1 retry) abort.
• Shared facts: hangs at different stages (kimi/dsr1 weight load, quark mxfp4 prefill); aiter unchanged across pass→fail; same runners failed Ensure VRAM is clear (zombie-KFD, ROCm/aiter#2061) Jul 6–9.
• Top hypothesis: [MEDIUM] driver-level VRAM-leak / zombie-KFD on MI35x runner pool. Disconfirming: some runs' pre-flight VRAM-clear passed, so GPU wasn't obviously pre-wedged.
• In-flight fix: ✅ #30431 (self-heal MI35x driver VRAM leaks, open since 2026-07-07) — chase reviewers.
• Suggested triage: reboot/drain the MI35x nodes, confirm rocm-smi shows no orphaned VRAM, re-run; land #30431 and re-check.

Cluster: 1-GPU diffusion suite fail (MI325) — rows 7–12

• Affected: mm-gen-1gpu shards 0/1/3 across both runs. Symptoms alternate: 60-min step timeout, "runner lost communication", and (shard 3 Jul 7 log) an InferenceMode weight-update 500. Most logs are BlobNotFound (run still finalizing).
• Shared facts: passing shard-0 runs already consume ~50–56 min of the 60-min step budget; HF cache reported "incomplete" → multi-GB re-downloads; lighter shard 2 passes every run.
• Top hypothesis: [MEDIUM] suite exceeds MI325 time/resource budget (download + load), starving/severing the runner. Disconfirming: today's logs unavailable, cannot separate resource-exhaustion from a genuine per-model hang; shard-3 has a distinct InferenceMode weight-update root.
• In-flight fix: ❌ none found.
• Suggested triage: pull runner dmesg/OOM + rocm-smi around the failure window; pre-stage HF weights or re-partition/raise timeout-minutes; separately repro shard-3's weights_updater.py InferenceMode copy.

Cluster: MI35x disagg GPU instability — rows 13–14

• Affected: stage-b-8gpu-mi35x-disagg both runs. test_disaggregation_basic/_pp/_nixl die with "No accelerator available" and/or GPU Hang → SIGABRT; test_mori passed Jul 11 (GPU recovered mid-job).
• Top hypothesis: [MEDIUM] intermittent runner GPU/NIC instability (green Jul 7/8, symptom varies day-to-day). Disconfirming: RDMA fabric was healthy this run.
• In-flight fix: ✅ #30431; related debug ✅ #29127 (isolate faulty rdma5 NIC).
• Suggested triage: re-run 3–5× to quantify flake rate; add a pre-flight torch.cuda.is_available() gate.

Cluster: DSV4-Flash GPU Hang (bring-up) — rows 15–16

• Affected: dsv4-flash-fp4-fp8 both runs; GPU Hang during weight load / decode graph capture; FP8/TBO files never reached.
• Top hypothesis: [LOW] DeepSeek-V4-Flash not yet stable on MI35x/gfx950 ROCm 7.2 (never-passed → no bisect window). Disconfirming: no stack frame pins a single kernel.
• In-flight fix: ⚠️ #27197 (DSV4 Flash on MI355X, draft/stale) and #30208 (load-tail diagnostics, draft).
• Suggested triage: relaunch FP4 server with --disable-cuda-graph / reduced --cuda-graph-max-bs to isolate the capture hang.

Cluster: Disagg _is_default array crash — rows 17–18

• Affected: mm-gen-2gpu (1) both runs; all 3 TestDisaggZImage* error in setUpClass from encoder-side ValueError: truth value of an array … is ambiguous at scheduler_mixin.py:226 → _is_default (:155 value == field_info.default).
• Top hypothesis: [HIGH/FACT] introduced by #30492 — _is_default crashes on numpy-array Req fields. Disconfirming: (none) — reproduced identically Jul 9–12.
• In-flight fix: ❌ none found — needs a new PR (guard _is_default for array/tensor values).
• Suggested triage: unit-test _is_default with an array value; guard with isinstance(value, np.ndarray) / np.array_equal.

Cluster: flux2 FP8 HIPBLAS + port cascade — rows 19–20

• Affected: mm-gen-2gpu (0) both runs. Genuine: [flux2_modelopt_fp8_tp2_t2i] → torch._scaled_mm HIPBLAS_STATUS_NOT_SUPPORTED (fp8_utils.py:1891). Cascade: a hung/leaked server (joy_echo Jul 11 / wan2_2 lora Jul 12) leaves scheduler port bound → 16–18 --strict-ports ERRORs.
• Top hypothesis: [HIGH] hipBLAS rejects FLUX.2 modelopt-FP8 e4m3fn GEMM on ROCm 7.2. Disconfirming: (none) — trace + fix PR agree.
• In-flight fix: ✅ #28889 (dequant→bf16 fallback) for flux2; ❌ none for the port-leak cascade.
• Suggested triage: land #28889; harden harness to kill the process group / free the scheduler port on startup timeout.

Cluster: Triton throughput <3500 gate — rows 21–22

• Affected: stage-b-1gpu-large (0) both runs; benchmark completes 500/500 but measures ~3150–3212 tok/s vs AMD gate 3500 at test_bench_serving_1gpu_part1.py:120.
• Top hypothesis: [MEDIUM] AMD 3500 gate set higher than MI325/ROCm7.2 Triton backend sustainably delivers (flat ~10% shortfall, no green in window). Disconfirming: no pre-Jul-6 green baseline queried, so a slow prior regression isn't excluded.
• In-flight fix: ❌ none found.
• Suggested triage: query Jul 5–8 runs for a last-green SHA; if none, re-calibrate the AMD gate.

Cluster: msgpack missing (ROCm image) — rows 23–24

• Affected: mm-gen-unit-test both runs; import msgpack at protocol.py:59 raises ModuleNotFoundError (1 failed / 766 passed).
• Top hypothesis: [HIGH] #30633 added the test + msgpack dep, but the prebuilt ROCm 7.2 image's pip install -e python[diffusion] never resolved/installed msgpack. Disconfirming: msgpack is correctly declared in the diffusion extra → gap is at the CI image/install layer.
• In-flight fix: ❌ none found.
• Suggested triage: docker exec ci_sglang pip show msgpack; add explicit install in amd_ci_install_dependency.sh or rebuild the image; interim pytest.importorskip("msgpack").

Cluster: GLM-Image invalid repo id — row 25

• Affected: mm-gen-2gpu (2), Jul 11 only. --model-path zai-org/GLM-Image/vision_language_encoder/ (3-segment) rejected by HF; retries until the 150-min job timeout.
• Top hypothesis: [HIGH] test concatenates repo id + subfolder (test_ar_models.py:60), from #25381. In-flight fix: ❌ none.
• Suggested triage: confirm whether GLM-Image weights should be pre-staged locally; if code fix, use HF subfolder= instead of string concat; consider disabling TestGLMImage on AMD until resolved.

Cluster: Remote TransferEngine zeroed params — row 26

• Affected: stage-b-2gpu-large (1), Jul 12 (passed Jul 7/10/11; failed Jul 8/12). transfer_engine backend reports success but leaves rank-1 dest params zeroed.
• Top hypothesis: [HIGH] AMD Mooncake transfer_engine flake (backend randomly selected). Disconfirming: (none) — failure predates the Jul 11→12 window and recurs across unrelated SHAs.
• In-flight fix: ✅ #30035 (pin AMD test to NCCL; draft since 2026-07-03).
• Suggested triage: chase #30035 out of draft; file a core investigation into the zeroed-transfer path.

Suggested triage order

Rank	Triage step	Targets	Why this first
1	Chase/land #30431 + reboot/drain MI35x nodes and re-run	C1 (rows 1–6), C2 (13–14)	8 jobs, largest blast radius; an in-flight infra fix already targets the zombie-KFD signature
2	Land #28889; harden 2-GPU harness port/teardown	C5 (rows 19–20)	In-flight fix exists for the genuine flux2 root; teardown fix removes the 18-case cascade
3	Instrument MI325 runner (dmesg/OOM, rocm-smi), pre-stage HF weights / re-partition suite	C8 (rows 7–12)	6 jobs; cheapest way to confirm resource-exhaustion vs hang once logs finalize
4	Open PR to guard _is_default against array-valued Req fields	C4 (rows 17–18)	Code-local [HIGH/FACT] root (#30492); deterministic, no bisect needed
5	Add msgpack to AMD diffusion install / image; query pre-Jul-6 history for the Triton gate	C7 (23–24), C6 (21–22)	Both are quick config/gate fixes with clear evidence

Rows 8, 10, 15–16, 25 remain data-limited (logs 404 or never-passed bring-up) — verify once run 29202664092 finalizes; C10 (row 26) is a known flake with in-flight #30035.

---

Job	Failed Steps	Started
stage-b-test-1-gpu-small-amd-mi35x-rocm720 (linux-mi35x-gpu-1)	Run test	2026-07-11T17:46
dsv4-flash-fp4-fp8-amd-rocm720	Accuracy Test MI35x ROCm 7.2 (8-GPU DeepSeek-V4-Flash FP4 + FP8)	2026-07-11T18:11
multimodal-gen-unit-test-amd-rocm720	Run diffusion unit tests	2026-07-11T21:51
stage-b-test-large-8-gpu-mi35x-disaggregation-amd-rocm720 (linux-mi35x-gpu-8.fabric)	Run test_disaggregation	2026-07-11T17:46
stage-c-test-large-8-gpu-amd-mi35x-rocm720 (linux-mi35x-gpu-8, 0)	Run test	2026-07-11T18:13
stage-b-test-1-gpu-large-amd-rocm720 (linux-mi325-1gpu-sglang, 0)	Run test	2026-07-11T19:59
multimodal-gen-test-2-gpu-amd-rocm720 (linux-mi325-2gpu-sglang, 1)	Run diffusion server tests (2-GPU)	2026-07-11T21:17
stage-c-test-large-8-gpu-amd-mi35x-rocm720 (linux-mi35x-gpu-8, 1)	Run test	2026-07-11T18:33
multimodal-gen-test-1-gpu-amd-rocm720 (linux-mi325-1gpu-sglang, 3)	N/A	2026-07-11T20:02
multimodal-gen-test-1-gpu-amd-rocm720 (linux-mi325-1gpu-sglang, 1)	N/A	2026-07-11T22:49
multimodal-gen-test-1-gpu-amd-rocm720 (linux-mi325-1gpu-sglang, 0)	N/A	2026-07-11T20:03
multimodal-gen-test-2-gpu-amd-rocm720 (linux-mi325-2gpu-sglang, 0)	Run diffusion server tests (2-GPU)	2026-07-11T22:58
multimodal-gen-test-2-gpu-amd-rocm720 (linux-mi325-2gpu-sglang, 2)	Run diffusion server tests (2-GPU)	2026-07-11T22:47
stage-b-test-large-8-gpu-mi35x-disaggregation-amd-rocm720 (linux-mi35x-gpu-8.fabric)	Run test_disaggregation	2026-07-12T17:48
stage-b-test-1-gpu-small-amd-mi35x-rocm720 (linux-mi35x-gpu-1)	Run test	2026-07-12T17:48
dsv4-flash-fp4-fp8-amd-rocm720	Accuracy Test MI35x ROCm 7.2 (8-GPU DeepSeek-V4-Flash FP4 + FP8)	2026-07-12T18:14
stage-c-test-large-8-gpu-amd-mi35x-rocm720 (linux-mi35x-gpu-8, 1)	Run test	2026-07-12T18:16
stage-c-test-large-8-gpu-amd-mi35x-rocm720 (linux-mi35x-gpu-8, 0)	Run test	2026-07-12T18:18
multimodal-gen-test-2-gpu-amd-rocm720 (linux-mi325-2gpu-sglang, 1)	Run diffusion server tests (2-GPU)	2026-07-12T20:19
stage-b-test-1-gpu-large-amd-rocm720 (linux-mi325-1gpu-sglang, 0)	Run test	2026-07-12T21:36
multimodal-gen-unit-test-amd-rocm720	Run diffusion unit tests	2026-07-12T21:43
multimodal-gen-test-2-gpu-amd-rocm720 (linux-mi325-2gpu-sglang, 0)	Run diffusion server tests (2-GPU)	2026-07-12T20:02
multimodal-gen-test-1-gpu-amd-rocm720 (linux-mi325-1gpu-sglang, 3)	N/A	2026-07-12T21:42
stage-b-test-2-gpu-large-amd-rocm720 (linux-mi325-2gpu-sglang, 1)	Run test	2026-07-12T22:16
multimodal-gen-test-1-gpu-amd-rocm720 (linux-mi325-1gpu-sglang, 1)	N/A	2026-07-12T21:41
multimodal-gen-test-1-gpu-amd-rocm720 (linux-mi325-1gpu-sglang, 0)	N/A	2026-07-12T21:41

Per-Job Analysis

stage-b-test-1-gpu-small-amd-mi35x-rocm720 (linux-mi35x-gpu-1) — failed step(s): Run test

Commit Info

• sglang: 9b4bb415dd
• aiter: 9127c94a1 — v0.1.17.dev110+g9127c94a1 (source: dev/patched image preserved; [CI-AITER-CHECK] reported "Dev/patched version detected → skipping rebuild")

Failed Tests

Test File	Test Function	Error	Log
N/A — suite crashed before any test output	N/A	HW Exception by GPU node-2 … reason :GPU Hang → exit code 134 (SIGABRT)	Log

The suite run_suite.py --suite stage-b-test-1-gpu-small-amd-mi35x was launched at 17:52:34 and the GPU hung ~17s later at 17:52:51, before any per-test output was emitted. A retry with HF_HUB_OFFLINE=1 hung the GPU again at 17:53:19. No test file/function can be pinned — the crash is at the GPU/driver level during the first test's model load/warmup. The suite contains test_mamba_state_scatter_triton.py, test_quark_mxfp4.py, test_dsa_skip_logits_clean.py, test_verify_splitkv.py, but none produced output.

Failure Summary

The stage-b-test-1-gpu-small-amd-mi35x-rocm720 job crashed with a hardware-level GPU Hang (HW Exception, exit code 134) immediately after the test suite launched, before any test ran. Both the initial attempt and the offline retry hung the same GPU agent (node-2). This is a GPU/driver-level failure on the MI35x (gfx950) ROCm 7.2 runner, not a test assertion failure.

Regression Status

Infrastructure issue (recurring, differing symptoms). This job was green through 2026-07-08 and has failed the last two completed scheduled runs — but with two different infra symptoms, not a single code regression.

Recent history of job stage-b-test-1-gpu-small-amd-mi35x-rocm720 (linux-mi35x-gpu-1) (branch main, event schedule, completed runs only):

Date	Run	Job Status	Symptom
Jul 11	29162225443	❌ Failed	GPU Hang (HW Exception, exit 134) — this run
Jul 10	29113205616	⏳ [IN-FLIGHT]	queued/incomplete — excluded
Jul 09	29039527644	❌ Failed	VRAM cleanup unsuccessful after 3 attempts (zombie KFD context, exit 1) — different symptom
Jul 08	28964460204	✅ Passed	—
Jul 07	28888292440	✅ Passed	—
Jul 06	28813230096	✅ Passed	—
Jul 05	28749537427	✅ Passed	—

First observed failure (this job): 2026-07-09. Last known passing: 2026-07-08. The GPU-Hang symptom specifically is first seen 2026-07-11.

Failure Cluster

GPU Hang / driver-level GPU fault on mi35x-rocm720 runner during suite launch (infra).

Facts (verified observations, no interpretation)

• HW Exception by GPU node-2 (Agent handle: 0x5a2c79a2e420) reason :GPU Hang at 17:52:51 (job.log:4326), then First attempt failed with exit code 134 (job.log:4327).
• Retry with HF_HUB_OFFLINE=1 also hung: HW Exception by GPU node-2 (Agent handle: 0x5ef5fb7a5ee0) reason :GPU Hang at 17:53:19 (job.log:4329), Process completed with exit code 134 (job.log:4330).
• Elapsed from suite launch (17:52:34, job.log:4317) to first hang: ~17s. Zero pytest/test output captured in between.
• Runner: linux-mi35x-gpu-1-qxll6-runner-h5lwg (job.log:2); GPU arch mi35x (gfx950), ROCm 7.2 image.
• aiter unchanged: dev image v0.1.17.dev110+g9127c94a1 preserved (job.log:4310-4311); sgl-kernel built cleanly (only -Wpass-failed loop-unroll warnings).
• Prior completed failure (Jul 09) was a different step: === FAILED: VRAM cleanup unsuccessful after 3 attempts === with the explicit hint "zombie KFD context (see [Bug] Custom all-reduce IPC buffers use fixed VA, conflict with torch_memory_saver pause/resume → zombie processes on MI300X ROCm/aiter#2061). The node will need to be rebooted before subsequent jobs can succeed." (prev_0709.log:324-332).

Hypothesised Causes (with confidence, with disconfirming evidence)

• [MEDIUM] Flaky/degraded MI35x GPU node — driver-level GPU hang independent of sglang code.
  • Supporting: crash is a GPU Hang HW exception with no test code in the stack; it recurs on retry on the same agent; the immediately prior failure on this same job was also an infra-level GPU state problem (zombie KFD VRAM leak requiring node reboot); job was green 4 consecutive runs through Jul 08; sgl-kernel and aiter both built/loaded fine.
  • Disconfirming: cannot 100% exclude that the first test's model warmup triggers a kernel that hangs the GPU (no test output means the exact trigger is unproven); a genuine code regression in a warmup-time kernel would present similarly.
  • How to verify: reboot/drain the mi35x node and re-run the suite; if it passes on a healthy node, infra confirmed. Cross-check whether other jobs on linux-mi35x-gpu-1 in the same window also hung.
• [LOW] Zombie KFD context from the Jul 09 VRAM-leak failure left the node in a bad state, causing the later hang.
  • Supporting: the Jul 09 hint explicitly says the node must be rebooted before subsequent jobs succeed; the Jul 11 hang is consistent with running on an un-rebooted, corrupted node.
  • Disconfirming: the Jul 11 run showed no "VRAM cleanup unsuccessful" pre-test failure (cleanup appears to have passed), so the node was not obviously dirty at start; runner pods are ephemeral (-qxll6-runner-h5lwg) though the physical GPU may be shared.
  • How to verify: check whether the Jul 11 job scheduled onto the same physical GPU as the Jul 09 failure and whether a reboot occurred between them.
• [LOW] sglang code regression in a warmup-time kernel merged in the Jul 08→Jul 11 window.
  • Supporting: green run existed on Jul 08; two failures since.
  • Disconfirming: the two failures have different infra-level symptoms (VRAM-leak vs GPU-hang), which is the signature of node flakiness, not a single deterministic code bug; no test-level stack trace implicates any sglang path.
  • How to verify: bisect 8d2b66fd90..9b4bb415dd only after infra is ruled out.

In-flight Fix Check

• In-flight fix: ✅ #30431 "[AMD] ci: self-heal driver-level VRAM leaks + graceful GPU cleanup on shutdown" — directly targets driver-level VRAM leaks and GPU cleanup on this AMD CI, matching the Jul 09 VRAM-leak symptom and plausibly the Jul 11 hang if node state is the cause. Chase reviewers; do NOT open a duplicate. (Search: is:pr is:open mi35x GPU Hang and mi35x rocm720.)
• No open PR found specifically for the GPU Hang exit-134 symptom.

Suggested Triage Steps

• Reboot/drain the affected mi35x node and re-run the stage-b-test-1-gpu-small-amd-mi35x-rocm720 job — if it goes green on a healthy node, the failure is infra, not code.
• Confirm whether the Jul 11 job ran on an un-rebooted GPU left dirty by the Jul 09 zombie-KFD VRAM leak; if so, land #30431 to self-heal these leaks and re-run.
• Check sibling mi35x jobs in runs 29162225443 and 29039527644 for the same GPU Hang signature to gauge node vs suite scope.
• Only if a healthy-node re-run still hangs: bisect 8d2b66fd90..9b4bb415dd and run each suite test individually to find which one triggers the hang.

Status

• Persistence: Job failing the last 2 completed scheduled runs (since 2026-07-09); green through 2026-07-08. The specific GPU Hang symptom is new on 2026-07-11; the Jul 09 failure was a distinct VRAM-leak/zombie-KFD infra symptom.
• Scope: stage-b-test-1-gpu-small-amd-mi35x-rocm720 (linux-mi35x-gpu-1) in pr-test-amd-rocm720.yml. No test-level failure — GPU/driver-level crash before any test output.
• Blocked work: Blocks the entire mi35x-rocm720 1-GPU small suite signal (mamba scatter, quark mxfp4, DSA skip-logits, verify-splitkv) — none of these tests were exercised.
• In-flight fix: ✅ #30431 (driver-level VRAM leak self-heal + GPU cleanup) — candidate remediation for the node-state hypothesis; awaiting merge.

dsv4-flash-fp4-fp8-amd-rocm720 — failed step(s): Accuracy Test MI35x ROCm 7.2 (8-GPU DeepSeek-V4-Flash FP4 + FP8)

Commit Info

• sglang: 9b4bb415d
• aiter: 9127c94a1 (v0.1.17.dev110+g9127c94a1) — source: Dockerfile default, dev image preserved ([CI-AITER-CHECK] Dev/patched version detected → skipping rebuild; no AITER_COMMIT_OVERRIDE)
• Runner: linux-mi35x-gpu-8-zf775-runner-8xhfg (mi35x / gfx950), 8-GPU

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/test_deepseek_v4_flash_fp4.py	server bring-up (setUp/model load — no pytest frame reached)	HW Exception ... reason: GPU Hang → Fatal Python error: Aborted → exit code 134 (SIGABRT)	Log

The two sibling tests in the same job (test_deepseek_v4_flash_fp8.py, test_deepseek_v4_flash_fp8_tbo.py) never executed today — the job aborted hard on the first test (fp4) on both attempts.

Failure Summary

Job dsv4-flash-fp4-fp8-amd-rocm720 (suite nightly-amd-8-gpu-mi35x-deepseek-v4-flash, nightly=True) runs 3 DeepSeek-V4-Flash tests. Today the first test test_deepseek_v4_flash_fp4.py crashed during model weight loading — immediately after the Execute dequant fp8 wo_a messages — with a hardware GPU Hang exception on all 8 GPU agents, aborting the process with SIGABRT (exit 134). The retry (HF_HUB_OFFLINE=1) hit the same GPU Hang during a second load attempt (node-5) and also exited 134.

Regression Status

Never-passed (no green run found) for test_deepseek_v4_flash_fp4.py in the queryable window. Lookback: pr-test-amd-rocm720.yml, event=schedule, branch=main, Jul 3–Jul 11 2026. The job has failed in every completed run, but the symptom varies run-to-run, so no single-commit regression is implicated.

Recent history of test_deepseek_v4_flash_fp4.py in job dsv4-flash-fp4-fp8-amd-rocm720 (same branch, same event, completed runs only):

Date	Run	Test File Status	Failure stage / symptom
Jul 11	29162225443	❌ Failed	Weight load → GPU Hang / SIGABRT (134), both attempts
Jul 9	29039527644	⚠️ Not reached	Job died at ensure_vram_clear.sh (VRAM not clearing, exit 1)
Jul 8	28964460204	⚠️ Not reached	Job died at ensure_vram_clear.sh (VRAM not clearing, exit 1)
Jul 7	28888292440	❌ Failed	Server init → Server process exited with code -9 (SIGKILL/OOM), 0/3 passed
Jul 6	28813230096	⚠️ Not reached	Job died at ensure_vram_clear.sh (VRAM not clearing, exit 1)

First observed failure for this test file: earliest in window (Jul 3). No passing run in the window. The Jul 6/8/9 runs never reached the test (aborted at the pre-test VRAM-clear guard), which is itself consistent with a GPU left in a bad state by a prior hang.

Failure Cluster

GPU Hang / server crash during DeepSeek-V4-Flash model load on mi35x (gfx950)

Facts (verified observations)

• [CI-AITER-CHECK] Runner GPU_ARCH=mi35x; Dockerfile expects AITER_COMMIT=9127c94a...; AITER version inside CI image: v0.1.17.dev110+g9127c94a1; Dev/patched version detected → skipping rebuild. No AITER_COMMIT_OVERRIDE (this is a normal scheduled pr-test-amd-rocm720 run, not an AITER Scout run — no Baseline A/B check applies).
• Verbatim crash sequence (attempt 1): HW Exception by GPU node-3 (Agent handle: ...) reason :GPU Hang (repeated across all agents) → Fatal Python error: Aborted → First attempt failed with exit code 134.
• Crash occurs during weight loading. Stack frames in the log: model_loader/weight_utils.py … default_weight_loader, layers/moe/fused_moe_triton/layer.py:519 _load_w13, :428 _load_model_weight_or_group_weight_scale, :1059 _weight_loader_impl. Last progress line before the hang: [TPx] Execute dequant fp8 wo_a (fp8 weight dequant at load time).
• Retry with HF_HUB_OFFLINE=1 began test_deepseek_v4_flash_fp4.py again, hit HW Exception by GPU node-5 … GPU Hang, ##[error]Process completed with exit code 134.
• Model config (server_args): model_path=deepseek-ai/DeepSeek-V4-Flash, tp_size=8, attention_backend='dsv4', kv_cache_dtype='fp8_e4m3', mem_fraction_static=0.9, page_size=256.
• Historical symptom drift (same test file, same job): SIGKILL/-9 OOM at server init (Jul 7) vs. VRAM-clear guard exit 1 (Jul 6/8/9) vs. GPU Hang/SIGABRT 134 (Jul 11). Different exit stages, same test never passing.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [LOW] DeepSeek-V4-Flash bring-up is not yet stable on mi35x (gfx950); the GPU hang/OOM cluster is an environment/model-bringup problem rather than a specific sglang commit regression.
  • Supporting: test file has never passed in the queryable window; failures span three distinct stages (VRAM guard, OOM at init, GPU hang at weight load) all on the same mi35x 8-GPU runner type; two open PRs explicitly target "DeepSeek V4 Flash on MI355X" and a "persistent DeepSeek-V4 weight-load tail" (see In-flight Fix).
  • Disconfirming: today's symptom (hard GPU Hang / SIGABRT during fp8 dequant at load) differs from Jul 7's SIGKILL/-9 (OOM), so a single root cause across all runs is not established; a hardware/runner-specific hang cannot be ruled out from the log alone.
  • How to verify: re-run the job on a freshly-reset mi35x node; check dmesg -T | grep -i oom and GPU dmesg/rocm-smi for the hang; A/B the fp8 dequant load path with SGLANG_SET_CPU_AFFINITY=1 (per PR #30208 diagnostics).
• [SPECULATION] fp8 weight-dequant path (_load_w13 in fused_moe_triton/layer.py) triggers the hang on gfx950.
  • Supporting: last log line before the hang is Execute dequant fp8 wo_a; the crash stack is inside _load_w13/default_weight_loader.
  • Disconfirming: log-line proximity is not causation — the hang is reported by the HW agent, and the printed frame may simply be where the CPU thread was when the async GPU op faulted. No code-level evidence ties a specific commit to this.
  • How to verify: enable kernel API logging / GPU coredump on the load path; bisect only if a green baseline can first be established.

No hypothesis reaches MEDIUM confidence: the test has no green baseline in the queryable window, so a regression window cannot be bounded and no commit can be implicated.

In-flight Fix Check

• In-flight fix: ✅ #30208 (draft, open since 2026-07-06) — "Experiment: DSV4 Flash load-only affinity diagnostics"; explicitly targets the persistent DeepSeek-V4-Flash weight-load tail on the ROCm720 Flash job and adds a load-only mode + CPU-affinity A/B. Directly relevant to today's weight-load hang — track it, do NOT open a duplicate.
• In-flight fix: ✅ #27197 (draft, open since 2026-06-04) — "Fix DeepSeek V4 Flash on MI355X"; addresses DeepSeek-V4-Flash runtime/correctness problems on MI355X (gfx950). Related; confirm whether it covers the load-time GPU-hang path.
• Search: is:pr is:open "DeepSeek-V4-Flash" (116 hits), filtered to the two above by title relevance.

Suggested Triage Steps

• Confirm the runner GPU state: pull dmesg/rocm-smi from linux-mi35x-gpu-8-zf775-runner-8xhfg around 18:22 UTC for the hang, and dmesg -T | grep -i oom for the Jul 7 SIGKILL — determine whether the hang is a persistent node-health issue vs. a model-load bug.
• Chase #30208 reviewers; its load-only + SGLANG_SET_CPU_AFFINITY A/B is the cheapest existing lever to isolate the weight-load tail on this exact job.
• Because there is no green baseline, do NOT bisect against commit 9b4bb415d — first establish any passing configuration (e.g. load-only mode from #30208), then bisect from there.
• If the node-health check shows a hardware GPU hang independent of sglang, quarantine/replace the runner and re-run before attributing to code.

Status

• Persistence: test_deepseek_v4_flash_fp4.py has never passed in the queryable window (Jul 3–Jul 11 2026); job dsv4-flash-fp4-fp8-amd-rocm720 failed every completed scheduled run in that window.
• Scope: single job on pr-test-amd-rocm720.yml (mi35x 8-GPU); blocks the DeepSeek-V4-Flash nightly accuracy signal on ROCm 7.2. Symptom is unstable across runs (GPU Hang / OOM / VRAM-clear guard).
• In-flight fix: ✅ #30208 (weight-load tail diagnostics) and #27197 (DSV4 Flash on MI355X) — both open drafts; no new PR needed yet.

multimodal-gen-unit-test-amd-rocm720 — failed step(s): Run diffusion unit tests

Commit Info

• sglang: 9b4bb415d
• aiter: 9127c94a1 — v0.1.17.dev110+g9127c94a1 (source: dev image; [CI-AITER-CHECK] reports "Dev/patched version detected → skipping rebuild", AITER_COMMIT_OVERRIDE empty)

Failed Tests

Test File	Test Function	Error	Log
python/sglang/multimodal_gen/test/unit/test_pi05_action_api.py	test_msgpack_roundtrip_preserves_string_keys_and_numpy_payloads	ModuleNotFoundError: No module named 'msgpack'	Log (step 7)

Result: 1 failed, 766 passed, 1 skipped, 5 deselected in 73.72s.

Failure Summary

The new unit test test_msgpack_roundtrip_preserves_string_keys_and_numpy_payloads fails at collection-of-runtime because pack_msgpack() (python/sglang/multimodal_gen/runtime/entrypoints/vla/protocol.py:59) does import msgpack, and msgpack is not installed in the ROCm 7.2 CI container. All other 766 diffusion unit tests pass; this is the sole failure and it is a missing-dependency error, not a logic/numerics regression.

Regression Status

New test — never passed on AMD ROCm 7.2. Both the test and the import msgpack code path were introduced by #30633 (649ce5dd3, merged 2026-07-11 07:50 +0800), which is a verified ancestor of the run commit. The test did not exist in any earlier scheduled run, so there is no prior green baseline for this test file.

Recent scheduled runs of pr-test-amd-rocm720.yml on main (event=schedule):

Date	Run	Test File Status	Note
Jul 11	29162225443	❌ Failed	first run containing this test (post-#30633)
Jul 10	29113205616	—	test file did not exist yet (head 7998fecfd1)
Jul 09	29039527644	—	test file did not exist yet
Jul 08	28964460204	—	test file did not exist yet

Failure Cluster

ModuleNotFoundError: No module named 'msgpack' (missing diffusion dependency in ROCm CI)

Facts (verified observations)

• test_pi05_action_api.py:237 calls unpack_msgpack(pack_msgpack(payload)); the crash is raised inside protocol.py:59 import msgpack (verified from traceback at log lines 5823–5825).
• msgpack is declared as a diffusion optional dependency at python/pyproject.toml:107, added by #30633.
• The install step ran pip install -e python[dev_hip,tracing,diffusion]. In its output every other diffusion extra (addict, av, cache-dit, cloudpickle, imageio, imageio-ffmpeg, moviepy, opencv-python-headless, remote-pdb, scikit-image, trimesh, xatlas, runai_model_streamer) is Requirement already satisfied from the prebuilt image — but msgpack appears nowhere (neither "Collecting msgpack" nor "already satisfied"). Verified: zero msgpack pip lines in the entire log (only unrelated 3rdparty/msgpack-c submodule and test lines).
• CI image: rocm/sgl-dev:v0.5.15-rocm720-mi30x-20260710 (built 2026-07-10, i.e. before #30633 merged 2026-07-11).
• This is pr-test-amd-rocm720.yml (not amd-aiter-scout.yml); aiter override is empty, so no aiter baseline A/B check applies.

Hypothesised Causes (with confidence)

• [FACT] The test requires msgpack, which is not importable in the ROCm 7.2 CI environment. Directly observable in the traceback.
• [MEDIUM] The pip install -e python[diffusion] step resolved against dependency metadata that predates #30633, so the newly-added msgpack entry was never seen and never installed.
  • Supporting: all other diffusion extras resolve as "already satisfied" from the 2026-07-10 image while msgpack is the only diffusion dep completely absent from pip's resolution; msgpack was the single diffusion dependency added by #30633.
  • Disconfirming: the editable build reports version sglang==0.0.0.dev1+g9b4bb415d.d20260711, implying setuptools-scm read the fresh checkout — which does contain msgpack at pyproject:107; under that reading pip should have emitted "Collecting msgpack". The absence is unexplained by a straightforward checkout-based resolution, so the exact resolution path is not fully pinned down.
  • How to verify: in the container run docker exec ci_sglang python -c "import msgpack" and pip show msgpack; inspect which pyproject.toml the -e python path points to (docker exec ci_sglang sed -n '95,120p' python/pyproject.toml).
• [LOW] Image staleness alone (msgpack simply not baked into the 07-10 image). Weakened by the fact that pip install -e python[diffusion] should top up any missing extra at install time, yet did not.

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs for msgpack (11 results, all unrelated IPC/serialization refactors, e.g. #30005, #29656) and pi05 (1 result, #24455, the original feature PR). None addresses installing msgpack in AMD ROCm CI.

Suggested Triage Steps

• Confirm the gap: docker exec ci_sglang python -c "import msgpack" and pip show msgpack inside the ROCm 7.2 image — expect ImportError / not-found.
• Determine why pip install -e python[diffusion] skipped msgpack: verify the python/pyproject.toml seen by the install actually contains line 107 ("msgpack",), and whether a stale image-bundled source tree or resolver cache shadows the fresh checkout.
• Short-term unblock options for maintainers to weigh: add an explicit msgpack install in scripts/ci/amd/amd_ci_install_dependency.sh (mirrors the existing cache-dit/accelerate top-up lines), rebuild the ROCm dev image to include msgpack, or guard the test with pytest.importorskip("msgpack") if msgpack is meant to be optional at runtime.

Status

• Persistence: First and only occurrence — first scheduled ROCm 7.2 run after #30633 landed (2026-07-11 21:51 UTC).
• Scope: 1 test in job multimodal-gen-unit-test-amd-rocm720; likely to recur every run until msgpack is present in the AMD CI environment. May also affect other AMD/ROCm diffusion jobs that exercise the Pi0.5 VLA protocol.py path.
• Blocked work: Pi0.5 VLA msgpack protocol coverage on AMD ROCm.
• In-flight fix: ❌ none found — needs a new PR (CI dependency install or image rebuild) or a runtime importorskip guard.

---

Generated by amd-bot using Claude Code CLI (last updated: 2026-07-12 23:21 UTC)

[amd-bot]

pr-test-amd-rocm720.yml — Per-Job Analysis (continued 2/6)

stage-b-test-large-8-gpu-mi35x-disaggregation-amd-rocm720 (linux-mi35x-gpu-8.fabric) — failed step(s): Run test_disaggregation

Commit Info

• sglang: 9b4bb415dd
• aiter: 9127c94a18 (v0.1.17.dev110+g9127c94a1, source: dev/patched image build preserved — AITER_COMMIT_OVERRIDE empty, not a scout run)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/disaggregation/test_disaggregation_basic.py	TestDisaggregationAccuracy.setUpClass	RuntimeError: Server process exited with code 1 — launched server subprocess died at startup with RuntimeError: No accelerator (CUDA, XPU, HPU, NPU, MUSA, MPS) or platform plugin is available	Log
test/registered/amd/disaggregation/test_disaggregation_pp.py	TestDisaggregationDecodePPAccuracy.setUpClass	RuntimeError: Server process exited with code 1 — same "No accelerator … available" in server subprocess	Log
test/registered/amd/disaggregation/test_nixl_transfer_engine_e2e.py	setUpClass (server crash during weight load)	HW Exception by GPU node-3 … reason :GPU Hang → Fatal Python error: Aborted (exit code -6, SIGABRT)	Log

test_mori_transfer_engine_e2e.py passed in the same job (Test Summary: 1/4 passed).

Failure Summary

Three of four disaggregation test files failed on the 8-GPU MI35x runner. test_disaggregation_basic and test_disaggregation_pp both died because their launched sglang.launch_server subprocesses hit RuntimeError: No accelerator … available — i.e. torch.cuda.is_available() returned False inside the child process, so get_device() fell through every branch to current_platform.get_device() which raised NotImplementedError (common.py:898). test_nixl_transfer_engine_e2e instead hit a hardware GPU Hang exception on GPU node-3 and aborted (SIGABRT). test_mori passed on the same GPUs, code, and image.

Regression Status

Known recurring / intermittent infrastructure instability on this runner. This job passed on Jul 7 and Jul 8 with the same test files, failed Jul 9 with a different symptom (all 4 killed with exit -9/SIGKILL), and fails today with "No accelerator" + "GPU Hang". The varying symptom day-to-day, plus test_mori passing in the very same job today, points to runner/GPU-runtime instability rather than a code regression in these test files.

Recent history of the disaggregation test files in job stage-b-test-large-8-gpu-mi35x-disaggregation-amd-rocm720 (branch main, event schedule, completed runs only):

Date	Run	Job	basic	pp	nixl	mori	Symptom
Jul 11	29162225443	job	❌	❌	❌	✅	No accelerator (basic/pp) + GPU Hang (nixl)
Jul 10	29113205616	—	—	—	—	—	[IN-FLIGHT] (run still queued — excluded)
Jul 9	29039527644	job	❌	❌	❌	❌	all 4 Server process exited with code -9 (SIGKILL) — different cluster
Jul 8	28964460204	job	✅	✅	✅	✅	clean pass
Jul 7	28888292440	job	✅	✅	✅	✅	clean pass

First observed failure for these test files in the queryable window: Jul 9 (different symptom). Last known passing: Jul 8. The specific "No accelerator + GPU Hang" symptom is new today.

Failure Cluster

MI35x disaggregation runner GPU instability — two related sub-symptoms: (a) GPU not visible to launched server subprocess (torch.cuda.is_available()==False → "No accelerator … available") for basic/pp; (b) hardware "GPU Hang" HW exception → SIGABRT for nixl.

Facts (verified observations)

• basic/pp server subprocesses: NotImplementedError from device_mixin.py:172 inside get_device → wrapped as RuntimeError: No accelerator (CUDA, XPU, HPU, NPU, MUSA, MPS) or platform plugin is available at server_args.py:3208 (_handle_missing_default_values → get_device()), during prepare_server_args, before any model/GPU work. Both failed in ~13 s.
• On ROCm, the branch that normally succeeds is torch.cuda.is_available() (HIP maps to torch.cuda); its returning False is what forces the fall-through — a runtime GPU-visibility failure, not a code-path bug.
• nixl: HW Exception by GPU node-3 … reason :GPU Hang printed 7× immediately before Fatal Python error: Aborted; thread dump shows the process was in hf_hub_download weight loading when the GPU-hang abort landed (exit -6).
• test_mori_transfer_engine_e2e ran full model load + CUDA graph capture + passed on the same 8 GPUs, same image, same commit — GPUs were functional for at least one test in this job.
• RDMA fabric healthy this run: all 8 rdma0..rdma7 devices ACTIVE @ 400 Gbps, INCLUDED.
• Jul 9 failure of the same job was a distinct symptom (exit -9, SIGKILL on all 4) — reinforces intermittent runner-level cause.

Hypothesised Causes (with confidence)

• [MEDIUM] MI35x runner GPU-runtime instability (driver-level state left dirty by a prior job / GPU cleanup race), making GPUs invisible to the first server subprocesses (torch.cuda.is_available()==False) and later producing a hardware GPU Hang.
  • Supporting: "No accelerator" occurs only at the start of the run (first two tests) while mori passes moments later; nixl fails with a HW "GPU Hang" (a hardware/driver signal, not a Python exception); symptom differs day-to-day (exit -9 Jul 9 vs No-accelerator/GPU-Hang Jul 11); the exact same test files pass on Jul 7/8 with comparable code. An open in-flight PR (#30431) explicitly targets "driver-level VRAM leaks + graceful GPU cleanup on shutdown" for AMD CI, matching this class of symptom.
  • Disconfirming: no direct proof of a leaked/held GPU from a prior job in this log (no rocm-smi/free-mem dump captured before the "No accelerator" error); cannot fully exclude a startup race in test orchestration that launches servers before the GPU is ready.
  • How to verify: add a pre-test rocm-smi / torch.cuda.is_available() gate in the job; re-run the disaggregation suite on linux-mi35x-gpu-8.fabric a few times to measure flake rate; check whether #30431's cleanup eliminates the "No accelerator" startup failures.
• [LOW] An sglang commit in …9b4bb415dd changed platform/device detection.
  • Supporting: 9b4bb415dd touches CUDA-graph shared-buffer sizing.
  • Disconfirming: the failing path (torch.cuda.is_available()==False) is environment-driven, not code-driven; mori passed with the identical code; the same test files passed Jul 7/8; Jul 9's failure on the same job had a completely different symptom on a different commit. No code-level evidence links the HEAD delta to device detection.
  • How to verify: only pursue if the failure reproduces deterministically across reruns (it does not appear to).

No hypothesis reaches HIGH; this is most consistent with runner/GPU instability, not a code regression.

In-flight Fix Check

• In-flight fix: ✅ #30431 "[AMD] ci: self-heal driver-level VRAM leaks + graceful GPU cleanup on shutdown" (open since 2026-07-07, non-draft) — directly targets the driver-level GPU-cleanup class of failure seen here; chase reviewers / verify it resolves the "No accelerator" startup failures rather than opening a duplicate.
• Related debug PR: #29127 "[DEBUG] Isolate faulty RDMA NIC (rdma5) on MI35x disaggregation runner" (open since 2026-06-24, draft) — same runner, ongoing instability investigation (RDMA was healthy this particular run, so likely not today's trigger).
• Searches run: is:pr is:open disaggregation mi35x, is:pr is:open "No accelerator available", is:pr is:open mi35x GPU Hang.

Suggested Triage Steps

• Re-run the disaggregation suite on linux-mi35x-gpu-8.fabric 3–5× to quantify flake rate and confirm intermittency (expected: some runs green, matching Jul 7/8).
• Add a pre-flight GPU health gate to the job (rocm-smi + python -c "import torch; assert torch.cuda.is_available()") so a dirty/invisible-GPU runner fails fast with a clear signal instead of a confusing "No accelerator" deep in server startup.
• Verify #30431 merges cleanly and re-run this job with it applied; check whether the "No accelerator" startup failures and GPU Hang disappear.
• If failures persist independent of code, treat as runner hardware/driver maintenance (drain/reset linux-mi35x-gpu-8.fabric) rather than an sglang code fix.

Status

• Persistence: Intermittent on this job — green Jul 7 & Jul 8, red Jul 9 (SIGKILL) and Jul 11 (No accelerator + GPU Hang); Jul 10 run still in-flight/queued.
• Scope: 3 of 4 test files in job stage-b-test-large-8-gpu-mi35x-disaggregation-amd-rocm720 (linux-mi35x-gpu-8.fabric); test_mori passed, isolating the failure to runner GPU state rather than the whole GPU set.
• Blocked work: PD-disaggregation (basic + PP + NIXL) CI signal on the ROCm 7.2 MI35x runner.
• In-flight fix: ✅ #30431 (driver-level VRAM leak / GPU cleanup) — chase reviewers, do not duplicate.

stage-c-test-large-8-gpu-amd-mi35x-rocm720 (linux-mi35x-gpu-8, 0) — failed step(s): Run test

Commit Info

• sglang: 9b4bb415d
• aiter: 9127c94a1 — v0.1.17.dev110+g9127c94a1 (source: dev image preserved; matches Dockerfile default, no override)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/test_kimi_k25_mxfp4_bcg_mi35x.py	TestKimiK25MXFP4BcgMI35x.test_bcg_gsm8k	HW Exception by GPU node reason: GPU Hang → SIGABRT (exit 134)	Log

The second registered test in this shard (test/registered/cuda_graph/breakable/test_breakable_cuda_graph.py) never ran — the process aborted on the first test.

Failure Summary

The job ran amd/Kimi-K2.5-MXFP4 (tp_size=4) for the test_bcg_gsm8k accuracy test. On the first attempt the process aborted with HW Exception by GPU node-3 ... reason: GPU Hang while streaming language weights during model load (kimi_k25.py → deepseek_v2.py:2888 load_weights), exiting with code 134. The harness retried with HF_HUB_OFFLINE=1; the second attempt aborted again with HW Exception by GPU node-5 ... reason: GPU Hang even earlier during model init, and the step completed with exit code 134.

Regression Status

Infrastructure issue (GPU/runner instability) — leading hypothesis. New crash-during-execution mode for this test file, on a runner that has been unable to clear VRAM for 3 consecutive prior days.

Recent history of test_kimi_k25_mxfp4_bcg_mi35x.py (test_bcg_gsm8k) in job stage-c-test-large-8-gpu-amd-mi35x-rocm720 (linux-mi35x-gpu-8, 0) (branch=main, event=schedule, completed runs only):

Date	Run	Job	Test File Status	Failed Function	Error
Jul 11	29162225443	job	❌ Failed (test ran)	test_bcg_gsm8k	GPU Hang during model load, exit 134
Jul 09	29039527644	job	⚪ Not run	—	Job died at "Ensure VRAM is clear" (exit 1) — test never reached
Jul 08	28964460204	job	⚪ Not run	—	Same VRAM cleanup failure (exit 1)
Jul 07	28888292440	job	⚪ Not run	—	Same VRAM cleanup failure (exit 1)
Jul 06	28813230096	job	✅ Passed	—	accuracy 0.927, ran 2572s
Jul 05	28749537427	job	✅ Passed	—	accuracy 0.930, ran 1951s

Last known passing (test executed): Jul 06. Jul 07–09 the test did not execute (runner-level VRAM cleanup failure). Jul 10 run (29113205616) is still queued — excluded [IN-FLIGHT].

Failure Cluster

GPU Hang (HW Exception) during Kimi-K2.5-MXFP4 model init on MI35x 8-GPU runner

Facts (verified observations, no interpretation)

• First attempt: HW Exception by GPU node-3 ... reason :GPU Hang at 18:23:43, log:4526. Thread dump shows the active frame in srt/models/kimi_k25.py / srt/models/deepseek_v2.py:2888 load_weights / srt/model_loader/weight_utils.py download_weights_from_hf — i.e. during weight loading, not graph capture or forward.
• First attempt failed with exit code 134 → retried HF_HUB_OFFLINE=1 (log:4725).
• Second attempt: HW Exception by GPU node-5 ... reason :GPU Hang at 18:24:29 (log:4767), aborting during model init right after aiter RoPE setup; Process completed with exit code 134 (log:4770).
• Two distinct GPU nodes hung across the two attempts (node-3, then node-5).
• watchdog_timeout=300 — the ~23 s crash is a hardware GPU Hang exception, not a watchdog timeout.
• The job's own "Ensure VRAM is clear" step succeeded this run (step 3 = success); the same job failed that exact step on Jul 07/08/09 ("VRAM cleanup unsuccessful after 3 attempts", exit 1).
• Same test executed and passed Jul 05 (acc 0.930) and Jul 06 (acc 0.927).
• 96 commits in window 3abdbab9b..9b4bb415d; HEAD 9b4bb415d is a breakable-CUDA-graph change (#30834).

Hypothesised Causes (with confidence, with disconfirming evidence)

• [MEDIUM] Infrastructure / GPU hardware instability on the linux-mi35x-gpu-8 runner pool.
  • Supporting: the same runner/job could not clear VRAM for 3 straight days (Jul 07–09) before any test started — a classic wedged-GPU signature; this run hung on two different GPU nodes (node-3, node-5) across two attempts; the hang occurs during weight load/init (before any sglang compute kernel runs); the test passed cleanly 5 days earlier with the same config.
  • Disconfirming: this run's VRAM-clear step passed, so the GPU was not obviously pre-wedged at job start; a genuinely bad GPU usually fails VRAM-clear too.
  • How to verify: check rocm-smi/dmesg on the affected runner node; re-run the job on a different MI35x host and see whether the hang follows the code or the machine.
• [LOW] sglang code regression in 3abdbab9b..9b4bb415d affecting Kimi-K2.5-MXFP4 load.
  • Supporting: last executed pass was Jul 06; 96 commits merged since (incl. DeepSeek-V2/V4 refactors and MoE reorder that Kimi-K2.5 inherits via deepseek_v2.py).
  • Disconfirming: the crash is a HW-level GPU Hang during weight loading, before attention/MoE/CUDA-graph kernels execute — inconsistent with a kernel/graph code bug; window is un-narrowable because Jul 07–09 never ran the test.
  • How to verify: not actionable until the runner-hardware hypothesis is ruled out; if ruled out, bisect the window on a known-good MI35x host.
• [LOW] aiter dev build v0.1.17.dev110+g9127c94a1.
  • Supporting: AMD-only path; aiter RoPE setup is the last log line before the 2nd-attempt hang.
  • Disconfirming: aiter is the Dockerfile default (no override); log-line proximity is not causation; hang also occurred on attempt 1 during pure weight loading before RoPE.

In-flight Fix Check

• In-flight fix: ❌ none found for this specific GPU Hang. Searched open PRs with keywords mi35x (6 results, all debug/routing PRs) and Kimi MXFP4 (0 results).
• Related active AMD investigation (not a fix): #30376 "[AMD] WIP - test(kimi-k26 mi35x)" (open since 2026-07-07), plus multiple debug branches (amd/kimik26-bisect, amd/kimik26-disagg-decode-repro, bingxche/pr29855-mi355x-kimi-test) indicate the team is already probing Kimi on MI35x.

Suggested Triage Steps

• Inspect the affected linux-mi35x-gpu-8 runner: rocm-smi, dmesg/kfd logs around 18:23–18:24 UTC for GPU node-3/node-5 hang/XID events; correlate with the Jul 07–09 VRAM-cleanup failures on the same host.
• Re-run the job on a different MI35x host to determine whether the hang follows the code or the machine (cheapest disconfirmation of the hardware hypothesis).
• If the hang follows the code: bisect 3abdbab9b..9b4bb415d running only test_kimi_k25_mxfp4_bcg_mi35x.py on a known-good MI35x node.
• Coordinate with the AMD engineers already tracking Kimi/MI35x (#30376) before opening a new issue, to avoid duplication.

Status (factual, no priority assignment)

• Persistence: This test's shard has produced no green signal since Jul 06 — Jul 07/08/09 blocked at VRAM-clear (test not run), Jul 11 crashed during execution (GPU Hang). Jul 10 run still queued.
• Scope: Confirmed on job stage-c-test-large-8-gpu-amd-mi35x-rocm720 (linux-mi35x-gpu-8, 0); the recurring VRAM-clear failures suggest a broader linux-mi35x-gpu-8 runner-health problem.
• Blocked work: Kimi-K2.5-MXFP4 accuracy signal (test_bcg_gsm8k) and the breakable-CUDA-graph tests co-located in this shard on AMD ROCm 7.2.
• In-flight fix: ❌ none found for this GPU Hang; related debug work in #30376.

stage-b-test-1-gpu-large-amd-rocm720 (linux-mi325-1gpu-sglang, 0) — failed step(s): Run test

Commit Info

• sglang: 9b4bb415
• aiter: 9127c94a (v0.1.17.dev110+g9127c94a1, source: dev image detected & preserved — AITER_COMMIT_OVERRIDE empty)

Failed Tests

Test File	Test Function	Error	Log
test/registered/perf/test_bench_serving_1gpu_part1.py	TestBenchServing1GPUPart1.test_offline_throughput_with_triton_attention_backend (line 120)	AssertionError: 3150.17 not greater than 3500 (retry once → still fails, retry() exceed maximum number of retries)	Job log

All 12 other test files in this job passed (test_streaming_session, test_zaya, test_adaptive_speculative, test_piecewise_cuda_graph_support_1_gpu, test_session_control, several attention unit tests). The job's exit code 255 is driven solely by this one perf assertion.

Failure Summary

The AMD MI325 / ROCm 7.2 offline-throughput benchmark with the Triton attention backend (Llama-3.1-8B, 500 prompts, context-length 8192) measured an output throughput of 3150.17 tok/s (initial attempt 3155.45), below the AMD gate of 3500 tok/s at test_bench_serving_1gpu_part1.py:120. The benchmark completed successfully (500/500 requests, no crash) — this is a pure performance-threshold miss, not a functional failure.

Regression Status

Known recurring failure / Never-passed in queryable window. This test function has been below the 3500 AMD threshold on every completed scheduled run where it was cleanly reached; no green run found in the last ~6 days.

Recent history of test_bench_serving_1gpu_part1.py (test_offline_throughput_with_triton_attention_backend) in job stage-b-test-1-gpu-large-amd-rocm720 (…, 0) (branch=main, event=schedule, completed runs only):

Date	Run	Test Fn Status	Measured throughput	Error
Jul 11	29162225443	❌ Failed	3150–3155	3150.17 not greater than 3500
Jul 9	29039527644	⚠️ N/A (different failure)	—	part1 failed earlier in test_offline_throughput_default + HF-download/server errors (Ran 0 tests); triton test not cleanly reached
Jul 8	28964460204	❌ Failed	3156–3167	3156.68 not greater than 3500
Jul 7	28888292440	❌ Failed	2563–2698	2698.47 not greater than 3500
Jul 6	28813230096	❌ Failed	3131–3141	3131.91 not greater than 3500

First observed failure in window: Jul 6 (earliest queried). No last-known-passing run found in Jul 6–11 window ⇒ treat as Never-passed for this hardware/config. Lookback range: 6 scheduled runs, 2026-07-06 → 2026-07-11.

Failure Cluster

Perf-threshold miss — Triton-backend offline throughput below AMD gate (3500 tok/s) on MI325/ROCm7.2

Facts (verified observations)

• Verbatim assertion (both attempts): AssertionError: 3155.4482780732505 not greater than 3500 then AssertionError: 3150.1719222453517 not greater than 3500, followed by Exception: retry() exceed maximum number of retries. (job log ~line 9811–9830).
• Benchmark completed cleanly: Successful requests: 500, Output token throughput (tok/s): 3150.17, Benchmark duration (s): 169.04 — no OOM, crash, or kernel fault.
• Threshold is AMD-specific: test_bench_serving_1gpu_part1.py:119-121 → if is_in_amd_ci(): self.assertGreater(res["output_throughput"], 3500) else: … 3700.
• Measured throughput on this job has ranged 2563–3167 tok/s across Jul 6–11, never reaching 3500.
• aiter unchanged across the window (Dockerfile-default dev image v0.1.17.dev110+g9127c94a1, no override) — same on passing sibling tests in the job.
• retry() gives only 1 retry; both samples were below gate, so run-to-run noise does not rescue it.

Hypothesised Causes (with confidence)

• [MEDIUM] The 3500 tok/s AMD gate is set higher than this MI325 / ROCm 7.2 + Triton-backend configuration sustainably delivers.
  • Supporting: throughput clusters tightly at ~3.1–3.2k across 4 distinct scheduled runs (different head SHAs) with no green run; failure is a stable margin (~10% below gate), not a spike; benchmark itself succeeds functionally.
  • Disconfirming: Jul 7 run dipped to ~2.6k, wider than pure gate-margin noise, hinting at additional run-to-run variance (possibly contention/thermals) rather than a fixed offset; a genuine perf regression predating the Jul 6 window cannot be excluded from this 6-day lookback.
  • How to verify: query scheduled runs older than Jul 6 for the last date this test passed on the same job; if it never passed since the Triton-backend case was added to AMD CI, the gate was mis-calibrated from the start.
• [LOW] A persistent Triton-attention-backend perf regression on ROCm 7.2 predating the queryable window.
  • Supporting: values are AMD-specific (NVIDIA gate is a separate 3700 path); Triton backend is explicitly selected here.
  • Disconfirming: no code-level evidence in the 6-day window (aiter static; no green→red transition captured); commit 9b4bb415 is just today's HEAD and is not implicated.
  • How to verify: extend the run history backward to locate a pass_sha..fail_sha, then git log that window for Triton-attention / scheduler / cuda-graph changes.

No hypothesis reaches HIGH; needs pre-Jul-6 history to distinguish "gate too high" from "old regression".

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs for test_offline_throughput_with_triton (0 results) and bench_serving throughput amd (6 unrelated results, none touching this gate). API: search/issues?q=repo:sgl-project/sglang+is:pr+is:open+test_offline_throughput_with_triton.

Suggested Triage Steps

• Extend the scheduled-run history for this job before 2026-07-06 to find the last date test_offline_throughput_with_triton_attention_backend passed; if none, the AMD 3500 gate was never met on MI325/ROCm7.2 and should be re-calibrated to observed p50 minus margin.
• If a passing baseline exists earlier, bisect the pass_sha..fail_sha window for Triton-attention-backend / cuda-graph / scheduler changes and re-run this single test on an MI325 ROCm7.2 runner.
• Cross-check the sibling pr-test-amd.yml (non-720) run of the same test to see whether the shortfall is ROCm-7.2-specific or general AMD, which narrows the cause between toolchain and kernel.

Status

• Persistence: Failing on every cleanly-reached completed scheduled run in the queried window (Jul 6, 7, 8, 11); no green run found — likely persistent for ≥6 days.
• Scope: 1 test function in 1 job (stage-b-test-1-gpu-large-amd-rocm720, shard 0) of pr-test-amd-rocm720.yml. All other tests in the job pass.
• Blocked work: Blocks a clean AMD ROCm 7.2 perf signal for the 1-GPU Triton-backend throughput gate; benchmark is functionally healthy.
• In-flight fix: ❌ none found in open PRs.

multimodal-gen-test-2-gpu-amd-rocm720 (linux-mi325-2gpu-sglang, 1) — failed step(s): Run diffusion server tests (2-GPU)

Commit Info

• sglang: 9b4bb415d
• aiter: v0.1.17.dev110+g9127c94a1 (source: dev image detected in CI — [CI-AITER-CHECK] Dev/patched version detected → skipping rebuild; Dockerfile default was 9127c94a1)

Failed Tests

Test File	Test Function	Error	Log
python/sglang/multimodal_gen/test/single_test_file/test_disagg_server.py	TestDisaggZImage1Rank::test_generates_image	RuntimeError: Warmup request failed for zimage_1rank: 500 Server Error (setUpClass)	Log
python/sglang/multimodal_gen/test/single_test_file/test_disagg_server.py	TestDisaggZImage2RankDenoiser::test_generates_image_with_sp2_denoiser	RuntimeError: Warmup request failed for zimage_sp2: 500 Server Error (setUpClass)	Log
python/sglang/multimodal_gen/test/single_test_file/test_disagg_server.py	TestDisaggZImageTracing::test_disagg_spans_share_trace_id	RuntimeError: Warmup request failed for zimage_trace: 500 Server Error (setUpClass)	Log

Failure Summary

All three TestDisaggZImage* classes in test_disagg_server.py errored in setUpClass because the disaggregated diffusion server returned 500 Internal Server Error on its warmup image-generation request, which then timed out after 120 s (DiffusionServer timeout). The server-side 500 originates in the encoder pool worker, which raised ValueError: The truth value of an array with more than one element is ambiguous while extracting transfer fields from the request result.

Regression Status

Never-passed (no green run found since the test was introduced 2026-07-09).

Recent history of test_disagg_server.py (TestDisaggZImage*) in job multimodal-gen-test-2-gpu-amd-rocm720 (linux-mi325-2gpu-sglang, 1) (branch main, event schedule, completed runs only):

Date	Run	Job	Test File Status	Failed Function	Error
Jul 11	29162225443	job	❌ Failed	TestDisaggZImage* (×3)	ValueError: ambiguous array → 500
Jul 09	29039527644	job	❌ Failed	TestDisaggZImage* (×3)	ValueError: ambiguous array → 500
Jul 08	28964460204	job	✅ Not present	— (job failed on test_server_2_gpu.py instead)	test_disagg_server.py did not exist

The Jul 08 run predates PR #30492 (e703f9e56), which added both test_disagg_server.py and the buggy scheduler_mixin.py. Every completed run that contains #30492 (Jul 09 onward) shows this identical failure — the test has never passed. Lookback window: Jul 02 → Jul 11 scheduled runs.

Failure Cluster

ValueError: ambiguous array truth value in disagg extract_transfer_fields → encoder 500

Facts (verified observations)

• Encoder pool traceback (verbatim from log): File ".../disaggregation/scheduler_mixin.py", line 1545, in _disagg_encoder_step → tensor_fields, scalar_fields = extract_transfer_fields(req_result) → line 226, in extract_transfer_fields → if _is_default(value, f): → ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all().
• The error retries 3× (error (attempt 1/3)) then the encoder produces no output; client sees 500 Server Error on /v1/images/generations, then RuntimeError: Model generation returned no output ... DiffusionServer timeout: request ... not completed within 120.0s.
• Source _is_default at scheduler_mixin.py:155: return value == field_info.default. When value is a numpy array with >1 element and field_info.default is a scalar, value == default returns an element-wise boolean array whose truthiness is ambiguous → ValueError.
• e703f9e56 (#30492, merged 2026-07-09) is the only commit in this file's history and added _is_default, extract_transfer_fields, and the test file (git log --diff-filter=A confirms both are new in that commit).
• Commit ancestry (verified via git merge-base --is-ancestor): Jul 09 run head 2e4d6368c3 and Jul 11 head 9b4bb415d contain #30492; Jul 08 head 8d2b66fd90 does not.
• Same failure reproduced in the Jul 09 sister run (job 86193013065): identical 3 TestDisaggZImage* ERRORs and identical ValueError message.

Hypothesised Causes (with confidence)

• [HIGH] _is_default in scheduler_mixin.py:155 (value == field_info.default) crashes when a Req field holds a numpy array — introduced by #30492 (e703f9e56).
  • Supporting: the failing stack trace terminates exactly at the two lines added by this commit (extract_transfer_fields:226 → _is_default:155); the failure appears in the first completed scheduled run containing the commit and is absent in the run immediately before it (which lacked the file); numpy array == scalar is a textbook trigger for this exact ValueError.
  • Disconfirming: (none found) — the same commit introduced the test, so there is no pre-#30492 green baseline for this test; however this strengthens rather than weakens the attribution, since the test has been red on 100% of runs that contain the code.
  • How to verify: reproduce on AMD MI325 2-GPU by running test_disagg_server.py::TestDisaggZImage1Rank against a build with #30492; inspect which Req field is a numpy array reaching _is_default (candidates: timesteps/sigmas/latent arrays), and guard _is_default with an isinstance(value, np.ndarray) / _is_tensor_like check before the == comparison.

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs for extract_transfer_fields (0), _is_default (0), test_disagg_server (0), disagg ambiguous (1 = #29193, unrelated — it concerns SRT Scheduler cur_batch/last_batch event-loop fields, not multimodal_gen transfer). Needs a new PR.

Suggested Triage Steps

• Add a numpy-array guard in _is_default (scheduler_mixin.py:153-159) — e.g. return False (or use np.array_equal) when value is an np.ndarray/tensor, before the scalar == comparison — then re-run test_disagg_server.py on AMD MI325 2-GPU.
• Instrument extract_transfer_fields to log f.name when _is_default raises, to confirm which Req field carries the numpy array (guides the minimal fix).
• Since the failure is fully reproducible and code-local to #30492, no bisect is needed; confirm the fix on both the 1-GPU and 2-GPU multimodal-gen disagg shards (the 1-GPU shards in the same runs should be checked for the same signature).

Status

• Persistence: Failing on every completed scheduled run since 2026-07-09 (introduced with #30492); test has never passed.
• Scope: test_disagg_server.py (3 TestDisaggZImage* tests) in multimodal-gen-test-2-gpu-amd-rocm720; likely also affects other multimodal-gen disagg shards exercising the encoder transfer path (verify 1-GPU shards).
• Blocked work: Blocks disaggregated Z-Image (encoder/denoiser/decoder) CI signal on AMD ROCm 7.2.
• In-flight fix: ❌ none found — needs a new PR.

stage-c-test-large-8-gpu-amd-mi35x-rocm720 (linux-mi35x-gpu-8, 1) — failed step(s): Run test

Commit Info

• sglang: 9b4bb415ddc315ebcf1f8fcc280537f0b30938dc
• aiter: 9127c94a18e4398e1eba91f6639e910f0994ad02 (source: dev/patched build detected in CI image — v0.1.17.dev110+g9127c94a1 matched the gfx950 Dockerfile default, rebuild skipped)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/test_deepseek_r1_mxfp4_8gpu.py	TestDeepseekR1MXFP4.setUpClass (server launch)	HW Exception ... reason :GPU Hang → Fatal Python error: Aborted → exit code 134	Log

test_kimi_k25_mxfp4.py and test_qwen3_coder_next_8gpu.py were scheduled after it in this partition but never ran — the whole test process aborted (exit 134) during the first test, on both the initial attempt and the HF_HUB_OFFLINE=1 retry.

Failure Summary

The partition-1 job of stage-c-test-large-8-gpu-amd-mi35x-rocm720 aborted while bringing up the amd/DeepSeek-R1-MXFP4-Preview server (TP=8) for test_deepseek_r1_mxfp4_8gpu.py. On the first attempt the process hung ~4 min after acquiring the model-download lock and the watchdog fired with a hardware-level GPU Hang on GPU node-3; the offline retry crashed within ~1 s of printing server_args, again with GPU Hang (node-5). Both attempts ended in Fatal Python error: Aborted / exit code 134 (SIGABRT).

Regression Status

Known recurring / intermittent failure — the MI35x-rocm720 partition-1 slot has been unstable for this test file all week, with node-health failures interleaved with passes.

Recent history of test_deepseek_r1_mxfp4_8gpu.py (TestDeepseekR1MXFP4.setUpClass) in job stage-c-test-large-8-gpu-amd-mi35x-rocm720 (linux-mi35x-gpu-8, 1) (branch main, event=schedule, completed runs only):

Date	Run	Job	Test File Status	Failed Function	Error
Jul 11	29162225443	job	❌ Failed	setUpClass	GPU Hang → Aborted → exit 134
Jul 9	29039527644	job	⚠️ Did not run	—	Pre-flight VRAM cleanup unsuccessful after 3 attempts (zombie KFD, exit 1)
Jul 8	28964460204	job	✅ Passed	—	—
Jul 7	28888292440	job	❌ Failed	setUpClass	Server process exited with code -9 (SIGKILL), elapsed 73s
Jul 6	28813230096	job	⚠️ Did not run	—	Pre-flight VRAM cleanup unsuccessful after 3 attempts (zombie KFD, exit 1)
Jul 5	28749537427	job	✅ Passed	—	—

First observed failure of this exact GPU Hang/exit-134 symptom: Jul 11 (this run). Last known pass: Jul 8. Note Jul 6 & Jul 9 were blocked before the test by a driver/KFD-level VRAM leak (the ROCm/aiter#2061 signature), so the test could not run — they are neither passes nor test-level failures.

Failure Cluster

GPU Hang / node instability on MI35x-rocm720 8-GPU runner during DeepSeek-R1-MXFP4 server startup

Facts (verified observations, no interpretation)

• Attempt 1: [CI Download] Process 3668 ACQUIRED lock for amd/DeepSeek-R1-MXFP4-Preview at 18:43:23; then at 18:47:16 (~3m53s later) many HW Exception by GPU node-3 ... reason :GPU Hang lines, Fatal Python error: Aborted, stack in python/sglang/srt/utils/watchdog.py:147 _watchdog_once / :127.
• First attempt failed with exit code 134 → harness retried with HF_HUB_OFFLINE=1.
• Attempt 2 (offline): server_args=... printed at 18:48:01; HW Exception by GPU node-5 ... reason :GPU Hang at 18:48:02; ##[error]Process completed with exit code 134.
• Exit 134 = 128 + 6 (SIGABRT). GPU Hang is a hardware/driver-level exception, not a Python assertion.
• aiter unchanged: dev image build v0.1.17.dev110+g9127c94a1 == gfx950 Dockerfile default 9127c94a… (verified from [CI-AITER-CHECK] markers, lines 4331–4333).
• Sister partition-0 job in this same run also failed ([86193... series pattern]); the MI35x-rocm720 node has shown zombie-KFD VRAM leaks on Jul 6 & Jul 9.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [MEDIUM] MI35x runner node-level GPU/KFD instability (hung GPU or leaked-VRAM/zombie-KFD state carried into this job), not an sglang code regression.
  • Supporting: the hang manifests as a hardware GPU Hang HW Exception, fires during weight download (CPU/IO-bound, no model compute yet) on attempt 1, and recurs within 1s on the retry — consistent with the GPU already being wedged. The same runner slot failed Jul 6 & Jul 9 with the [Bug] Custom all-reduce IPC buffers use fixed VA, conflict with torch_memory_saver pause/resume → zombie processes on MI300X ROCm/aiter#2061 zombie-KFD VRAM-leak signature requiring a reboot.
  • Disconfirming: the exact exit-134 GPU Hang symptom is new to Jul 11 (Jul 6/9 failed at the pre-flight VRAM-clear step, a different, earlier stage); the test passed cleanly Jul 5 & Jul 8 on presumably the same hardware, so hardware alone is not deterministic.
  • How to verify: check rocm-smi VRAM-occupancy diagnostics in this job's pre-flight step; reboot/drain the MI35x node and re-run test_deepseek_r1_mxfp4_8gpu.py in isolation.
• [LOW] A hang inside DeepSeek-R1-MXFP4 (Quark MXFP4 MoE) init/weight-load path specific to gfx950.
  • Supporting: crash localizes to deepseek_weight_loader/model_runner init in the attempt-1 traceback; Jul 7 also crashed this same test at startup (SIGKILL -9).
  • Disconfirming: exit signals differ across days (134 vs -9), and the attempt-1 hang occurred during download before compute — more consistent with a pre-wedged GPU than a model-code hang. aiter is unchanged vs the passing Jul 8 run.
  • How to verify: bisect is low-value until node health is ruled out; if node is clean, run the server standalone with --attention-backend aiter and capture a py-spy/rocgdb backtrace at hang time.

In-flight Fix Check

• In-flight fix: ✅ #30431 "[AMD] ci: self-heal driver-level VRAM leaks + graceful GPU cleanup on shutdown" (open since 2026-07-07, updated 2026-07-11) — directly targets the [Bug] Custom all-reduce IPC buffers use fixed VA, conflict with torch_memory_saver pause/resume → zombie processes on MI300X ROCm/aiter#2061 zombie-KFD/VRAM-leak MI35x incident that blocked the Jul 6/Jul 9 runs; self-heals the node instead of failing pre-flight. Chase reviewers; do NOT open a duplicate. (Search: is:pr is:open zombie+KFD and mi35x+GPU+Hang.)
• Note: #30431 fixes the pre-flight VRAM-clear path; whether it also prevents the in-test GPU Hang seen here is unverified — it may recover the node faster but not stop an already-wedged GPU mid-run.

Suggested Triage Steps

• Inspect this job's Ensure VRAM is clear pre-flight rocm-smi output to confirm whether the node entered the test already carrying leaked VRAM ([Bug] Custom all-reduce IPC buffers use fixed VA, conflict with torch_memory_saver pause/resume → zombie processes on MI300X ROCm/aiter#2061 signature).
• Drain/reboot the specific MI35x-rocm720 node, then re-run test_deepseek_r1_mxfp4_8gpu.py alone to see if the GPU Hang reproduces on a clean node (distinguishes infra from model-code hang).
• Track #30431 to merge; after it lands, re-run this partition and confirm the pre-flight self-heal clears the node.
• If the hang reproduces on a freshly rebooted node, capture a py-spy dump + rocgdb backtrace at hang time and bisect only then — treat as a possible gfx950 MXFP4 init hang.

Status

• Persistence: This exact GPU Hang/exit-134 symptom first seen Jul 11 (this run). The broader "MI35x-rocm720 partition unstable for this test" pattern spans Jul 6–11 (2 infra blocks, 2 test crashes, 2 passes).
• Scope: stage-c-test-large-8-gpu-amd-mi35x-rocm720 partition 1 (and partition 0 also failing this run) in pr-test-amd-rocm720.yml; the underlying zombie-KFD VRAM-leak family has affected multiple AMD workflows per #30431's description (~37 jobs, 7+ days).
• Blocked work: Blocks the MI35x-rocm720 large 8-GPU signal for DeepSeek-R1-MXFP4, Kimi-K2.5-MXFP4, and Qwen3-Coder-Next (latter two never ran this partition).
• In-flight fix: ✅ #30431 (open since 2026-07-07) — awaiting merge.

multimodal-gen-test-1-gpu-amd-rocm720 (linux-mi325-1gpu-sglang, 3) — failed step(s): N/A

Commit Info

• sglang: 9b4bb415d
• aiter: N/A — this job's log blob is missing (BlobNotFound), so [CI-AITER-CHECK] markers could not be read. Not a scout run.

Failed Tests

Test File	Test Function	Error	Log
log unavailable for this job	—	Job completed/failure but log blob is BlobNotFound; step 9 "Run diffusion server tests (1-GPU)" stuck in_progress, steps 10/20 never ran → abnormal termination, not a clean pytest exit	Job

Most recent retrievable shard-3 failure (Jul 7, run 28888292440, same deterministic partition --partition-id 3 --total-partitions 4), shown as the likely-persistent failure this job hits:

Test File	Test Function	Error	Log
python/sglang/multimodal_gen/test/single_test_file/test_update_weights_from_disk.py	TestUpdateWeightsFromDisk::test_update_weights_specific_modules[Qwen-Image]	AssertionError: Update failed: ... 'Inplace update to inference tensor outside InferenceMode is not allowed' (500≠200)	Log
same	TestUpdateWeightsFromDisk::test_update_weights_from_disk_default[Qwen-Image]	assert 500 == 200 (same InferenceMode cause)	same
same	TestUpdateWeightsFromDisk::test_corrupted_weights_rollback[Qwen-Image]	assert (500 == 200)	same
same	TestUpdateWeightsFromDiskWithOffload::test_update_weights_with_offload_enabled[Qwen-Image]	assert 500 == 200	same
same	TestUpdateWeightsFromDiskWithOffload::test_update_weights_with_offload_enabled[FLUX.2-klein-base-4B]	setup ValueError: Could not find model ... FLUX.2-klein-base-4B (model not cached)	same

(Jul 7 result line: 5 failed, 1 passed, 9 warnings, 4 errors in 826.99s.)

Failure Summary

The target job multimodal-gen-test-1-gpu-amd-rocm720 (…, 3) is marked completed/failure but its log was never uploaded (BlobNotFound) and its test step is frozen in_progress — the runner/job was terminated abnormally at ~26 min (started 20:02:20Z, ended 20:28:21Z, well under the 60-min step timeout). Shard 3 of this job has failed in every completed scheduled run from Jul 4 through Jul 11; the last log that survived (Jul 7) shows the diffusion weight-update suite test_update_weights_from_disk.py failing on Qwen-Image with an "Inplace update to inference tensor outside InferenceMode" server error, plus FLUX.2-klein-base-4B setup errors from a missing model cache.

Regression Status

Known recurring failure (persistent ≥ 8 days). Shard 3 has never passed in the queryable window.

Recent history of shard 3 of job multimodal-gen-test-1-gpu-amd-rocm720 (branch main, event schedule, completed runs only):

Date	Run	Shard-3 Status	Duration	Note
Jul 11	29162225443	❌ Failed	26m	log BlobNotFound (abnormal term.)
Jul 9	29039527644	❌ Failed	31m	log BlobNotFound
Jul 8	28964460204	❌ Failed	21m	—
Jul 7	28888292440	❌ Failed	24m	log retrievable → test_update_weights_from_disk.py
Jul 6	28813230096	❌ Failed	—	—
Jul 5	28749537427	❌ Failed	—	—
Jul 4	28714582765	❌ Failed	—	—

No passing shard-3 run found back to Jul 4; the multimodal-gen job was not present in the Jul 1 runs, so no green baseline is queryable. First-observed-failure/last-green cannot be pinned within this window.

Failure Cluster

Diffusion weight-update: "Inplace update to inference tensor outside InferenceMode" (shard 3). Distinct from the shard-0/1 sibling cluster (60-min step timeout / DiffusionServer request timeout).

Facts (verified observations)

• Target job 86569165351: status=completed, conclusion=failure; step 9 in_progress, steps 10 (upload artifacts) & 20 (post-checkout) pending; log endpoint returns BlobNotFound. Ran 20:02:20Z→20:28:21Z (~26m). [FACT]
• The three 1-GPU shards that failed today (0, 1, 3) all return BlobNotFound; sibling jobs that finished cleanly (2-GPU shard, unit-test) have full logs. [FACT]
• Sibling multimodal-gen-test-2-gpu (…,1) [86569165339] failed cleanly with RuntimeError: ... DiffusionServer timeout: request ... not completed within 120.0s → 500 → warmup failure in test_disagg_server.py. [FACT]
• Shards 0/1 today ran ~72–77 min (hit the timeout-minutes: 60 test step); shard 3 died at ~26 min (a different, earlier failure mode). Shard 2 passed. [FACT]
• Jul 7 shard-3 log: param.data.copy_() path in weights_updater.py:222 (called from _load_weights_into_module, _apply_weights at line 339) raised "Inplace update to inference tensor outside InferenceMode is not allowed"; scheduler returned 500 → test asserts 500 == 200. [FACT]
• FLUX.2-klein-base-4B setup errors (Jul 7) are a separate cause: ValueError: Could not find model at /sgl-data/hf-cache/.../FLUX.2-klein-base-4B/... (model not cached). [FACT]
• Overall run 29162225443 still queued at snapshot (2 jobs in_progress, 4 queued) — run-level log archive 404s. [FACT]

Hypothesised Causes (with confidence)

• [LOW→MEDIUM] Weight-update writes into inference tensors without disabling inference mode. _load_weights_into_module does param.data.copy_(...) (weights_updater.py:222) with no torch.inference_mode(False) guard, while diffusion module params are created under inference mode (cf. the existing with torch.inference_mode(False): guard at runtime/layers/lora/linear.py:125).
  • Supporting: error message matches this exact code path; the LoRA linear path already needed the same inference_mode(False) workaround, indicating a known class of issue in this runtime.
  • Disconfirming: (a) confirmed only from the Jul 7 log — today's job log is missing, so I cannot verify today's shard-3 failed on this same line vs. a harder GPU hang/runner death (the abnormal-termination symptom differs from Jul 7's clean pytest exit). (b) weights_updater.py has no recent commit touching this line (last change e703f9e56 #30492, a get_parallel() refactor), so no obvious regression commit; may be an AMD/ROCm-specific or never-passed port gap.
  • How to verify: run python3 sglang/multimodal_gen/test/run_suite.py --suite 1-gpu --partition-id 3 --total-partitions 4 -k "not flux_2" on an MI325 ROCm 7.2 container; if it reproduces, wrap the copy in torch.inference_mode(False) on a branch and re-run.
• [LOW] Runner/GPU-level instability causing abnormal job termination (log never uploaded). Today's shard-3 (and shards 0/1) died without finalizing logs; the 2-GPU sibling shows a GPU-side "not fully cleared" warning (ROCm GPU memory not fully cleared after 60s).
  • Supporting: three 1-GPU shards simultaneously fail to upload logs; step frozen mid-run.
  • Disconfirming: shard 2 (same runner pool) passed and uploaded normally; the persistent deterministic ~24m shard-3 failure across days points to a test defect, not pure infra flake.
  • How to verify: re-run shard 3 alone; check runner dmesg/GPU health and whether the log uploads on a clean runner.
• [FACT] FLUX.2-klein-base-4B setup errors are a model-cache miss, independent of the InferenceMode failures. (Note: this shard runs with -k "not flux_2", but the node-id "FLUX.2" ≠ keyword "flux_2", so FLUX.2 params are not filtered out.)

In-flight Fix Check

• Searched open PRs: InferenceMode → only #30759 (dspark target-verify buffers — unrelated, spec decode not diffusion). update weights diffusion → 30 PRs, none targeting test_update_weights_from_disk / weights_updater inference-tensor copy.
• In-flight fix: ❌ none found in open PRs — needs new PR or investigation.

Suggested Triage Steps

• Confirm today's shard-3 failure is the same cluster: re-run job 86569165351 so a log uploads; the current BlobNotFound blocks direct test-level confirmation.
• Reproduce locally on MI325 ROCm 7.2: run_suite.py --suite 1-gpu --partition-id 3 --total-partitions 4 -k "not flux_2".
• If the InferenceMode error reproduces, test wrapping the copy in weights_updater.py _load_weights_into_module with torch.inference_mode(False) (mirroring runtime/layers/lora/linear.py:125) on a branch and re-run.
• Separately, seed/cache black-forest-labs/FLUX.2-klein-base-4B on the runner (or exclude it correctly by node-id) to clear the 4 setup errors.
• Investigate why 1-GPU shards 0/1/3 fail to upload logs (runner/GPU health) vs shard 2 which passes — rule out a runner-level kill.

Status

• Persistence: shard 3 failing on every completed scheduled run Jul 4→Jul 11 (≥ 8 days); no green shard-3 in the queryable window.
• Scope: pr-test-amd-rocm720.yml multimodal-gen 1-GPU shard 3 (this job). Related-but-distinct diffusion failures today in shards 0/1 (60-min timeout) and the 2-GPU sibling (DiffusionServer 120s request timeout); the whole workflow has been red for 6+ consecutive completed scheduled runs.
• Data limitation: today's exact test-level failure is unconfirmed — job log is BlobNotFound and the run is still in progress. Test-level attribution is inferred from the deterministic partition and the last retrievable (Jul 7) shard-3 log.
• In-flight fix: ❌ none found.

---

Generated by amd-bot using Claude Code CLI (last updated: 2026-07-12 23:21 UTC)

[amd-bot]

pr-test-amd-rocm720.yml — Per-Job Analysis (continued 3/6)

multimodal-gen-test-1-gpu-amd-rocm720 (linux-mi325-1gpu-sglang, 1) — failed step(s): N/A

Commit Info

• sglang: 9b4bb415d
• aiter: N/A — the job log blob returns BlobNotFound (never uploaded because the runner died), so no [CI-AITER-CHECK] markers are recoverable.

Failed Tests

Test File	Test Function	Error	Log
N/A (diffusion 1-GPU suite step)	N/A	Runner lost communication with the server during step 9 Run diffusion server tests (1-GPU)	Job

Test-file granularity is unavailable for this failure: the self-hosted runner lost contact with GitHub while step 9 was still executing (step 9 is frozen at in_progress, steps 10/20 never ran), so no test log was ever finalized or uploaded. The failing unit is the whole suite invocation python3 sglang/multimodal_gen/test/run_suite.py --suite 1-gpu --partition-id 1 --total-partitions 4 -k "not flux_2" (pr-test-amd-rocm720.yml:750).

Failure Summary

This is an infrastructure failure, not a test assertion failure. The MI325 self-hosted runner running shard 1 of the 1-GPU diffusion suite lost communication with the GitHub server ~72 min into the job, during the heavy diffusion server tests (Wan2.1-T2V, HunyuanVideo, Qwen-Image, FLUX.1). The job's log blob was never uploaded, so no per-test result exists.

Regression Status

Known recurring infrastructure failure (intermittent). This is not tied to commit 9b4bb415d — the same symptom spans multiple head SHAs.

Recent history of the 1-GPU diffusion suite in job multimodal-gen-test-1-gpu-amd-rocm720 (…, 1) (shard 1, main, event=schedule, completed runs only):

Date	Run	Job	Test File Status	Failure Mode
Jul 11	29162225443	shard 1	❌ Failed	Runner lost communication (step 9 frozen in_progress)
Jul 09	29039527644	shard 1	❌ Failed	Step 9 timed out after 60 min
Jul 08	28964460204	shard 1	❌ Failed	Runner lost communication
Jul 07	28888292440	shard 1	✅ Passed	—
Jul 06	28813230096	shard 1	❌ Failed	Runner lost communication
Jul 05	28749537427	shard 1	❌ Failed	Step 9 timed out after 60 min

Last known pass: Jul 07. Failing 5 of the last 6 completed scheduled runs, alternating between two symptoms (60-min step timeout and runner-communication-loss). Both point at the same underlying resource/time-budget problem on the diffusion 1-GPU suite.

Failure Cluster

Runner lost communication during 1-GPU diffusion server tests (MI325 resource/time-budget exhaustion)

Facts (verified observations)

• FACT — Job 86569165357 concluded failure; step 9 Run diffusion server tests (1-GPU) is stuck at status=in_progress, conclusion=None; steps 10 (Upload diffusion failure artifacts) and 20 (Post Checkout) never ran. (Verified via jobs API.)
• FACT — The job's check-run annotation reads verbatim: "The self-hosted runner lost communication with the server. … Anything in your workflow that terminates the runner process, starves it for CPU/Memory, or blocks its network access can cause this error."
• FACT — The log blob returns <Error><Code>BlobNotFound></Error> (HTTP 404), consistent with the runner dying before finalizing logs.
• FACT — In the same run 29162225443, 3 of 4 1-GPU shards failed with the identical "runner lost communication" annotation: shard 0 86569165363, shard 1 86569165357, shard 3 86569165351. Shard 2 86569165352 passed (only a Node.js-deprecation warning).
• FACT — Job ran 22:49:04 → 00:01:08 UTC (~72 min). Step 9 has timeout-minutes: 60 (pr-test-amd-rocm720.yml:728).
• FACT — Historical shard-1 failures alternate between annotation "runner lost communication" (Jul 11/08/06) and "The action 'Run diffusion server tests (1-GPU)' has timed out after 60 minutes" (Jul 09/05); Jul 07 passed.
• FACT — The suite loads heavy video/image diffusion models (Wan2.1-T2V-1.3B, HunyuanVideo, Qwen-Image, FLUX.1) — see pr-test-amd-rocm720.yml:714.

Hypothesised Causes

• [MEDIUM] The 1-GPU diffusion suite exceeds the MI325 runner's time/resource budget, causing either a 60-min step timeout or CPU/memory starvation severe enough to sever the runner's connection to GitHub.
  • Supporting: two failure signatures (60-min timeout and runner-comm-loss) alternate on the same shard across days; the comm-loss annotation itself names CPU/memory starvation as a cause; the suite loads multiple multi-GB diffusion models; 3/4 shards in one run died the same way while one shard finished fast (17 min) — consistent with per-shard test-mix/model-download variance pushing some shards over budget.
  • Disconfirming: no runner-side host metrics (dmesg/OOM logs) are available to prove memory starvation vs a network blip; shard 2 passing shows the runner fleet is not uniformly down, so a pure GitHub/network outage is unlikely but a single-node network issue on the affected runners cannot be fully excluded.
  • How to verify: on the affected MI325 runner, correlate the job window with host dmesg/OOM-killer logs and rocm-smi/free-memory sampling; check whether the failing shards were downloading uncached models (the "NOT CACHED" path in step 8) and running longer as a result.
• [LOW] A specific diffusion test hangs the GPU/process, indirectly killing the runner agent.
  • Supporting: step 9 froze rather than erroring; a hung HIP kernel could wedge the process.
  • Disconfirming: no logs survived to confirm a hang location; the alternate signature is a clean 60-min timeout, which is more consistent with slow-but-progressing tests than a hard hang.
  • How to verify: reduce shard size / raise timeout-minutes, capture py-spy dump on the server process near the 60-min mark.

This is not a code regression attributable to commit 9b4bb415d; the symptom predates it across multiple SHAs, so there is no meaningful pass_sha..fail_sha window to bisect.

In-flight Fix Check

• In-flight fix: ❌ none found directly targeting the 1-GPU diffusion runner-comm-loss / 60-min timeout. Keyword searches (diffusion+timeout, diffusion+1-gpu, multimodal_gen+timeout, runner+communication) returned only feature/refactor PRs, not an infra fix for this symptom. Candidate worth a human glance: #29584 ("ci(amd): fix three pre-existing MI325 CI failures") — verify whether it covers the diffusion 1-GPU suite before assuming overlap.

Suggested Triage Steps

• Pull host-side logs for the affected MI325 runner(s) around 22:49–00:01 UTC on Jul 11 (dmesg -T, OOM-killer output, agent Runner_*.log) to confirm whether the process was OOM-killed vs a network drop.
• Compare the passing shard 2 vs failing shards 0/1/3 in run 29162225443: check step-8 "NOT CACHED" model output to see if the failing shards paid extra multi-GB download time and blew the 60-min budget.
• If resource-bound is confirmed, evaluate re-partitioning the suite (more than 4 shards) or raising step-9 timeout-minutes at pr-test-amd-rocm720.yml:728 so slow-but-healthy runs stop tripping the timeout/comm-loss.
• Re-run this job once in isolation; if it passes (as Jul 07 did), treat as an intermittent infra/resource issue rather than a functional test break.

Status

• Persistence: Recurring/intermittent since at least Jul 05 — failed 5 of last 6 completed scheduled runs on shard 1 (last pass Jul 07), alternating 60-min-timeout and runner-comm-loss signatures.
• Scope: Broad within the 1-GPU diffusion matrix — 3 of 4 shards (0, 1, 3) failed identically in today's run 29162225443; the same suite also times out on prior days. Confined to the MI325 diffusion 1-GPU suite on pr-test-amd-rocm720.
• Blocked work: Blocks reliable AMD ROCm 7.2 diffusion (T2V/T2I/I2V/LoRA) CI signal on the 1-GPU suite.
• In-flight fix: ❌ none found targeting this symptom; check #29584 for possible overlap.

multimodal-gen-test-1-gpu-amd-rocm720 (linux-mi325-1gpu-sglang, 0) — failed step(s): N/A

Commit Info

• sglang: 9b4bb415
• aiter: N/A — the job log blob was never finalized (BlobNotFound), so no [CI-AITER-CHECK] markers are retrievable. The job installs diffusion deps via amd_ci_install_dependency.sh diffusion.

Failed Tests

Test File	Test Function	Error	Log
python/sglang/multimodal_gen/test/server/test_server_1_gpu.py (partition 0 of 4)	TestDiffusionServerOneGpu::test_diffusion_generation[...] — exact case not identifiable	Step "Run diffusion server tests (1-GPU)" exceeded its timeout-minutes: 60 and was hard-killed (no test-level assertion)	Job

The exact failing/hanging test case cannot be named: the run is still in-flight and this job was killed mid-step, so its log was never uploaded (BlobNotFound on the logs API). Partition 0 of this suite exercises test_diffusion_generation cases (e.g. zimage_image_t2i, qwen_image_edit_ti2i, wan2_1_t2v_1.3b, fast_hunyuan_video, cosmos3_nano_t2v, …), observed from a prior run's log.

Failure Summary

The multimodal-gen-test-1-gpu-amd-rocm720 shard 0 job hit the 60-minute step timeout on "Run diffusion server tests (1-GPU)" and was terminated. [FACT] Step 9 started 2026-07-11T20:14:19Z, never recorded a conclusion (in_progress), and the job was killed at 21:20:28Z (~66 min in-step); the downstream "Upload diffusion failure artifacts" and "Post Checkout" steps were left pending. [FACT] The sister shard 1 (job 86569165357, 72.1 min, step also in_progress) timed out identically, while shard 2 passed (17.4 min) and shard 3 failed fast (26.0 min — a distinct, non-timeout failure).

Regression Status

Known recurring / chronic — but note two distinct failure modes on this shard. This run's failure is a 60-min step timeout; prior completed-run failures were clean test errors (step9=completed/failure), not timeouts.

Recent history of shard 0 (multimodal-gen-test-1-gpu-amd-rocm720 (…, 0)) — same branch (main), same event (schedule), completed runs only (job-level; note the caveat below):

Date	Run	Job	Job Status	step9 mode	Total dur
Jul 11	29162225443 (in-flight run)	job	❌ Failed	timeout (in_progress)	77.1m
Jul 9	29039527644	job	✅ Passed	completed/success	67.2m
Jul 8	28964460204	job	❌ Failed	completed/failure (cosmos3_nano_t2v)	69.2m
Jul 7	28888292440	job	✅ Passed	completed/success	54.1m
Jul 6	28813230096	job	❌ Failed	completed/failure	69.5m
Jul 5	28749537427	job	❌ Failed	completed/failure	69.1m

Caveat: partition assignment is dynamic (run_suite.py --partition-id), so the specific test in shard 0 varies run-to-run; the table is job-level because this run's log is unavailable for test-file granularity. Key observation: even passing shard-0 runs take ~54–67 min total (~50–56 min in-step), i.e. right at the 60-min step budget — the suite has essentially no timeout headroom.

Failure Cluster

Diffusion 1-GPU server test suite exceeds 60-min step timeout (partition killed mid-run)

Facts (verified observations)

• [FACT] Job 86569165363 conclusion failure; step "Run diffusion server tests (1-GPU)" status in_progress/no conclusion; started 20:14:19Z, killed 21:20:28Z (~66 min > timeout-minutes: 60).
• [FACT] Step config: .github/workflows/pr-test-amd-rocm720.yml:727-728 sets timeout-minutes: 60 on this step; command runs run_suite.py --suite 1-gpu --partition-id 0 --total-partitions 4 -k "not flux_2".
• [FACT] Same run: shard 1 also timed out (72.1m, step in_progress); shard 2 passed (17.4m); shard 3 failed fast (26.0m, non-timeout). Only 2 of 4 partitions exceed the budget.
• [FACT] Overall run 29162225443 is still in-flight (multiple queued/in_progress jobs), which is why the killed job's log blob is unavailable.
• [FACT] Prior partition-0 content (from Jul 8 job log) includes heavy cases: fast_hunyuan_video (~12 min alone), wan2_2_ti2v_5b (~8.5 min), plus ~10 others; that run reached ~50 min in-step before its last cases.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [MEDIUM] Partition-0 (and -1) suite runtime has crossed the fixed 60-min step budget — capacity/timeout, not a functional bug.
  • Supporting: passing shard-0 runs already consume ~50–56 min in-step; two partitions in this run exceeded 60 min while the two lighter partitions finished; the killed step never emitted an assertion.
  • Disconfirming: partitioning is dynamic, so a single uncached model download or transient runner slowness (not steady runtime growth) could equally explain a one-off overrun; without the log this cannot be separated from a genuine hang in one case.
  • How to verify: re-run shard 0 with SGLANG_TEST_NUM_INFERENCE_STEPS unchanged and inspect per-case timings; or temporarily raise timeout-minutes to capture where time is spent.
• [LOW] Newly added CI test cases in the Jul 10–11 window inflated per-partition load. Window 2e4d6368..9b4bb415 added diffusion cases: 649ce5dd3 (Pi0.5, #30633), edd91cbdd (cosmos3 action output, #27168), e9493a015 (Z-Image-Turbo CI, #30584).
  • Supporting: more cases → more per-partition wall time on an already-marginal budget.
  • Disconfirming: cannot confirm any of these landed in shard 0's partition this run (no log); Z-Image was already being tested before this window, so #30584 is not net-new coverage.
  • How to verify: diff the collected test list per partition against the last green run; check whether new cases route into partitions 0/1.
• [LOW] Genuine hang in one diffusion case (e.g. server never becomes ready / stalls during denoise).
  • Supporting: only some partitions exceed budget; a single stuck case would consume the whole 60 min.
  • Disconfirming: no stack trace/log available; prior runs show these cases completing.
  • How to verify: re-run with per-case timeout and capture server logs.

In-flight Fix Check

In-flight fix: ❌ none found — searched open PRs for diffusion+timeout, multimodal-gen+timeout, diffusion+CI+partition; results are unrelated diffusion features (e.g. #19114 parallelize module loading, #24200). No open PR addresses the 1-GPU diffusion partition timeout on ROCm 7.2.

Suggested Triage Steps

• Re-run once the parent run completes and confirm whether shard 0 times out again or was a one-off overrun (distinguish steady runtime growth vs. transient slow model download).
• If reproducible: capture per-case timings for partition 0 (raise timeout-minutes temporarily, or add per-test timing) to find whether one case hangs or the aggregate simply exceeds 60 min.
• If it's aggregate load: consider increasing --total-partitions for the 1-GPU suite (more, smaller shards) or raising timeout-minutes above 60 — maintainer's call.
• Note #19114 (parallelize pipeline module loading) as a possible runtime-reduction lever if load, not a hang, is confirmed.

Status

• Persistence: shard 0 has failed in 4 of the last 6 completed scheduled runs (Jul 4–11), but the timeout mode (this run) is distinct from the test-error mode of prior failures; the suite has run at/near the 60-min budget for ≥1 week.
• Scope: 2 of 4 partitions (shards 0 and 1) in run 29162225443 timed out; shard 3 failed for a different (non-timeout) reason; shard 2 passed. Limited to the AMD ROCm 7.2 1-GPU diffusion suite in this snapshot.
• In-flight fix: ❌ none found in open PRs.
• Data limitation: this run is in-flight and the job was killed mid-step, so no job log is retrievable and the exact failing/hanging test_diffusion_generation[...] case cannot be pinned; re-analyze once the run completes and logs finalize.

multimodal-gen-test-2-gpu-amd-rocm720 (linux-mi325-2gpu-sglang, 0) — failed step(s): Run diffusion server tests (2-GPU)

Commit Info

• sglang: 9b4bb415
• aiter: 9127c94a (v0.1.17.dev110+g9127c94a1, source: dev/patched image build preserved — [CI-AITER-CHECK] Dev/patched version detected → skipping rebuild; matches Dockerfile default)

Failed Tests

All failures are in python/sglang/multimodal_gen/test/server/test_server_2_gpu.py, class TestDiffusionServerTwoGpu, param of test_diffusion_generation. This shard (shard 0) had 1 genuine FAILED + 17 ERROR (18 total).

Test File	Test Function (param)	Error	Log
test/server/test_server_2_gpu.py	test_diffusion_generation[flux2_modelopt_fp8_tp2_t2i]	RuntimeError: CUDA error: HIPBLAS_STATUS_NOT_SUPPORTED in torch._scaled_mm	Log
test/server/test_server_2_gpu.py	test_diffusion_generation[joy_echo_t2v_2gpu]	ERROR — server startup hang (~20 min) → timeout; JoyAI-Echo not in HF cache	Log
test/server/test_server_2_gpu.py	16× cascade victims (see below)	RuntimeError: Scheduler port 5555 is unavailable and --strict-ports is enabled	Log

The 16 cascade victims: wan2_1_t2v_1.3b_cfg_parallel, wan2_1_t2v_1_3b_cache_dit_sp_only_2gpu, fsdp-inference, mova_360p_tp2, mova_360p_ring1_uly2, ltx_2_two_stage_t2v, ltx_2_3_two_stage_ti2v_2gpus, wan2_1_i2v_14b_480P_2gpu, ltx_2.3_two_stage_t2v_2gpus, wan2_1_i2v_14b_lora_2gpu, wan2_1_i2v_14b_720P_2gpu, zimage_image_t2i_2_gpus, zimage_image_t2i_2_gpus_non_square, flux_image_t2i_2_gpus, flux_2_image_t2i_2_gpus, ltx_2.3_one_stage_ti2v.

Failure Summary

The job has two distinct real failures plus a cascade. (1) flux2_modelopt_fp8_tp2_t2i reached inference and failed in torch._scaled_mm (FP8 GEMM) with HIPBLAS_STATUS_NOT_SUPPORTED during warmup denoising. (2) joy_echo_t2v_2gpu hung during server startup because the jdopensource/JoyAI-Echo model was not in the HF cache and the HF-Hub download stalled past the ~20-min startup timeout; the killed server left a child process holding scheduler port 5555, so the 16 subsequent testcases all ERRORed with Scheduler port 5555 is unavailable under --strict-ports (68 occurrences in the log).

Regression Status

Known recurring failures (both genuine failures + the cascade are pre-existing).

Recent history of test_diffusion_generation[flux2_modelopt_fp8_tp2_t2i] (tracked across shards; test-to-shard mapping is dynamic), same branch/event, completed runs only:

Date	Run	Job	Test Status	Detail
Jul 11	29162225443	shard 0 job	❌ Failed	HIPBLAS_STATUS_NOT_SUPPORTED (reached inference)
Jul 9	29039527644	shard 0 job	❌ Error	startup-cascade victim; never reached inference (HIPBLAS not observable)
Jul 8	28964460204	shard 1 job	❌ Failed	HIPBLAS_STATUS_NOT_SUPPORTED, identical torch._scaled_mm trace (29 hits)

No green run of this test reaching inference found in the queryable window (Jul 7–11); whenever it reaches denoising it HIPBLAS-fails, otherwise it is a cascade victim.

Recent history of test_diffusion_generation[joy_echo_t2v_2gpu] (cascade root), same branch/event, completed runs only:

Date	Run	Job	Test Status	Detail
Jul 11	29162225443	shard 0 job	❌ Error	startup hang → timeout (model not cached)
Jul 9	29039527644	shard 0 job	❌ Error	startup hang → timeout, port cascade (68 hits)
Jul 8	28964460204	shard 0 job	❌ Error	startup hang → timeout, port cascade (38 hits)
Jul 7	28888292440	shard 0 job	❌ Error	startup hang → timeout, port cascade (38 hits)

joy_echo_t2v_2gpu fails in every completed run in the window (Never-passed in queryable history, lookback Jul 7–11).

Failure Cluster

Two clusters: (A) "FP8 torch._scaled_mm HIPBLAS_STATUS_NOT_SUPPORTED on ROCm 7.2" (flux2), and (B) "Leaked scheduler port 5555 after joy_echo HF-download startup timeout" (joy_echo + 16 cascade ERRORs).

Facts (verified observations)

• flux2_modelopt_fp8_tp2_t2i stack: denoising.py → flux_2.py:77 _get_qkv_projections → linear.py:427 → modelopt_quant.py:432 apply_fp8_linear → fp8_utils.py:1891 torch._scaled_mm → RuntimeError: HIPBLAS_STATUS_NOT_SUPPORTED (verbatim, log ~line 4671). Confirmed torch._scaled_mm at python/sglang/srt/layers/quantization/fp8_utils.py:1891 (per-tensor GEMM path).
• Same HIPBLAS failure with identical apply_fp8_linear/torch._scaled_mm trace on Jul 8 run 28964460204 (29 occurrences) — not new today.
• joy_echo last log line before the 20-min gap: Model not found in cache, will download from HF Hub → Downloading model snapshot from HF Hub for jdopensource/JoyAI-Echo (attempt 1/5)... (23:57:30) → FAILED during server startup (00:17:03). ~20 min ≈ startup timeout.
• After joy_echo, the very next test (wan2_1_t2v_1.3b_cfg_parallel) fails with RuntimeError: Scheduler port 5555 is unavailable and --strict-ports is enabled (log line 7929); this message repeats 68× for all remaining tests.
• AITER_COMMIT_OVERRIDE is empty → this is a regular pr-test-amd-rocm720.yml scheduled run, not an AITER Scout run.

Hypothesised Causes (with confidence)

• [HIGH] flux2 failure: hipBLASLt on ROCm 7.2 / gfx942 (MI325) does not support the FP8 e4m3fn torch._scaled_mm configuration used by the FLUX.2 ModelOpt-FP8 checkpoint.
  • Supporting: error returns directly from torch._scaled_mm in apply_fp8_linear; AMD-only symptom; recurs on multiple days with identical trace; an open AMD PR (#28889) titled exactly "Fix FP8 e4m3fn GEMM on ROCm ... (fixes flux2 modelopt-fp8 HIPBLAS_STATUS_NOT_SUPPORTED)" confirms the code path.
  • Disconfirming: (none found) — the trace is unambiguous.
  • How to verify: apply #28889 on a branch and re-run only [flux2_modelopt_fp8_tp2_t2i] on MI325 rocm720.
• [HIGH] joy_echo failure + cascade: an infrastructure/test-harness issue — jdopensource/JoyAI-Echo is not pre-cached, the HF-Hub download stalls past the server-startup timeout, and on timeout the server's child process is not fully reaped, leaving scheduler port 5555 bound; every later --strict-ports startup then fails.
  • Supporting: Model not found in cache … Downloading … (attempt 1/5) immediately precedes the 20-min gap; first post-timeout test fails on port 5555; 68 identical port errors; same pattern in all 4 prior runs.
  • Disconfirming: cannot see inside the child-process teardown from this log alone; the stall could also be network/HF-Hub slowness on the day rather than a genuine hang — but the every-run recurrence points to a missing-cache/cleanup problem, not transient network.
  • How to verify: pre-cache jdopensource/JoyAI-Echo into /sgl-data/hf-cache on the runner (or mark the test to require a cached model), re-run; separately, confirm the server-startup-timeout path kills the whole process group so port 5555 is released.

In-flight Fix Check

• flux2 HIPBLAS: ✅ #28889 "[AMD] Fix FP8 e4m3fn GEMM on ROCm: dequant->bf16 fallback in apply_fp8_linear (fixes flux2 modelopt-fp8 HIPBLAS_STATUS_NOT_SUPPORTED)" — open since 2026-06-22, last updated 2026-07-11, not a draft. Chase reviewers; do NOT open a duplicate. (query: is:pr is:open flux2 fp8)
• joy_echo startup / port 5555 cascade: ❌ none found — needs a new PR or an infra fix (pre-cache the model + ensure process-group teardown on startup timeout). (queries: is:pr is:open JoyAI Echo → 0; is:pr is:open strict-ports scheduler port → no relevant match.)

Suggested Triage Steps

• Verify #28889 merges cleanly and re-run [flux2_modelopt_fp8_tp2_t2i] on MI325 rocm720 to confirm it clears cluster A.
• For cluster B (bigger blast radius — 17 of 18 failures): pre-cache jdopensource/JoyAI-Echo on the rocm720 runners (or gate the testcase behind a cache-present check) so its startup no longer times out; this alone should recover the 16 cascade victims.
• Independently harden the diffusion server-test harness: on startup timeout, kill the whole process group (or free scheduler port 5555) before starting the next case, so one hung model can't poison the rest of the shard. Consider retiring --strict-ports for these serial tests or randomizing the scheduler port.
• If the JoyAI-Echo download is genuinely slow rather than absent-from-cache, raise the server-startup timeout for that one model or move it to a nightly-only suite.

Status

• Persistence: joy_echo_t2v_2gpu + port-5555 cascade — failing every completed run Jul 7–11 (Never-passed in window). flux2_modelopt_fp8_tp2_t2i HIPBLAS — recurring whenever it reaches inference (observed Jul 8 and Jul 11).
• Scope: this job multimodal-gen-test-2-gpu-amd-rocm720 (shard 0); the joy_echo/port-cascade pattern also appears on the same job in prior runs; flux2 HIPBLAS roams across shards. AMD rocm720 only (AITER_COMMIT_OVERRIDE empty → not a scout run).
• Blocked work: 2-GPU diffusion server CI signal on AMD ROCm 7.2 — 18/25 assigned cases red, of which 16 are collateral from one hung model.
• In-flight fix: flux2 → ✅ #28889 (open, awaiting merge); joy_echo/cascade → ❌ none.

multimodal-gen-test-2-gpu-amd-rocm720 (linux-mi325-2gpu-sglang, 2) — failed step(s): Run diffusion server tests (2-GPU)

Commit Info

• sglang: 9b4bb415
• aiter: 9127c94a — v0.1.17.dev110+g9127c94a1 (source: dev image preserved; [CI-AITER-CHECK] reported Dev/patched version detected → skipping rebuild)

Failed Tests

Test File	Test Function	Error	Log
python/sglang/multimodal_gen/test/single_test_file/test_ar_models.py	TestGLMImage::test_generates_image (fails at setUpClass)	RuntimeError: AR model failed to start for glmimage → root OSError/HFValidationError: Repo id must be in the form 'namespace/repo_name': 'zai-org/GLM-Image/vision_language_encoder/'	Log

The job itself ended with ##[error]The action 'Run diffusion server tests (2-GPU)' has timed out after 150 minutes (step 9, line 5794).

Failure Summary

The GLM-Image AR-stage server never starts: sglang serve --model-path zai-org/GLM-Image/vision_language_encoder/ passes a 3-segment string that HuggingFace rejects as an invalid repo id. The test harness retries the cluster launch 3× per pytest cycle and pytest re-runs the cycle repeatedly, so no test ever completes and the job hits the 150-minute step timeout. multimodal-gen-unit-test in the same workflow passed; only the 2-GPU (and 1-GPU) diffusion server shards are affected.

Regression Status

Never-passed (test is new; it has failed in every scheduled run since it was introduced).

Recent history of test_ar_models.py (TestGLMImage::test_generates_image) in job multimodal-gen-test-2-gpu-amd-rocm720, same branch (main), same event (schedule), completed runs only:

Date	Run	Job	Test File Status	Failed Function	Error
Jul 11	29162225443	shard 2	❌ Failed	TestGLMImage::test_generates_image	invalid HF repo id → timeout
Jul 09	29039527644	shard 2	❌ Failed	TestGLMImage::test_generates_image	invalid HF repo id (identical)
Jul 08	28964460204	shard 2	⚪ N/A	—	test did not exist yet

The test file was added 7aab39a1 (2026-07-09, PR #25381 "[Diffusion] SGLang backend for GLM Image AR. Step 1"). The first scheduled run after that commit (Jul 09) already fails with the byte-identical error, so there is no known passing run for this test on AMD ROCm 7.2. Lookback window: 2026-07-04 → 2026-07-11 (the 07-10 run 29113205616 is still queued, excluded).

Failure Cluster

Invalid HF repo id from concatenated GLM-Image model subpath during server startup

Facts (verified observations)

• Verbatim error (repeated every attempt): OSError: Repo id must be in the form 'repo_name' or 'namespace/repo_name': 'zai-org/GLM-Image/vision_language_encoder/'. Use repo_type argument if needed. (line 4490).
• Startup stack trace originates in server_args.py → ModelConfig.from_server_args → config.py:84 _try_load_longcat_config → config.py:63 PretrainedConfig.get_config_dict(model) (the frame that raises), which calls transformers cached_files → HFValidationError.
• The model path is built by the test itself: test_ar_models.py:60 → f"{self.model}/vision_language_encoder/", with model = DEFAULT_AR_MODEL_NAME_FOR_TEST = "zai-org/GLM-Image" (test_utils.py:154). Concatenation yields the invalid 3-segment repo id.
• Reproducibility: identical failure on shards 0, 1, 2 of the 2-GPU job in the current run, and on the 07-09 run — deterministic, not flaky.
• The failure is at setUpClass, so test_generates_image never executes; the actual assertion body is never reached.
• aiter version is a preserved dev build, unchanged between the 07-09 and 07-11 runs (both use the Dockerfile default image; not an override) — so aiter is not a variable here.

Hypothesised Causes (with confidence)

• [HIGH] Test constructs an invalid HuggingFace repo id by concatenating the repo id with subfolders (zai-org/GLM-Image + /vision_language_encoder/), and --model-path only accepts a valid namespace/repo id or an existing local directory.
  • Supporting: the exact rejected string in the log equals f"{self.model}/vision_language_encoder/" from test_ar_models.py:60; HF's HFValidationError is raised on repo-id format validation, before any download; failure is byte-identical across runs and shards.
  • Disconfirming: none found — the traceback points directly at the constructed path. (Whether this ever passed elsewhere, e.g. NVIDIA where the model may be pre-downloaded to a local path so the string resolves as a real directory instead of a repo id, was not checked and would explain why the same test can be green on other backends.)
  • How to verify: on an AMD MI325 ROCm 7.2 container, run sglang serve --model-path zai-org/GLM-Image/vision_language_encoder/; confirm it reproduces the HFValidationError. Then check whether the GLM-Image weights are expected to be pre-staged to a local cache path on the diffusion runners (the test likely assumes a local checkout so the multi-segment path is a filesystem directory, not a repo id).
• [LOW] The _try_load_longcat_config probe (config.py:62, calls get_config_dict unconditionally before the normal AutoConfig path) surfaces the error.
  • Supporting: it is the top raising frame.
  • Disconfirming: AutoConfig.from_pretrained on the same string would fail identically — the probe is not the cause, only the messenger; the invalid path is the real issue.
  • How to verify: not actionable; addressing the path (or pre-staging the model locally) resolves it regardless of the probe.

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs for GLM-Image vision_language_encoder (2 hits: #30683 NPU dynamic batching, #20359 VLE offload — neither touches the model-path/repo-id construction) and glmimage AR (0 hits).

Suggested Triage Steps

• Confirm whether GLM-Image weights are meant to be pre-downloaded to a local directory on the ROCm 7.2 diffusion runners; if the runner is missing the local checkout, the concatenated path is treated as a remote repo id and fails — this may be a runner/model-staging gap rather than a code bug.
• If a code fix is intended, the model/tokenizer paths in test_ar_models.py:60,62 need to reference a valid local path or use the HF subfolder= mechanism instead of string concatenation — verify with a single sglang serve repro before changing.
• Independently, consider why the failing setup retries long enough to hit the 150-minute timeout (3 attempts × repeated pytest cycles) — a fast-fail on unrecoverable startup errors would surface this test-level failure in minutes instead of timing out the whole job.
• Since this is Never-passed and new (PR #25381, 2 days old), consider disabling TestGLMImage on AMD ROCm 7.2 until the model path / staging is resolved, to stop the 150-min job timeout from masking other diffusion signal.

Status

• Persistence: Failing since introduced 2026-07-09 (PR #25381); every completed scheduled run since has failed with the identical error. No green run exists.
• Scope: multimodal-gen-test-2-gpu-amd-rocm720 shards 0, 1, 2 in the current run; the multimodal-gen-test-1-gpu shards in the same and prior runs also show failures. Confined to pr-test-amd-rocm720.yml diffusion server tests; the unit-test job passes.
• Impact: The unrecoverable startup error is retried until the 150-minute step timeout, consuming a full 2-GPU runner slot per run and blocking all GLM-Image AR diffusion signal on AMD ROCm 7.2.
• In-flight fix: ❌ none found in open PRs.

stage-b-test-large-8-gpu-mi35x-disaggregation-amd-rocm720 (linux-mi35x-gpu-8.fabric) — failed step(s): Run test_disaggregation

Commit Info

• sglang: 96a04cb1 (Fix DeepEP CI test registration, #30873)
• aiter: 9127c94a v0.1.17.dev110+g9127c94a1 (source: Dockerfile default, AITER_COMMIT_OVERRIDE empty)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/disaggregation/test_disaggregation_basic.py	(whole file — server-based e2e, crashed before/after a subtest)	HW Exception … reason: GPU Hang → process aborted, exit code 134 (SIGABRT); retry aborted with RuntimeError: No accelerator … is available	Log

The remaining 3 files in this suite (test_disaggregation_pp.py, test_mori_transfer_engine_e2e.py, test_nixl_transfer_engine_e2e.py) were never reached — the retry of test_disaggregation_basic.py hard-crashed the runner (GPU became undetectable) and terminated the job.

Failure Summary

The 8-GPU MI35x disaggregation suite aborted while running test_disaggregation_basic.py. The first attempt progressed all the way through the gsm8k eval (200/200) and the retraction test, then the ROCm runtime emitted HW Exception by GPU node-3 … reason: GPU Hang and the process died with exit code 134 (SIGABRT). The automatic retry hit the same GPU Hang on node-2 and then could not even initialize — get_device() raised RuntimeError: No accelerator (CUDA, XPU, HPU, NPU, MUSA, MPS) or platform plugin is available, indicating the physical GPU was left in an unusable state.

Regression Status

Known recurring failure on this job (intermittent, symptom varies day-to-day). This specific GPU Hang / exit-134 signature is new vs. Jul 11, but the job has been red since Jul 9.

Recent history of test_disaggregation_basic.py in job stage-b-test-large-8-gpu-mi35x-disaggregation-amd-rocm720 (branch=main, event=schedule, completed runs only):

Date	Run	Job	Test File Status	Failed Function	Error
Jul 12	run	job	❌ Failed	whole file	GPU Hang → exit 134; retry No accelerator
Jul 11	run	job	❌ Failed	whole file	RuntimeError: No accelerator … available at job start (exit 1)
Jul 9	run	job	❌ Failed	whole file	ImportError: cannot import name 'get_attention_cp_size' (different cluster)
Jul 8	run	job	✅ Passed	—	—

First observed failure of this file: Jul 9. Last known pass: Jul 8 (8d2b66fd). Note the Jul 9 failure was a genuine but different code issue (get_attention_cp_size ImportError), not the GPU-hang symptom seen Jul 11–12. (Jul 10 run 29113205616 was still queued/excluded.)

Failure Cluster

GPU Hang (HW Exception) on MI35x 8-GPU fabric disaggregation runner → SIGABRT / "No accelerator available" on retry.

Facts (verified observations, no interpretation)

• HW Exception by GPU node-3 … reason :GPU Hang immediately follows a green gsm8k eval (100%|██████████| 200/200) and First attempt failed with exit code 134 (log:14116-14118).
• Retry (HF_HUB_OFFLINE=1) hit HW Exception by GPU node-2 … GPU Hang (log:14160), then RuntimeError: No accelerator … is available from get_device() at python/sglang/srt/utils/common.py:900 during ServerArgs.__post_init__ (log:14167-14195); job ended exit code 134.
• exit 134 = 128 + 6 = SIGABRT.
• aiter unchanged (Dockerfile default 9127c94a, no override); this is pr-test-amd-rocm720.yml, not an AITER Scout run.
• Same runner class showed the identical No accelerator available crash at job start on Jul 11 with a different sglang SHA (9b4bb415); on that run test_mori_transfer_engine_e2e.py passed after basic failed, i.e. the GPU recovered mid-job.
• The failure is non-deterministic within a single job: on Jul 12 the crash happened deep into basic (after full gsm8k + retraction), not at import/startup.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [MEDIUM] Infrastructure / hardware GPU hang on the MI35x 8-GPU fabric runner (not an sglang code regression).
  • Supporting: GPU Hang is a ROCm driver-emitted HW Exception, not an sglang assertion; the test ran fully green (gsm8k 200/200) before hanging; the retry left the GPU undetectable (No accelerator), a classic bricked-device signature; the same runner showed the same signature Jul 11 on a different SHA; test_mori passed on Jul 11 after basic failed, proving intermittent hardware recovery; there is an active DEBUG PR isolating a faulty NIC on this exact runner (#29127).
  • Disconfirming: Jul 8 passed cleanly, so the runner is not permanently broken — consistent with flakiness but does not fully rule out a code path that occasionally triggers the hang.
  • How to verify: re-run the job; check runner/GPU dmesg + rocm-smi for XGMI/HBM ECC or NIC faults during the hang window; correlate with #29127 NIC-isolation findings.
• [LOW] An sglang commit in the 8d2b66fd..96a04cb1 window introduced a code path that hangs the GPU during disaggregation.
  • Supporting: last clean pass was Jul 8; window is bounded.
  • Disconfirming: no code-level evidence links any commit to the GPU Hang; symptom is a driver-level fault; failure is non-deterministic and recovered mid-job on Jul 11; the intervening Jul 9 red was an unrelated get_attention_cp_size ImportError (already resolved), not this symptom.
  • How to verify: only pursue if re-runs consistently reproduce; then bisect the window on the MI35x disaggregation runner.

In-flight Fix Check

• In-flight fix: ✅ #29127 "[DEBUG] Isolate faulty RDMA NIC (rdma5) on MI35x disaggregation runner" — directly targets hardware faults on this runner; track its findings.
• In-flight fix: ✅ #30431 "[AMD] ci: self-heal driver-level VRAM leaks + graceful GPU cleanup on shutdown" — addresses the driver-state / GPU-cleanup class that produces the No accelerator retry crash.
• (Searched is:pr is:open with keywords GPU Hang mi35x, disaggregation mi35x, disaggregation GPU hang.)

Suggested Triage Steps

• Re-run the job once; if it passes, this confirms the intermittent-hardware hypothesis (the [MEDIUM] cluster).
• Inspect the MI35x fabric runner during the next hang: rocm-smi, dmesg for XGMI/HBM ECC/NIC errors in the 18:02 UTC window; feed results into #29127.
• If re-runs reproduce deterministically on the same SHA, bisect 8d2b66fd..96a04cb1 on the MI35x disaggregation runner to test the [LOW] code-regression hypothesis.
• Consider chasing #30431 to merge, so a bricked GPU triggers graceful cleanup instead of a bare No accelerator abort that masks the real hang.

Status

• Persistence: test_disaggregation_basic.py failing since Jul 9 (3 consecutive completed runs), but with two distinct causes — Jul 9 was an unrelated ImportError; the GPU-Hang/exit-134 signature appears Jul 11–12.
• Scope: single job stage-b-test-large-8-gpu-mi35x-disaggregation-amd-rocm720 on the MI35x 8-GPU fabric runner; the crash also blocks the other 3 disaggregation test files in the suite (never reached).
• Blocked work: full MI35x disaggregation (PD) CI signal on rocm720.
• In-flight fix: ✅ #29127 (runner NIC isolation) and ✅ #30431 (GPU self-heal/cleanup) — both address this failure class; no new PR needed until re-runs indicate a code cause.

---

Generated by amd-bot using Claude Code CLI (last updated: 2026-07-12 23:21 UTC)

[amd-bot]

nightly-test-amd.yml — 47 failure(s)

Run started (UTC): 2026-07-11 17:47 UTC
Last scanned (UTC): 2026-07-12 23:22 UTC
Commits: sglang 96a04cb

Summary

Counts: 47 failures · 5 clusters

#	Cluster	Job	Test File	Test Function	Status	Hypothesis (confidence)
1	MoRI gtest_discover build break (cmake 4.4)	nightly-2-gpu-mi35x-glm51-mxfp4	N/A (install step)	N/A	new today	[HIGH] cmake 4.4 → PR #30942
2	MoRI gtest_discover build break (cmake 4.4)	nightly-2-gpu-mi35x-deepseek-r1-mxfp4-tp2	N/A (install step)	N/A	new today	[HIGH] cmake 4.4 → PR #30942
3	MoRI gtest_discover build break (cmake 4.4)	nightly-test-1-gpu-mi35x	N/A (install step)	N/A	new today	[HIGH] cmake 4.4 → PR #30942
4	MoRI gtest_discover build break (cmake 4.4)	nightly-accuracy-8-gpu-mi35x-deepseek-v32	test_deepseek_v32_eval_mi35x.py	crash on load	GPU-Hang 3d; today MoRI	[HIGH] cmake 4.4 → PR #30942
5	MoRI gtest_discover build break (cmake 4.4)	nightly-8-gpu-mi35x-qwen35-triton-dcp	N/A (install step)	N/A	new today	[HIGH] cmake 4.4 → PR #30942
6	MoRI gtest_discover build break (cmake 4.4)	nightly-accuracy-8-gpu-mi35x-deepseek-v32-mtp	N/A (install step)	N/A	new today	[HIGH] cmake 4.4 → PR #30942
7	MoRI gtest_discover build break (cmake 4.4)	nightly-perf-8-gpu-mi35x-deepseek-v32-basic	N/A (install step)	N/A	new today	[HIGH] cmake 4.4 → PR #30942
8	MoRI gtest_discover build break (cmake 4.4)	nightly-8-gpu-mi35x-deepseek-r1-mxfp4	N/A (install step)	N/A	new today	[HIGH] cmake 4.4 → PR #30942
9	MoRI gtest_discover build break (cmake 4.4)	nightly-8-gpu-mi35x-deepseek-r1-mxfp4-tp4	test_deepseek_r1_mxfp4_tp4_mtp_mi35x.py	crash on load	GPU-Hang + MoRI (2 bugs)	[HIGH] cmake 4.4 → PR #30942
10	MoRI gtest_discover build break (cmake 4.4)	nightly-4-gpu-mi35x-minimax-m25	N/A (install step)	N/A	new today	[HIGH] cmake 4.4 → PR #30942
11	MoRI gtest_discover build break (cmake 4.4)	nightly-perf-8-gpu-mi35x-deepseek-v32-mtp	test_deepseek_v32_mtp_perf_mi35x.py	crash on load	GPU-Hang 3d (not MoRI today)	[LOW] DSA/MoE window bisect
12	MoRI gtest_discover build break (cmake 4.4)	nightly-8-gpu-mi35x-kimi-k26	N/A (install step)	N/A	new today	[HIGH] cmake 4.4 → PR #30942
13	MoRI gtest_discover build break (cmake 4.4)	[nightly-8-gpu-mi35x-glm5-mxfp4](https://github.com/sgl-project/sglang/actions

---

Job	Failed Steps	Started
nightly-2-gpu-mi35x-deepseek-r1-mxfp4-tp2	Accuracy Test MI35x (2-GPU DeepSeek-R1-MXFP4 TP2)	2026-07-11T17:47
nightly-test-1-gpu-mi35x	Nightly Test MI35x (1-GPU)	2026-07-11T17:47
nightly-accuracy-2-gpu	N/A	2026-07-11T21:15
nightly-8-gpu-mi35x-grok2	Accuracy Test MI35x (8-GPU Grok2)	2026-07-11T18:38
nightly-2-gpu-mi35x-glm51-mxfp4	Accuracy Test MI35x (2-GPU GLM-5.1-MXFP4 GSM8K)	2026-07-11T17:47
nightly-accuracy-2-gpu-vlm	Nightly Accuracy Test (2-GPU VLM MMMU)	2026-07-11T21:22
nightly-8-gpu-mi35x-deepseek-r1-mxfp4-tp4	Accuracy Test MI35x (DeepSeek-R1-MXFP4 TP4 MTP), Accuracy Test MI35x (DeepSeek-R1-MXFP4 TP4)	2026-07-11T18:48
nightly-accuracy-8-gpu-mi35x	Accuracy Test MI35x (8-GPU GPT-OSS)	2026-07-11T18:54
nightly-accuracy-8-gpu-mi35x-deepseek-v32-mtp	Accuracy Test MI35x (8-GPU DeepSeek-V3.2 TP+MTP)	2026-07-11T18:52
nightly-8-gpu-mi35x-deepseek-r1-mxfp4-kv-fp8	Accuracy Test MI35x (8-GPU DeepSeek-R1-MXFP4 KV FP8)	2026-07-11T19:02
nightly-8-gpu-mi35x-kimi-k26	Accuracy Test MI35x (8-GPU Kimi-K2.6)	2026-07-11T18:50
nightly-8-gpu-mi35x-deepseek-r1-hicache	Accuracy Test MI35x (8-GPU DeepSeek-R1 HiCache)	2026-07-11T19:16
nightly-4-gpu	Nightly Test (4-GPU)	2026-07-11T17:49
nightly-accuracy-8-gpu-mi35x-deepseek-v32	Accuracy Test MI35x (8-GPU DeepSeek-V3.2)	2026-07-11T18:59
nightly-8-gpu-mi35x-grok1-int4	Accuracy Test MI35x (8-GPU Grok1-INT4)	2026-07-11T18:55
nightly-8-gpu-mi35x-qwen3-235b-mxfp4	Accuracy Test + Performance Test MI35x (8-GPU Qwen3-235B-MXFP4)	2026-07-11T19:05
nightly-4-gpu-mi35x-minimax-m25	Accuracy Test MI35x (4-GPU MiniMax-M2.5)	2026-07-11T19:06
nightly-perf-8-gpu-mi35x-deepseek-v32-mtp	Performance Test MI35x (8-GPU DeepSeek-V3.2 MTP)	2026-07-11T19:16
nightly-8-gpu-mi35x-glm51	Accuracy Test MI35x (8-GPU GLM-5.1 DSA)	2026-07-11T19:04
nightly-8-gpu-mi35x-qwen35-triton-dcp	Accuracy Test MI35x (8-GPU Qwen 3.5 Triton DCP)	2026-07-11T19:10
nightly-8-gpu-mi35x-qwen35	Accuracy Test MI35x (8-GPU Qwen 3.5)	2026-07-11T19:18
nightly-8-gpu-mi35x-glm5-mxfp4	Accuracy Test MI35x (8-GPU GLM-5-MXFP4)	2026-07-11T19:10
nightly-perf-8-gpu-mi35x-deepseek-v32-basic	Performance Test MI35x (8-GPU DeepSeek-V3.2 Basic)	2026-07-11T19:13
nightly-2-gpu-mi35x-glm51-mxfp4	Install dependencies	2026-07-12T17:48
nightly-2-gpu-mi35x-deepseek-r1-mxfp4-tp2	Install dependencies	2026-07-12T17:48
nightly-test-1-gpu-mi35x	Install dependencies	2026-07-12T17:48
nightly-accuracy-8-gpu-mi35x-deepseek-v32	Accuracy Test MI35x (8-GPU DeepSeek-V3.2)	2026-07-12T18:40
nightly-8-gpu-mi35x-qwen35-triton-dcp	Install dependencies	2026-07-12T18:46
nightly-accuracy-8-gpu-mi35x-deepseek-v32-mtp	Install dependencies	2026-07-12T19:04
nightly-perf-8-gpu-mi35x-deepseek-v32-basic	Install dependencies	2026-07-12T19:00
nightly-8-gpu-mi35x-deepseek-r1-mxfp4	Install dependencies	2026-07-12T19:08
nightly-8-gpu-mi35x-deepseek-r1-mxfp4-tp4	Accuracy Test MI35x (DeepSeek-R1-MXFP4 TP4 MTP), Install dependencies	2026-07-12T18:44
nightly-4-gpu-mi35x-minimax-m25	Install dependencies	2026-07-12T18:55
nightly-perf-8-gpu-mi35x-deepseek-v32-mtp	Performance Test MI35x (8-GPU DeepSeek-V3.2 MTP)	2026-07-12T18:52
nightly-8-gpu-mi35x-kimi-k26	Install dependencies	2026-07-12T18:51
nightly-8-gpu-mi35x-glm5-mxfp4	Install dependencies	2026-07-12T19:00
nightly-accuracy-8-gpu-mi35x	Install dependencies	2026-07-12T18:51
nightly-8-gpu-mi35x-grok1-int4	Install dependencies	2026-07-12T18:46
nightly-8-gpu-mi35x-grok2	Install dependencies	2026-07-12T19:04
nightly-8-gpu-mi35x-deepseek-r1-mxfp4-ar-fusion	Install dependencies	2026-07-12T19:09
nightly-8-gpu-mi35x-glm51	Install dependencies	2026-07-12T19:12
nightly-8-gpu-mi35x-qwen3-235b-mxfp4	Install dependencies	2026-07-12T19:08
nightly-8-gpu-mi35x-qwen35	Install dependencies	2026-07-12T19:14
nightly-8-gpu-mi35x-deepseek-r1-hicache	Install dependencies	2026-07-12T19:13
nightly-8-gpu-mi35x-deepseek-r1-mxfp4-kv-fp8	Install dependencies	2026-07-12T19:13
nightly-accuracy-2-gpu	N/A	2026-07-12T21:01
nightly-accuracy-2-gpu-vlm	Nightly Accuracy Test (2-GPU VLM MMMU)	2026-07-12T22:11

Per-Job Analysis

nightly-2-gpu-mi35x-deepseek-r1-mxfp4-tp2 — failed step(s): Accuracy Test MI35x (2-GPU DeepSeek-R1-MXFP4 TP2)

Commit Info

• sglang: 9b4bb415d
• aiter: 9127c94a1 (v0.1.17.dev110+g9127c94a1) — source: dev image ([CI-AITER-CHECK] Dev/patched version detected → skipping rebuild, matches Dockerfile gfx950 default)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/accuracy/mi35x/test_deepseek_r1_mxfp4_tp2_mi35x.py	TestDeepSeekR1MXFP4TP2MI35x.setUpClass (server launch)	HW Exception ... reason :GPU Hang → Fatal Python error: Aborted → exit 134	Log

Failure Summary

The single accuracy test for DeepSeek-R1-MXFP4 (TP2) on MI35x never reached execution — the server subprocess aborted during model startup. On the first attempt the crash occurred at 70% of Multi-thread loading shards (GPU node-2); the automatic retry with HF_HUB_OFFLINE=1 crashed ~17s into startup (GPU node-3). Both aborts are HW Exception ... reason :GPU Hang → Fatal Python error: Aborted → exit code 134. The main-thread stack is in weight loading (model_runner.py:1455 load_model ← initialize:707 ← __init__:583 ← tp_worker.py ← scheduler.py).

Regression Status

Known recurring failure — but the symptom differs each failing day, so this is an unstable/recurring failure rather than one clean regression.

Recent history of test_deepseek_r1_mxfp4_tp2_mi35x.py (setUpClass) in job nightly-2-gpu-mi35x-deepseek-r1-mxfp4-tp2 (branch main, event=schedule, completed runs only):

Date	Run	Test File Status	Failed Function	Error
Jul 11	29162254653	❌ Failed	setUpClass	GPU Hang during weight load (70%) → exit 134
Jul 10	29113206892	❌ Failed	setUpClass	GPU Hang after 100% load, stack in fused_moe_triton/layer.py → exit 134
Jul 09	29039566002	❌ Failed	setUpClass	ImportError: cannot import name 'get_attention_cp_size' + watchdog timeout → server exit -9
Jul 08	28964511688	✅ Passed	—	—
Jul 05	28749559519	✅ Passed	—	—
Jul 04	28714602438	✅ Passed	—	—

First observed failure: Jul 09. Last known pass: Jul 08 (run 28964511688). Note: Jul 06/07 and Jul 09–11 workflow runs were largely cancelled at the workflow level; the per-job results above are the completed data points. The current workflow run is queued at the top level but this job completed with failure.

Failure Cluster

GPU Hang / HW Exception during DeepSeek-R1-MXFP4 model startup on MI35x (exit 134)

Facts (verified observations)

• Verbatim: HW Exception by GPU node-2 ... reason :GPU Hang then Fatal Python error: Aborted, First attempt failed with exit code 134 (log:4520–5300); retry ##[error]Process completed with exit code 134. at log:5341–5344.
• Both attempts crashed, on different GPU nodes (node-2 then node-3), at different startup phases (70% shard load vs ~17s pre-load) — the failure is not tied to a single shard/tensor.
• Stack (main thread) is entirely in the load path: model_runner.py:1455 load_model; no sglang compute frame present today.
• aiter version unchanged (dev image v0.1.17.dev110+g9127c94a1) — not rebuilt.
• The Jul 09 ImportError: get_attention_cp_size is not reproducible today: grep "def get_attention_cp_size" python/sglang/srt/layers/dp_attention.py returns nothing in the current tree (that symptom has been resolved and is a different failure from today's GPU hang).

Hypothesised Causes (with confidence + disconfirming evidence)

• [LOW] Runner/GPU-state (infra) hang, not a code regression.
  • Supporting: crash reproduces on two attempts on two different GPU nodes at two different phases; pure GPU Hang HW exception with no sglang compute frame; symptom is inconsistent across the three failing days.
  • Disconfirming: the job passed cleanly Jul 04/05/08 and has failed every completed run since Jul 09 — a persistent 3-day failure is more than a one-off flake, so a code/model/image change is not excluded.
  • How to verify: re-run this job alone on a known-good MI35x runner; check dmesg/GPU reset counters on the runner around 17:59–18:00 UTC.
• [LOW] sglang change in the Jul 08→Jul 09 window (8d2b66fd90..2e4d6368c3) affecting DeepSeek/MXFP4 load or MoE.
  • Supporting: last pass Jul 08, first fail Jul 09; Jul 10's crash stack referenced fused_moe_triton/layer.py.
  • Disconfirming: today's stack is only in generic weight loading, not MoE; the Jul 09 ImportError culprit is already gone yet the job still fails, so no single commit explains all three days.
  • How to verify: git log --oneline 8d2b66fd90..2e4d6368c3 and bisect for MXFP4/MoE/loader changes; re-run pinned to the Jul 08 pass SHA.

No hypothesis reaches MEDIUM; needs a runner-health check plus a bisect of 8d2b66fd90..2e4d6368c3.

In-flight Fix Check

• In-flight fix: ✅ #30431 "[AMD] ci: self-heal driver-level VRAM leaks + graceful GPU cleanup on shutdown" — directly targets AMD GPU-hang/VRAM-leak-on-shutdown, the symptom cluster here; worth landing/testing before opening a duplicate. (found via search q=repo:sgl-project/sglang is:pr is:open mxfp4 hang)
• In-flight fix: ⚠️ possibly related #30407 / #30355 — AMD DeepSeek MLA GPU-fault fixes on MI355; only relevant if the hang is in the MLA/attention path (not confirmed today).
• No open PR references test_deepseek_r1_mxfp4_tp2_mi35x specifically.

Suggested Triage Steps

• Re-run job nightly-2-gpu-mi35x-deepseek-r1-mxfp4-tp2 in isolation on a freshly-reset MI35x runner to separate infra hang from a code regression.
• Check the runner's GPU state/dmesg for a hardware reset around 2026-07-11 17:59–18:00 UTC (two hangs on nodes 2 and 3).
• Bisect 8d2b66fd90..2e4d6368c3 for MXFP4/MoE/model-loader changes; re-run pinned to Jul 08 pass SHA 8d2b66fd90 to confirm the window.
• Evaluate #30431 against this cluster; if it addresses the shutdown VRAM-leak/hang, chase reviewers rather than filing a new issue.

Status

• Persistence: Failing every completed run since 2026-07-09 (3 consecutive failing days; last pass 2026-07-08).
• Scope: 1 job / 1 test file (nightly-2-gpu-mi35x-deepseek-r1-mxfp4-tp2), MI35x runner; blocks the DeepSeek-R1-MXFP4 TP2 accuracy signal on AMD MI35x.
• Symptom stability: unstable — GPU hang during load (Jul 11), GPU hang in MoE forward (Jul 10), ImportError (Jul 09, already resolved in tree).
• In-flight fix: ✅ candidate #30431 (AMD GPU cleanup / VRAM-leak self-heal).

nightly-test-1-gpu-mi35x — failed step(s): Nightly Test MI35x (1-GPU)

Commit Info

• sglang: 9b4bb415ddc315ebcf1f8fcc280537f0b30938dc
• aiter: 9127c94a18e4398e1eba91f6639e910f0994ad02 (v0.1.17.dev110+g9127c94a1, source: dev image preserved — matches Dockerfile default, no rebuild)

Failed Tests

Test File	Test Function	Error	Log
test/registered/unit/batch_invariant_ops/test_batch_invariant_ops.py	N/A (process aborted before pytest result)	HW Exception by GPU node-2 … reason :GPU Hang → SIGABRT (exit 134)	Log

This is not a pytest assertion failure — the worker process was killed by a GPU hardware exception. The single enabled test (test_batch_invariant_ops.py) had only just begun executing when GPU node-2 hung.

Failure Summary

The nightly-test-1-gpu-mi35x job failed at step 6 ("Nightly Test MI35x (1-GPU)"). The suite nightly-amd-1-gpu-mi35x enabled exactly one test (test_batch_invariant_ops.py). During execution, GPU node-2 raised HW Exception … reason :GPU Hang, aborting the process with exit code 134 (SIGABRT). The launcher retried once with HF_HUB_OFFLINE=1; the retry hit the same GPU Hang and failed again. Total job runtime was ~6 min.

Regression Status

New failure mode today (first GPU-Hang occurrence for this test file in the queryable window). Not a code regression — the exact same test passed cleanly the prior night on a near-identical toolchain.

Recent history of test_batch_invariant_ops.py in job nightly-test-1-gpu-mi35x (branch main, event schedule, completed runs only):

Date	Run	Job	Test File Status	Failed Function	Error
Jul 11	run	nightly-test-1-gpu-mi35x	❌ Failed	— (GPU Hang, aborted)	GPU Hang → exit 134
Jul 10	run	nightly-test-1-gpu-mi35x	✅ Passed (1/1)	—	—
Jul 9	run	nightly-test-1-gpu-mi35x	⚠️ Not reached	—	job died at VRAM-clear step (exit 1) — different symptom
Jul 8	run	nightly-test-1-gpu-mi35x	✅ Passed	—	—
Jul 5	run	nightly-test-1-gpu-mi35x	⚠️ Not reached	—	job died at VRAM-clear step (exit 1) — different symptom
Jul 4	run	nightly-test-1-gpu-mi35x	✅ Passed	—	—
Jul 3	run	nightly-test-1-gpu-mi35x	✅ Passed	—	—
Jul 2	run	nightly-test-1-gpu-mi35x	⚠️ Not reached	—	job died at VRAM-clear step (exit 1) — different symptom

First observed GPU-Hang failure for this test file: 2026-07-11. Last known passing: 2026-07-10 (1/1). Note the prior job-level failures (Jul 9/5/2) are a distinct cluster — they aborted at the ensure_vram_clear.sh pre-test step ("VRAM cleanup unsuccessful after 3 attempts", exit 1) and never ran any test, so they are passing runs for this test file.

Failure Cluster

GPU Hang hardware exception during single-GPU MI35x test execution (SIGABRT / exit 134).

Facts (verified observations)

• ensure_vram_clear.sh reported VRAM clean at job start: GPU Memory Allocated (VRAM%): 0 and ✓ VRAM is already clear; skipping cleanup (step 3, line 131/140).
• Fault at line 4411: HW Exception by GPU node-2 (Agent handle: 0x615f118b59d0) reason :GPU Hang → First attempt failed with exit code 134.
• Retry with HF_HUB_OFFLINE=1 hit two more GPU Hang exceptions on node-2 (lines 4433-4434) → Process completed with exit code 134.
• Exit 134 = 128 + SIGABRT(6) — process aborted by hardware exception, not a test assertion.
• The same test file + same single-test suite passed 1/1 in 22 s the prior night (Jul 10, line 4457).
• aiter version unchanged (dev image v0.1.17.dev110+g9127c94a1, matches Dockerfile default) — verified via [CI-AITER-CHECK] markers (lines 4453-4458).
• A non-fatal Local docker registry pull failed; falling back to public registry occurred at setup — unrelated infra hiccup, image pulled successfully.

Hypothesised Causes (with confidence)

• [MEDIUM] Hardware/driver-level GPU Hang on the specific MI35x runner (node-2), not a code defect.
  • Supporting: fault is a ROCm HW exception (reason :GPU Hang) on a named agent, reproduced on immediate retry (same node stays wedged); the identical test+commit-context passed the previous night; VRAM was verified clean at start so it is not a leaked-allocation OOM; failure appears within seconds of process start (torch/model init), not deep in test logic.
  • Disconfirming: only one occurrence so far, so a transient device-state issue vs. a persistent bad runner cannot yet be distinguished; the test does run real GPU ops, so a genuine kernel-level hang in batch_invariant_ops cannot be fully excluded from a single log.
  • How to verify: re-run nightly-test-1-gpu-mi35x (or run only test_batch_invariant_ops.py) on the MI35x pool; if it passes, this was a transient/node-specific hang. If it recurs on the same physical node, cordon that runner. Check dmesg/rocm-smi on the runner for XGMI/ring-timeout entries.
• [LOW] Regression in batch_invariant_ops kernel path landing in main on Jul 11.
  • Supporting: this is the only enabled test and the hang happens during its execution.
  • Disconfirming: GPU-Hang HW exceptions are almost always device/driver-level, not Python-test-level; the test passed one day earlier; no code-level evidence links any recent commit to a hang.
  • How to verify: git log --oneline 7998fecfd114..9b4bb415ddc3 and check for changes under batch-invariant ops / attention; re-run to establish reproducibility before bisecting.

In-flight Fix Check

• In-flight fix: ⚠️ partial — #30431 "[AMD] ci: self-heal driver-level VRAM leaks + graceful GPU cleanup on shutdown" (open since 2026-07-07). It targets driver-level VRAM leaks + graceful GPU cleanup on the MI35x pool (branch amd/ci-vram-leak-selfheal). It directly addresses the Jul 9/5/2 VRAM-cleanup failures and its "graceful GPU cleanup" may reduce hung-GPU carryover, but it does not specifically claim to fix in-test GPU Hangs. Search ran: is:pr is:open GPU Hang mi35x (8 hits) and batch_invariant_ops AMD (0 hits).
• In-flight fix (test-specific): ❌ none found for a batch_invariant_ops GPU-hang.

Suggested Triage Steps

• Re-run the job (or just test_batch_invariant_ops.py) on the MI35x pool to establish reproducibility — a single transient GPU Hang is most likely infrastructure.
• If it recurs on the same physical runner, inspect that node's dmesg/rocm-smi for GPU ring/XGMI timeout, and cordon the runner.
• If it reproduces across multiple MI35x nodes, then treat as a code regression: git log --oneline 7998fecfd114..9b4bb415ddc3 and bisect commits touching batch-invariant-ops / attention kernels; run the isolated test in container rocm/sgl-dev:v0.5.15-rocm700-mi35x-20260710.
• Track #30431 — it should eliminate the separate Jul 9/5/2 VRAM-cleanup failure mode on this same job and may help GPU-hang recovery.

Status

• Persistence: First GPU-Hang failure for this test file (2026-07-11); test passed the prior night (Jul 10, 1/1). Reproduced on the in-job retry but not yet across separate runs.
• Scope: 1 job (nightly-test-1-gpu-mi35x) in nightly-test-amd.yml. The overall run 29162254653 is still queued/in progress; other MI35x jobs not assessed here.
• Blocked work: Blocks the single-GPU MI35x batch-invariant-ops nightly signal. Distinct from the recurring VRAM-cleanup failures (Jul 9/5/2) on the same job.
• In-flight fix: ⚠️ partial — #30431 (open since 2026-07-07) covers AMD GPU/VRAM cleanup; no test-specific fix for the GPU Hang.

nightly-accuracy-2-gpu — failed step(s): N/A

Commit Info

• sglang: 9b4bb415dd
• aiter: 9127c94a1 (v0.1.17.dev110+g9127c94a1, source: dev image detected, rebuild skipped — Dockerfile default matches)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/accuracy/mi30x/test_gsm8k_eval_amd.py	test_nightly_gsm8k_eval (model Qwen/Qwen3-4B, TP=1)	Status: ❌ FAILED — Score 0.914 but threshold: N/A (no threshold configured)	Log

Possible secondary (unconfirmed — log truncated): mistralai/Mixtral-8x7B-Instruct-v0.1 (TP=2) first-pass gsm8k score 0.291 vs threshold 0.69, entered the retry loop; final status not present in the retrieved log (log ends 22:01:53, job ended 22:20:35). See Facts.

Failure Summary

The nightly-accuracy-2-gpu job runs run_suite.py --suite nightly-amd, which drives the AMD gsm8k accuracy harness test/registered/amd/accuracy/mi30x/test_gsm8k_eval_amd.py. The model Qwen/Qwen3-4B is evaluated (TP=1), achieves a healthy gsm8k score of 0.914, but is marked ❌ FAILED because its accuracy threshold prints as N/A. The model score is fine; the failure is a missing threshold-config entry, which forces an automatic FAIL and also causes the eval to run 3× (the retry loop never satisfies its break condition), inflating eval time to 239.5 s.

Regression Status

New regression — first failing nightly run was Jul 10 (first scheduled run containing the culprit commit).

Recent history of test_gsm8k_eval_amd.py (Qwen/Qwen3-4B, TP=1) in job nightly-accuracy-2-gpu (branch main, event=schedule, completed jobs only):

Date	Run	Job	Test File Status	Failed Model	Error
Jul 11	29162254653	nightly-accuracy-2-gpu	❌ Failed	Qwen/Qwen3-4B	Score 0.914, threshold: N/A → FAILED
Jul 10	29113206892	nightly-accuracy-2-gpu	❌ Failed	Qwen/Qwen3-4B	Score 0.903, threshold: N/A → FAILED
Jul 09	29039566002	nightly-accuracy-2-gpu	➖ Not run	—	Qwen3-4B not in eval list (pre-merge); job died earlier with exit code 255 after Llama-3.1-8B

First observed (this failure): Jul 10 nightly. Last run where Qwen3-4B was not configured to fail: n/a — the model was only added to the AMD list on Jul 10. Lookback: last 3 completed scheduled runs.

Failure Cluster

Nightly gsm8k model marked FAILED due to missing accuracy threshold (threshold: N/A)

Facts (verified observations)

• Log (current run): Testing: Qwen/Qwen3-4B (TP=1, FP8=False) → Score: 0.914 (threshold: N/A) → Status: ❌ FAILED (step:6:12626). All other TP1/TP2 models passed with numeric thresholds: Llama-3.1-8B 0.816≥0.80 ✅, Qwen3-8B 0.916≥0.76 ✅, Llama-3.1-70B 0.928≥0.89 ✅.
• Source test/registered/amd/accuracy/mi30x/test_gsm8k_eval_amd.py:37-66: MODEL_SCORE_THRESHOLDS contains Qwen/Qwen3-8B (0.76) and Qwen/Qwen3-30B-A3B-Thinking-2507 (0.86) but no Qwen/Qwen3-4B entry.
• Source same file :273 threshold = MODEL_SCORE_THRESHOLDS.get(model) → None; :288 threshold_str = f"{threshold:.2f}" if threshold else "N/A"; :289 passed = threshold and score >= threshold → falsy → :301 prints Status: ❌ FAILED.
• Retry loop :274-283 for attempt in range(3): ... if threshold and score >= threshold: break — with threshold=None the break never fires, so the eval runs 3× (log shows Score 0.912, 0.912, 0.914; Evaluation 239.5 s vs ~80–105 s for single-pass models).
• Source python/sglang/test/test_utils.py:153-155: DEFAULT_MODEL_NAME_FOR_NIGHTLY_EVAL_TP1 = "meta-llama/Llama-3.1-8B-Instruct,Qwen/Qwen3-8B,Qwen/Qwen3-4B" — includes Qwen3-4B; consumed by the AMD harness at test_gsm8k_eval_amd.py:209.
• git show b76dd0be6 --stat: PR #27757 (b76dd0be6, "Fix Mistral GSM8K chat eval", merged 2026-07-10 04:08 UTC) changed test_utils.py to add Qwen/Qwen3-4B to the TP1 list; it did not touch test_gsm8k_eval_amd.py, leaving the AMD threshold dict without a Qwen3-4B entry.
• git merge-base --is-ancestor b76dd0be6 2e4d6368c3 → false: the Jul 09 run predates the merge, confirming Qwen3-4B was not evaluated pre-merge.
• aiter unchanged/irrelevant: AITER_COMMIT_OVERRIDE empty, dev image v0.1.17.dev110+g9127c94a1 matches Dockerfile default; failure is in Python test config, not a kernel path.
• Log truncation caveat: retrieved job log ends at 22:01:53 (during Mixtral gsm8k retry) though the job completed 22:20:35; Mixtral's final result is not in the retrieved log.

Hypothesised Causes

• [FACT] Qwen/Qwen3-4B is evaluated by the AMD nightly TP1 group but has no entry in MODEL_SCORE_THRESHOLDS, so the harness assigns threshold=None, prints threshold: N/A, and hard-codes the result to FAILED regardless of the (passing) 0.914 score.
  • Supporting: threshold dict verified to lack Qwen3-4B (test_gsm8k_eval_amd.py:37-66); pass/fail logic verified (:288-301); model present in eval list (test_utils.py:154); identical threshold: N/A → FAILED on Jul 10 (0.903) and Jul 11 (0.914).
  • Disconfirming: (none found) — the score exceeds every plausible Qwen3-4B threshold, so this is purely a config gap, not a real accuracy regression.
  • How to verify: add "Qwen/Qwen3-4B": <threshold> to MODEL_SCORE_THRESHOLDS and re-run; the model should report a numeric threshold and PASS.
• [HIGH] PR #27757 introduced the regression by adding Qwen3-4B to the shared DEFAULT_MODEL_NAME_FOR_NIGHTLY_EVAL_TP1 without adding the corresponding AMD threshold.
  • Supporting: commit diff adds Qwen3-4B to test_utils.py:154 and does not modify the AMD threshold file (git show b76dd0be6 --stat); first failing nightly (Jul 10) is the first scheduled run containing this commit; pre-merge Jul 09 did not test Qwen3-4B.
  • Disconfirming: the PR title targets a CUDA Mistral eval fix; the AMD breakage is a side effect of editing the shared constant, so intent was unrelated — but the causal chain to the AMD job is direct.
  • How to verify: git revert b76dd0be6 (or just re-add Qwen3-4B threshold) on a branch and re-run nightly-accuracy-2-gpu.
• [LOW] Mixtral-8x7B TP2 may be a separate genuine accuracy failure (first-pass 0.291 < 0.69).
  • Supporting: current-run first-pass gsm8k score 0.291 well below its 0.69 threshold; entered retry loop.
  • Disconfirming: 0.291 is likely a partial/first-attempt score; the retry loop may recover; final result is missing due to log truncation, so this is unconfirmed.
  • How to verify: fetch the complete job log (or GITHUB_STEP_SUMMARY artifact) and read the final Results for mistralai/Mixtral-8x7B-Instruct-v0.1 block.

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs for Qwen3-4B threshold (3 unrelated hits: #30204, #16638, #28045), MODEL_SCORE_THRESHOLDS (0 hits), nightly gsm8k threshold (no matching config-fix PR). No PR adds the missing Qwen3-4B AMD threshold.

Suggested Triage Steps

• Add "Qwen/Qwen3-4B": <threshold> to MODEL_SCORE_THRESHOLDS in test/registered/amd/accuracy/mi30x/test_gsm8k_eval_amd.py (measured AMD score ≈0.90–0.92 across Jul 10/11; a 5%-below value ≈0.85 matches the existing convention). Then re-run nightly-accuracy-2-gpu.
• Consider hardening the harness so a missing threshold is an explicit config error (e.g. SKIPPED/ERROR: no threshold) rather than silently FAILED — this also stops the wasted 3× retry when threshold is None.
• Separately, pull the complete log / step-summary for run 29162254653 to confirm whether mistralai/Mixtral-8x7B-Instruct-v0.1 (score 0.291 first-pass) is a real accuracy failure; if so, triage it independently (it has a valid threshold and is a different cluster).

Status

• Persistence: Failing every completed nightly run since Jul 10 (2 consecutive: Jul 10 + Jul 11).
• Scope: Qwen/Qwen3-4B in nightly-accuracy-2-gpu (nightly-test-amd.yml). The shared constant is also consumed by CUDA nightly eval paths, but only the AMD harness lacks the threshold, so impact is AMD-only for this symptom.
• Blocked work: Blocks a green AMD nightly accuracy signal; the failure masks whether other models in the same job (e.g. Mixtral) have genuine regressions.
• In-flight fix: ❌ none found — needs a one-line threshold addition (new PR).

nightly-8-gpu-mi35x-grok2 — failed step(s): Accuracy Test MI35x (8-GPU Grok2)

Commit Info

• sglang: 9b4bb415ddc315ebcf1f8fcc280537f0b30938dc
• aiter: 9127c94a18e4398e1eba91f6639e910f0994ad02 — v0.1.17.dev110+g9127c94a1 (source: Dev/patched version detected → preserved; AITER_COMMIT_OVERRIDE empty, matches Dockerfile default)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/accuracy/mi35x/test_grok2_eval_mi35x.py	(whole-suite launch; server never became ready)	Model weight download aborted → Process completed with exit code 134	Log

The failure is not an accuracy/assertion failure — the xai-org/grok-2 server never finished loading, so the eval never ran.

Failure Summary

The nightly-8-gpu-mi35x-grok2 job launches an 8-GPU (TP=8, fp8) xai-org/grok-2 server on MI35x. On the first attempt the scheduler process aborted (exit 134 / SIGABRT) while downloading weights from HuggingFace — the faulthandler dump shows the main thread blocked in snapshot_download (huggingface_hub/_snapshot_download.py:513) via ci_weight_validation.py:1941 → weight_utils.py:610 download_weights_from_hf → grok.py:960 _prepare_presharded_weights. The harness then retried with HF_HUB_OFFLINE=1, which failed because the grok-2 weights/config were never cached — LocalEntryNotFoundError / OSError: We couldn't connect to 'https://huggingface.co' — ending in exit code 134.

Regression Status

Known recurring failure — started 2026-07-09, failing every completed run since.

Recent history of test_grok2_eval_mi35x.py in job nightly-8-gpu-mi35x-grok2 (branch main, event=schedule, completed runs only):

Date	Run	Job	Test File Status	Error
Jul 11	29162254653	nightly-8-gpu-mi35x-grok2	❌ Failed	HF download abort (exit 134) → offline retry LocalEntryNotFoundError
Jul 10	29113206892	nightly-8-gpu-mi35x-grok2	❌ Failed	First attempt exit 134 → HF_HUB_OFFLINE retry failed
Jul 09	29039566002	nightly-8-gpu-mi35x-grok2	❌ Failed	First attempt exit 255 → HF_HUB_OFFLINE retry failed
Jul 08	28964511688	nightly-8-gpu-mi35x-grok2	✅ Passed	—
Jul 05	28749559519	nightly-8-gpu-mi35x-grok2	✅ Passed	—
Jul 04	28714602438	nightly-8-gpu-mi35x-grok2	✅ Passed	—
Jul 03	28676359018	nightly-8-gpu-mi35x-grok2	✅ Passed	—

First observed failure: 2026-07-09. Last known passing: 2026-07-08. (Jul 6/7 runs were cancelled at the run level and are excluded.) The Jul 9/10 logs were downloaded and confirmed to share the same signature: first-attempt abort during weight fetch + HF_HUB_OFFLINE retry failure.

Failure Cluster

grok-2 weight download failure / HF-offline retry cannot find cached weights (MI35x nightly accuracy).

Facts (verified observations, no interpretation)

• First attempt aborts inside weight download; faulthandler top frames (verbatim): huggingface_hub/_snapshot_download.py:513 snapshot_download → ci_weight_validation.py:1941 ci_download_with_validation_and_retry → weight_utils.py:610 download_weights_from_hf → grok.py:960 _prepare_presharded_weights.
• Harness message: First attempt failed with exit code 134 then Retrying with HF_HUB_OFFLINE=1 (offline mode to use cached models)....
• Offline retry error (verbatim): huggingface_hub.errors.LocalEntryNotFoundError: Cannot find the requested files in the disk cache and outgoing traffic has been disabled. and OSError: We couldn't connect to 'https://huggingface.co' to load the files, and couldn't find them in the cached files. → Process completed with exit code 134.
• [2026-07-11 18:50:27] Failed to load generation config for xai-org/grok-2: We couldn't connect to 'https://huggingface.co' (offline attempt).
• Independent network signal on the same runner: [local-pull] Error response from daemon: Get "https://10.44.14.109:5000/v2/": ... Client.Timeout exceeded (docker registry pull fell back to public registry).
• Config: model_path='xai-org/grok-2', tokenizer_path='alvarobartt/grok-2-tokenizer', tp_size=8, quantization='fp8', watchdog_timeout=300 (only ~2 min elapsed before the abort, so the soft watchdog did not fire).
• The does not appear to have a file named generation_config.json message is benign/pre-existing (present in passing runs' code path too, "Proceeding without generation config") — not the failure.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [MEDIUM] HuggingFace connectivity/download instability on the MI35x runner — the first attempt aborts mid-snapshot_download and the offline retry then has nothing cached to fall back to.
  • Supporting: main thread is inside snapshot_download at abort time; retry explicitly fails with "couldn't connect to huggingface.co" and "Cannot find the requested files in the disk cache"; an unrelated docker-registry pull on the same runner also timed out (independent network symptom).
  • Disconfirming: exit code differs across days (134 on Jul 10/11, 255 on Jul 09), which is more consistent with an intermittent abort than a single deterministic error; the same test passed Jul 3–8, so the model repo itself is fetchable when the network cooperates.
  • How to verify: re-run the job; check runner egress to huggingface.co; confirm whether a pre-populated grok-2 cache exists on the MI35x runners so the HF_HUB_OFFLINE=1 retry can succeed.
• [LOW] sglang code regression in the weight-download / CI-validation path between Jul 8 (pass) and Jul 9 (first fail).
  • Supporting: temporal boundary — last green Jul 8, first red Jul 9.
  • Disconfirming: the failing frames are in huggingface_hub/network, not in changed sglang logic; the symptom is a connectivity error, not a Python exception in sglang code; no green-run baseline exists after Jul 8 to A/B against. Regression window: 8d2b66fd9071..2e4d6368c35f.
  • How to verify: git log --oneline 8d2b66fd9071..2e4d6368c35f -- python/sglang/srt/model_loader/ci_weight_validation.py python/sglang/srt/models/grok.py python/sglang/srt/model_loader/weight_utils.py and inspect any download-path changes.
• Not aiter-related: aiter is unchanged (dev build preserved, override empty) and the failure occurs before any kernel execution, during weight fetch.

In-flight Fix Check

• In-flight fix: ⚠️ [#23400](https://github.com/sgl-project/sglang/pull/23400) (open) — "ci: prefetch HF snapshots and skip redundant metadata calls to survive 429s" — same problem family (CI HF-download robustness) but scoped to 429 rate-limits, not confirmed to cover this abort/offline-cache path. Worth checking against this failure rather than opening a duplicate.
• In-flight fix: ❌ none found specifically for grok-2 MI35x download / HF_HUB_OFFLINE cache miss.
• Queries run: search/issues q=repo:sgl-project/sglang is:pr is:open grok-2+download (none), grok2+mi35x, HF_HUB_OFFLINE.

Suggested Triage Steps

• Re-run job 86569231233 to confirm intermittency vs. deterministic breakage (exit code varied 134/255 across days ⇒ likely intermittent network).
• Verify the MI35x runners have a pre-seeded xai-org/grok-2 + alvarobartt/grok-2-tokenizer HF cache so the HF_HUB_OFFLINE=1 fallback can actually succeed; if absent, that fallback can never work once the first download aborts.
• Check runner egress/DNS to huggingface.co around 2026-07-09 (the pass→fail boundary) and whether HF rate-limiting (429) is involved — cross-reference #23400.
• Only if network is ruled out: git log --oneline 8d2b66fd9071..2e4d6368c35f and inspect changes to ci_weight_validation.py / grok.py / weight_utils.py.

Status (factual, no priority assignment)

• Persistence: Failing since 2026-07-09 (3 consecutive completed scheduled runs: Jul 9, 10, 11); last green Jul 8.
• Scope: nightly-8-gpu-mi35x-grok2 (MI35x). The sibling nightly-8-gpu-grok2 job was cancelled in the recent runs, so its comparable signal is unavailable in this window.
• Blocked work: grok-2 MI35x nightly accuracy signal is dark until model weights load.
• In-flight fix: ⚠️ #23400 possibly relevant (HF-download robustness / 429s); no exact-match PR found.

---

Generated by amd-bot using Claude Code CLI (last updated: 2026-07-12 23:22 UTC)

[amd-bot]

nightly-test-amd.yml — Per-Job Analysis (continued 2/8)

nightly-2-gpu-mi35x-glm51-mxfp4 — failed step(s): Accuracy Test MI35x (2-GPU GLM-5.1-MXFP4 GSM8K)

Commit Info

• sglang: 9b4bb415d
• aiter: 9127c94a1 (v0.1.17.dev110+g9127c94a1, source: dev image preserved — [CI-AITER-CHECK] Dev/patched version detected → skipping rebuild)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/accuracy/mi35x/test_glm51_mxfp4_tp2_gsm8k_mi35x.py	TestGLM51MXFP4TP2GSM8KMI35x.setUpClass (model/server init)	HW Exception … reason :GPU Hang → Fatal Python error: Aborted → exit code 134	Log

Failure Summary

The single-test job (GLM-5.1-MXFP4 GSM8K accuracy, TP2, MI35x) aborted during model weight loading. After Load weight begin. avail mem=279.75 GB and acquiring the HF download lock, the ROCm/HSA runtime raised HW Exception by GPU node-2 … reason :GPU Hang ~2 minutes later, triggering Fatal Python error: Aborted (exit 134). The faulthandler main-thread stack sits in deepseek_weight_loader.py:206 do_load_weights → deepseek_v2.py:2888 load_weights (the model loads as DeepseekV3ForCausalLM with DSA attention + Quark MXFP4). The offline retry (HF_HUB_OFFLINE=1) hit the identical GPU Hang and exit 134.

Regression Status

Recurring failure (heterogeneous adjacent failure modes) — the GPU-Hang-during-weight-load signature specifically is new since Jul 10.

Recent history of test_glm51_mxfp4_tp2_gsm8k_mi35x.py in job nightly-2-gpu-mi35x-glm51-mxfp4 (branch main, event=schedule, completed runs only):

Date	Run	Test File Status	Failure Mode	Detail
Jul 11	29162254653	❌ Failed	GPU Hang → exit 134	during weight load
Jul 10	29113206892	❌ Failed	GPU Hang → exit 134	same signature
Jul 09	29039566002	❌ Failed	TimeoutError: Server failed to start	elapsed 5430s
Jul 08	28964511688	❌ Failed	VRAM usage exceeds threshold (pre-test guard)	test did not run
Jul 05	28749559519	✅ Passed	—	score 0.937, elapsed 1747s
Jul 04	28714602438	✅ Passed	—	score 0.943, elapsed 2450s
Jul 03	28676359018	❌ Failed	VRAM usage exceeds threshold	test did not run
Jul 02	28611376093	❌ Failed	VRAM usage exceeds threshold	test did not run

• First GPU-Hang failure: Jul 10 (29113206892).
• Last clean pass: Jul 5 (28749559519, sglang 48ba79c11).

Failure Cluster

GPU Hang (HW exception) during GLM-5.1-MXFP4 weight loading on MI35x → exit 134.

Facts (verified observations)

• Job runs exactly ONE test file (✅ Enabled 1 test(s)), so job status = test-file status.
• Current run: the Ensure VRAM is clear step passed — so this is NOT a residual-VRAM guard failure; the crash is a genuine in-process GPU Hang.
• Both the online attempt and the HF_HUB_OFFLINE=1 retry aborted with GPU Hang → exit 134 (log lines 4530, 4699–4702).
• Main-thread faulthandler stack at abort: run_scheduler_process → Scheduler.__init__ → init_model_worker → TpModelWorker → ModelRunner.load_model → loader.load_weights_and_postprocess → deepseek_v2.py:2888 load_weights → deepseek_weight_loader.py:206 do_load_weights.
• Model loads as DeepseekV3ForCausalLM, quant_method=QuarkFusedMoEMethod, Shared experts fusion optimization enabled, DSA attention (page_size=1, attention_backend='dsa'), kv_cache_dtype='fp8_e4m3'.
• aiter is a preserved dev build (v0.1.17.dev110+g9127c94a1), matching the Dockerfile default AITER_COMMIT=9127c94a1 — no override.
• Adjacent nights failed for different reasons (Jul 8/3/2: VRAM guard infra; Jul 9: server-start timeout), indicating MI35x runner health is a live confounder.

Hypothesised Causes (with confidence)

• [LOW] MI35x runner / driver health issue (GPU Hang is a hardware-level HSA exception).
  • Supporting: three distinct failure modes in the last 8 nights (VRAM leak guard, server-start timeout, GPU hang) point to unstable runner state; an in-flight CI self-heal PR for exactly this class of VRAM-leak/GPU-cleanup problem exists (#30431).
  • Disconfirming: the GPU Hang reproduced on both the online attempt and the offline retry within the same run, and again the next night (Jul 10 → Jul 11), which is more consistent than pure runner flake; the Ensure VRAM is clear guard passed this run.
  • How to verify: re-run the job on a freshly-rebooted MI35x runner; if it passes, this is infra.
• [LOW] sglang code regression in the DeepSeek/MXFP4 weight-load or shared-experts-fusion path merged in 48ba79c11..9b4bb415d (74 commits; window narrows to Jul5..Jul10 = 48ba79c11..7998fecfd). Candidate touch-points: #30460 (DeepSeek-V2 dual-stream MoE reorder), #25519 (Quark E8M0 NaN-sentinel detection), #25694 (Quark scheme check), #30044 (triton_ops kernel migration).
  • Supporting: crash is inside do_load_weights; Quark MXFP4 + shared-experts fusion run weight-preprocessing kernels that could hang.
  • Disconfirming: no single commit is confirmed reachable from the hang; the hardware-level GPU Hang and heterogeneous adjacent failures are equally consistent with infra. No code-level evidence ties any of these commits to the fault.
  • How to verify: git bisect the window on an MI35x runner around test_glm51_mxfp4_tp2_gsm8k_mi35x.py.
• No hypothesis above LOW confidence; needs git bisect between 48ba79c11..7998fecfd on healthy MI35x hardware to separate infra from code regression.

In-flight Fix Check

• In-flight fix: ⚠️ partial — #30431 ("[AMD] ci: self-heal driver-level VRAM leaks + graceful GPU cleanup on shutdown") targets the VRAM-leak/GPU-cleanup infra class (relevant to Jul 8/3/2 guard failures and possibly the hangs), but is not a fix for a GPU Hang during weight loading specifically.
• In-flight fix: ❌ none found for a GLM-5.1-MXFP4 weight-load GPU-hang code regression. (Searched open PRs for GLM-5.1+MXFP4, GPU+Hang+MI35x, mxfp4+deepseek, glm51+mxfp4.)

Suggested Triage Steps

• Re-run nightly-2-gpu-mi35x-glm51-mxfp4 on a freshly-rebooted MI35x runner to test the runner-health hypothesis; if green, attribute to infra and track #30431.
• If it still hangs, git bisect 48ba79c11..7998fecfd running only test/registered/amd/accuracy/mi35x/test_glm51_mxfp4_tp2_gsm8k_mi35x.py on MI35x.
• Capture a GPU coredump / rocm-smi state at hang time (weight-load phase) to identify the hanging kernel (Quark MXFP4 dequant vs shared-experts fusion vs DSA setup).

Status

• Persistence: GPU-Hang signature on 2 consecutive completed nights (Jul 10 → Jul 11); the test file has been red every completed run since the Jul 5 pass, but via 3 different failure modes.
• Scope: 1 job / 1 workflow (nightly-test-amd.yml, nightly-2-gpu-mi35x-glm51-mxfp4). Single test file.
• Blocked work: GLM-5.1-MXFP4 GSM8K accuracy signal on MI35x (2-GPU).
• In-flight fix: ⚠️ #30431 covers the VRAM-leak/GPU-cleanup infra class only; ❌ none for a weight-load GPU-hang code regression.

nightly-accuracy-2-gpu-vlm — failed step(s): Nightly Accuracy Test (2-GPU VLM MMMU)

Commit Info

• sglang: 9b4bb415ddc3
• aiter: 9127c94a1 v0.1.17.dev110+g9127c94a1 (source: dev image detected — rebuild skipped; Dockerfile default is the same commit)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/accuracy/mi30x/test_vlms_mmmu_eval_amd.py	TestNightlyVLMMmmuEvalAMD.test_mmmu_vlm_models	AssertionError: The following models failed: — 5 VLMs below MMMU threshold	Log

Per-model MMMU scores this run (100 samples, 3 retries each, exit code 1):

Model	TP	Score	Threshold	Result
Qwen/Qwen2-VL-7B-Instruct	1	0.2799	0.30	❌
Qwen/Qwen2.5-VL-7B-Instruct	1	0.2802	0.33	❌
Qwen/Qwen3-VL-30B-A3B-Instruct	2	0.2703	0.29	❌
openbmb/MiniCPM-o-2_6	1	0.2802	0.32	❌
deepseek-ai/deepseek-vl2-small	1	0.2802	0.31	❌
OpenGVLab/InternVL2_5-2B	1	0.2902	0.29	✅
openbmb/MiniCPM-v-2_6	1	0.3102	0.25	✅
deepseek-ai/Janus-Pro-7B	1	0.2901	0.28	✅
moonshotai/Kimi-VL-A3B-Instruct	1	0.3399	0.26	✅
XiaomiMiMo/MiMo-VL-7B-RL	1	0.31	0.27	✅
deepseek-ai/DeepSeek-OCR-2	1	0.3002	0.25	✅

Failure Summary

The job runs a single test file whose one test (test_mmmu_vlm_models) evaluates 11 VLMs on 100 MMMU samples and asserts each clears a per-model accuracy threshold. 5 of 11 models scored below threshold and the test raised AssertionError at test_vlms_mmmu_eval_amd.py:380 (_check_results, called from line 314). The other 6 models passed. Each failing model was retried 3× (loop at lines 241-250) and produced a stable score each time, so this is not a transient flake.

Regression Status

Known recurring failure (accuracy-threshold cluster). The same 5 models have been below threshold in every completed scheduled run that reached the eval, across the full queryable window (Jul 2 – Jul 11). No green run for these models exists in the window → Never-passed within lookback for this specific 5-model set. Two other completed runs in the window failed the same job for unrelated reasons (an ImportError and a 2 h timeout), not the accuracy assertion.

Recent history of test_vlms_mmmu_eval_amd.py (test_mmmu_vlm_models) in job nightly-accuracy-2-gpu-vlm (branch=main, event=schedule, completed runs only):

Date	Run	Test File Status	Failing detail
Jul 11	29162254653	❌ Failed	Same 5 models below threshold (this run)
Jul 10	29113206892	❌ Failed	Same 5 models below threshold
Jul 9	29039566002	❌ Failed	Different cause: ImportError: cannot import name 'get_attention_cp_size'
Jul 8	28964511688	⚪ Cancelled	Job cancelled — excluded
Jul 7	28888348134	⚪ Cancelled	Job cancelled — excluded
Jul 5	28749559519	❌ Failed	Same 5 models below threshold
Jul 4	28714602438	❌ Failed	Different cause: 2 h timeout (elapsed=7200)
Jul 3	28676359018	❌ Failed	Same 5 models below threshold
Jul 2	28611376093	❌ Failed	Same 5 models below threshold

First observed (accuracy cluster) within window: Jul 2. Last known passing: none in Jul 2–11 window.

Failure Cluster

VLM MMMU accuracy below per-model threshold on AMD MI30x (Qwen2-VL / Qwen2.5-VL / Qwen3-VL-30B / MiniCPM-o-2_6 / deepseek-vl2-small)

Facts (verified observations)

• AssertionError: The following models failed: raised at test_vlms_mmmu_eval_amd.py:380; test file returned exit code 1, job step exit code 255 (log lines 15142, 15207, 15224).
• The 5 failing scores (0.2703–0.2802) sit just below thresholds of 0.29–0.33; all are stable across the 3 retries (e.g. Qwen3-VL logged 0.2601, 0.2601, 0.2703).
• Failing thresholds have been unchanged since the file was last edited dc4380e33ac9 (2026-03-11); the file has only 3 commits ever, none in the Jul window — so the failure is not a threshold change.
• Cross-run: identical 5-model failure set reproduced on Jul 2, 3, 5, 10, 11 (5 independent completed runs). Score for a given model wobbles slightly run-to-run (e.g. Qwen2-VL 0.2799 on Jul 11 vs 0.2901 on Jul 5, threshold 0.30) — borderline around the threshold.
• Most VLM scores across the whole matrix cluster in a narrow 0.27–0.34 band, close to the MMMU 4-option random-chance floor (~0.25).
• aiter version unchanged (dev image v0.1.17.dev110+g9127c94a1, rebuild skipped) — same aiter across passing and failing runs.

Hypothesised Causes (with confidence)

• [MEDIUM] Per-model accuracy thresholds are set higher than the models' actual MMMU-100 accuracy on AMD MI30x, so the assertion fails by design.
  • Supporting: thresholds (0.30–0.33) exceed stable observed scores (0.27–0.28) by a consistent margin; scores are reproducible across 3 retries and 5 runs; thresholds unchanged since March while failing consistently; 6 sibling models with lower thresholds pass.
  • Disconfirming: several distinct models converge on nearly identical scores (0.2802 for Qwen2.5-VL, MiniCPM-o-2_6, deepseek-vl2-small), which is suspicious and could instead point to a shared eval-pipeline degradation rather than genuine per-model accuracy — if so, raising thresholds would mask a real bug.
  • How to verify: run the same models on the MMMU harness on a known-good backend (or an earlier sglang/aiter combo) and compare scores; if scores are ~0.28 everywhere, thresholds are simply miscalibrated.
• [LOW] Systematic VLM output/answer-parsing degradation on AMD (e.g. max_tokens=30 truncating answers, or a decode/attention change) pushing multiple models toward random-chance.
  • Supporting: whole-matrix scores hug the ~0.25 MC random floor; multiple unrelated models land on the exact same 0.2802.
  • Disconfirming: 6 models still clear their thresholds (up to 0.3399 for Kimi-VL), so decoding is not globally broken; max_tokens=30 is committed config, not a recent change; no code change to the eval path in the Jul window.
  • How to verify: inspect a few generated /tmp/mmmu_*.json responses for truncated/unparseable answers; A/B max_tokens=30 vs a larger value on one failing model.
• Note: this is not a fresh regression tied to 9b4bb415ddc3 — the accuracy cluster predates it by ≥9 days.

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs test_vlms_mmmu (0 results) and VLM MMMU accuracy (19 results, none touching this test file or AMD VLM thresholds). No open PR adjusts these thresholds or targets AMD VLM MMMU accuracy.

Suggested Triage Steps

• Confirm whether the borderline scores reflect true model accuracy or an eval-pipeline issue: inspect the emitted /tmp/mmmu_<model>.json / .html reports for the 5 failing models on this run for truncated or unparseable answers (ties directly to the max_tokens=30 / answer-extraction path).
• A/B one failing model (e.g. Qwen2.5-VL-7B, largest gap: 0.2802 vs 0.33) with a larger max_tokens to rule out answer truncation before touching thresholds.
• If scores are confirmed genuine and stable, re-calibrate the 5 per-model accuracy_threshold values in test_vlms_mmmu_eval_amd.py:45-133 to reflect measured MI30x accuracy (maintainer decision — do not raise thresholds to mask a real regression).
• Separately track the two unrelated failures of this job in the window (ImportError get_attention_cp_size on Jul 9; 2 h timeout on Jul 4) — they are distinct clusters, not part of this accuracy failure.

Status

• Persistence: Accuracy-threshold failure on the same 5 models in every completed scheduled run that reached the eval since 2026-07-02 (Jul 2, 3, 5, 10, 11); no passing run in the queryable window.
• Scope: Single job nightly-accuracy-2-gpu-vlm in nightly-test-amd.yml; 5 of 11 VLMs affected. Not observed to block other jobs.
• Blocked work: AMD nightly VLM MMMU accuracy signal is red; the persistent red obscures any genuine future VLM accuracy regression on these 5 models.
• In-flight fix: ❌ none found — needs a new PR (threshold re-calibration) or eval-pipeline investigation.

nightly-8-gpu-mi35x-deepseek-r1-mxfp4-tp4 — failed step(s): Accuracy Test MI35x (DeepSeek-R1-MXFP4 TP4 MTP), Accuracy Test MI35x (DeepSeek-R1-MXFP4 TP4)

Commit Info

• sglang: 9b4bb415ddc3 (#30834)
• aiter: 9127c94a18e4 — v0.1.17.dev110+g9127c94a1 (source: dev/patched image build; [CI-AITER-CHECK] detected the dev version and skipped rebuild; matches Dockerfile default AITER_COMMIT)
• CI image: rocm/sgl-dev:v0.5.15-rocm700-mi35x-20260710 (auto-resolved "newest by date" in scripts/ci/amd/amd_ci_start_container.sh; tag date = Jul 10)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/accuracy/mi35x/test_deepseek_r1_mxfp4_tp4_mi35x.py	N/A — crash during weight load (before any test assertion)	Fatal Python error: HW Exception by GPU node-3 ... reason :GPU Hang → exit code 134 (SIGABRT)	Log
test/registered/amd/accuracy/mi35x/test_deepseek_r1_mxfp4_tp4_mtp_mi35x.py	N/A — crash during model init (before any test assertion)	HW Exception by GPU node-4 ... reason :GPU Hang → exit code 134 (SIGABRT)	Log

Failure Summary

Both test scripts in this job crashed with an AMD HW Exception ... reason :GPU Hang and process exit code 134 (SIGABRT). The hang fires ~15 s after Load weight begin — during weight loading, before warmup / CUDA-graph capture / any GSM8K request. Each script was retried once with HF_HUB_OFFLINE=1 and hung again on a different GPU node (node-3→node-5 for the first, node-4→node-2 for the second), so all 4 attempts died the same way.

Regression Status

Symptom changed — new "GPU Hang" crash since Jul 10 (the job itself has been red longer, but for a different, non-crash reason before Jul 10).

Recent history of test_deepseek_r1_mxfp4_tp4_mi35x.py + ..._mtp_mi35x.py in job nightly-8-gpu-mi35x-deepseek-r1-mxfp4-tp4 (nightly-test-amd.yml, branch main, event=schedule, completed runs only):

Date	Run	Job	Test File Status	Symptom
Jul 11	29162254653	job	❌ Both crash	GPU Hang → exit 134 (during weight load)
Jul 10	29113206892	job	❌ Both crash	GPU Hang → exit 134
Jul 09	29039566002	job	❌ Failed (exit 1)	accuracy-type failure — no hang, server ran to completion
Jul 08	28964511688	job	❌ Failed (exit 1)	accuracy-type failure — no hang
Jul 04	28714602438	job	⚠️ tp4_mi35x ✅ PASS (acc=0.948) · mtp ❌ accuracy assert (acc=0.939 < GSM8K_MTP_ACCURACY_THRESHOLD, line 198)	no hang

• First observed GPU-Hang crash: Jul 10 run 29113206892.
• Last hang-free run: Jul 09 run 29039566002 (weight load completed; failure was accuracy/exit-1).
• Last green for test_deepseek_r1_mxfp4_tp4_mi35x.py: Jul 04 (acc=0.948). The MTP script has not passed in the queryable window.
• The current run 29162254653 is queued at the workflow level but this job completed (failed); treated as a completed data point.
• Lookback window searched: Jun 30 – Jul 11 (runs Jul 01, 06, 07, 08, 10 were cancelled at workflow level and excluded/partial).

Failure Cluster

GPU Hang HW Exception during weight load on mi35x (exit 134)

Facts (verified observations)

• Verbatim: HW Exception by GPU node-3 (Agent handle: 0x...) reason :GPU Hang and Fatal Python error: HW Exception by GPU node-3 ... reason :GPU Hang, followed by ##[error]Process completed with exit code 134.
• Timing: Load weight begin. avail mem=279 GB at 18:57:29 → GPU Hang at 18:57:44 (≈15 s later), i.e. mid-weight-load, before warmup/graph capture.
• Reproducibility: 4/4 attempts in this job hung (2 scripts × initial + HF_HUB_OFFLINE=1 retry), each on a different GPU node (node-3, node-5, node-4, node-2).
• Environment: CI image rocm/sgl-dev:v0.5.15-rocm700-mi35x-20260710; amd_ci_start_container.sh find_latest_image() resolves the newest dated tag, so the image content changed at the Jul 10 boundary without any workflow-file edit.
• aiter version is v0.1.17.dev110+g9127c94a1 (unchanged Dockerfile-default dev build).
• On Jul 09 and earlier, weight load completed (avail mem dropped 283→198 GB) and the model served requests — the hang is genuinely new to Jul 10+.
• Broader context: nearly all *-mi35x-* jobs in the Jul 10 and Jul 11 runs are red (systemic across the mi35x fleet), while several passed on Jul 04/05.

Hypothesised Causes (with confidence)

• [MEDIUM] The mi35x CI base image rolled to v0.5.15-rocm700-mi35x-20260710 (auto-selected newest-by-date) at the exact Jul 10 regression boundary; the GPU Hang HW Exception during weight load is a driver/runtime-level fault consistent with an image/driver/firmware change rather than an application bug.
  • Supporting: image tag date (20260710) coincides precisely with the first hang run (Jul 10); the fault is a hardware-level HW Exception, not a Python traceback in sglang code; the prior image completed weight load fine (Jul 09); the crash recurs across retries on different GPU nodes; the failure is systemic across most mi35x jobs simultaneously.
  • Disconfirming: cannot diff image contents from the log alone; a fleet-wide hardware/runner-health degradation could produce the identical symptom independent of the image roll; no direct proof the 20260710 image (vs an earlier tag) is the trigger.
  • How to verify: pin/override the container to the last known-good dated tag (e.g. the image used on the Jul 09 run) via amd_ci_start_container.sh --mi35x-base-tag <old_tag> and re-run test_deepseek_r1_mxfp4_tp4_mi35x.py on an mi35x-8gpu runner; if it loads weights and runs, the newer image is implicated.
• [LOW] An sglang commit in the Jul09→Jul10 window 2e4d6368c35f..7998fecfd114 (57 commits) touching mi35x/mxfp4/weight-load paths.
  • Supporting: window contains AMD/quant/MoE/DeepSeek changes (e.g. 789bc3995 #30587, fef5eda4f #30711).
  • Disconfirming: the hang is a HW Exception occurring before model forward/graph capture, so application-level code is an unlikely trigger; #30587 only edits the miles (nightly-test-amd-miles-rocm700.yml) workflows, not nightly-test-amd.yml, so it does not gate this job; the same symptom hits nearly all mi35x jobs regardless of which code path they exercise.
  • How to verify: only pursue if the image A/B check above exonerates the image; then bisect the window with the image pinned constant.

In-flight Fix Check

• Possible mitigation: #30431 "[AMD] ci: self-heal driver-level VRAM leaks + graceful GPU cleanup on shutdown" — open, directly targets driver-level GPU state corruption/cleanup on AMD CI; may reduce GPU-Hang recurrence but is not a confirmed fix for this specific hang. (found via search/issues?q=...mi35x+hang)
• Image pin/revert: In-flight fix: ❌ none found — no open PR pins or rolls back the mi35x-20260710 image for nightly-test-amd.yml.

Suggested Triage Steps

• Confirm scope: check whether the other mi35x jobs in run 29162254653 (e.g. nightly-8-gpu-mi35x-deepseek-r1-mxfp4, -ar-fusion, -hicache) also show reason :GPU Hang — if fleet-wide, this is an image/runner issue, not a per-test bug.
• Image A/B: pin the container to the last hang-free dated tag (the one on the Jul 09 run) via amd_ci_start_container.sh --mi35x-base-tag <old_tag> and re-run test_deepseek_r1_mxfp4_tp4_mi35x.py; compare against the 20260710 image.
• If the image is exonerated, bisect 2e4d6368c35f..7998fecfd114 with the image held constant.
• Check mi35x runner/host health (GPU reset state, driver/firmware version on linux-mi35x-gpu-8-* runners); a GPU Hang during weight load across multiple nodes can indicate a degraded runner.
• Track #30431 as a candidate mitigation for driver-level GPU cleanup.

Status

• Persistence: GPU-Hang crash for 2 consecutive completed runs (Jul 10 → Jul 11); the job has been red longer (since ≥ Jul 02) but for a different accuracy/exit-1 reason before Jul 10.
• Scope: both test scripts in this job; appears systemic across most *-mi35x-* jobs in the Jul 10/11 runs.
• Blocked work: DeepSeek-R1-MXFP4 TP4 (± MTP) accuracy signal on mi35x is fully blocked — the server cannot even load weights, so no accuracy data is produced.
• In-flight fix: no image pin/revert PR open; #30431 is a possible driver-cleanup mitigation, unconfirmed.

nightly-accuracy-8-gpu-mi35x — failed step(s): Accuracy Test MI35x (8-GPU GPT-OSS)

Commit Info

• sglang: 9b4bb415ddc315ebcf1f8fcc280537f0b30938dc
• aiter: 9127c94a18e4398e1eba91f6639e910f0994ad02 — v0.1.17.dev110+g9127c94a1 (source: dev image preserved; [CI-AITER-CHECK] reports "Dev/patched version detected → skipping rebuild", matches Dockerfile gfx950 default)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/accuracy/mi35x/test_gpt_oss_w4a8_mxfp4_eval_mi35x.py	N/A (script-style eval; model amd/gpt-oss-120b-w-mxfp4-a-fp8)	HW Exception by GPU node-3 reason :GPU Hang → Fatal Python error: Aborted → exit code 134	Log

Failure Summary

The nightly-accuracy-8-gpu-mi35x job aborted with exit code 134 (SIGABRT). The bf16 test_gpt_oss_eval_mi35x.py file passed cleanly (gpt-oss-20b 0.585≥0.47, gpt-oss-120b 0.860≥0.79). The crash occurred in the next file, test_gpt_oss_w4a8_mxfp4_eval_mi35x.py, while loading weights for amd/gpt-oss-120b-w-mxfp4-a-fp8: a GPU Hang HW exception fired ~14s into weight loading, aborting all 8 TP workers. The automatic HF_HUB_OFFLINE=1 retry re-ran the suite from the start and hung again (GPU node-5) on the previously-passing gpt-oss-20b, indicating the GPU was left wedged by the first hang.

Regression Status

Known recurring failure (GPU Hang cluster; 2nd consecutive completed run). For this specific test file it is new today — it passed Jul 10.

Recent history of test_gpt_oss_w4a8_mxfp4_eval_mi35x.py in job nightly-accuracy-8-gpu-mi35x (branch=main, event=schedule, completed runs only):

Date	Run	Job	Test File Status	Detail
Jul 11	29162254653	job	❌ GPU Hang	Aborted during load_gptoss_weight_quark weight load, exit 134
Jul 10	29113206892	job	✅ Passed (0.870)	Job still failed — same GPU Hang cluster hit test_qwen3_coder_next_eval_mi35x.py instead
Jul 9	29039566002	job	⚪ Did not run	Infra: No docker containers found on host, exit 1
Jul 8	28964511688	job	⚪ Did not run	Infra: No docker containers found on host, exit 1
Jul 5	28749559519	job	⚪ Did not run	Infra: No docker containers found on host, exit 1
Jul 3	28676359018	job	⚪ Did not run	Infra: exit 1
Jul 2	28611376093	job	✅ Passed (0.860)	Full job success (all 4 files green)

Last full-green run: Jul 2 (cba3801f5214). Note Jul 3–9 base-job failures were an unrelated infra cluster ("No docker containers found on host", exit 1), NOT the GPU Hang.

Failure Cluster

GPU Hang during multi-threaded weight loading (HW Exception ... reason :GPU Hang → Fatal Python error: Aborted, exit 134, mi35x/gfx950)

Facts (verified observations)

• Verbatim: HW Exception by GPU node-3 (Agent handle: ...) reason :GPU Hang followed by Fatal Python error: Aborted (log ~19:08:58), then First attempt failed with exit code 134, retry, then HW Exception by GPU node-5 ... reason :GPU Hang and ##[error]Process completed with exit code 134.
• Main-thread fault stack (from faulthandler dump): model_runner.py:707 initialize → tp_worker.py init → loader.py load_model/load_weights_and_postprocess → gpt_oss.py:889 load_weights → quark/weights.py load_gptoss_weight_quark. Background threads were in download_weights_from_hf/filelock.
• The hang fired only ~14 s after "Using Quark MXFP4-W/FP8-A MoE scheme" — during weight load, before any accuracy eval ran.
• Same-symptom recurrence Jul 10 (job): identical GPU Hang → exit 134 → retry → GPU Hang pattern, but on a different model (Qwen/Qwen3-Coder-Next, "Multi-thread loading shards: 50% Completed | 20/40" then GPU Hang). On that run the w4a8 gpt-oss test itself passed (0.870).
• In both Jul 10 and Jul 11 the post-crash retry hung on a model that had passed moments earlier → GPU not recovered between attempts.
• aiter version unchanged (dev image v0.1.17.dev110+g9127c94a1) across the passing (Jul 2/10) and failing (Jul 11) runs.

Hypothesised Causes (with confidence)

• [MEDIUM] Runner/driver-level GPU instability on the mi35x (gfx950) host — a GPU Hang during weight loading that also wedges the GPU for the subsequent retry.
  • Supporting: the specific model that hangs changes run-to-run (Jul 11 = gpt-oss-120b-w-mxfp4-a-fp8; Jul 10 = qwen3-coder-next); both hangs occur during multi-thread shard loading; the retry hangs on a previously-passing model, showing the fault is not model-code-specific but a persistent GPU state; symptom is a hardware HW Exception ... GPU Hang, not a Python/accuracy assertion.
  • Disconfirming: no GPU counter/dmesg evidence captured in the log to prove a driver fault vs a specific kernel deadlock; the w4a8 file itself passed the day before, so a same-day weight-loading path change cannot be fully excluded.
  • How to verify: re-run the job in isolation on a freshly-reset mi35x node; check host dmesg/rocm-smi for XGMI/HBM/driver errors around the hang; run test_gpt_oss_w4a8_mxfp4_eval_mi35x.py alone (not after other 8-GPU tests) to see if it reproduces without a wedged GPU.
• [LOW] A weight-loading code path exercised by load_gptoss_weight_quark / Quark MXFP4 dequant hangs the GPU.
  • Supporting: the main-thread fault frame is inside quark/weights.py load_gptoss_weight_quark during shard load.
  • Disconfirming: the very same file+model passed Jul 10; the Jul 10 hang was in a non-Quark model (qwen3-coder-next), so the hang is not exclusive to the Quark path — weakens a Quark-specific code cause.
  • How to verify: bisect cba3801f5214..9b4bb415ddc3 for changes under quark/ / gpt_oss.py / weight loader, but only after ruling out the GPU-instability hypothesis.
• No hypothesis reaches HIGH confidence; the varying victim model and retry-wedge strongly favor infra/GPU-state over a code regression.

In-flight Fix Check

• In-flight fix: ⚠️ candidate #30431 "[AMD] ci: self-heal driver-level VRAM leaks + graceful GPU cleanup on shutdown" (open) — directly targets driver-level GPU cleanup/wedging on AMD CI, which matches the retry-hangs-on-wedged-GPU symptom. Not a confirmed fix for the initial hang, but relevant to the retry cascade. (found via is:pr is:open GPU Hang mi35x)
• In-flight fix: ❌ none found specifically for a gpt-oss-120b-w-mxfp4-a-fp8 weight-loading hang.

Suggested Triage Steps

• Re-run job 86569231256 on a freshly-reset mi35x node; if a different model hangs (as Jul 10 vs Jul 11), that confirms the GPU-instability hypothesis over a model-specific regression.
• Inspect the runner host dmesg / rocm-smi / ROCm driver logs around 19:08 UTC for HBM/XGMI/driver faults; check for prior-test GPU state not being reset between the 4 accuracy files.
• Evaluate whether #30431 (graceful GPU cleanup on shutdown) would prevent the retry from inheriting a wedged GPU; if so, chase reviewers rather than opening a duplicate.
• Only if the hang reproduces on a clean GPU while running test_gpt_oss_w4a8_mxfp4_eval_mi35x.py in isolation, bisect cba3801f5214..9b4bb415ddc3 for Quark/gpt-oss weight-loader changes.

Status

• Persistence: GPU Hang cluster failing 2 consecutive completed runs (Jul 10 → Jul 11). This specific test file (test_gpt_oss_w4a8_mxfp4_eval_mi35x.py) first hung Jul 11; last passed Jul 10. Base nightly-accuracy-8-gpu-mi35x job red since Jul 3, but Jul 3–9 were an unrelated docker-host infra failure.
• Scope: nightly-accuracy-8-gpu-mi35x job only (this analysis). The GPU Hang aborts the whole 8-GPU accuracy suite, so it also blocks the qwen3-coder-next and qwen3-moe evals in the same job.
• Blocked work: full gpt-oss / qwen3 accuracy signal on mi35x (gfx950).
• In-flight fix: ⚠️ candidate #30431 for the GPU-cleanup/retry-wedge aspect; no dedicated PR for the initial weight-load hang.

nightly-accuracy-8-gpu-mi35x-deepseek-v32-mtp — failed step(s): Accuracy Test MI35x (8-GPU DeepSeek-V3.2 TP+MTP)

Commit Info

• sglang: 9b4bb415
• aiter: 9127c94a (v0.1.17.dev110+g9127c94a1, source: Dev/patched image detected → preserved, no rebuild)
• Docker image: rocm/sgl-dev:v0.5.15-rocm700-mi35x-20260711 · Runner: linux-mi35x-gpu-8-zf775-runner-4rdzf

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/accuracy/mi35x/test_deepseek_v32_mtp_eval_mi35x.py	N/A — crashed in server startup before eval (script-style test)	HW Exception ... reason :GPU Hang → Fatal Python error: Aborted → exit 134	Log

Failure Summary

The DeepSeek-V3.2 TP+MTP accuracy test on MI35x crashed during model weight loading (before any accuracy eval ran). A GPU-level HW Exception ... reason :GPU Hang fired while loading MoE weights, aborting the Python process with SIGABRT (exit code 134). Both the first attempt and the HF_HUB_OFFLINE=1 retry crashed identically. No accuracy score was produced.

Regression Status

Known recurring failure (this test file has been red since 2026-07-06; last green 2026-07-05). The specific GPU-Hang / exit-134 symptom is new as of 2026-07-10 and has recurred every run since (2/2). Earlier failures in the window had different symptoms (VRAM-cleanup infra failures, exit 255).

Recent history of test_deepseek_v32_mtp_eval_mi35x.py in job nightly-accuracy-8-gpu-mi35x-deepseek-v32-mtp (branch=main, event=schedule, completed runs only):

Date	Run	Test File Status	Symptom	Image
Jul 11	29162254653	❌ Failed	GPU Hang, exit 134	v0.5.15-...20260711
Jul 10	29113206892	❌ Failed	GPU Hang, exit 134	v0.5.15-...20260710
Jul 09	29039566002	❌ Failed	exit 255	v0.5.14-...20260709
Jul 08	28964511688	❌ Failed	VRAM cleanup unsuccessful	v0.5.14
Jul 07	28888348134	❌ Failed	exit 255	v0.5.14-...20260707
Jul 06	28813271245	❌ Failed	VRAM cleanup unsuccessful	v0.5.14
Jul 05	28749559519	✅ Passed	Accuracy: 0.965	v0.5.14-...20260705
Jul 04	28714602438	❌ Failed	VRAM cleanup unsuccessful	v0.5.14

First GPU-Hang occurrence: 2026-07-10. Last green (any symptom): 2026-07-05. Lookback: 8 scheduled runs (Jul 04–11).

Failure Cluster

GPU Hang HW exception during DeepSeek-V3.2 MoE weight loading on MI35x (exit 134)

Facts (verified observations)

• Verbatim: HW Exception by GPU node-3 (Agent handle: ...) reason :GPU Hang then Fatal Python error: Aborted; process ends ##[error]Process completed with exit code 134. (retry crashed on GPU node-5).
• Crash stack (first attempt) is in MoE weight loading: fused_moe_triton/layer.py → _weight_loader_impl / _load_w13 / _load_w2, called from deepseek_v2.py:2888 load_weights → loader.py:829/794 → model_runner.py:1455 load_model.
• First attempt hung at Multi-thread loading shards: 6% Completed | 9/163; retry hung ~15 s after test start (even earlier).
• Both post-bump runs (Jul 10, Jul 11) use image v0.5.15-rocm700-mi35x-*; the last passing run (Jul 05) used v0.5.14-rocm700-mi35x-20260705.
• aiter is a dev build baked into the image (v0.1.17.dev110+g9127c94a1), not an override. This is not a scout run (AITER_COMMIT_OVERRIDE empty), so no Baseline A/B / Failure Origin applies.

Hypothesised Causes (with confidence)

• [MEDIUM] The rocm700-mi35x base-image bump — PR #30587 / commit 789bc3995 (merged Jul 10) — introduced the GPU-Hang symptom.
  • Supporting: exact temporal alignment — the image tag switched v0.5.14→v0.5.15 between the last non-hang run (Jul 09, still v0.5.14) and the first GPU-Hang run (Jul 10, v0.5.15); both post-bump runs hang identically at the same code path; the crash is a HW-level GPU Hang, not Python logic. The commit changes only workflow YAML (image tag + rename), so the real variable is the base-image contents (ROCm runtime + bundled aiter/kernels).
  • Disconfirming: the test was already unstable before the bump (VRAM-cleanup + exit-255 across Jul 04–09; only Jul 05 passed in 8 days), so the bump did not break a stable test. The hang reproduced on two different GPU nodes (node-3, node-5) on the same physical runner, which is also consistent with an unhealthy runner rather than the image.
  • How to verify: re-run this job pinned to the previous image v0.5.14-rocm700-mi35x-20260705 on a known-healthy MI35x runner; if it passes, the image is implicated. Cross-check whether other jobs on the v0.5.15 image also GPU-Hang.
• [LOW] Unhealthy GPU/driver on runner linux-mi35x-gpu-8-zf775-runner-4rdzf.
  • Supporting: GPU Hang is a hardware-level symptom; the same runner family showed VRAM-cleanup driver failures earlier in the week.
  • Disconfirming: hang reproduced across two attempts on two different GPU nodes → not a single dead GPU; more consistent with a systemic (image/driver) issue than one bad card.

In-flight Fix Check

• In-flight fix: ❌ none found directly targeting the MI35x GPU-Hang during weight load. Search: is:pr is:open GPU Hang mi35x and deepseek-v32 mi35x — closest is #30431 "[AMD] ci: self-heal driver-level VRAM leaks + graceful GPU cleanup on shutdown", which addresses the sibling VRAM-cleanup symptom (Jul 04/06/08), not this GPU Hang.

Suggested Triage Steps

• Re-run the job with the image pinned to v0.5.15 vs the last-green v0.5.14-rocm700-mi35x-20260705 to A/B the #30587 image bump; if v0.5.14 passes, escalate the new base image (ROCm/aiter/kernel contents) to the miles image owner.
• Check runner linux-mi35x-gpu-8-zf775-runner-4rdzf GPU/driver health (dmesg/rocm-smi) and try the same commit on a different MI35x runner to rule out hardware.
• Grep other Jul 10–11 jobs on the v0.5.15 image for reason :GPU Hang to determine whether the hang is image-wide or specific to this DeepSeek-V3.2 MTP weight-load path.

Status

• Persistence: GPU-Hang symptom failing 2/2 completed runs since 2026-07-10; test file red since 2026-07-06 (last green 2026-07-05, Accuracy 0.965).
• Scope: nightly-accuracy-8-gpu-mi35x-deepseek-v32-mtp (this job runs only this one test file). Sibling VRAM-cleanup symptom seen on the same runner family earlier in the window.
• Blocked work: no DeepSeek-V3.2 TP+MTP accuracy signal on MI35x.
• In-flight fix: ❌ none for the GPU Hang; #30431 covers only the VRAM-cleanup sibling symptom.

nightly-8-gpu-mi35x-deepseek-r1-mxfp4-kv-fp8 — failed step(s): Accuracy Test MI35x (8-GPU DeepSeek-R1-MXFP4 KV FP8)

Commit Info

• sglang: 9b4bb415d
• aiter: 9127c94a1 v0.1.17.dev110+g9127c94a1 (source: dev image preserved — [CI-AITER-CHECK] Dev/patched version detected → skipping rebuild, matches Dockerfile default)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/accuracy/mi35x/test_deepseek_r1_mxfp4_kv_fp8_eval_mi35x.py	N/A — crashed during model weight load, before eval	HW Exception ... reason: GPU Hang → Fatal Python error: Aborted → exit 134	Log

Failure Summary

The single test test_deepseek_r1_mxfp4_kv_fp8_eval_mi35x.py crashed while loading model weight shards (Multi-thread loading shards: 0% Completed | 0/73) with a HW Exception ... reason: GPU Hang across all 8 TP ranks, aborting the process with exit code 134 (SIGABRT). The model never finished loading, so the accuracy eval itself never ran. The automatic offline retry (HF_HUB_OFFLINE=1) also hit GPU Hang on a different GPU node.

Regression Status

Infrastructure issue (hardware GPU hang on the mi35x runner) — not a test-logic regression.

Recent history of test_deepseek_r1_mxfp4_kv_fp8_eval_mi35x.py in job nightly-8-gpu-mi35x-deepseek-r1-mxfp4-kv-fp8 (same branch main, event=schedule, completed runs only):

Date	Run	Job	Test File Status	Failure Symptom	Error
Jul 11	29162254653	job	❌ Failed	GPU Hang (this run)	HW Exception: GPU Hang → exit 134
Jul 10	29113206892	job	✅ Passed	—	—
Jul 9	29039566002	job	❌ Failed (test did not run)	pre-test VRAM cleanup	VRAM cleanup unsuccessful after 3 attempts → exit 1
Jul 8	28964511688	job	✅ Passed	—	—
Jul 7	28888348134	job	❌ Failed (test did not run)	pre-test VRAM cleanup	VRAM cleanup unsuccessful after 3 attempts → exit 1
Jul 5	28749559519	job	✅ Passed	—	—

Last clean pass: Jul 10. Today's GPU Hang symptom is new today and is distinct from the Jul 7/9 failures, which were pre-test VRAM-cleanup failures where the eval never started. The job alternates pass/fail day-to-day, all failures rooting in mi35x runner GPU/driver state rather than the test code. (Jul 6 run was cancelled at run level; excluded.)

Failure Cluster

GPU Hang / HW Exception during DeepSeek-R1 weight-shard loading on mi35x runner

Facts (verified observations, no interpretation)

• First attempt: HW Exception by GPU node-3 ... reason :GPU Hang on all 8 ranks at Multi-thread loading shards: 0% Completed | 0/73 (log line 4682), immediately after Load weight begin and Found local HF snapshot ... skipping download. Process aborts: Fatal Python error: Aborted, First attempt failed with exit code 134.
• Retry with HF_HUB_OFFLINE=1: HW Exception by GPU node-5 ... reason :GPU Hang again → Process completed with exit code 134.
• Weight load never progressed past 0/73 shards; the accuracy eval never executed → no accuracy/score numbers were produced.
• aiter version unchanged from Dockerfile default ([CI-AITER-CHECK] AITER version matches-equivalent: dev build g9127c94a1 == Dockerfile 9127c94a02), so no aiter delta between this run and prior passing runs.
• Two prior failing runs (Jul 7, Jul 9) had a different symptom — VRAM usage exceeds threshold / VRAM cleanup unsuccessful in the pre-test cleanup step (exit 1), i.e. leftover VRAM from a previous job on the runner.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [MEDIUM] Hardware/driver-level GPU hang on the shared mi35x runner (not caused by this PR's code).
  • Supporting: HW Exception ... reason :GPU Hang is a driver-reported hardware fault, raised at 0/73 shards before any model forward/kernel of this test runs; it recurs on a different GPU node (node-3 then node-5) across two independent process launches; sibling prior failures on the same runner are VRAM-state issues, pointing to persistent runner GPU state corruption rather than test code.
  • Disconfirming: HEAD commit 9b4bb415d touches cuda-graph shared-buffer sizing, which runs during capture/warmup — after weight load — so it cannot explain a crash at 0/73 shards; but the full nightly-to-nightly sglang delta was not exhaustively bisected, so a load-path regression cannot be 100% excluded.
  • How to verify: reset/drain the mi35x runner (reboot GPUs, clear VRAM) and re-run the job; if it passes on the same sglang SHA, hardware state is confirmed.
• [LOW] sglang regression in the weight-loading / Quark MXFP4 path landing since the Jul 10 pass.
  • Supporting: crash is at Load weight begin for a QuarkFusedMoEMethod MXFP4 model; last pass was Jul 10.
  • Disconfirming: a GPU Hang HW exception is a driver fault, not a Python exception you'd expect from a code bug; the retry hangs on a different GPU node before any sglang load logic diverges; no stack trace pointing into sglang loading code.
  • How to verify: git log --oneline 7998fecfd114..9b4bb415d -- python/sglang/srt/model_loader python/sglang/srt/layers/quantization/quark and inspect any diffs to the MXFP4/Quark load path; re-run on a known-healthy mi35x runner.

In-flight Fix Check

• In-flight fix: ✅ #30431 [AMD] ci: self-heal driver-level VRAM leaks + graceful GPU cleanup on shutdown — directly targets AMD-runner driver-level VRAM leaks / GPU cleanup, the same class of runner-state fault behind both today's GPU Hang and the Jul 7/9 VRAM-cleanup failures. Chase reviewers; do NOT open a duplicate. (Searched repo:sgl-project/sglang is:pr is:open mi35x GPU Hang.)

Suggested Triage Steps

• Reset the mi35x runner (reboot/drain GPUs, verify VRAM is clear via rocm-smi) and re-run job 86569231271; a pass on the same SHA confirms an infra/hardware root cause.
• Review and land #30431, then watch whether the alternating pass/fail pattern on this job stops.
• Only if the hang reproduces on a freshly-reset runner: bisect the weight-load / Quark MXFP4 path in 7998fecfd114..9b4bb415d and re-run the eval in the AMD mi35x container.

Status (factual, no priority assignment)

• Persistence: New GPU Hang symptom in this run (Jul 11); last clean pass Jul 10. Job alternates pass/fail day-to-day; today's is a distinct symptom from the Jul 7/9 VRAM-cleanup failures.
• Scope: 1 job / 1 test file (nightly-8-gpu-mi35x-deepseek-r1-mxfp4-kv-fp8) on the mi35x runner. Both the primary and offline-retry attempts hung, on GPU nodes 3 and 5 respectively.
• Blocked work: DeepSeek-R1-MXFP4 KV-FP8 accuracy signal on AMD mi35x for this nightly.
• In-flight fix: ✅ #30431 (AMD CI driver-level VRAM/GPU self-heal) — awaiting merge.

---

Generated by amd-bot using Claude Code CLI (last updated: 2026-07-12 23:22 UTC)

[amd-bot]

nightly-test-amd.yml — Per-Job Analysis (continued 3/8)

nightly-8-gpu-mi35x-kimi-k26 — failed step(s): Accuracy Test MI35x (8-GPU Kimi-K2.6)

Commit Info

• sglang: 9b4bb415d
• aiter: 9127c94a1 — v0.1.17.dev110+g9127c94a1 (source: dev image preserved — [CI-AITER-CHECK] Dev/patched version detected → skipping rebuild; AITER_COMMIT_OVERRIDE empty)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/accuracy/mi35x/test_kimi_k26_eval_mi35x.py	TestKimiK26EvalMI35x.test_kimi_k26_gsm8k_accuracy	HW Exception ... reason :GPU Hang during weight load → exit code 134 (SIGABRT)	Log

Failure Summary

The single test test_kimi_k26_gsm8k_accuracy (Kimi-K2.6, TP=8, MI35x) crashed during server launch / model weight loading, before the GSM8K eval ran. Both the initial attempt and the retry aborted with a ROCm HW Exception ... reason :GPU Hang, terminating the process with exit code 134. The accuracy assertion (ACCURACY_THRESHOLD = 0.92) was never reached.

Regression Status

Known recurring failure (test file red since Jul 4); the GPU-Hang-during-load symptom specifically is 2 days old (Jul 10–11). The prior 3 failures (Jul 5/8/9) were a different symptom — pre-test VRAM cleanup failing.

Recent history of test_kimi_k26_eval_mi35x.py (test_kimi_k26_gsm8k_accuracy) in job nightly-8-gpu-mi35x-kimi-k26 (branch main, event schedule, completed runs only):

Date	Run	Job	Test File Status	Failure symptom	Exit
Jul 11	29162254653	nightly-8-gpu-mi35x-kimi-k26	❌ Failed	GPU Hang during weight load	134
Jul 10	29113206892	nightly-8-gpu-mi35x-kimi-k26	❌ Failed	GPU Hang during weight load	134
Jul 9	29039566002	nightly-8-gpu-mi35x-kimi-k26	❌ Failed	VRAM cleanup unsuccessful (pre-test)	1
Jul 8	28964511688	nightly-8-gpu-mi35x-kimi-k26	❌ Failed	VRAM cleanup unsuccessful (pre-test)	1
Jul 5	28749559519	nightly-8-gpu-mi35x-kimi-k26	❌ Failed	VRAM cleanup unsuccessful (pre-test)	1
Jul 4	28714602438	nightly-8-gpu-mi35x-kimi-k26	✅ Passed	—	0

(Jul 6–7 schedule runs were cancelled, excluded. First observed failure of this test file: Jul 5. Last known pass: Jul 4, head 854b46be99. GPU-Hang cluster first observed Jul 10.)

Failure Cluster

"GPU Hang HW exception during Kimi-K2.6 multi-thread weight loading (MI35x 8-GPU)"

Facts (verified observations, no interpretation)

• Test invocation: python3 run_suite.py --hw amd --suite nightly-amd-accuracy-8-gpu-mi35x-kimi-k26 --nightly --timeout-per-file 7200 --continue-on-error (log line ~4379).
• Attempt 1: crash during Multi-thread loading shards: 8% Completed | 5/64, printing HW Exception by GPU node-3 (Agent handle: ...) reason :GPU Hang ×14 and Fatal Python error (log ~4756–4770). Stack frames at crash are in model_loader/loader.py and the watchdog thread (srt/utils/watchdog.py).
• Attempt 2 (retry): [CI Test Method] TestKimiK26EvalMI35x.test_kimi_k26_gsm8k_accuracy re-run, crashed with HW Exception by GPU node-5 ... reason :GPU Hang ×3 during startup → ##[error]Process completed with exit code 134 (log 7122–7141).
• Config: TP=8, mem_fraction_static=0.826, prefill_attention_backend='aiter', decode_attention_backend='triton', enable_multithread_load=true, SGLANG_ROCM_FUSED_DECODE_MLA=0.
• Today's pre-test ensure_vram_clear.sh passed (no VRAM error), but Jul 5/8/9 runs failed that same check 3× (ERROR: VRAM usage exceeds threshold (5%) → FAILED: VRAM cleanup unsuccessful after 3 attempts) — evidence the MI35x runner GPUs have been in a leaked/bad state around this window.
• aiter unchanged today (dev image preserved, no override).

Hypothesised Causes (with confidence, with disconfirming evidence)

• [MEDIUM] MI35x 8-GPU runner hardware/driver-level fault — a GPU Hang HW exception fired during pure weight-load (host→device copies, no model compute yet), on two different GPU nodes across the two attempts (node-3, then node-5).
  • Supporting: crash is in the loader path, not in any kernel/attention code; the immediately preceding 3 nightlies failed the pre-test VRAM-clear check on the same runner, indicating persistent leaked GPU state; two distinct GPU nodes hung.
  • Disconfirming: today's own VRAM-clear check passed, so the GPU was nominally clean at job start; a pure infra fault does not by itself explain why Jul 4 passed cleanly.
  • How to verify: check runner linux-mi35x-gpu-8-zf775-runner-7k2d2 dmesg / GPU health; re-run the job on a different MI35x host; confirm whether other MI35x jobs in the same nightly also hung.
• [LOW] sglang code change between 854b46be99 (Jul 4 pass) and 9b4bb415d affecting the multithread weight loader.
  • Supporting: temporal — GPU-Hang symptom appeared Jul 10, within this window.
  • Disconfirming: the symptom is a HW-level GPU Hang, not a Python exception in loader code; the intervening Jul 5/8/9 failures were VRAM/infra, not loader crashes; no code-level evidence links any commit to the hang.
  • How to verify: bisect the loader-related commits in 854b46be99..9b4bb415d on a healthy MI35x host.

No hypothesis reaches HIGH; the two attempts hung on different GPU nodes during load, which most strongly implicates runner/driver state over a specific commit.

In-flight Fix Check

• VRAM-leak / GPU cleanup (infra cluster): ✅ #30431 "[AMD] ci: self-heal driver-level VRAM leaks + graceful GPU cleanup on shutdown" (open since 2026-07-07, updated 2026-07-11) — directly targets the driver-level VRAM leak / GPU-state symptom underlying this cluster; chase reviewers, do NOT open a duplicate.
• This test specifically: ✅ #30376 "[AMD] WIP - test(kimi-k26 mi35x)" (draft, open since 2026-07-07) — WIP on this exact test; track it.
• (Searched: is:pr is:open kimi mi35x and is:pr is:open VRAM clear.)

Suggested Triage Steps

• Inspect runner linux-mi35x-gpu-8-zf775-runner-7k2d2 health (dmesg, rocm-smi, GPU reset counters) around 2026-07-11 19:04–19:05 UTC; confirm whether GPU node-3/node-5 logged a hardware fault.
• Verify whether #30431 (self-heal VRAM/GPU cleanup) addresses the leaked-state precondition; if so, land it and re-run this nightly.
• Re-run nightly-8-gpu-mi35x-kimi-k26 on a different/known-healthy MI35x host to confirm the hang is runner-specific (infra) vs reproducible (code).
• Only if the hang reproduces on a healthy host: bisect loader-touching commits in 854b46be99..9b4bb415d.

Status (factual)

• Persistence: test file red since 2026-07-04 (last green); GPU-Hang symptom on 2 consecutive completed runs (Jul 10, Jul 11). Preceded by 3 VRAM-cleanup infra failures (Jul 5/8/9).
• Scope: nightly-8-gpu-mi35x-kimi-k26 job in nightly-test-amd.yml (schedule). Crash occurs at model-load, so accuracy signal for Kimi-K2.6 on MI35x is fully blocked.
• In-flight fix: ✅ #30431 (VRAM/GPU self-heal, infra) + ✅ #30376 (draft, this test).

nightly-8-gpu-mi35x-deepseek-r1-hicache — failed step(s): Accuracy Test MI35x (8-GPU DeepSeek-R1 HiCache)

Commit Info

• sglang: 9b4bb415d
• aiter: 9127c94a1 — v0.1.17.dev110+g9127c94a1 (source: Dockerfile default; dev/patched image build detected and preserved, AITER_COMMIT_OVERRIDE empty)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/test_deepseek_r1_hicache_mi35x.py	TestDeepSeekR1HiCacheMI35x.setUpClass (server launch / weight load)	HW Exception ... reason: GPU Hang → Fatal Python error: Aborted → exit 134	Log

The server crashed during model initialization (weight loading), so the actual accuracy test test_gsm8k never executed. This is a server-launch crash, not a test-assertion failure.

Failure Summary

Step 6 ("Accuracy Test MI35x (8-GPU DeepSeek-R1 HiCache)") aborted while the DeepSeek-R1-0528 server was loading weights across all 8 TP ranks. At 19:26:10 all ranks logged Load weight begin / Shared experts fusion optimization enabled, then ~4 minutes later (19:30:19) every GPU agent on node-3 raised HW Exception ... reason: GPU Hang, killing the process with SIGABRT (exit 134). The automatic retry with HF_HUB_OFFLINE=1 crashed the same way on node-5 (exit 134), failing the job.

Regression Status

Known recurring failure — but with different failure signatures each run, consistent with infrastructure/hardware instability on the MI35x 8-GPU runners rather than a single code regression. No green run for this test file in the queryable window (Never-passed in the lookback range Jul 3–11).

Recent history of test_deepseek_r1_hicache_mi35x.py (TestDeepSeekR1HiCacheMI35x) in job nightly-8-gpu-mi35x-deepseek-r1-hicache (branch main, event schedule, completed runs only):

Date	Run	Job	Test File Status	Failure Signature
Jul 11	29162254653	job	❌ Failed	GPU Hang (node-3, node-5), exit 134
Jul 10	29113206892	job	❌ Failed	GPU Hang, exit 134, 8× Aborted
Jul 9	29039566002	job	❌ Failed	VRAM cleanup unsuccessful after 3 attempts (pre-test infra step)
Jul 5	28749559519	job	❌ Failed	8× Memory access fault

Failure Cluster

GPU Hang HW exception during 8-GPU DeepSeek-R1 weight load on MI35x (broader cluster: MI35x 8-GPU runner instability — GPU Hang / memory access fault / VRAM-leak cleanup failures)

Facts (verified observations, no interpretation)

• HW Exception by GPU node-3 (Agent handle: 0x...) reason :GPU Hang printed for all 8 GPU agents, immediately followed by Fatal Python error: Aborted (log ~19:30:19).
• First attempt: First attempt failed with exit code 134; retry with HF_HUB_OFFLINE=1 also hit HW Exception by GPU node-5 ... reason :GPU Hang, then ##[error]Process completed with exit code 134.
• Exit code 134 = 128 + 6 (SIGABRT).
• The Python traceback printed at abort points into model_loader/weight_utils.py:_prepare_weights / loader.py / deepseek_v2.py — these are worker threads parked in weight download at the moment the abort signal was delivered, not a crash originating in that code.
• ~4-minute gap between last progress log (Shared experts fusion optimization enabled, 19:26:10) and the GPU Hang (19:30:19).
• aiter unchanged vs Dockerfile default ([CI-AITER-CHECK] confirms v0.1.17.dev110+g9127c94a1, no override); NVIDIA nightlies not affected — AMD-only.
• Historical runs of this exact job show three distinct infra symptoms (GPU Hang, Memory access fault, VRAM cleanup failure), not a consistent code-path crash.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [MEDIUM] MI35x 8-GPU runner hardware/driver instability (GPU Hang), not an sglang or aiter code regression.
  • Supporting: HW-level GPU Hang exception raised by the GPU agents themselves (not a Python-level error); reproduced on two different GPU nodes (node-3 then node-5) within the same job; the sibling runs fail with unrelated infra symptoms (VRAM leak, memory access fault) rather than a stable code crash; AMD-only.
  • Disconfirming: a genuine hang during weight-load could in principle be triggered by a specific kernel (e.g. shared-experts-fusion or aiter allreduce-fusion init) that only this workload exercises — cannot fully rule out a workload-specific trigger without a clean-hardware repro.
  • How to verify: re-run the job on a freshly power-cycled MI35x node; check dmesg/rocm-smi for XGMI/ring-hang faults; if it still hangs on clean hardware at the same phase, escalate to a workload-specific kernel investigation.
• [LOW] Workload-specific trigger during DeepSeek-R1 init on gfx950 (e.g. aiter allreduce-fusion / shared-experts fusion path).
  • Supporting: hang consistently occurs right after Shared experts fusion optimization enabled and Enable Aiter AllReduce Fusion for DeepseekV3ForCausalLM.
  • Disconfirming: log-line proximity is not causation — the hang is 4 minutes after that log line; other runs of the same job fail before this phase (VRAM cleanup step) entirely, arguing for a hardware root cause.
  • How to verify: bisect is not useful for an intermittent HW hang; instead pin the runner and re-run with --disable-cuda-graph / fusion flags off to see if the hang phase moves.

In-flight Fix Check

• In-flight fix: ✅ #30431 "[AMD] ci: self-heal driver-level VRAM leaks + graceful GPU cleanup on shutdown" (open since 2026-07-07) — directly targets driver-level VRAM leaks / GPU cleanup on the AMD runners, the same infra family as the Jul 9 VRAM-cleanup failure and plausibly the GPU-Hang leftovers. Chase reviewers rather than opening a duplicate. (API: search/issues?q=repo:sgl-project/sglang+is:pr+is:open+GPU+Hang+mi35x)
• No open PR specifically targeting a "GPU Hang during DeepSeek-R1 weight load" code path was found.

Suggested Triage Steps

• Re-run nightly-8-gpu-mi35x-deepseek-r1-hicache on a freshly power-cycled/known-healthy MI35x 8-GPU runner; if it passes, this is confirmed transient runner instability.
• On the affected runner, inspect dmesg and rocm-smi around 19:30 UTC for XGMI/ring hang or ECC faults on node-3/node-5.
• Verify #30431 merges cleanly and re-run this job to see whether the pre-test VRAM-leak/GPU-cleanup path resolves the recurring aborts.
• If the hang persists on clean hardware at the same Load weight/fusion phase, escalate to a workload-specific investigation (re-run with aiter allreduce-fusion and shared-experts fusion disabled to isolate).

Status (factual, no priority assignment)

• Persistence: nightly-8-gpu-mi35x-deepseek-r1-hicache has failed in every completed scheduled run in the lookback window (Jul 3–11); no passing run found for this test file.
• Scope: AMD MI35x 8-GPU nightly only; NVIDIA nightlies unaffected. Symptom varies run-to-run (GPU Hang / memory access fault / VRAM cleanup), pointing at runner-level instability.
• Blocked work: blocks the DeepSeek-R1 HiCache accuracy signal on MI35x (test_gsm8k never runs).
• In-flight fix: ✅ #30431 (open since 2026-07-07) — infra self-heal for driver-level VRAM leaks / GPU cleanup.

nightly-4-gpu — failed step(s): Nightly Test (4-GPU)

Commit Info

• sglang: 9b4bb415d
• aiter: 9127c94a1 — v0.1.17.dev110+g9127c94a1 (source: Dockerfile default; dev/patched image detected, rebuild skipped — AITER_COMMIT_OVERRIDE empty)

Failed Tests

Test File	Test Function	Error	Log
test/registered/vlm/test_encoder_dp.py	test_vlm_mmmu_benchmark (TestVLMEncoderDP)	Server process exited with code -9 following Memory access fault by GPU node-8 … Write access to a read-only page → retry() exceed maximum number of retries	Log

Failure Summary

The single test test_vlm_mmmu_benchmark in test/registered/vlm/test_encoder_dp.py failed. The test launches a VLM server with --mm-enable-dp-encoder --tp=4; shortly after startup (right after /model_info returns 200, during warmup/first health_generate) the GPU raises Memory access fault … Write access to a read-only page, the server dies, health checks return 503 for the full timeout, and the launcher reaps the process with code -9. Both randomly-selected models on the two retry attempts (Qwen/Qwen3-VL-32B-Instruct and Qwen/Qwen2.5-VL-72B-Instruct) hit the identical fault, so the crash is model-agnostic.

Regression Status

Known recurring / flaky failure (not introduced by the HEAD commit).

Recent history of test/registered/vlm/test_encoder_dp.py (test_vlm_mmmu_benchmark) in job nightly-4-gpu (branch main, event schedule, completed runs only):

Date	Run	Job	Test File Status	Failed Function	Error
Jul 11	29162254653	nightly-4-gpu	❌ Failed	test_vlm_mmmu_benchmark	code -9 / GPU mem fault
Jul 10 (PM)	29113206892	nightly-4-gpu	✅ Passed (1/1)	—	mem fault occurred but retry passed
Jul 10 (AM)	29039566002	nightly-4-gpu	❌ Failed	test_vlm_mmmu_benchmark	code -9 (Qwen2.5-VL-72B)
Jul 08	28964511688	nightly-4-gpu	⚪ Cancelled (excluded)	—	job cancelled
Jul 05	28749559519	nightly-4-gpu	❌ Failed	test_vlm_mmmu_benchmark	code -9 / GPU mem fault
Jul 04	28714602438	nightly-4-gpu	❌ Failed	test_vlm_mmmu_benchmark	code -9 / GPU mem fault

The identical Memory access fault … Write access to a read-only page symptom is present in the logs of Jul 04, Jul 05, Jul 10 (both runs), and Jul 11. It is intermittent: on Jul 10 PM the fault still occurred but the single retry succeeded, yielding a green job. First observed failure in the queryable window: Jul 04 2026 (earliest run inspected — the fault predates HEAD commit 9b4bb415d by ~1 week). No fully-clean run of this test file was found; the only "pass" (Jul 10 PM) was a retry-recovered flake.

Failure Cluster

GPU memory access fault (write to read-only page) during VLM DP-encoder server warmup

Facts (verified observations)

• Log (Jul 11): Memory access fault by GPU node-8 … on address 0x7ef49e800000. Reason: Write access to a read-only page. at 18:01:11, immediately after GET /model_info … 200 OK; server then answers GET /health_generate … 503 until timeout, then Server process exited with code -9.
• Test config: TestVLMEncoderDP.other_args = ["--mm-enable-dp-encoder", "--tp=4", "--cuda-graph-max-bs-decode", "32"] (test/registered/vlm/test_encoder_dp.py:22-28).
• CI picks one random model per attempt (random.choice(MODELS), test_encoder_dp.py:37); the retry harness re-invoked the method, so two distinct models (Qwen3-VL-32B, Qwen2.5-VL-72B) were exercised and both faulted.
• Same fault symptom + same test file reproduced across Jul 04 / 05 / 10 / 11 (verified from each run's downloaded log).
• aiter is a dev/patched image build (v0.1.17.dev110+g9127c94a1), unchanged vs the Dockerfile default; no AITER_COMMIT_OVERRIDE.
• Ran 1 test in 1576.237s … FAILED (errors=1); harness reported Test Summary: 0/1 passed, exit code 255.

Hypothesised Causes (with confidence)

• [LOW] GPU memory access fault in the VLM DP-encoder path (--mm-enable-dp-encoder + --tp=4) during warmup/first forward.
  • Supporting: fault fires only in this DP-encoder test, model-agnostic (2 models), reproducible on the exact same code path across ≥4 dates; occurs right after startup during the first generation.
  • Disconfirming: no captured host/Python stack trace pinpoints an sglang or aiter frame (the fault is an async GPU write-to-read-only-page, so the crashing kernel is not identified in the log); it is intermittent (Jul 10 PM passed on retry), which is atypical of a deterministic code bug.
  • How to verify: reproduce on AMD MI30x with --mm-enable-dp-encoder --tp=4 for Qwen2.5-VL-72B/Qwen3-VL-32B under HSA_ENABLE_SDMA=0 / AMD_SERIALIZE_KERNEL=3 to surface the faulting kernel; or bisect nightly runs before Jul 04 to find the first faulting date.
• [SPECULATION] HEAD commit 9b4bb415d (cuda-graph breakable-graph shared buffer sizing) as cause.
  • Supporting: touches cuda-graph capture, and the fault occurs after cuda-graph capture completes.
  • Disconfirming: the identical fault exists on Jul 04/05, well before this commit merged — it cannot be the root cause. Listed only to explicitly rule it out.

No hypothesis reaches MEDIUM confidence; the crashing kernel is not identifiable from the log alone. Needs a repro with kernel serialization/coredump on AMD.

In-flight Fix Check

• In-flight fix: ⚠️ candidate #18721 "[BUG] fix mm_enable_dp_encoder hang for Qwen3-VL models" — targets the exact mm_enable_dp_encoder path and Qwen3-VL, but describes a hang, not a memory-access-fault; may or may not address this symptom. Also related: #14977 "Reduce GPU memory usage in DP vision encoder", #25951 "[VLM][DP encoder] shard pixel_values per-rank".
• In-flight fix: ❌ none found that explicitly targets the GPU memory-access-fault symptom in test_encoder_dp.py.
• (Searched: is:pr is:open dp-encoder, mm-enable-dp-encoder, memory access fault.)

Suggested Triage Steps

• Reproduce on AMD MI30x with --mm-enable-dp-encoder --tp=4 for the two Qwen-VL models under AMD_SERIALIZE_KERNEL=3 (and/or HSA_ENABLE_SDMA=0) to capture the faulting kernel/stack, then enable a ROCm coredump to identify the write-to-read-only-page site.
• Bisect nightly nightly-4-gpu runs before Jul 04 to find the first date this fault appeared, then git log --oneline <last_pass_sha>..<first_fail_sha> filtered to VLM / DP-encoder / mm-processor / cuda-graph paths.
• Check whether the fault is aiter-linked by rerunning the test against a non-dev aiter image (the current run used a dev/patched aiter build).
• Chase #18721 reviewers and verify whether its fix also resolves the memory fault (not just the hang); if not, a new issue/PR is needed for the DP-encoder GPU fault.
• Given intermittency, consider quarantining test_encoder_dp.py on AMD nightly until root-caused, to unblock the rest of the nightly-4-gpu signal.

Status

• Persistence: Recurring/flaky since at least Jul 04 2026 (failed Jul 04, 05, 10-AM, 11; passed only on Jul 10-PM via retry recovery). Fault symptom present in every inspected run.
• Scope: nightly-4-gpu job of nightly-test-amd.yml; single test file test/registered/vlm/test_encoder_dp.py. Fault is model-agnostic across the VLM model list.
• Blocked work: Blocks the AMD nightly 4-GPU VLM DP-encoder signal; the whole nightly-4-gpu job goes red whenever the retry does not recover.
• In-flight fix: ⚠️ candidate #18721 (targets mm_enable_dp_encoder Qwen3-VL hang — symptom mismatch, needs verification); no dedicated fix for the memory-access-fault found.

nightly-accuracy-8-gpu-mi35x-deepseek-v32 — failed step(s): Accuracy Test MI35x (8-GPU DeepSeek-V3.2)

Commit Info

• sglang: 9b4bb415d
• aiter: 9127c94a1 (v0.1.17.dev110+g9127c94a1, source: dev image detected → rebuild skipped)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/accuracy/mi35x/test_deepseek_v32_eval_mi35x.py	deepseek-ai/DeepSeek-V3.2 (basic) eval driver (script-level, no pytest fn)	HW Exception by GPU node-3 reason :GPU Hang → Fatal Python error: Aborted (exit 134)	Log

Failure Summary

The single-test job nightly-accuracy-8-gpu-mi35x-deepseek-v32 crashed while bringing up the DeepSeek-V3.2 server (TP=8, DSA attention backend, fp8 checkpoint, bf16 KV cache) on the MI35x/gfx950 runner. All 8 TP ranks reached Load weight begin (~277 GB free), and ~20 s later every rank raised HW Exception by GPU node-3 reason :GPU Hang, aborting with exit code 134. The eval never produced an accuracy score. Both the primary attempt and the HF_HUB_OFFLINE=1 retry crashed identically.

Regression Status

Recurring failure — GPU Hang for 2 consecutive completed runs (07-10, 07-11). The test file has been red or infra-blocked every completed run since 07-07; last clean pass was 07-06.

Recent history of test_deepseek_v32_eval_mi35x.py in job nightly-accuracy-8-gpu-mi35x-deepseek-v32 (branch main, event schedule, completed runs only):

Date	Run	Test File Status	Failure signature
Jul 11	29162254653	❌ Failed	GPU Hang during weight load (exit 134)
Jul 10	29113206892	❌ Failed	GPU Hang in _load_w13 (fused_moe_triton/layer.py:519) (exit 134)
Jul 09	29039566002	❌ Failed	VRAM precheck: "VRAM usage exceeds threshold" — test never ran (exit 1)
Jul 08	28964511688	❌ Failed	rerun churn: RuntimeError: ninja exited with status 1 + weight-load tracebacks
Jul 07	28888348134	❌ Failed	VRAM precheck: "VRAM usage exceeds threshold" — test never ran (exit 1)
Jul 06	28813271245	✅ Passed	—
Jul 05	28749559519	✅ Passed	—
Jul 04	28714602438	✅ Passed	—
Jul 03	28676359018	❌ Failed	exit 1 (pre-test env)

First observed GPU-Hang failure: 2026-07-10. Last known clean pass: 2026-07-06 (3abdbab9b). Note: today's run 29162254653 is workflow-status queued, but this job itself completed (failure) — not in-flight.

Failure Cluster

GPU Hang during DeepSeek-V3.2 fp8 MoE weight load on MI35x (gfx950)

Facts (verified observations)

• All 8 TP ranks logged Load weight begin. avail mem=~277 GB at 19:09:29–30; HW Exception by GPU node-3 ... reason :GPU Hang fired across ranks at 19:09:50, then Fatal Python error: Aborted. Watchdog threads (srt/utils/watchdog.py) appear in the abort stacks (they are the reporting threads, not the fault site).
• Exit code 134 (SIGABRT) on both attempts; retry with HF_HUB_OFFLINE=1 reproduced the same GPU Hang (node-5) at 19:10:37 → ##[error]Process completed with exit code 134.
• Config: attention_backend='dsa', dsa_prefill_backend=tilelang, dsa_decode_backend=tilelang, kv_cache_dtype='bfloat16', page_size=1, mem_fraction_static=0.85, fp8 checkpoint (Fp8MoEMethod).
• The 07-10 run's GPU Hang stack top-frame is _load_w13 (moe/fused_moe_triton/layer.py:519) reached via do_load_weights (deepseek_common/deepseek_weight_loader.py:434) — i.e. the hang is in the MoE weight-loading path, consistent with today's crash timing (immediately after Load weight begin).
• Container image in use: rocm/sgl-dev:v0.5.15-rocm700-mi35x-20260710.
• aiter version unchanged (dev image v0.1.17.dev110+g9127c94a1).

Hypothesised Causes (with confidence)

• [LOW] GPU Hang triggered during DeepSeek-V3.2 fp8 MoE weight load on gfx950 — possibly a bad/hung GPU state on the runner rather than a code regression.
  • Supporting: 07-07 and 07-09 completed runs failed the pre-test "Ensure VRAM is clear" step ("VRAM usage exceeds threshold"), meaning a prior job left the GPU dirty; a physically hung GPU explains both the leftover-VRAM prechecks and the mid-load hang. Symptom is confined to one runner/GPU node (node-3, node-5).
  • Disconfirming: the hang is reproducible across two attempts and two consecutive nights, which is more deterministic than a random runner flake; a purely transient GPU state would not persist 07-10→07-11.
  • How to verify: check MI35x runner GPU health (rocm-smi, dmesg) at run time; re-run the isolated test on a known-clean gfx950 node.
• [LOW] A change in the 07-06..07-10 window regressed DeepSeek-V3.2 startup on mi35x. The window has 74 commits; candidates touching this path include 789bc3995 (#30587, "Bump miles rocm700-mi35x base image and migrate nightly test onto it" — matches the new ...-mi35x-20260710 image), and MoE/DSA changes 87992eeec (#30460), bda1dc0d9 (#30645).
  • Supporting: last clean pass 07-06 precedes these; the base-image bump directly changes the mi35x runtime/driver stack the hang occurs on.
  • Disconfirming: the window is muddied — the 07-07..07-09 runs were infra-blocked (VRAM precheck / ninja build), so no clean "first failing commit" can be pinned; the hang could predate 07-10 but was masked by the earlier infra failures.
  • How to verify: bisect 3abdbab9b..7998fecfd on a clean mi35x node; specifically A/B the base-image bump #30587.

No hypothesis reaches MEDIUM confidence: the fault site is masked by a GPU Hang (no Python exception), and the regression window is obscured by intervening infra failures.

In-flight Fix Check

• In-flight fix: ✅ [#30431](https://github.com/sgl-project/sglang/pull/30431) (open since 2026-07-07) — "[AMD] ci: self-heal driver-level VRAM leaks + graceful GPU cleanup on shutdown" — addresses the dirty-GPU / VRAM-leak symptom (the 07-07/07-09 precheck failures) but not necessarily the weight-load GPU Hang root cause; worth landing to stop the runner-state cascade.
• In-flight fix: ❌ none found specifically targeting the DeepSeek-V3.2 mi35x GPU Hang (searched deepseek+v32+mi35x, mi35x+GPU+Hang, dsv32+mi35x). Note debug branches exist upstream (fix/mi35x-deepseek-v32-regression, bingxche/debug-dsv32-mi35x-mtp-mem-access-fault) but no open PR.

Suggested Triage Steps

• Inspect MI35x runner GPU health at run time (rocm-smi, kernel dmesg for XGMI/HW-exception entries) to determine whether node-3/node-5 is physically hung vs. a software regression.
• Land/verify #30431 to stop hung GPUs from cascading into VRAM-precheck failures on subsequent nightly jobs.
• Bisect 3abdbab9b..7998fecfd on a clean gfx950 node, prioritizing an A/B of base-image bump #30587; re-run test_deepseek_v32_eval_mi35x.py in container rocm/sgl-dev:v0.5.15-rocm700-mi35x-20260710.
• If the hang reproduces on a clean node with the pre-07-06 image, treat as a genuine mi35x DeepSeek-V3.2 startup regression; if it only reproduces on the new image, escalate the image/driver bump.

Status

• Persistence: GPU Hang failing 2 consecutive completed nights (2026-07-10, 2026-07-11); test file red or infra-blocked every completed run since 2026-07-07; last clean pass 2026-07-06.
• Scope: nightly-accuracy-8-gpu-mi35x-deepseek-v32 (this analysis). Sibling mi35x-deepseek-v32 jobs (-mtp, perf-basic, perf-mtp) are also failing in the same recent runs — likely the same runner/GPU cluster, worth cross-checking.
• Blocked work: DeepSeek-V3.2 accuracy signal on AMD MI35x.
• In-flight fix: ✅ #30431 for the VRAM-leak/cleanup symptom; ❌ none for the weight-load GPU Hang itself.

nightly-8-gpu-mi35x-grok1-int4 — failed step(s): Accuracy Test MI35x (8-GPU Grok1-INT4)

Commit Info

• sglang: 9b4bb415ddc315ebcf1f8fcc280537f0b30938dc
• aiter: 9127c94a18e4398e1eba91f6639e910f0994ad02 — version v0.1.17.dev110+g9127c94a1 (source: Dockerfile default; dev/patched image detected → rebuild skipped, no override)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/accuracy/mi35x/test_grok1_int4_eval_mi35x.py	server launch / model weight load (script-level, not an assertion)	HW Exception … reason: GPU Hang → Fatal Python error: Aborted → exit 134	Log

Failure Summary

The grok1-int4 MI35x accuracy test crashed while launching the server. During weight loading (Load weight begin at 19:04:32) the runtime hit a GPU Hang HW exception on GPU node-3 at 19:05:09, aborting the process (SIGABRT / exit 134). The harness then retried with HF_HUB_OFFLINE=1, but the weights were not yet in the disk cache (the hung first attempt never finished downloading amd/grok-1-W4A8KV8), so the retry failed with a secondary OSError: We couldn't connect to 'https://huggingface.co' and again exit 134. The accuracy eval never ran.

Regression Status

Known recurring failure (this GPU-Hang signature) — recurring on Jul 10 & Jul 11, with one clean pass on Jul 08.

Recent history of test_grok1_int4_eval_mi35x.py in job nightly-8-gpu-mi35x-grok1-int4 (branch=main, event=schedule, completed runs only):

Date	Run	Job	Test File Status	Failed Function	Error
Jul 11	29162254653	job	❌ Failed	server/model load	GPU Hang → exit 134
Jul 10	29113206892	job	❌ Failed	server/model load	GPU Hang → exit 134 (same cluster)
Jul 09	29039566002	job	❌ Failed	pre-test GPU check	VRAM usage exceeds threshold → exit 1 (different cluster)
Jul 08	28964511688	job	✅ Passed	—	—
Jul 07	28888348134	job	❌ Failed	pre-test GPU check	VRAM usage exceeds threshold → exit 1 (different cluster)

First observed (GPU-Hang cluster): Jul 10 2026. Last known pass: Jul 08 2026 (SHA 8d2b66fd9071). Note the Jul 09/Jul 07 failures are a separate infra cluster (leftover VRAM on the shared runner), not this GPU hang.

Failure Cluster

GPU Hang HW exception during grok1-int4 model weight loading (exit 134), with secondary HF-offline cache-miss on retry.

Facts (verified observations)

• HW Exception by GPU node-3 (Agent handle: …) reason :GPU Hang printed 19:05:09, immediately followed by Fatal Python error: Aborted across multiple TP threads (verbatim from log).
• Fatal thread stacks at abort time were in download_weights_from_hf (weight_utils.py:571) → _prepare_presharded_weights (grok.py:960) → _get_weights_iterator (loader.py:548), several ranks blocked on a filelock … __enter__ for the HF download.
• First attempt failed with exit code 134 (line ~5031) → Retrying with HF_HUB_OFFLINE=1 (offline mode to use cached models)... (line ~5032).
• Retry failed: huggingface_hub.errors.LocalEntryNotFoundError … outgoing traffic has been disabled → OSError: We couldn't connect to 'https://huggingface.co', then more GPU Hang on node-5, ##[error]Process completed with exit code 134.
• AITER_COMMIT_OVERRIDE is empty; aiter is the Dockerfile default dev image — not a scout run, so no aiter override in play.
• Server config: tp_size=8, attention_backend='aiter', quantization='fp8', model amd/grok-1-W4A8KV8 on gfx950 (MI35x).
• Same GPU-Hang + exit-134 signature verified in the Jul 10 run (86430243477); Jul 09/Jul 07 failures are a different VRAM usage exceeds threshold signature.

Hypothesised Causes (with confidence)

• [LOW] GPU/driver-level hang on the MI35x (gfx950) runner during grok1-int4 weight load — an infrastructure/hardware hang rather than a specific code change.
  • Supporting: HW Exception … reason: GPU Hang is a driver-reported hardware hang; it fires during weight download while ranks are blocked on a filelock (I/O wait), not during compute; the same test passed Jul 08 on a comparable config, and Jul 09/07 failures on the same job were a related GPU-state/VRAM-leak infra symptom.
  • Disconfirming: cannot fully exclude a code path — the hang recurs on two consecutive days (Jul 10, 11) which could indicate a regression landing after the Jul 08 pass; the abort stack goes through grok.py weight-prep code, so a grok1-int4 / fp8 load path issue is not ruled out.
  • How to verify: re-run the job on a freshly reset MI35x node; if it passes, confirms transient GPU hang. If it reproduces, bisect 8d2b66fd9071..7998fecfd114 (Jul 08 pass → Jul 10 first GPU-hang), focusing on grok / weight-loader / fp8-on-gfx950 commits.
• [FACT] The secondary OSError/huggingface.co failure is a consequence, not the cause — it only appears on the HF_HUB_OFFLINE=1 retry because the first (hung) attempt never finished caching the weights. Do not treat the HF error as the root failure.

No hypothesis reaches MEDIUM; the primary crash is a GPU hang with no code-level evidence. Needs a clean-node re-run and/or bisect of 8d2b66fd9071..7998fecfd114.

In-flight Fix Check

• In-flight fix: ⚠️ candidate #30431 "[AMD] ci: self-heal driver-level VRAM leaks + graceful GPU cleanup on shutdown" — targets AMD driver-level GPU-state cleanup/leaks on shared runners; plausibly relevant to both this GPU hang and the Jul 09/07 VRAM-threshold cluster, but not a confirmed fix for the hang itself. (Query: is:pr is:open GPU hang amd.)
• In-flight fix: ❌ none found for a grok1-int4-specific weight-load fix (query: is:pr is:open grok mi35x → only unrelated #20894).

Suggested Triage Steps

• Re-run nightly-8-gpu-mi35x-grok1-int4 on a freshly power-cycled / reset MI35x node to test the transient-GPU-hang hypothesis.
• If it reproduces, bisect 8d2b66fd9071..7998fecfd114, prioritizing commits touching grok.py, the weight loader, or fp8/gfx950 quant paths.
• Track #30431 — if merged, re-check whether the GPU-Hang recurrence clears alongside the VRAM-threshold cluster.
• Consider pre-caching amd/grok-1-W4A8KV8 weights on the runner so the HF_HUB_OFFLINE=1 retry can actually succeed and produce a cleaner primary-failure signal (removes the misleading secondary HF OSError).

Status

• Persistence: GPU-Hang signature failing 2 consecutive scheduled runs (Jul 10 06:41-ish, Jul 11); last clean pass Jul 08. Broader job has been red most days this week (Jul 04–11) across two distinct clusters.
• Scope: nightly-8-gpu-mi35x-grok1-int4 job in nightly-test-amd.yml (MI35x/gfx950 only). The non-MI35x nightly-8-gpu-grok1-int4 sister job was cancelled, not comparable.
• Blocked work: no grok1-int4 accuracy signal on MI35x for the last 2 nights.
• In-flight fix: candidate #30431 (AMD GPU cleanup); no grok-specific fix open.

nightly-8-gpu-mi35x-qwen3-235b-mxfp4 — failed step(s): Accuracy Test + Performance Test MI35x (8-GPU Qwen3-235B-MXFP4)

Commit Info

• sglang: 9b4bb415ddc315ebcf1f8fcc280537f0b30938dc
• aiter: 9127c94a1 — v0.1.17.dev110+g9127c94a1 (source: dev image preserved; [CI-AITER-CHECK] reports "Dev/patched version detected → skipping rebuild", Dockerfile default was the same SHA 9127c94a1)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/test_qwen3_instruct_mxfp4.py	TestQwen3Instruct2507MXFP4.setUpClass (server launch)	HW Exception by GPU node reason :GPU Hang → Fatal Python error: Aborted → exit 134	Log

The test body (test_a_gsm8k, accuracy assert >0.93) never ran — the crash occurred during model weight loading in setUpClass.

Failure Summary

The job launches an amd/Qwen3-235B-A22B-Instruct-2507-mxfp4 server (TP=4, EP=2, aiter backend) on an 8-GPU MI35x (gfx950) runner. The server process aborted with exit code 134 (SIGABRT) during model weight loading — the first attempt crashed at 4% of shard loading (Multi-thread loading shards: 1/26) with HW Exception by GPU node-3 ... reason :GPU Hang; the offline-mode retry crashed even earlier on GPU node-5 with the same GPU Hang. The GSM8K accuracy test was never reached.

Regression Status

Never-passed (no green run found) — lookback window Jul 4 – Jul 11 2026 (schedule event, main, completed runs only).

Recent history of test/registered/amd/test_qwen3_instruct_mxfp4.py (TestQwen3Instruct2507MXFP4) in job nightly-8-gpu-mi35x-qwen3-235b-mxfp4:

Date	Run	Job	Test File Status	Failed Function	Error
Jul 11	29162254653	job	❌ Failed	setUpClass	GPU Hang during weight load → exit 134
Jul 10	29113206892	job	❌ Failed	setUpClass	GPU Hang during model load → exit 134 (same cluster)
Jul 9	29039566002	job	❌ Failed (test never ran)	— (pre-test)	VRAM cleanup pre-check → exit 1
Jul 8	28964511688	job	❌ Failed (test never ran)	— (pre-test)	VRAM cleanup pre-check → exit 1
Jul 7	28888348134	job	❌ Failed	setUpClass	Scheduler never ready (wait_for_ready) → exit 255
Jul 5	28749559519	job	❌ Failed (test never ran)	— (pre-test)	VRAM cleanup pre-check → exit 1
Jul 4	28714602438	job	❌ Failed (test never ran)	— (pre-test)	VRAM cleanup pre-check → exit 1

No passing run for this test file exists in the queryable window. Jul 6/1/Jun-30 runs were cancelled at the workflow level (not comparable). First observed failure: earliest queryable run (Jul 4). No last-known-passing date exists.

Failure Cluster

GPU Hang HW exception during Qwen3-235B weight loading on MI35x (SIGABRT / exit 134)

Facts (verified observations, no interpretation)

• Verbatim (attempt 1, line 4612-4619): Multi-thread loading shards: 4% Completed | 1/26 ... HW Exception by GPU node-3 (Agent handle: ...) reason :GPU Hang followed by Fatal Python error: HW Exception ... reason :GPU Hang and Fatal Python error: Aborted.
• Verbatim (attempt 2, offline retry): HW Exception by GPU node-5 (Agent handle: ...) reason :GPU Hang → ##[error]Process completed with exit code 134.
• Both attempts crashed on different physical GPU nodes (node-3, then node-5) at the model-load stage, before any forward pass.
• Reproducibility: 2/2 attempts in this run; the same GPU-Hang→exit-134 signature also in Jul 10's run.
• aiter version unchanged (v0.1.17.dev110+g9127c94a1) between Jul 10 and Jul 11 — both GPU-Hang runs used the identical aiter dev image ([CI-AITER-CHECK] markers).
• The 5 non-GPU-Hang historical failures fall into two other clusters: a pre-test VRAM cleanup guard tripping (ERROR: VRAM usage exceeds threshold (5%) ... → exit 1, test never launched) on Jul 4/5/8/9, and a scheduler-ready timeout (_wait_for_scheduler_ready / wait_for_ready, exit 255) on Jul 7.
• Test config (test_qwen3_instruct_mxfp4.py:34-41): TP=4, EP=2, --attention-backend aiter, SGLANG_USE_AITER=1, SERVER_LAUNCH_TIMEOUT=3600.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [MEDIUM] Runner/hardware or GPU-state health issue on the MI35x (gfx950) fleet, not an sglang code regression.
  • Supporting: crash is a hardware-level GPU Hang HW exception during weight load (before any model forward pass / kernel dispatch by test code); it reproduced on two different physical GPU nodes in one run; the same job stack shows unrelated failure modes on other days (VRAM-not-clean pre-check exit 1 on Jul 4/5/8/9, scheduler-ready timeout exit 255 on Jul 7), indicating an unstable runner environment rather than one code path.
  • Disconfirming: a GPU Hang can also be triggered by a bad kernel/allocation from sglang or aiter during weight materialization; the log does not include a device-side fault address to confirm it is purely hardware.
  • How to verify: check dmesg/rocm-smi GPU health and prior-job residue on the runner; re-run the job on a freshly-reset MI35x node; try --attention-backend triton and/or SGLANG_USE_AITER=0 to rule out the aiter path.
• [LOW] mxfp4 weight-loading / gfx950 code path defect (sglang or aiter) hangs the GPU during shard load.
  • Supporting: this is a large (235B) mxfp4 model on newer gfx950 hardware; the crash is deterministic at the load stage.
  • Disconfirming: aiter version is unchanged across the two GPU-Hang runs, so an aiter regression cannot explain a newly-appearing hang; there is no green baseline to establish a regression window (test never passed), so no code change can be pinned; the earlier failures on this job were unrelated (VRAM/timeout), consistent with environmental flakiness.
  • How to verify: attempt a local load of amd/Qwen3-235B-A22B-Instruct-2507-mxfp4 on a known-good gfx950 box with the same sglang+aiter SHAs.
• [LOW] Suite-registration anomaly. Log prints Warning: Unknown suite nightly-8-gpu-mi35x-qwen3-235b-mxfp4 for backend AMD, nightly=True, yet the test is still enabled and launched — so this is a warning, not the failure cause.
  • Disconfirming: the test clearly launched (server_args printed, weights began loading), so registration is functional.

No hypothesis above MEDIUM confidence for a code root cause; because the test has no green run in the queryable window, there is no pass→fail commit window to bisect.

In-flight Fix Check

• In-flight fix: ❌ none found — no open PR directly targets the MI35x Qwen3-235B mxfp4 GPU Hang. Searched search/issues?q=repo:sgl-project/sglang+is:pr+is:open+mi35x, +Qwen3-235B+mxfp4, +gfx950+hang.
• Related (not a fix): #30639 "[AMD][CI] Rebalance nightly 8-GPU jobs off MI325: move DeepSeek-V3.1 / Qwen3-235…" touches placement of this job's family — worth flagging to the AMD CI owner in case the intent is to relocate this suite.

Suggested Triage Steps

• Inspect the MI35x runner's GPU health and residual state (prior-job VRAM residue is already flagged by the Jul 4/5/8/9 pre-check exit-1 failures) — rocm-smi, dmesg for GPU Hang / XGMI / ECC events at ~19:17–19:18 UTC on Jul 11.
• Re-run the job on a freshly-reset MI35x node; if it passes, the GPU Hang is environmental.
• If it reproduces on clean hardware, A/B the attention path: re-run once with --attention-backend triton (and SGLANG_USE_AITER=0) to isolate aiter vs sglang loader.
• Attempt a standalone load of amd/Qwen3-235B-A22B-Instruct-2507-mxfp4 (TP=4, EP=2) on a known-good gfx950 box with SHAs 9b4bb415 + aiter 9127c94a1 to confirm/deny a code-path defect.
• Since the test has never passed on this runner, consider whether it is a not-yet-validated port to MI35x; if hardware is exonerated, coordinate with the owner of #30639.

Status (factual, no priority assignment)

• Persistence: Failing in every completed run in the queryable window (Jul 4 – Jul 11, 7 runs). The GPU-Hang→exit-134 variant specifically appears on Jul 10 and Jul 11 (2 consecutive runs); earlier runs failed with different signatures (VRAM pre-check exit 1, scheduler-ready timeout exit 255).
• Scope: Single job nightly-8-gpu-mi35x-qwen3-235b-mxfp4 in nightly-test-amd.yml. No green baseline exists for this test file.
• Blocked work: Blocks Qwen3-235B mxfp4 accuracy/perf CI signal on MI35x (gfx950).
• In-flight fix: ❌ none found; related placement PR #30639.

---

Generated by amd-bot using Claude Code CLI (last updated: 2026-07-12 23:22 UTC)

[amd-bot]

nightly-test-amd.yml — Per-Job Analysis (continued 4/8)

nightly-4-gpu-mi35x-minimax-m25 — failed step(s): Accuracy Test MI35x (4-GPU MiniMax-M2.5)

Commit Info

• sglang: 9b4bb415ddc315ebcf1f8fcc280537f0b30938dc
• aiter: 9127c94a18e4398e1eba91f6639e910f0994ad02 (v0.1.17.dev110+g9127c94a1, source: dev image preserved — Dev/patched version detected → skipping rebuild)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/accuracy/mi35x/test_minimax_m25_tp4_eval_mi35x.py	N/A (server process crash during model load, no test body reached)	HW Exception by GPU node-3/node-5 reason: GPU Hang → Fatal Python error: Aborted → exit 134	Log

Failure Summary

The MiniMax-M2.5 TP4 accuracy job on MI35x (gfx950) crashed with exit code 134 (SIGABRT). On the first attempt the process aborted at ~2/125 weight shards loaded (Multi-thread loading shards: 2% Completed) when the runtime raised HW Exception by GPU node-3 ... reason :GPU Hang; the abort surfaced inside the MoE weight loader (fused_moe_triton/layer.py → minimax_m2.py:load_weights). The offline retry (HF_HUB_OFFLINE=1) hit the identical GPU Hang on node-5 and aborted again. The test body (accuracy eval) never executed.

Regression Status

Known recurring failure (GPU Hang cluster began Jul 10; a different infra failure occupied Jul 8-9). Last green run for this test file: Jul 5 (accuracy=0.933 threshold=0.93 PASS).

Recent history of test_minimax_m25_tp4_eval_mi35x.py in job nightly-4-gpu-mi35x-minimax-m25 (branch=main, event=schedule, completed runs only):

Date	Run	Job	Test File Status	Failed Function	Error
Jul 11	29162254653	job	❌ Failed	N/A (crash on load)	GPU Hang → exit 134
Jul 10	29113206892	job	❌ Failed	N/A (crash on load)	GPU Hang → exit 134
Jul 9	29039566002	job	⚠️ Failed (different cluster)	pre-test VRAM check	VRAM usage exceeds threshold (5%) → exit 1 (infra)
Jul 8	28964511688	job	⚠️ Failed (different cluster)	pre-test VRAM check	VRAM usage exceeds threshold (5%) → exit 1 (infra)
Jul 5	28749559519	job	✅ Passed	—	accuracy=0.933 PASS

(Jul 6-7 scheduled runs were cancelled — job did not run. First GPU-Hang failure: Jul 10. Last known pass: Jul 5.)

Failure Cluster

GPU Hang HW exception during model weight loading on MI35x (exit 134)

Facts (verified observations)

• Log line 4148: Multi-thread loading shards: 2% Completed | 2/125 [00:14<12:20, 6.02s/it]HW Exception by GPU node-3 (Agent handle: ...) reason :GPU Hang, immediately followed by Fatal Python error: Aborted (lines 4152-4166).
• Abort traceback top frames: minimax_m2.py:1379 load_weights → fused_moe_triton/layer.py weight_loader / _load_w13 / _weight_loader_impl → loader.py:829 load_weights_and_postprocess → model_runner.py:1455 load_model. The crash is in the weight-copy path, before any forward pass.
• First attempt exit code 134 (line 4507); offline retry also hit GPU Hang on node-5 and exited 134 (lines 4547-4550).
• The GPU Hang manifested on two different physical nodes across the two attempts (node-3, then node-5).
• aiter version unchanged: dev image v0.1.17.dev110+g9127c94a1 preserved ([CI-AITER-CHECK] lines 3944-3946), matching the Dockerfile gfx950 default.
• Same job on Jul 8-9 failed with an unrelated VRAM usage exceeds threshold (5%) pre-test cleanup error (exit 1) — the mi35x runner pool has shown GPU-state hygiene problems for several days.
• Docker image: rocm/sgl-dev:v0.5.15-rocm700-mi35x-20260710.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [MEDIUM] Runner/hardware GPU-state issue on the MI35x pool (wedged/hung GPU), not a code regression.
  • Supporting: the HW Exception ... reason :GPU Hang fires during trivial host→device weight copy at 2/125 shards (no model compute yet); it recurs on two different physical nodes (node-3, node-5) within one job; the same runner pool threw VRAM usage exceeds threshold infra failures on Jul 8-9; GPU Hang is a driver/hardware-level fault, not a Python exception.
  • Disconfirming: the test passed cleanly on Jul 5 on the same pool, so the pool is not permanently broken; a code path that corrupts GPU state during load could in principle also produce a hang (not ruled out).
  • How to verify: re-run the job on a freshly-reset MI35x node; check rocm-smi / dmesg on node-3 and node-5 for XGMI/HBM ECC or ring-timeout errors around the run timestamp.
• [LOW] sglang code regression in the weight-loading / MoE path between Jul 5 and Jul 10.
  • Supporting: last pass Jul 5 (48ba79c11e1f), first GPU-Hang failure Jul 10 (7998fecfd114); the abort traceback runs through fused_moe_triton/layer.py.
  • Disconfirming: a GPU Hang HW exception during a plain weight memcpy is not a typical software-bug symptom (those raise Python/HIP errors, not hardware hangs); the intervening Jul 8-9 failures were infra VRAM leaks, not this crash, so no clean "code changed → hang appeared" boundary exists.
  • How to verify: bisect 48ba79c11e1f..7998fecfd114 only after the hardware hypothesis is ruled out by a clean-node rerun.

In-flight Fix Check

• In-flight fix: ✅ #30431 [AMD] ci: self-heal driver-level VRAM leaks + graceful GPU cleanup on shutdown — directly targets AMD driver-level GPU-state leaks/cleanup, which is the most probable class of this failure (and the Jul 8-9 VRAM failures on the same pool). Chase reviewers; do NOT open a duplicate.
• In-flight fix: ❌ none found specifically for MiniMax-M2.5 MI35x GPU-Hang-on-load. (Searched open PRs for minimax+mi35x and GPU Hang; call: search/issues?q=repo:sgl-project/sglang+is:pr+is:open+....)

Suggested Triage Steps

• Re-run nightly-4-gpu-mi35x-minimax-m25 on a freshly power-cycled / reset MI35x node to confirm whether the GPU Hang is node-specific (fastest disconfirmation of the hardware hypothesis).
• Inspect dmesg / rocm-smi --showreset on nodes node-3 and node-5 around 2026-07-11 19:15 UTC for HBM ECC, XGMI, or ring-timeout events.
• Verify #30431 applies to the MI35x runner path and land it, then watch the next 2-3 nightly runs of this job.
• Only if a clean-node rerun still hangs: bisect 48ba79c11e1f..7998fecfd114, focusing on fused_moe_triton/ and model_loader/loader.py changes.

Status

• Persistence: GPU-Hang cluster failing 2 consecutive completed runs (Jul 10 → Jul 11); test file last green Jul 5. The job has produced no green run since Jul 5 (Jul 8-9 failed on a separate infra VRAM issue).
• Scope: single job nightly-4-gpu-mi35x-minimax-m25 in nightly-test-amd.yml; symptom is confined to the MI35x (gfx950) runner pool, which also showed VRAM-leak infra failures Jul 8-9.
• Blocked work: MiniMax-M2.5 TP4 accuracy signal on MI35x is unavailable.
• In-flight fix: ✅ #30431 (AMD driver-level VRAM/GPU cleanup) — likely relevant; awaiting merge.

nightly-perf-8-gpu-mi35x-deepseek-v32-mtp — failed step(s): Performance Test MI35x (8-GPU DeepSeek-V3.2 MTP)

Commit Info

• sglang: 9b4bb415ddc315ebcf1f8fcc280537f0b30938dc
• aiter: 9127c94a18e4398e1eba91f6639e910f0994ad02 (v0.1.17.dev110+g9127c94a1) — source: dev/patched image build detected, rebuild skipped (Dockerfile default is the same SHA)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/perf/mi35x/test_deepseek_v32_mtp_perf_mi35x.py	N/A — server crash before any perf assertion	HW Exception ... reason: GPU Hang → Fatal Python error: Aborted → exit 134 (both attempts)	Job log

Failure Summary

The DeepSeek-V3.2 MTP perf test never reached a benchmark assertion. The server process crashed during model initialization / multithread weight loading with a hardware GPU Hang exception on node-3 (first attempt) and node-5 (retry with HF_HUB_OFFLINE=1), both terminating with exit code 134 (SIGABRT). All 8 TP ranks finished Load weight end, and the abort thread-dump shows the main thread parked in monitored_barrier (model_runner.py:4903) while thread-pool workers were still inside the fused-MoE fp8 weight loader (fused_moe_triton/layer.py _load_w13/_load_w2).

Regression Status

Recurring failure — GPU-Hang signature is 2 consecutive days old (Jul 10 → Jul 11). The job has been red since Jul 05, but with a different symptom almost every night, so the GPU-Hang cluster specifically is new as of Jul 10.

Recent history of test_deepseek_v32_mtp_perf_mi35x.py in job nightly-perf-8-gpu-mi35x-deepseek-v32-mtp (branch main, event=schedule, completed runs only):

Date	Run	Test File Status	Failure signature
Jul 11	run	❌ Failed	GPU Hang → Aborted, exit 134 (this run)
Jul 10	run	❌ Failed	GPU Hang → Aborted, exit 134 (same cluster)
Jul 09	run	❌ Failed	Server process exited -9 (SIGKILL/OOM), exit 255
Jul 08	run	❌ Failed	VRAM usage exceeds threshold (5%), exit 1
Jul 07	run	❌ Failed	VRAM usage exceeds threshold (5%), exit 1
Jul 05	run	❌ Failed	VRAM usage exceeds threshold (5%), exit 1
Jul 04	run	✅ Passed	—
Jul 03	run	✅ Passed	—
Jul 02	run	❌ Failed	exit 255

First GPU-Hang occurrence: Jul 10. Last fully-green run: Jul 04. Lookback window: 8 completed scheduled runs (Jul 02–11).

Failure Cluster

GPU Hang during DeepSeek-V3.2 fp8 multithread weight loading (MI35x/gfx950)

Facts (verified observations)

• Verbatim: HW Exception by GPU node-3 (Agent handle: 0x...) reason :GPU Hang immediately followed by Fatal Python error: Aborted; process ##[error]Process completed with exit code 134.
• Both attempts crashed the same way (attempt 1 on node-3, retry on node-5); First attempt failed with exit code 134 then a HF_HUB_OFFLINE=1 retry that also GPU-Hung.
• All 8 ranks logged Load weight end ... type=DeepseekV32ForCausalLM, quant=fp8 before the crash; the crash struck ~1.5 min after the last rank finished loading — i.e. at the transition out of weight loading (thread-pool workers still finalizing MoE weights, main thread at monitored_barrier model_runner.py:4903).
• Jul 10 GPU Hang was also on node-3, and printed mid Multi-thread loading shards: 33% — same node, same weight-loading phase as today.
• Server config (from the test): --tp 8 --dsa-prefill-backend tilelang --dsa-decode-backend tilelang --speculative-algorithm EAGLE --speculative-num-steps 3 ... --mem-fraction-static 0.7 --model-loader-extra-config '{"enable_multithread_load": true}'.
• The neighboring MI35x jobs in the same nightly run (nightly-accuracy-8-gpu-mi35x-deepseek-v32, nightly-perf-8-gpu-mi35x-deepseek-v32-basic, several deepseek-r1-mxfp4 jobs) were also red across these dates — this test config is not the only unstable one on this runner.
• aiter version is a dev build identical to the Dockerfile default and unchanged across the passing (Jul 03/04) and failing runs → not an aiter version delta.

Hypothesised Causes (with confidence)

• [MEDIUM] Driver/GPU left in a bad state by a prior job on the MI35x runner (node-3), not an sglang code regression.
  • Supporting: GPU Hang is a hardware-level exception; it recurs on the same physical node-3 across Jul 10 & 11; the same runner shows a rotating set of resource symptoms night-to-night (VRAM-threshold Jul 05/07/08, SIGKILL -9 Jul 09, GPU Hang Jul 10/11); an in-flight PR (#30431) exists specifically for "driver-level VRAM leaks + graceful GPU cleanup on shutdown" on AMD CI.
  • Disconfirming: two consecutive GPU-Hangs on the same node could instead be a genuine content-dependent hang in weight loading; no A/B on a clean-state runner performed.
  • How to verify: re-run this single job on a freshly-reset MI35x node; if it passes on clean hardware, the hang is environmental. Check whether #30431 landing clears the VRAM/hang cluster.
• [LOW] enable_multithread_load concurrent H2D copies stress the GPU during fp8 MoE weight staging.
  • Supporting: the abort thread-dump shows worker threads inside fused_moe_triton/layer.py (_load_w13/_load_w2) at hang time; the crash is at the weight-loading→init boundary.
  • Disconfirming: enable_multithread_load was already enabled on Jul 09 (which failed with -9, not a hang) and presumably on the Jul 03/04 green runs; log-frame proximity is not causation (workers being in the loader at abort time only means they hadn't exited yet).
  • How to verify: re-run with enable_multithread_load: false and compare.
• [LOW] sglang code regression in the Jul 09→Jul 10 window (57 commits, 2e4d6368c35f..7998fecfd114).
  • Supporting: the GPU-Hang symptom first appears Jul 10.
  • Disconfirming: a code regression would not normally manifest as a HW GPU Hang; the prior nights already failed (for other reasons), so there is no clean green→hang transition to bisect against; no commit in the window is tied to the weight-load path by evidence.
  • How to verify: only worthwhile after ruling out the environmental hypothesis; bisect the 57-commit window on a known-good MI35x node.

In-flight Fix Check

• In-flight fix: ✅ #30431 "[AMD] ci: self-heal driver-level VRAM leaks + graceful GPU cleanup on shutdown" (open since 2026-07-07) — directly targets the VRAM-leak/GPU-cleanup family on MI35x that plausibly covers this hang cluster; ping reviewers rather than opening a duplicate. Ran: search/issues?q=...is:pr+is:open+GPU+Hang+mi35x.
• No open PR found matching deepseek-v32 mtp perf GPU-Hang specifically.

Suggested Triage Steps

• Re-run this single job on a freshly-reset/known-good MI35x (gfx950) node; if green, confirm the hang is node-3 driver state, not code.
• Cross-check whether #30431's GPU-cleanup logic, once merged, clears the VRAM-threshold + GPU-Hang cluster on this runner.
• If the hang reproduces on clean hardware, re-run with --model-loader-extra-config '{"enable_multithread_load": false}' to test the multithread-load hypothesis.
• Only if both above fail to explain it: bisect 2e4d6368c35f..7998fecfd114 (57 commits) on a stable MI35x node.

Status

• Persistence: GPU-Hang signature on 2 consecutive nights (Jul 10–11); the job itself has been red every scheduled run since Jul 05 (varied symptoms), last fully-green Jul 04.
• Scope: nightly-perf-8-gpu-mi35x-deepseek-v32-mtp in nightly-test-amd.yml; sibling MI35x DeepSeek jobs on the same runner are also red across these dates, consistent with a runner-level rather than test-specific problem.
• Blocked work: DeepSeek-V3.2 MTP perf signal on MI35x/gfx950.
• In-flight fix: ✅ #30431 (open since 2026-07-07) — candidate cleanup fix for the VRAM/GPU-Hang cluster.

nightly-8-gpu-mi35x-glm51 — failed step(s): Accuracy Test MI35x (8-GPU GLM-5.1 DSA)

Commit Info

• sglang: 9b4bb415ddc3
• aiter: 9127c94a1 — dev build v0.1.17.dev110+g9127c94a1 (source: dev image preserved; [CI-AITER-CHECK] says "Dev/patched version detected → skipping rebuild", Dockerfile default was 9127c94a1, AITER_COMMIT_OVERRIDE empty)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/accuracy/mi35x/test_glm51_eval_mi35x.py	(whole-file run of zai-org/GLM-5.1-FP8 dsa)	HW Exception … reason: GPU Hang → Fatal Python error: Aborted → exit 134 (SIGABRT)	Log

Failure Summary

The 8-GPU MI35x GLM-5.1 accuracy test hung the GPUs while bringing up zai-org/GLM-5.1-FP8 in DSA (sparse-attention) mode. After weight load began (19:14:33), all 8 ranks logged no further progress and ~70 s later the ROCm runtime raised HW Exception … reason :GPU Hang on GPU node-3, the watchdog (python/sglang/srt/utils/watchdog.py:147) aborted the process (exit 134), and the --continue-on-error retry hung again on node-5. The Performance Test step was skipped as a result.

Regression Status

Known recurring failure — job red every completed run since 2026-07-05; last test-level pass 2026-07-04. The GPU-Hang symptom on this test is directly confirmed on 07-10 and 07-11. On 07-05→07-09 the job died earlier, at the pre-test "Ensure VRAM is clear" runner precheck (=== FAILED: VRAM cleanup unsuccessful after 3 attempts ===, exit 1), so the accuracy test never actually ran those nights (no Testing: zai-org line).

Recent history of test_glm51_eval_mi35x.py in job nightly-8-gpu-mi35x-glm51 (branch=main, event=schedule, completed runs only):

Date	Run	Test File Status	Failed Function	Error
Jul 11	29162254653	❌ Failed (test ran)	GLM-5.1-FP8 dsa	GPU Hang → exit 134
Jul 10	29113206892	❌ Failed (test ran)	GLM-5.1-FP8 dsa	GPU Hang → Aborted
Jul 09	29039566002	⚠️ Test not reached	—	VRAM precheck fail (exit 1)
Jul 08	28964511688	⚠️ Test not reached	—	VRAM precheck fail (exit 1)
Jul 07	28888348134	⚠️ Test not reached	—	VRAM precheck fail (exit 1)
Jul 06	28813271245	⚠️ Test not reached	—	VRAM precheck fail (exit 1)
Jul 05	28749559519	⚠️ Test not reached	—	VRAM precheck fail (exit 1)
Jul 04	28714602438	✅ Passed	—	—

First observed failure (job): 2026-07-05. Last test-level pass: 2026-07-04. Lookback: last 12 scheduled runs.

Failure Cluster

GPU Hang during GLM-5.1-FP8 (DSA) model bring-up on MI35x 8-GPU

Facts (verified observations)

• Verbatim: HW Exception by GPU node-3 (Agent handle: …) reason :GPU Hang (15 occurrences, line 4734), followed by Fatal Python error: Aborted with python/sglang/srt/utils/watchdog.py, line 147 in the traceback.
• First attempt failed with exit code 134 (log 5041); final ##[error]Process completed with exit code 134 (log 5120). 134 = 128+6 = SIGABRT.
• Timeline: Load weight begin. avail mem=277 GB at 19:14:33 (log 4713) → [CI Download] … ACQUIRED lock for zai-org/GLM-5.1-FP8 19:14:33 (log 4733) → GPU Hang 19:15:43 (log 4734). ~70 s of silence before the hang.
• DSA path note (log 4673): ROCm DSA indexer: aiter preshuffle paged-MQA path is unavailable (needs Triton>=3.5.0 or AITER_ENABLE_AOT_GLUON_PA_MQA_LOGITS=1); falling back to legacy page_size=1 / KVBlockSize=1 path. — the test exercises the DSA indexer legacy fallback on gfx950.
• Reproducibility: same GPU-Hang symptom on this exact test on 07-10 and 07-11; both --continue-on-error attempts hung this run (node-3, then node-5).
• aiter unchanged in symptom window: dev build v0.1.17.dev110+g9127c94a1 preserved ([CI-AITER-CHECK] at log ~3426).

Hypothesised Causes (with confidence)

• [LOW] DSA/indexer sparse-attention code change in the 07-04→07-10 window regressed GLM-5.1 DSA on gfx950.
  • Supporting: last test-level pass was 07-04; several DSA/indexer commits landed 07-09/07-10 right when the confirmed GPU-Hang first appears (07-10): 336b64ecc "[AMD] fix dsv4 indexer dtype dispatch on gfx950" (#29479), bda1dc0d9 "[DSA] Fix top-k v2 … (IMA in FA3 sparse decode)" (#30645), fef5eda4f (#30711), ecb7fb398 "Fix CuTe DSL DSA paged MQA export" (#30627); the run uses the DSA legacy fallback path (log 4673).
  • Disconfirming: I could NOT confirm the test ran (and passed/failed) on 07-05→07-09 — those nights died at the VRAM precheck before the test started — so the true regression window is the full 96-commit span 07-04..07-10, not narrowed to these commits. No stack trace pins any of them; the hang has no Python frame beyond the watchdog. None modifies a line provably on the failing path.
  • How to verify: on an MI35x (gfx950) 8-GPU box, run python3 test/registered/amd/accuracy/mi35x/test_glm51_eval_mi35x.py at 9b4bb415ddc3, then git revert each candidate on a branch and re-run; or bisect 854b46be9977..9b4bb415ddc3.
• [LOW] Runner / driver-level GPU hang (infra), not an sglang code regression.
  • Supporting: GPU Hang is a ROCm HW exception; the VRAM-precheck failures on 07-05→07-09 show this MI35x runner was left with dirty VRAM (orphaned processes) for days — consistent with a poisoned/flaky node; the retry hung on a different node (node-5) too.
  • Disconfirming: the hang lands on the same test/model two nights running (07-10, 07-11) at the same phase (post weight-load), which points at the workload rather than random hardware.
  • How to verify: check whether other 8-GPU MI35x jobs on the same runner pool are healthy in these runs; drain/reset the node and re-run only this test.
• [LOW] VRAM-precheck failures (07-05→07-09) and the GPU Hang are the same underlying issue (test hangs → leaves VRAM dirty → next run's precheck fails).
  • Supporting: precheck failures begin the night immediately after the last pass (07-05) and persist until the hang becomes directly visible (07-10).
  • Disconfirming: the "Ensure VRAM is clear" step inspects VRAM left by a prior job on the shared runner, which may be a neighbor job, not this test — cannot attribute without per-runner job history.

No hypothesis reaches MEDIUM: the hang produces no informative Python frame and the confirmed regression window is the full 96-commit span. Needs a GPU bisect between 854b46be9977..9b4bb415ddc3.

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs for GLM-5.1 hang, glm51 mi35x (via search/issues); top hits (#23970 HiSparse ROCm long-prompt swap-in, #29508 quickreduce cudagraph, #27209 gfx950 Dockerfile) do not target this GLM-5.1 DSA GPU-Hang.

Suggested Triage Steps

• Confirm scope on the runner: check whether sibling 8-GPU MI35x nightly jobs on the same node pool are also hanging in runs 29113206892 / 29162254653; if isolated to GLM-5.1 DSA, it's a workload regression, not the node.
• Reproduce on MI35x (gfx950): run test/registered/amd/accuracy/mi35x/test_glm51_eval_mi35x.py at 9b4bb415ddc3; capture a py-spy / rocm coredump at the hang to get a real frame (the watchdog trace alone is not the cause).
• Bisect 854b46be9977..9b4bb415ddc3, prioritizing the DSA/indexer commits above (#29479, #30645, #30711, #30627); if implicated, revert on a branch and re-run this test.
• Test the DSA path hypothesis directly: the run fell back to the legacy page_size=1 DSA indexer path — try AITER_ENABLE_AOT_GLUON_PA_MQA_LOGITS=1 (or the preshuffle paged-MQA path) to see if the hang is specific to the legacy fallback.
• Drain/reset the affected MI35x node before re-runs so a stale-VRAM precheck failure doesn't mask the actual test result.

Status

• Persistence: job red every completed scheduled run since 2026-07-05; GPU-Hang on the test directly confirmed 07-10 and 07-11 (2 consecutive nights). Last test-level pass 2026-07-04.
• Scope: nightly-8-gpu-mi35x-glm51 job in nightly-test-amd.yml; blocks the GLM-5.1-FP8 DSA accuracy signal on MI35x and skips its Performance Test.
• In-flight fix: ❌ none found in open PRs.

nightly-8-gpu-mi35x-qwen35-triton-dcp — failed step(s): Accuracy Test MI35x (8-GPU Qwen 3.5 Triton DCP)

Commit Info

• sglang: 9b4bb415d
• aiter: 9127c94a1 (v0.1.17.dev110+g9127c94a1, source: dev image preserved — [CI-AITER-CHECK] Dev/patched version detected → skipping rebuild)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/test_qwen3p5_triton_dcp.py	TestQwen35TritonDCPGsm8k.test_a_gsm8k	Fatal Python error: HW Exception by GPU node reason :GPU Hang → Aborted → exit 134	Log (step 6)

Failure Summary

The single test in this job (TestQwen35TritonDCPGsm8k.test_a_gsm8k, model Qwen/Qwen3.5-397B-A17B-FP8, tp=8 / dcp=2 / triton backend / context 1,048,576 with yarn 4×) never reached the GSM8K eval. The server crashed during model bring-up with a hardware-level GPU Hang (Fatal Python error: HW Exception by GPU node ... reason :GPU Hang → Aborted, exit 134). This happened on both attempts — the first during weight loading (load_weights in qwen3_5.py), and the retry (offline mode) during server init/warmup right after the ROCm DSA-indexer fallback message.

Regression Status

Flaky / intermittent — heterogeneous infra/hardware symptoms. This job runs only the one Qwen3.5 test file, so job status ≈ test-file status. The test passed cleanly on Jul 5 and Jul 6 (score 0.95). The GPU Hang / exit 134 signature specifically appeared Jul 10 and Jul 11; adjacent days failed with different (but related) resource symptoms.

Recent history of test_qwen3p5_triton_dcp.py (TestQwen35TritonDCPGsm8k.test_a_gsm8k) in job nightly-8-gpu-mi35x-qwen35-triton-dcp (branch main, event schedule, completed runs only):

Date	Run	Test File Status	Failure signature
Jul 11	29162254653	❌ Failed	GPU Hang → exit 134 (this run)
Jul 10	29113206892	❌ Failed	GPU Hang → exit 134
Jul 9	29039566002	❌ Failed	Server process exited code -9 (SIGKILL) → exit 255
Jul 8	28964511688	❌ Failed	VRAM cleanup unsuccessful after 3 attempts → exit 1
Jul 7	28888348134	❌ Failed	VRAM usage exceeds threshold (pre-flight) → exit 1
Jul 6	28813271245	✅ Passed	—
Jul 5	28749559519	✅ Passed	score=0.9505
Jul 4	28714602438	❌ Failed	exit 1

First observed failing (this signature): Jul 10. Last known passing: Jul 6. (Current run 29162254653 is queued at the workflow level, but this job itself completed with exit 134.)

Failure Cluster

GPU Hang during Qwen3.5 model bring-up on MI35x (driver/KFD instability)

Facts (verified observations)

• Verbatim (current run): HW Exception by GPU node-5 ... reason :GPU Hang at 19:25:06 (retry), and Fatal Python error: HW Exception by GPU node-3 ... reason :GPU Hang / Aborted at 19:24:19 (first attempt). Both end in Process completed with exit code 134.
• The first-attempt faulthandler dump shows threads in load_weights (python/sglang/srt/models/qwen3_5.py) → load_model (model_runner.py:1455) → weight download/filelock — i.e. the hang struck during weight loading.
• The retry hung during server init, immediately after ROCm DSA indexer: aiter preshuffle paged-MQA path is unavailable ... falling back to legacy page_size=1 / KVBlockSize=1 path.
• This run's captured log has no VRAM pre-flight errors (0 VRAM usage exceeds threshold lines) — the node appears to have started clean this run.
• Same job, adjacent days, same test: Jul 7/8 = VRAM cleanup failures; Jul 9 = server SIGKILL (-9); Jul 10 = GPU Hang. Jul 5/6 = clean pass (score 0.95). All non-passing failures are at server-launch/model-load time, none are accuracy failures.
• Not a scout run (nightly-test-amd.yml, event schedule); aiter unchanged (dev image preserved), so no aiter override is in play.

Hypothesised Causes (with confidence + disconfirming evidence)

• [MEDIUM] MI35x runner driver/KFD-level VRAM leak / zombie-KFD node degradation leaves GPUs in a bad state, causing the GPU Hang while bringing up this very large (397B, 1M-context) model.
  • Supporting: open PR #30431 documents an active fleet-wide MI35x zombie-KFD/VRAM-leak incident (7+ consecutive days, ~37 jobs across all four AMD workflows, ROCm/aiter#2061 signature). The Jul 7–11 failure streak on this job (VRAM-cleanup → OOM SIGKILL → GPU Hang) is exactly the multi-symptom pattern of a degraded node; the test passed clean Jul 5–6.
  • Disconfirming: this specific run's log shows no VRAM pre-flight failure, so the node was not obviously pre-loaded with leaked VRAM at start — the hang occurred later during load, which a pure pre-existing leak wouldn't fully explain.
  • How to verify: check whether Jul 10 & Jul 11 runs landed on the same physical MI35x node (runner label / node id in setup logs); reboot/drain that node and re-run the job.
• [LOW] Genuine load-time hang bringing up a 397B-A17B MoE at 1,048,576 context (yarn 4×) across 8 GPUs — memory-pressure / kernel stall during weight load or warmup.
  • Supporting: crash is during load_weights (attempt 1) and warmup after the DSA-indexer legacy fallback (attempt 2); this is the most resource-intensive config in the nightly matrix.
  • Disconfirming: identical config passed Jul 5 and Jul 6, so the code path itself is capable of succeeding — points away from a deterministic code bug.
• No hypothesis reaches HIGH; there is no clean single-commit regression window (Jul 6 pass → Jul 7 fail, but Jul 7's failure was a pre-flight VRAM-cleanup infra error, not a code path). Symptoms vary run-to-run, which argues against a code regression.

In-flight Fix Check

• In-flight fix: ✅ #30431 "[AMD] ci: self-heal driver-level VRAM leaks + graceful GPU cleanup on shutdown" (open, non-draft, since 2026-07-07, last updated 2026-07-11) — directly targets this MI35x zombie-KFD/VRAM-leak incident. Chase reviewers; do NOT open a duplicate.
• Searched: search/issues?q=...is:pr+is:open with keywords Qwen3.5 GPU Hang, qwen3.5 dcp, GPU Hang mi35x, triton dcp hang.

Suggested Triage Steps

• Identify the physical MI35x node for runs 29113206892 (Jul 10) and this run — inspect the job setup logs for the runner/node id; if shared, drain/reboot and re-run to confirm node-locality.
• Land / prioritize #30431 (VRAM self-heal + graceful GPU cleanup) and observe whether the Jul-7-onward failure streak on this job clears.
• If the hang persists on a known-clean node, capture a GPU coredump / rocm-smi snapshot during model load to determine whether the hang is genuine load-time (memory pressure at 397B / 1M context) rather than pre-existing node degradation.
• If it proves node-only flakiness, consider a retry/quarantine policy for this nightly job until the driver-leak fix is merged.

Status

• Persistence: Failing since Jul 7 (5 consecutive completed runs), with the specific GPU Hang / exit 134 signature on Jul 10 and Jul 11. Last green Jul 6.
• Scope: This job only within this analysis, but PR #30431 reports the underlying MI35x VRAM-leak incident affecting ~37 jobs across all four AMD workflows.
• Blocked work: Blocks the Qwen3.5 Triton-DCP nightly accuracy signal on AMD MI35x.
• In-flight fix: ✅ #30431 open since 2026-07-07 — awaiting merge.

nightly-8-gpu-mi35x-qwen35 — failed step(s): Accuracy Test MI35x (8-GPU Qwen 3.5)

Commit Info

• sglang: 9b4bb415
• aiter: 9127c94a — v0.1.17.dev110+g9127c94a1 (source: dev/patched image, rebuild skipped per [CI-AITER-CHECK])

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/accuracy/mi35x/test_qwen35_eval_mi35x.py	TestQwen35EvalMI35x.test_lm_eval	AssertionError: extra_buffer needs CUDA/MUSA/NPU (FLA). (server fails to launch → Server process exited with code 1)	Log

Failure Summary

The single accuracy test test_lm_eval failed because the sglang serve process for Qwen/Qwen3.5-397B-A17B (TP8, --attention-backend aiter) exits with code 1 during argument parsing — before any model load or GPU kernel runs. The crash is an AssertionError: extra_buffer needs CUDA/MUSA/NPU (FLA). raised at python/sglang/srt/server_args.py:4581 in _validate_mamba_extra_buffer. The test retries the launch once, both attempts crash identically, then reports FAILED (errors=1).

Regression Status

Never-passed (no green run found in queryable window). The extra_buffer needs CUDA/MUSA/NPU crash is present in every completed run that actually reached the test (Jul 8–11). The Jul 4 run failed earlier at the "Ensure VRAM is clear" step (infra), so the test did not run there. No passing run of this test file exists in the queryable window (shallow clone, 96 commits).

Recent history of test_qwen35_eval_mi35x.py (test_lm_eval) in job nightly-8-gpu-mi35x-qwen35 (branch=main, event=schedule, completed runs only):

Date	Run	Job	Test File Status	Failed Function	Error
Jul 11	29162254653	job	❌ Failed	test_lm_eval	extra_buffer needs CUDA/MUSA/NPU
Jul 10	29113206892	job	❌ Failed	test_lm_eval	extra_buffer needs CUDA/MUSA/NPU (verified)
Jul 9	29039566002	job	❌ Failed	test_lm_eval	extra_buffer needs CUDA/MUSA/NPU (verified)
Jul 8	28964511688	job	❌ Failed	test_lm_eval	extra_buffer needs CUDA/MUSA/NPU (verified)
Jul 4	28714602438	job	⚠️ Not run	—	Job failed earlier at VRAM-clear step (infra); test did not execute

Failure Cluster

Mamba radix-cache extra_buffer strategy rejected on ROCm at server startup

Facts (verified observations, no interpretation)

• Verbatim error (repeated on both launch attempts): AssertionError: extra_buffer needs CUDA/MUSA/NPU (FLA). at log line 4608.
• Full startup stack: prepare_server_args → ServerArgs.from_cli_args → __post_init__ (server_args.py:2802) → _handle_model_specific_adjustments (:4539) → _handle_mamba_radix_cache (:4610) → _validate_mamba_extra_buffer (:4581).
• Launch command: sglang serve --model-path Qwen/Qwen3.5-397B-A17B --tp 8 --attention-backend aiter --trust-remote-code --model-loader-extra-config '{"enable_multithread_load": true}' --watchdog-timeout 1200 --device cuda.
• Platform is ROCm: log shows Automatically detected platform rocm and runner GPU_ARCH=mi35x.
• Crash is pre-GPU: it happens during ServerArgs.__post_init__ argument resolution, before model load — so aiter kernels are never reached.
• The assertion in server_args.py:4581 is literally assert (is_cuda() or is_musa() or is_npu()), "extra_buffer needs CUDA/MUSA/NPU (FLA).". On ROCm all three are False (is_hip() is the true predicate).
• supports_mamba_cache_extra_buffer (arg_groups/overrides.py:1098) returns True for the Qwen3.5 arch when linear_attn_backend == "triton" (the default) — with no platform check.
• _mamba_radix_cache_resolution (overrides.py:1107) auto-selects "extra_buffer" when strategy is "auto" and (wants_overlap OR wants_paging) AND arch supports it. Overlap schedule is on by default, so on ROCm it resolves to extra_buffer then fails validation.
• Same error verified in the Jul 8, 9, 10 job logs (2 occurrences each). aiter version unchanged across these (dev build preserved).

Hypothesised Causes (with confidence, with disconfirming evidence)

• [HIGH] Platform-gating bug: the auto mamba-radix-cache resolution selects the extra_buffer strategy for Qwen3.5 hybrid-mamba models on ROCm, but extra_buffer requires FLA (flash-linear-attention) which is CUDA/MUSA/NPU-only, so _validate_mamba_extra_buffer aborts the server at startup.
  • Supporting: the exact stack frames (_mamba_radix_cache_resolution → _validate_mamba_extra_buffer) appear verbatim in the crash; supports_mamba_cache_extra_buffer is arch+backend gated with no is_hip() guard; the failing assertion checks only is_cuda()/is_musa()/is_npu(); ROCm is detected as the platform.
  • Disconfirming: (none found) — the crash is deterministic, reproduced on 4 consecutive nights with identical stack, and is independent of the aiter version.
  • How to verify: launch Qwen/Qwen3.5-397B-A17B on an MI35x with --mamba-radix-cache-strategy no_buffer --disable-overlap-schedule (or --disable-radix-cache); server should start, confirming the auto→extra_buffer path is the trigger.

This is not an amd-aiter-scout.yml run, so no Baseline A/B / Failure Origin section applies. The failure is a pure sglang config-validation assertion; aiter is not implicated.

In-flight Fix Check

• In-flight fix: ✅ #29965 "[AMD] Fix mamba radix cache auto strategy crashing on ROCm (fall back to no_buffer)" (open since 2026-07-02, 1 file) — directly targets this crash. Chase reviewers; do NOT open a duplicate.
• Related / alternative approaches also open:
  • #29391 "Fix mamba radix cache auto strategy crashing on non-FLA platforms (MPS/MLX)" (open 2026-06-26) — same bug class for MPS/MLX.
  • #29832 "[AMD] Enable mamba extra_buffer on ROCm + shared-prefix donate correctness test" (open 2026-07-01) and #27141 "[ROCm] Support radix-cache with mamba-extra-buffer for Qwen3.5" — the opposite fix direction (actually enable extra_buffer on ROCm rather than fall back).
• Search used: is:pr is:open extra_buffer and is:pr is:open mamba+radix+ROCm via the GitHub search API.

Suggested Triage Steps

• Verify #29965 merges cleanly against HEAD 9b4bb415 and re-run nightly-8-gpu-mi35x-qwen35; it should unblock this test if the fall-back-to-no_buffer approach is accepted.
• Maintainers must first decide the intended behavior on ROCm — fall back to no_buffer (#29965/#29391) vs enable extra_buffer on ROCm (#29832/#27141) — since these are competing fixes for the same crash.
• As a standalone repro on MI35x: launch the model with --mamba-radix-cache-strategy no_buffer --disable-overlap-schedule to confirm the server starts (isolates the auto-resolution path from any downstream Qwen3.5 accuracy issue this test may still surface once it can launch).

Status (factual, no priority assignment)

• Persistence: extra_buffer crash present every completed run that reached the test — Jul 8, 9, 10, 11 (4 consecutive nights); the test has no recorded green run in the queryable window (Never-passed).
• Scope: 1 job (nightly-8-gpu-mi35x-qwen35) in nightly-test-amd.yml. The sibling nightly-8-gpu-mi35x-qwen35-triton-dcp and nightly-8-gpu-qwen35 jobs are also failing on the same nights and likely share this cluster (not individually verified here).
• Blocked work: Qwen3.5 8-GPU accuracy signal on AMD MI35x.
• In-flight fix: ✅ #29965 (open since 2026-07-02) — direct fix awaiting merge; competing approach in #29832/#27141.

nightly-8-gpu-mi35x-glm5-mxfp4 — failed step(s): Accuracy Test MI35x (8-GPU GLM-5-MXFP4)

Commit Info

• sglang: 9b4bb415dd
• aiter: 9127c94a18 — v0.1.17.dev110+g9127c94a1 (source: dev/patched image build, preserved; rebuild skipped)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/accuracy/mi35x/test_glm5_mxfp4_eval_mi35x.py	TestGLM5MXFP4EvalMI35x.test_glm5_mxfp4_accuracy	HW Exception ... reason :GPU Hang → Fatal Python error: Aborted → exit code 134 (SIGABRT) during model-worker init	Log

Failure Summary

The single test in this job (test_glm5_mxfp4_eval_mi35x.py, 8-GPU TP eval of amd/GLM-5-MXFP4) crashed while the server subprocess was still initializing the model worker. Both the first attempt and the HF_HUB_OFFLINE=1 retry died the same way: HW Exception by GPU node-N ... reason :GPU Hang, followed by Fatal Python error: Aborted and Process completed with exit code 134. The crash never reached the GSM8K benchmark — it aborts inside scheduler.py → init_tp_model_worker → tp_worker.py:init_model_worker (model load / warmup).

Regression Status

Known recurring failure — test file broken since 2026-07-06 (last green 2026-07-05), but the symptom has shifted twice. The current "GPU Hang → exit 134" cluster is the third phase and has appeared in the last 2 completed runs (07-10, 07-11).

Recent history of test_glm5_mxfp4_eval_mi35x.py (test_glm5_mxfp4_accuracy) in job nightly-8-gpu-mi35x-glm5-mxfp4 (branch main, event=schedule, completed runs only; this job runs exactly one test file, so job status == test-file status):

Date	Run	Test File Status	Symptom / Exit
Jul 11	29162254653	❌ Failed	GPU Hang → exit 134
Jul 10	29113206892	❌ Failed	GPU Hang → exit 134
Jul 09	29039566002	❌ Failed	accuracy AssertionError: Failed models: ['amd/GLM-5-MXFP4 (mxfp4)'] → exit 255
Jul 08	28964511688	❌ Failed	VRAM precheck (zombie KFD ctx) → exit 1
Jul 07	28888348134	❌ Failed	VRAM precheck (zombie KFD ctx) → exit 1
Jul 06	28813271245	❌ Failed	VRAM precheck (zombie KFD ctx) → exit 1
Jul 05	28749559519	✅ Passed	GSM8K PASS
Jul 04	28714602438	✅ Passed	—
Jul 03	28676359018	❌ Failed	exit 1

First observed failure of the current GPU-Hang symptom: 2026-07-10. Last known green for the test file: 2026-07-05. (The current run 29162254653 is still queued at the run level, but this job already completed as failed.)

Failure Cluster

GPU Hang / Fatal Python error: Aborted (SIGABRT, exit 134) during GLM-5-MXFP4 model-worker init on MI35x.

Facts (verified observations, no interpretation)

• Verbatim: HW Exception by GPU node-5 (Agent handle: ...) reason :GPU Hang then ##[error]Process completed with exit code 134. (log tail, both attempts).
• Crash stack frames (interleaved but readable): run_scheduler_process → Scheduler.__init__ (scheduler.py:583/767/847) → init_tp_model_worker → tp_worker.py:273 init_model_worker, with watchdog.py on another thread.
• aiter version is identical in the last green (07-05) and both GPU-Hang runs (07-10, 07-11): v0.1.17.dev110+g9127c94a1 (verified via [CI-AITER-CHECK] markers).
• Docker base image changed across the pass→fail boundary: 07-05 = rocm/sgl-dev:v0.5.14-rocm700-mi35x-20260705; 07-10 = ...v0.5.15-...-20260710; 07-11 = ...v0.5.15-...-20260711.
• The 3 hanging/failing runs ran on 3 different physical nodes in the same pool: ...zf775-runner-vndw7 (07-11), ...cl9bp (07-10), ...ngb6g (07-06). Current run's VRAM precheck passed (all GPUs 0% VRAM at start).
• 07-06/07/08 logs explicitly print: "almost certainly a zombie KFD context (see [Bug] Custom all-reduce IPC buffers use fixed VA, conflict with torch_memory_saver pause/resume → zombie processes on MI300X ROCm/aiter#2061). The node will need to be rebooted" — i.e. node-health degradation predates the current symptom.
• Commit window 48ba79c11e..9b4bb415dd (07-05 pass → current) = 96 commits.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [MEDIUM] MI35x zf775 pool node-health degradation (driver-level VRAM / zombie-KFD-context leak) — the test-file failure is a chain: VRAM-precheck-fail (07-06/07/08) → accuracy fail (07-09) → GPU Hang on init (07-10/11).
  • Supporting: 07-06/07/08 logs name the zombie KFD context ([Bug] Custom all-reduce IPC buffers use fixed VA, conflict with torch_memory_saver pause/resume → zombie processes on MI300X ROCm/aiter#2061) + node-reboot requirement; a hung GPU context left by a prior job cleanly explains a "GPU Hang" the moment the model touches the device; failures span the whole pool, consistent with a cross-job driver leak.
  • Disconfirming: the current run's VRAM precheck passed (0% VRAM) so no allocated leak was detected at start; three distinct nodes hung, which a single-node zombie context alone would not explain.
  • How to verify: reboot/drain the zf775 MI35x nodes, then re-run test_glm5_mxfp4_eval_mi35x.py; check dmesg/KFD state on vndw7 after this run.
• [LOW] Base-image bump #30587 (789bc3995, "[AMD-miles] Bump miles rocm700-mi35x base image and migrate nightly test onto it") introduced a ROCm-runtime regression producing init-time GPU hangs.
  • Supporting: merged 2026-07-10 15:47 UTC; the 07-10 nightly (start 18:03 UTC) was the first run to include it, and that is exactly when the exit-134 GPU-Hang symptom first appeared alongside the v0.5.14→v0.5.15 image change.
  • Disconfirming: the test file was already failing (VRAM precheck) since 07-06, before this commit; the symptom shift could be coincidental with ongoing node degradation rather than caused by the image.
  • How to verify: pin the nightly back to v0.5.14-...-mi35x on a branch and re-run; if it passes, the image bump is implicated.
• [LOW] GLM-5-MXFP4 kernel/graph-capture hang during warmup on gfx950 (sglang code in the 96-commit window).
  • Supporting: hang is inside init_model_worker (warmup / CUDA-graph capture); several open GLM-5 gfx950 kernel PRs exist (e.g. #30506, #30715, #30808).
  • Disconfirming: no code-level evidence ties any specific commit to the failing frame; the test/model config and aiter are unchanged vs the 07-05 green run.

No hypothesis reaches HIGH; a specific culprit commit cannot be pinned. Needs node-health verification first, then (if clean) a git bisect across 48ba79c11e..9b4bb415dd.

In-flight Fix Check

• In-flight fix: ✅ #30431 "[AMD] ci: self-heal driver-level VRAM leaks + graceful GPU cleanup on shutdown" (open since 2026-07-07) — directly targets the driver-level VRAM-leak / zombie-KFD state behind the [MEDIUM] hypothesis; chase reviewers rather than opening a duplicate. (searched is:open mi35x hang, KFD VRAM.)
• In-flight fix: ❌ none found specifically for a GLM-5-MXFP4 init GPU-Hang (searched is:open GLM-5 MXFP4).

Suggested Triage Steps

• Reboot/drain the MI35x zf775 runner pool (vndw7, cl9bp, ngb6g), then re-run test_glm5_mxfp4_eval_mi35x.py — confirms/denies the node-health hypothesis cheaply.
• Land or fast-track #30431 and observe whether the VRAM-precheck + GPU-Hang chain clears on subsequent nightlies.
• If hangs persist on a freshly-rebooted node: pin the nightly image back to v0.5.14-rocm700-mi35x on a branch and re-run to isolate base-image bump #30587.
• If image is exonerated: bisect 48ba79c11e..9b4bb415dd, focusing on gfx950 GLM-5 / MoE-MXFP4 / warmup-graph-capture commits, running the test in the mi35x container.

Status (factual)

• Persistence: test file red every completed scheduled run since 2026-07-06 (last green 07-05); the current GPU-Hang/exit-134 symptom in the last 2 runs (07-10, 07-11).
• Scope: 1 job (nightly-8-gpu-mi35x-glm5-mxfp4) in nightly-test-amd.yml; symptom localized to the MI35x zf775 pool, spanning ≥3 physical nodes.
• Blocked work: GLM-5-MXFP4 8-GPU accuracy signal on MI35x.
• In-flight fix: ✅ #30431 (VRAM-leak self-heal, open since 2026-07-07) for the node-state hypothesis; no GLM-5-MXFP4-specific fix found.

---

Generated by amd-bot using Claude Code CLI (last updated: 2026-07-12 23:22 UTC)

[amd-bot]

nightly-test-amd.yml — Per-Job Analysis (continued 5/8)

nightly-perf-8-gpu-mi35x-deepseek-v32-basic — failed step(s): Performance Test MI35x (8-GPU DeepSeek-V3.2 Basic)

Commit Info

• sglang: 9b4bb415d
• aiter: 9127c94a1 — v0.1.17.dev110+g9127c94a1 (source: Dev image detected — [CI-AITER-CHECK] reports "Dev/patched version detected → skipping rebuild", so the image's pre-baked aiter was preserved, not the Dockerfile default)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/perf/mi35x/test_deepseek_v32_basic_perf_mi35x.py	TestNightlyDeepseekV32BasicPerformance.test_bench_one_batch	HW Exception ... reason :GPU Hang → Fatal Python error: Aborted (exit 134)	Log

Failure Summary

The DeepSeek-V3.2 basic perf benchmark on MI35x (gfx950, TP=8) loaded all 163 weight shards successfully, then the GPUs threw HW Exception ... reason :GPU Hang on node-3 (and later node-5) and the Python process aborted with exit code 134 (SIGABRT). The interleaved fatal-error traceback's top frames are all in python/sglang/srt/layers/moe/fused_moe_triton/layer.py, reached via deepseek_v2.py:load_weights → deepseek_weight_loader.py:do_load_weights → load_weights_and_postprocess — i.e. the hang occurs during MoE weight post-processing right after loading, before the benchmark itself runs.

Regression Status

Known recurring failure — GPU Hang cluster is new since Jul 10 (2 consecutive completed runs). The job has been red since Jul 5, but the earlier reds are a different symptom (VRAM-cleanup precondition failure, exit 1 — the model never loaded). The last run where this test file actually executed and passed was Jul 4.

Recent history of test_deepseek_v32_basic_perf_mi35x.py (test_bench_one_batch) in job nightly-perf-8-gpu-mi35x-deepseek-v32-basic (branch main, event=schedule, completed runs only):

Date	Run	Test File Status	Symptom	Error
Jul 11	29162254653	❌ Failed	GPU Hang (exit 134)	Fatal Python error: Aborted in fused_moe_triton/layer.py
Jul 10	29113206892	❌ Failed	GPU Hang (exit 134)	same — 14× GPU Hang, fused_moe_triton/layer.py
Jul 9	29039566002	❌ Failed	VRAM cleanup (exit 1)	VRAM cleanup unsuccessful — model never loaded
Jul 8	28964511688	❌ Failed	VRAM cleanup (exit 1)	VRAM cleanup unsuccessful — model never loaded
Jul 5	28749559519	❌ Failed	VRAM cleanup (exit 1)	ERROR: VRAM usage exceeds threshold (5%)
Jul 4	28714602438	✅ Passed	—	—
Jul 3	28676359018	✅ Passed	—	—

(Jul 6–7 runs 28813271245 / 28888348134 were run-level cancelled → excluded. Today's parent run 29162254653 is still queued overall, but this job completed with failure.)

Failure Cluster

GPU Hang during MoE weight post-processing (DeepSeek-V3.2, MI35x/gfx950)

Facts (verified observations, no interpretation)

• Weight loading completed 163/163 shards, then HW Exception by GPU node-3 ... reason :GPU Hang printed 10+ times, followed by Fatal Python error: Aborted; step ended Process completed with exit code 134 (job.log:4791–4804, 8196).
• Fatal-error traceback top frames are all in fused_moe_triton/layer.py; call chain includes load_weights (deepseek_v2.py), do_load_weights / load_weights_and_postprocess (deepseek_weight_loader.py), loader.py (job.log:4815–4870).
• Jul 10 run reproduced the identical symptom: 14× GPU Hang, same fused_moe_triton/layer.py top frame, exit 134.
• Jul 5/8/9 reds are a distinct precondition failure: ERROR: VRAM usage exceeds threshold (5%) / VRAM cleanup unsuccessful after 3 attempts, exit 1, zero GPU Hang lines — the benchmark process never started.
• aiter is a dev build baked into the CI image (v0.1.17.dev110+g9127c94a1), unchanged by any override (AITER_COMMIT_OVERRIDE empty).
• The harness logged Warning: Unknown suite nightly-perf-8-gpu-mi35x-deepseek-v32-basic for backend AMD but still executed the test file — not the failure cause.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [LOW] A change in the MoE weight-post-processing / quantization-shuffle path (or the gfx950 aiter kernel it dispatches) hangs the GPU during process_weights_after_loading for the DeepSeek-V3.2 checkpoint.
  • Supporting: every top stack frame is in fused_moe_triton/layer.py, reached from the weight loader; the hang is deterministic across Jul 10 + Jul 11 on the same MI35x runner.
  • Disconfirming: no single commit is implicated — the regression window is confounded. The test did not execute Jul 5–9 (VRAM precondition aborted it), so the true "last real pass" is Jul 4 and the GPU-Hang could have been latent before it first became observable on Jul 10. Also, aiter is a fixed dev image, so an aiter-side kernel regression cannot be ruled in or out from this log alone.
  • How to verify: on an MI35x/gfx950 box, load deepseek-ai/DeepSeek-V3.2 with TP=8 and break in process_weights_after_loading to catch the hang; then bisect the sglang range below.
• [SPECULATION] MI35x base-image bump #30587 (789bc3995) landed in the window.
  • Supporting: touches MI35x nightly config; a driver/image change is a classic GPU-hang trigger.
  • Disconfirming (strong): the diff only edits nightly-test-amd-miles-rocm700.yml and a docker-release workflow — a different (amd-miles) pipeline. It does not touch nightly-test-amd.yml, the mi35x deepseek test, or any kernel code. Effectively ruled out for this job.

No hypothesis above LOW confidence for the GPU Hang. The Jul 9→Jul 10 window is 57 commits (2e4d6368c35f..7998fecfd114) and is confounded by the Jul 5–9 VRAM masking; a bisect anchored on the last real pass 854b46be9977 is required.

In-flight Fix Check

• VRAM-cleanup precondition failures (Jul 5/8/9 symptom): In-flight fix: ✅ #30431 "[AMD] ci: self-heal driver-level VRAM leaks + graceful GPU cleanup on shutdown" — directly targets the ERROR: VRAM usage exceeds threshold / cleanup path. Chase reviewers; do NOT open a duplicate. (Query: search/issues?q=...+mi35x+GPU+Hang.)
• GPU Hang in MoE post-processing (Jul 10/11 symptom): In-flight fix: ❌ none found matching deepseek v32 mi35x / mi35x GPU Hang / deepseek-v32 perf — needs new investigation.

Suggested Triage Steps

• Bisect the sglang range 854b46be9977..7998fecfd114 (last real pass → first observed GPU Hang), re-running test_deepseek_v32_basic_perf_mi35x.py on an MI35x/gfx950 8-GPU box. Prioritize commits touching MoE / fused_moe_triton / quantization-shuffle / gfx950 kernels.
• Reproduce locally with a process_weights_after_loading breakpoint to capture a clean single-thread traceback (the CI dump is interleaved across threads) and confirm the exact hanging kernel.
• Since aiter is a pinned dev image (v0.1.17.dev110+g9127c94a1), try an A/B with a known-good earlier aiter build in the MI35x image to determine whether the hang is sglang-side or aiter-kernel-side.
• Track the VRAM-cleanup reds separately under #30431; once merged, confirm whether the GPU Hang is the sole remaining failure.

Status

• Persistence: GPU Hang cluster failing for 2 consecutive completed runs (since Jul 10 ~18:28 UTC). Broader job red since Jul 5, but Jul 5–9 were a separate VRAM-cleanup precondition failure. Last green: Jul 4.
• Scope: nightly-perf-8-gpu-mi35x-deepseek-v32-basic in nightly-test-amd.yml. The sibling MI35x deepseek-v32 jobs (accuracy, mtp) have also been red across this window — worth checking whether they share the GPU-Hang symptom.
• Blocked work: DeepSeek-V3.2 basic perf signal on MI35x.
• In-flight fix: ✅ #30431 for the VRAM-cleanup symptom; ❌ none for the GPU Hang symptom.

nightly-2-gpu-mi35x-glm51-mxfp4 — failed step(s): Install dependencies

Commit Info

• sglang: 96a04cb13f9c3ed86028e090784a9eb059cf5318
• aiter: N/A — job failed in the MORI build inside amd_ci_install_dependency.sh (before the [CI-AITER-CHECK] / aiter-detection section at line 226+ was reached; no aiter markers present).

Failed Tests

Test File	Test Function	Error	Log
N/A — build failure (not a test)	N/A	ninja: build stopped: subcommand failed → CalledProcessError: cmake --build . returned exit status 1	Job log

The nightly GLM5.1-mxfp4 accuracy test never ran. The job died during dependency installation while (re)building the ROCm/mori library at commit e31d426a.

Failure Summary

The Run bash scripts/ci/amd/amd_ci_install_dependency.sh step reinstalls MORI ([MORI] Reinstalling MORI e31d426a…, log line 2973). The MORI CMake build fails at the gtest_discover_tests post-link phase for 7 umbp test executables (test_client_registry_external_kv, test_tier_priority_route_get, test_external_kv_block_index, test_peer_dram_allocator, test_peer_ssd_eviction, test_router_dedup, test_client_registry). CMake's GoogleTest/ParseTestList.cmake:96 fails to parse an empty JSON string (Syntax error: value, object or array expected at Line 1, Column 1), which stops ninja and aborts setup.py develop with exit 1.

Regression Status

Infrastructure / build issue (not an sglang code regression, not a test failure).

Clean test-file-level history is not available: the specific test never executed, and the prior scheduled nightly-test-amd.yml runs (same branch, event=schedule) are almost all cancelled, so they are not comparable passing/failing baselines:

Date	Run	Status (run-level)	Head SHA
Jul 12	29202683625	this run (MORI build fail)	96a04cb1
Jul 11	29162254653	cancelled	9b4bb415
Jul 10	29113206892	cancelled	7998fecf
Jul 09	29039566002	cancelled	2e4d6368
Jul 05	28749559519	failure	48ba79c1

Failure Cluster

MORI gtest_discover_tests CMake JSON parse failure during ROCm dependency install (cmake 4.4).

Facts (verified observations)

• The CI image is rocm/sgl-dev:v0.5.15-rocm700-mi35x-20260712 (built today, 2026-07-12) — log line 1444.
• MORI is reinstalled fresh at MORI_COMMIT=e31d426a…, MORI_GPU_ARCHS=gfx950 — log line 2973; logic at scripts/ci/amd/amd_ci_install_dependency.sh:205-217.
• The CMake in use is cmake 4.4: error path is .../cmake/data/share/cmake-4.4/Modules/GoogleTest/ParseTestList.cmake:96.
• 7 FAILED: [code=1] entries, each an identical string sub-command JSON failed parsing json string … Syntax error: value, object or array expected at the gtest_discover_tests_impl discovery step (log lines ~4325-4434).
• Terminal chain: ninja: build stopped: subcommand failed (line 4482) → setup.py:434 build_extension → CalledProcessError: 'cmake --build . -j 128' returned non-zero exit status 1 (line 4508) → ##[error]Process completed with exit code 1 (line 4518).
• AITER_COMMIT_OVERRIDE is empty; the run is event=schedule.

Hypothesised Causes (with confidence)

• [MEDIUM] cmake 4.4 in the freshly-built (2026-07-12) CI image breaks MORI's gtest_discover_tests JSON parsing.
  • Supporting: the failing stack is entirely inside cmake-4.4/Modules/GoogleTest/ParseTestList.cmake; the empty-JSON parse error is a known behavior change in newer CMake's GoogleTest discovery; an in-flight fix (#30942) explicitly pins cmake==4.3.4 to fix exactly this "MoRI gtest_discover build break"; the image is dated today, consistent with a fresh cmake bump.
  • Disconfirming: no green baseline was captured to prove cmake 4.3.x previously succeeded on this exact MORI commit (prior nightlies are cancelled, not passing); the empty JSON could alternatively stem from the built test binary failing to run at discovery time (e.g. missing runtime lib / no GPU during build) rather than a pure cmake-version issue.
  • How to verify: rebuild MORI in the same container with cmake==4.3.4 vs 4.4 and compare; or run one of the built binaries (e.g. ./test_client_registry --gtest_list_tests --gtest_output=json) manually to see whether it emits valid JSON.
• [LOW] MORI commit e31d426a is itself incompatible with the image's toolchain.
  • Supporting: MORI is checked out to a pinned commit that the baked image was not built against.
  • Disconfirming: the C++ compile/link steps all succeed (128/146 → link OK); only the cmake-side test discovery fails, pointing at cmake behavior, not MORI source.
  • How to verify: covered by the cmake A/B above.

In-flight Fix Check

• In-flight fix: ✅ [#30942](https://github.com/sgl-project/sglang/pull/30942) — "[AMD] Pin cmake==4.3.4 in ROCm Dockerfile to fix MoRI gtest_discover build break" (opened 2026-07-12 18:13 UTC by yctseng0211, currently draft). Directly targets this failure — chase to un-draft/merge; do NOT open a duplicate.
  • Search run: search/issues?q=repo:sgl-project/sglang+is:pr+is:open+mori+gtest.

Suggested Triage Steps

• Un-draft and land #30942 (pin cmake==4.3.4), then re-run nightly-2-gpu-mi35x-glm51-mxfp4 to confirm the MORI build passes and the GLM test actually executes.
• Before merging, verify the A/B: build MORI e31d426a in rocm/sgl-dev:v0.5.15-rocm700-mi35x-20260712 with cmake 4.3.4 vs 4.4 to confirm 4.3.4 resolves the gtest_discover_tests JSON parse error.
• If pinning cmake is only a stopgap, consider disabling MORI test-target discovery in the CI reinstall (-DBUILD_TESTING=OFF or GTEST_DISCOVERY_MODE=PRE_TEST) so a discovery-time issue can't block dependency install — for maintainer decision, not a directive.

Status

• Persistence: First observed in this run (29202683625, 2026-07-12); a clean prior baseline is unavailable because recent scheduled nightlies were cancelled. The dated (2026-07-12) image + cmake 4.4 suggests a newly-introduced break.
• Scope: Blocks the entire amd_ci_install_dependency.sh step, so every AMD job that reinstalls MORI on this image is at risk — not just nightly-2-gpu-mi35x-glm51-mxfp4. The GLM5.1-mxfp4 nightly accuracy signal is fully blocked (test never ran).
• In-flight fix: ✅ #30942 (draft, opened 2026-07-12) — pin cmake==4.3.4.

nightly-2-gpu-mi35x-deepseek-r1-mxfp4-tp2 — failed step(s): Install dependencies

Commit Info

• sglang: 96a04cb13
• aiter: N/A — no [CI-AITER-CHECK] markers in the log. The job died in the MORI build inside "Install dependencies", which runs before aiter detection (amd_ci_install_dependency.sh:226), so aiter was never resolved.

Failed Tests

Test File	Test Function	Error	Log
N/A — dependency build failure (not a test)	N/A	CMake Error … ParseTestList.cmake:96 (string): JSON failed parsing … Line 1, Column 1 Syntax error while running gtest_discover_tests on src/umbp/tests/test_peer_dram_allocator	Job log

The actual accuracy test Accuracy Test MI35x (2-GPU DeepSeek-R1-MXFP4 TP2) (step 6) was skipped — it never ran. The failure is entirely in step 5 "Install dependencies".

Failure Summary

The "Install dependencies" step reinstalls MoRI from the Dockerfile-pinned commit (e31d426a) via python3 setup.py develop. The MoRI CMake build reached [133/146], successfully linked test_peer_dram_allocator, then gtest_discover_tests ran that executable at build time to enumerate its test cases and failed to parse the output as JSON (ninja: build stopped: subcommand failed → setup.py CalledProcessError → step exit 1). No sglang test executed.

Regression Status

New regression (in the Install/MoRI-build step). The "Install dependencies" step for this job passed in every prior scheduled run and first broke today.

Recent history of the MoRI build in "Install dependencies" for job nightly-2-gpu-mi35x-deepseek-r1-mxfp4-tp2 (branch main, event schedule, completed runs only):

Date	Run	Install step (MoRI build)	Note
Jul 12	29202683625	❌ Failed	MoRI gtest_discover JSON parse error (cmake 4.4)
Jul 11	29162254653	✅ Passed	job failed later (accuracy)
Jul 10	29113206892	✅ Passed	job failed later (accuracy)
Jul 9	29039566002	✅ Passed	job failed later (accuracy)
Jul 8	28964511688	✅ Passed	job succeeded

Install/MoRI-build first observed failing: 2026-07-12; last passing: 2026-07-11. (Note: the overall job was already red Jul 9–11 for a separate, downstream accuracy-test reason; today's failure is a distinct, earlier build break.)

Failure Cluster

MoRI gtest_discover_tests JSON parse break under cmake 4.4 (ROCm install-dependencies)

Facts (verified observations, no interpretation)

• Verbatim error (log): CMake Error at /opt/venv/lib/python3.10/site-packages/cmake/data/share/cmake-4.4/Modules/GoogleTest/ParseTestList.cmake:96 (string): string sub-command JSON failed parsing json string: * Line 1, Column 1 Syntax error: value, object or array expected. then ninja: build stopped: subcommand failed. and subprocess.CalledProcessError: Command '['cmake', '--build', '.', '-j', '128']' returned non-zero exit status 1.
• The cmake in use is 4.4 — path cmake/data/share/cmake-4.4/... (FACT, from log).
• MoRI reinstall marker (log): [MORI] Reinstalling MORI e31d426a13e96e1cbff96a1c904d291aefe8c46a (MORI_GPU_ARCHS=gfx950).
• MORI_COMMIT is identical (e31d426a) in both the last-passing SHA 9b4bb415dd (Jul 11) and today's 96a04cb13 — verified via GitHub contents API. The MoRI source being built did not change.
• The failed cmake target is the test-discovery of src/umbp/tests/test_peer_dram_allocator; the executable linked fine and other test executables in the same batch also linked — only the discovery-time JSON parse failed.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [HIGH] Unpinned cmake auto-upgraded from 4.3.4 → 4.4 (pulled from PyPI at image/install time); cmake 4.4's GoogleTest/ParseTestList.cmake rejects the output MoRI's gtest_discover_tests feeds it, breaking the MoRI build.
  • Supporting: log shows cmake-4.4 path in the exact failing module; MoRI source (MORI_COMMIT) unchanged between Jul 11 pass and Jul 12 fail, so cmake is the moved variable; in-flight fix PR #30942 documents "Last green image build 07-10 — cmake 4.3.4; first red 07-11 — cmake 4.4" and fixes it by pinning cmake==4.3.4.
  • Disconfirming: (none found) — the version delta, timing, and the maintainer's own root-cause writeup all converge.
  • How to verify: rebuild the ROCm image / rerun install-dependencies with cmake==4.3.4 pinned (as in PR #30942) and confirm MoRI gtest_discover passes.

In-flight Fix Check

• In-flight fix: ✅ #30942 "[AMD] Pin cmake==4.3.4 in ROCm Dockerfile to fix MoRI gtest_discover build break" (open draft, created 2026-07-12 18:13 UTC by yctseng0211) — directly targets this break. Chase review/merge and un-draft; do NOT open a duplicate. (Found via search/issues?q=repo:sgl-project/sglang+is:pr+is:open+gtest_discover.)

Suggested Triage Steps

• Land in-flight PR #30942 (pin cmake==4.3.4) — since the CI image is not rebuilt on every run, also confirm the install-dependencies MoRI reinstall path picks up the pin (the reinstall uses the container's existing cmake, so the pin must be present in the image or the reinstall block).
• As a confirmation A/B: in a container, pip install 'cmake==4.3.4' then rerun cd /sgl-workspace/mori && python3 setup.py develop on an mi35x/gfx950 node; expect the test_peer_dram_allocator discovery to pass.
• After the cmake pin lands, re-run this nightly job to expose the previously-masked downstream accuracy-test status (the job was already red Jul 9–11 for a separate reason once install succeeded).

Status (factual, no priority assignment)

• Persistence: MoRI-build break first seen 2026-07-12 (this run); the underlying cmake-4.4 regression has broken the nightly ROCm image build since 2026-07-11 per PR #30942.
• Scope: Affects the "Install dependencies" step of any AMD job that reinstalls MoRI (ENABLE_MORI images), plus the release-docker-amd-nightly.yml image build — not isolated to this one job.
• Blocked work: Blocks all downstream testing for MoRI-enabled AMD jobs (the DeepSeek-R1-MXFP4 TP2 accuracy test never runs).
• In-flight fix: ✅ #30942 (open draft) — pin cmake==4.3.4.

nightly-test-1-gpu-mi35x — failed step(s): Install dependencies

Commit Info

• sglang: 96a04cb1
• aiter: N/A — the job failed during the MoRI dependency build, before the [CI-AITER-CHECK] block ran; no [CI-AITER-CHECK] markers are present in the log.
• MoRI: e31d426a (pinned in docker/rocm.Dockerfile:113, authored 2026-06-25, unchanged across all recent runs)

Failed Tests

This is not a test failure — it is a build failure of the MoRI dependency during the Install dependencies step (step 5, scripts/ci/amd/amd_ci_install_dependency.sh → python3 setup.py develop). No sglang test ever ran.

Component	Failing target	Error	Log
MoRI umbp gtest discovery	test_peer_ssd_eviction	CMake ParseTestList.cmake:96 — string JSON parse failed: "Syntax error: value, object or array expected"	Log
MoRI umbp gtest discovery	test_peer_dram_allocator	same	Log
MoRI umbp gtest discovery	test_ssd_read_lease_gating	same	Log
MoRI umbp gtest discovery	test_tier_priority_route_get	same	Log

Terminal errors: ninja: build stopped: subcommand failed. → subprocess.CalledProcessError: Command '['cmake', '--build', '.', '-j', '128']' returned non-zero exit status 1. → ##[error]Process completed with exit code 1.

Failure Summary

The nightly-test-1-gpu-mi35x job never reached any test. During dependency install it rebuilds MoRI (MORI_GPU_ARCHS=gfx950) from the Dockerfile-pinned commit. The MoRI build fails at the gtest_discover_tests post-link step for four umbp SSD/peer/tier test binaries: CMake runs each freshly-built binary to enumerate its tests as JSON but receives empty output, so ParseTestList.cmake aborts with a JSON parse error, ninja stops, and setup.py develop exits 1.

Regression Status

New regression (first observed today, 2026-07-12). The MoRI gtest-discovery build break does not appear in any prior run of this job in the queryable window; the recent red runs of nightly-test-1-gpu-mi35x all failed for different, unrelated reasons.

Recent history of the MoRI build failure in job nightly-test-1-gpu-mi35x (branch main, event=schedule, completed runs only):

Date	Run	MoRI build	Actual failure cause that run
Jul 12	29202683625	❌ Failed	MoRI gtest_discover_tests JSON parse (this report)
Jul 11	29162254653	✅ Built OK	GPU Hang during test run (HW Exception … GPU Hang, exit 134)
Jul 10	29113206892	✅ Built OK	— (job passed)
Jul 9	29039566002	✅ (not reached)	VRAM cleanup unsuccessful (pre-install, exit 1)
Jul 5	28749559519	✅ (not reached)	VRAM cleanup unsuccessful (pre-install, exit 1)

First observed (MoRI build break): 2026-07-12. Last known MoRI build success: 2026-07-11 (run 29162254653, same pinned MoRI commit).

Failure Cluster

MoRI gtest_discover_tests build break — empty JSON from umbp test binaries (cmake 4.4)

Facts (verified observations)

• Today's run uses cmake 4.4 — 8 references to .../cmake/data/share/cmake-4.4/... in the log; the error originates in cmake-4.4/Modules/GoogleTest/ParseTestList.cmake:96.
• The Jul 11 run built MoRI successfully from the identical pinned commit e31d426a — no MoRI or sglang code change is involved.
• Only 4 of the ~12 umbp test binaries fail discovery (test_peer_ssd_eviction, test_peer_dram_allocator, test_ssd_read_lease_gating, test_tier_priority_route_get); siblings like test_external_kv_block_index, test_router_dedup, test_ssd_copy_pipeline discover fine.
• docker/rocm.Dockerfile installs cmake unpinned, so each daily image pull can drift the cmake version.
• Today's container image tag: rocm/sgl-dev:v0.5.15-rocm700-mi35x-20260712 (built today, ships cmake 4.4).

Hypothesised Causes (with confidence)

• [HIGH] cmake 4.4 broke MoRI's gtest_discover_tests JSON output, which is now parsed as empty by ParseTestList.cmake.
  • Supporting: today's build runs under cmake 4.4 (verified in log); the error is inside cmake-4.4's ParseTestList.cmake; the same MoRI commit built fine on Jul 11 under the prior image; in-flight fix #30942 documents the identical root cause and timeline (last green image 29093958349 on 07-10 with cmake 4.3.4; first red on 07-11 with cmake 4.4).
  • Disconfirming: (none found) — the version-drift timeline and PR evidence are mutually consistent.
  • How to verify: rebuild MoRI in the mi35x-20260712 image with pip install cmake==4.3.4 and re-run python3 setup.py develop; expect the four discoveries to succeed.
• [LOW] The four failing binaries crash at --gtest_list_tests time (SSD/peer/DRAM/tier static init touching host resources).
  • Supporting: only these 4 binaries fail; empty stdout is consistent with an early crash/abort.
  • Disconfirming: they discovered fine under cmake 4.3.4 on the same runner class the day before, so the binaries themselves are not the trigger — the cmake version is.
  • How to verify: run one binary manually with --gtest_list_tests under both cmake versions.

In-flight Fix Check

• In-flight fix: ✅ #30942 "[AMD] Pin cmake==4.3.4 in ROCm Dockerfile to fix MoRI gtest_discover build break" (open, created 2026-07-12 by yctseng0211) — pins cmake==4.3.4 in docker/rocm.Dockerfile. Chase reviewers; do NOT open a duplicate. (Query: search/issues?q=repo:sgl-project/sglang+is:pr+is:open+gtest+discover.)

Suggested Triage Steps

• Land in-flight fix #30942 (pin cmake==4.3.4), then re-run nightly-test-1-gpu-mi35x to confirm MoRI reinstalls cleanly.
• Since the CI image is baked with an unpinned cmake, also confirm the nightly image rebuild (release-docker-amd-nightly.yml) picks up the pin so the baked image ships cmake 4.3.4 — otherwise the install-step reinstall keeps hitting cmake 4.4 until the image refreshes.
• Longer term: report the JSON-output regression upstream to cmake / MoRI's gtest_discover_tests usage so the pin can eventually be lifted.

Status

• Persistence: First occurrence 2026-07-12; MoRI built successfully as recently as 2026-07-11 (same pinned commit).
• Scope: Affects every AMD job that reinstalls MoRI in the Install dependencies step on the cmake-4.4 image (this is mi35x/gfx950; gfx942 mi30x/mi325 jobs run the same reinstall path and are likely affected too). Blocks all downstream nightly tests in this job.
• Blocked work: Entire nightly-test-1-gpu-mi35x suite (no tests executed).
• In-flight fix: ✅ #30942 (open since 2026-07-12) — pins cmake==4.3.4.

nightly-accuracy-8-gpu-mi35x-deepseek-v32 — failed step(s): Accuracy Test MI35x (8-GPU DeepSeek-V3.2)

Commit Info

• sglang: 96a04cb1
• aiter: 9127c94a — v0.1.17.dev110+g9127c94a1 (source: dev/patched CI image preserved; AITER_COMMIT_OVERRIDE empty; Dockerfile default is the same SHA)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/accuracy/mi35x/test_deepseek_v32_eval_mi35x.py	test_deepseek_v32_accuracy (crash in server subprocess during weight load, before assertion)	HW Exception by GPU node-3 reason :GPU Hang → Fatal Python error: Aborted → exit code 134 (SIGABRT)	Log

Failure Summary

The nightly-accuracy-8-gpu-mi35x-deepseek-v32 job crashed while loading DeepSeek-V3.2 weights (TP=8) on the MI35x runner. At ~66% of shard loading (108/163), a HW Exception by GPU node-3 (reason: GPU Hang) fired, causing every model-loader worker thread to abort (Fatal Python error: Aborted, exit code 134). Both the primary attempt and the HF_HUB_OFFLINE=1 retry crashed identically. The crash occurred during weight loading — before inference, accuracy evaluation, or CUDA-graph capture ran.

Regression Status

Known recurring failure — GPU-Hang cluster failing for 3 consecutive completed scheduled runs (since 2026-07-10).

Recent history of test_deepseek_v32_eval_mi35x.py (job nightly-accuracy-8-gpu-mi35x-deepseek-v32, branch=main, event=schedule, completed runs only):

Date	Run	Test File Status	Symptom
Jul 12	29202683625	❌ Failed	GPU Hang → exit 134
Jul 11	29162254653	❌ Failed	GPU Hang → exit 134
Jul 10	29113206892	❌ Failed	GPU Hang → exit 134
Jul 09	29039566002	❌ Failed	different — exit code 1 (setup)
Jul 08	28964511688	❌ Failed	different — JIT cooperative_groups.h not found + watchdog timeout → accuracy AssertionError
Jul 07	28888348134	❌ Failed	different — exit code 1 (setup)
Jul 06	28813271245	✅ Passed	—
Jul 05	28749559519	✅ Passed	—

First observed GPU-Hang failure: 2026-07-10. Last GREEN run for this test file: 2026-07-06. Note the test file has been red every run since 07-07, but for three distinct symptoms — only the 07-10 → 07-12 window is the GPU-Hang cluster. (Today's run 29202683625 is still queued at the workflow level, but this job itself completed with conclusion=failure.)

Failure Cluster

GPU Hang HW exception during model weight loading (reason :GPU Hang on GPU node-3, faulthandler abort inside fused_moe_triton/layer.py weight-loader threads).

Facts (verified observations)

• Verbatim trigger (log ~line 4749): HW Exception by GPU node-3 (Agent handle: 0x...) reason :GPU Hang, printed 15× (one per loader thread), immediately followed by Fatal Python error: Aborted.
• All faulthandler stack traces top out in python/sglang/srt/layers/moe/fused_moe_triton/layer.py at _load_w13 (line 519) / _load_w2 (line 589) → _load_model_weight_or_group_weight_scale → _weight_loader_impl → weight_loader, driven by concurrent.futures worker threads (the enable_multithread_load: true loader). This is the weight-load phase, not a compute kernel.
• Crash point: Multi-thread loading shards: 66% Completed | 108/163 — mid weight load.
• Both attempts failed identically (primary + HF_HUB_OFFLINE=1 retry), exit code 134 = 128 + SIGABRT.
• Infra instability signal in the same run: local docker registry pull failed (context deadline exceeded), fell back to public registry (log line 1470-1471).
• aiter is an unchanged pinned dev image across passing (07-05/06) and failing (07-10/11/12) runs — see [CI-AITER-CHECK] markers.
• This is nightly-test-amd.yml (not aiter-scout) → no Failure-Origin / Baseline A/B check required.

Hypothesised Causes (with confidence)

• [MEDIUM] GPU hardware/driver hang on the MI35x runner (node-3), not an sglang code fault.
  • Supporting: the hang fires during pure weight-load tensor copies (no custom kernel, no graph capture); it names a specific hardware agent (GPU node-3); the same run also shows unrelated infra flakiness (registry timeout); "GPU Hang" is a driver/HW-level HSA exception, not a Python/CUDA API error.
  • Disconfirming: it has reproduced 3 consecutive runs (07-10 → 07-12), which is more persistent than a typical one-off bad-node flake — could instead indicate a sticky bad runner in the pool, or a genuine software regression in the load path.
  • How to verify: check which physical runner served each of the 3 failing runs (are they the same host?); rerun the job pinned to a different MI35x node; check dmesg/amd-smi XID/hang counters on node-3.
• [LOW] Regression in the DeepSeek-V3.2 / MoE weight-load path landing in the Jul 06..Jul 10 window.
  • Supporting: last GREEN was 07-06; the GPU-Hang symptom is new as of 07-10; the stack sits in fused_moe_triton/layer.py weight loaders.
  • Disconfirming: no code-level evidence yet; weight loading is largely host→device copy and unlikely to hang the GPU by itself; the intervening 07-07/08/09 failures were unrelated symptoms (JIT cooperative_groups.h missing, setup exit-1), suggesting the runner/image environment was already unstable before the hang appeared.
  • How to verify: git log --oneline the range between the 07-06 pass SHA (3abdbab9) and the 07-10 fail SHA (7998fecf), grep for changes under layers/moe/fused_moe_triton/, DeepSeek-V3.2 loading, or aiter preshuffle; A/B by reverting suspects and rerunning on a known-good node.

No hypothesis reaches HIGH; the two must be disambiguated by node identity + a clean-node rerun before blaming code.

In-flight Fix Check

• In-flight fix: ⚠️ [#30431](https://github.com/sgl-project/sglang/pull/30431) "[AMD] ci: self-heal driver-level VRAM leaks + graceful GPU cleanup on shutdown" — a candidate mitigation for driver-level GPU state hangs on AMD CI, not a confirmed fix for this specific weight-load hang. Query: search/issues?q=...+is:open+GPU+Hang+mi35x.
• In-flight fix: ❌ none found for a direct DeepSeek-V3.2 weight-load GPU-hang fix.

Suggested Triage Steps

• Identify the physical MI35x host for runs 29113206892, 29162254653, 29202683625 — if all three are the same node, drain/reset node-3 and rerun.
• Rerun the job on a different MI35x node; if it passes, this is a bad-runner/hardware issue, not a code regression.
• If the hang follows the code across nodes: bisect 3abdbab9..7998fecf, focusing on layers/moe/fused_moe_triton/layer.py and DeepSeek-V3.2 multithread load; try enable_multithread_load: false to isolate the threaded loader.
• Track #30431 as a possible CI self-heal mitigation; confirm whether it clears the node before rerun.

Status

• Persistence: GPU-Hang cluster failing 3 consecutive completed scheduled runs (2026-07-10 → 2026-07-12); test file red every run since 2026-07-07 across 3 distinct symptoms.
• Scope: nightly-accuracy-8-gpu-mi35x-deepseek-v32. The sister ...-deepseek-v32-mtp accuracy job also failed on 07-10/07-11 — worth checking whether it shares the GPU-Hang symptom (blast radius on the MI35x pool).
• Blocked work: DeepSeek-V3.2 accuracy signal on MI35x (gfx950) nightly CI.
• In-flight fix: ⚠️ #30431 candidate CI mitigation; no direct fix PR found.

nightly-8-gpu-mi35x-qwen35-triton-dcp — failed step(s): Install dependencies

Commit Info

• sglang: 96a04cb1
• aiter: N/A — the failure occurred in the Install dependencies step (MORI build) before aiter was set up; no [CI-AITER-CHECK] markers are present in the log.
• MORI: e31d426a (Dockerfile default, docker/rocm.Dockerfile:113)
• Base image: rocm/sgl-dev:v0.5.15-rocm700-mi35x-20260712

Failed Tests

This is not a pytest failure — the accuracy test (Step 6) never ran. The job failed in Step 5 "Install dependencies" while rebuilding the MORI dependency.

Component	Stage	Error	Log
MORI (umbp gtest targets: test_external_kv_hit_index, test_peer_ssd_eviction, test_client_registry, test_router_dedup)	gtest_discover_tests (post-build)	CMake Error … ParseTestList.cmake:96 (string): string sub-command JSON failed parsing json string: Line 1, Column 1 — Syntax error: value expected → ninja: build stopped → subprocess.CalledProcessError … exit status 1	Step 5 log

Failure Summary

Step 5 "Install dependencies" failed while rebuilding the MORI library. After MORI's umbp unit-test executables were linked, CMake's gtest_discover_tests ran each binary to enumerate its tests and got empty/invalid output for 4 targets, causing ParseTestList.cmake to abort with a JSON parse error. Ninja stopped, setup.py develop raised CalledProcessError, and the step exited 1. The 8-GPU Qwen 3.5 Triton DCP accuracy test itself never executed.

Regression Status

New regression (this failure signature is new today). The same job has been red every night since Jul 8, but at different steps each night — today's MORI-build break at Step 5 is distinct from the earlier failures.

Recent history of job nightly-8-gpu-mi35x-qwen35-triton-dcp (nightly-test-amd.yml, branch main, event schedule, completed runs only):

Date	Run	Failing Step	Signature
Jul 12	29202683625	Step 5 Install dependencies	❌ MORI gtest_discover JSON parse (this failure)
Jul 11	29162254653	Step 6 Accuracy Test	⚠️ Different: GPU Hang / exit 134 (MORI built fine)
Jul 10	29113206892	Step 6 Accuracy Test	⚠️ Different (MORI built fine)
Jul 9	29039566002	Step 6 Accuracy Test	⚠️ Different (MORI built fine)
Jul 8	28964511688	Step 3 Ensure VRAM is clear	⚠️ Different: infra
Jul 5	28749559519	—	✅ Passed

For the MORI-build (Step 5) signature specifically: first observed today Jul 12; MORI built successfully every prior day in the window. Lookback range: Jul 5 – Jul 12 (scheduled runs).

Failure Cluster

MORI gtest_discover_tests JSON parse failure under cmake 4.4 (ROCm image dependency build break)

Facts (verified observations)

• MORI commit is unchanged across the passing (Jul 11) and failing (Jul 12) runs: both e31d426a (from docker/rocm.Dockerfile:113, reinstalled by scripts/ci/amd/amd_ci_install_dependency.sh:205). So the MORI source did not change. [FACT]
• The base image changed: Jul 11 used …-mi35x-20260711, today uses …-mi35x-20260712. [FACT]
• Today's build uses cmake 4.4 (…/cmake/data/share/cmake-4.4/Modules/GoogleTest/ParseTestList.cmake). [FACT]
• Exactly 4 of MORI's umbp test binaries fail discovery; all other targets ([135/146]…[142/146]) linked successfully. The error is at test discovery (running the binary to list tests), not compilation/linking. [FACT]
• The accuracy test step (Step 6) did not run; no aiter/model-load logs are present. [FACT]

Hypothesised Causes

• [HIGH] The new 20260712 ROCm dev image ships cmake 4.4, whose GoogleTest/ParseTestList.cmake breaks MORI's gtest_discover_tests (empty/invalid JSON → "value expected").
  • Supporting: cmake 4.4 in the failing log; base image bumped 20260711→20260712 while MORI commit is identical; an open sglang PR (#30942, opened 2026-07-12 18:13 UTC — minutes before this run) is titled "Pin cmake==4.3.4 in ROCm Dockerfile to fix MoRI gtest_discover build break", describing this exact symptom.
  • Disconfirming: I could not read the exact cmake version in the Jul 11 log (MORI built cleanly there, so no cmake path was printed) — the cmake-4.4 vs earlier delta is inferred from the image-tag change, not directly diffed. Only 4 of many gtest targets fail, which the cmake-version hypothesis alone doesn't fully explain (may be a JSON-output nuance in those 4 binaries).
  • How to verify: rebuild MORI e31d426a in the 20260712 image with cmake pinned to 4.3.4 (i.e. apply #30942) and confirm gtest_discover_tests passes.

This is an image/toolchain dependency break, not an sglang source regression — no code change in the sglang delta is implicated (docker/rocm.Dockerfile MORI pin is unchanged).

In-flight Fix Check

In-flight fix: ✅ #30942 (open since 2026-07-12) — "[AMD] Pin cmake==4.3.4 in ROCm Dockerfile to fix MoRI gtest_discover build break". Directly targets this failure; chase reviewers/merge, do not open a duplicate. (Search run: search/issues?q=repo:sgl-project/sglang+is:pr+is:open+mori+in:title.)

Suggested Triage Steps

• Verify/merge in-flight fix #30942; after merge, re-run this job on the 20260712 (or newer) image and confirm Step 5 passes.
• If a faster unblock is needed, pin cmake==4.3.4 in scripts/ci/amd/amd_ci_install_dependency.sh before the MORI reinstall (matches the Dockerfile fix).
• Independently, the pre-existing Step 6 accuracy-test failures (Jul 9–11 GPU Hang, exit 134) and the Jul 8 VRAM-clear infra failure are separate issues that will resurface once the MORI build is fixed — track them separately.

Status

• Persistence: MORI-build (Step 5) signature: 1 run (new today, Jul 12). Job overall: red every scheduled night since Jul 8; last green Jul 5.
• Scope: Affects any AMD job on the 20260712 ROCm image that rebuilds MORI (MORI is rebuilt by amd_ci_install_dependency.sh for ENABLE_MORI=1 images), so likely broader than this single job.
• Blocked work: Blocks the 8-GPU MI35x Qwen 3.5 Triton DCP nightly accuracy signal; the accuracy test cannot run until the dependency build is fixed.
• In-flight fix: ✅ #30942 (open since 2026-07-12) — awaiting merge.

nightly-accuracy-8-gpu-mi35x-deepseek-v32-mtp — failed step(s): Install dependencies

Commit Info

• sglang: 96a04cb1 — "Fix DeepEP CI test registration" (#30873); touches only .github/workflows/pr-test-extra.yml, test/run_suite.py, and two test/registered/ files — unrelated to the MoRI build.
• aiter: N/A — no [CI-AITER-CHECK] markers in this nightly-accuracy job log.
• MoRI: e31d426a (MORI_GPU_ARCHS=gfx950, reinstalled by amd_ci_install_dependency.sh) — unchanged across all runs in the window.

Failed Tests

Test File	Test Function	Error	Log
N/A — build error (dependency install)	N/A	CMake Error … string(JSON) … Syntax error: value, object or array expected → ninja: build stopped → CalledProcessError … exit status 1	Log

The target test test/registered/amd/accuracy/mi35x/test_deepseek_v32_mtp_eval_mi35x.py never ran — the job died during the amd_ci_install_dependency.sh step while rebuilding MoRI.

Failure Summary

The job failed while building the MoRI dependency during environment setup, before the DeepSeek-V3.2 MTP accuracy test executed. CMake 4.4's GoogleTest gtest_discover_tests ran the freshly-linked umbp test binaries (test_external_kv_hit_index, test_client_registry, test_client_registry_external_kv, test_external_kv_block_index, test_tier_priority_route_get, test_ssd_copy_pipeline) for POST-BUILD test discovery; the JSON output was empty, so ParseTestList.cmake:96 errored, ninja stopped, and MoRI's setup.py develop exited 1.

Regression Status

New failure signature today (2026-07-12) — distinct from the failure that has affected this job on prior days.

Recent history of the job nightly-accuracy-8-gpu-mi35x-deepseek-v32-mtp (branch main, event schedule, completed runs only):

Date	Run	Job Status	Failure Signature	Exit
Jul 12	29202683625	❌ Failed	MoRI build break — cmake 4.4 gtest_discover JSON parse (test never ran)	1
Jul 11	29162254653	❌ Failed	GPU Hang during accuracy test (fused_moe_triton/layer.py weight load)	134
Jul 10	29113206892	❌ Failed	(accuracy test ran; not MoRI build)	—
Jul 08	28964511688	❌ Failed	(accuracy test ran; not MoRI build)	—
Jul 07	28888348134	❌ Failed	accuracy test crash	255
Jul 06	28813271245	❌ Failed	accuracy test	1
Jul 05	28749559519	✅ Passed	—	—

The MoRI-build break is new to today's run (verified: Jul 11 and Jul 07 logs contain zero ParseTestList/gtest_discover markers and completed all 146 MoRI build steps before running the test). The underlying accuracy test has been red since Jul 06 (last green Jul 05) for a separate GPU-Hang reason, which today's earlier build break masks.

Note: the parent run 29202683625 still shows workflow-level queued, but this job is completed/failure, so it is validly analyzed.

Failure Cluster

MoRI gtest_discover_tests build break under CMake 4.4 (empty-JSON parse error)

Facts (verified observations)

• Error verbatim (log.txt:4373): CMake Error at .../cmake-4.4/Modules/GoogleTest/ParseTestList.cmake:96 (string): string sub-command JSON failed parsing json string: * Line 1, Column 1 Syntax error: value, object or array expected.
• CMake in use today: cmake-4.4 (pip-installed, /opt/venv/lib/python3.10/site-packages/cmake/data/share/cmake-4.4).
• Six umbp test targets FAILED at link/discovery (log.txt:4288,4306,4324,4345,4363,4381); then ninja: build stopped: subcommand failed (log.txt:4440) → subprocess.CalledProcessError: Command '['cmake', '--build', '.', '-j', '128']' returned non-zero exit status 1 (log.txt:4472) → ##[error]Process completed with exit code 1 (log.txt:4476).
• MoRI source commit e31d426a identical on Jul 12, Jul 11, Jul 07 (confirmed from [MORI] Reinstalling MORI … lines in each log).
• Jul 11 / Jul 07 logs: MoRI built to [146/146] with 0 gtest-discovery invocations; today discovery is invoked and fails.
• sglang commit 96a04cb1 modifies no build/C++/MoRI code.

Hypothesised Causes

• [HIGH] CMake 4.4 (pip) breaks MoRI's gtest_discover_tests: the stricter string(JSON …) parser in cmake-4.4's GoogleTest module errors on the empty test-list output from the umbp test binaries, whereas the previously-used cmake tolerated it.
  • Supporting: error originates entirely inside cmake-4.4/Modules/GoogleTest/ParseTestList.cmake; MoRI source and sglang code are unchanged; a maintainer independently filed #30942 "Pin cmake==4.3.4 in ROCm Dockerfile to fix MoRI gtest_discover build break" (open since 2026-07-12), naming the identical root cause and fix.
  • Disconfirming: (none found) — the fix PR title, the in-log cmake path, and the unchanged MoRI/sglang sources all converge.
  • How to verify: rebuild MoRI in the MI35x container with cmake==4.3.4 pinned (exactly what #30942 does) and confirm the umbp test discovery succeeds.
• [LOW] umbp test binaries themselves fail to execute at discovery time (e.g. missing runtime lib on the runner), producing empty JSON.
  • Supporting: "Line 1, Column 1" empty-string parse is consistent with a binary that emitted nothing.
  • Disconfirming: the same MoRI commit + same runner class discovered/ran fine when an older cmake tolerated empty output on prior days; the cmake-version hypothesis fully explains the change.
  • How to verify: run one umbp test binary manually with --gtest_list_tests --gtest_output=json.

In-flight Fix Check

• In-flight fix: ✅ #30942 "[AMD] Pin cmake==4.3.4 in ROCm Dockerfile to fix MoRI gtest_discover build break" (open since 2026-07-12, author yctseng0211) — chase reviewers; do not open a duplicate. (Searched is:pr is:open mori build and is:pr is:open cmake gtest.)

Suggested Triage Steps

• Land / expedite review of #30942 (pin cmake==4.3.4 in the ROCm Dockerfile), then re-run nightly-accuracy-8-gpu-mi35x-deepseek-v32-mtp to confirm the MoRI build succeeds.
• After the build break clears, note the underlying accuracy failure will likely resurface: the test has been red since Jul 06 with a GPU-Hang during weight loading (fused_moe_triton/layer.py _weight_loader_impl, exit 134). Bisect that separately in the sglang window between the Jul 05 pass (48ba79c1 region) and Jul 06 first-fail.
• Optionally verify by rebuilding MoRI locally with cmake 4.4 vs 4.3.4 to A/B-confirm the discovery break.

Status

• Persistence: MoRI build break is new as of 2026-07-12; the job has been red since 2026-07-06 (last green 2026-07-05), but for a different (GPU-Hang) reason.
• Scope: Any MI35x/ROCm job whose setup rebuilds MoRI via amd_ci_install_dependency.sh is at risk of this identical cmake-4.4 build break, not just this one accuracy job.
• Blocked work: DeepSeek-V3.2 MTP accuracy signal on MI35x; the test cannot even start until MoRI builds.
• In-flight fix: ✅ #30942 (pin cmake==4.3.4) — awaiting merge.

---

Generated by amd-bot using Claude Code CLI (last updated: 2026-07-12 23:22 UTC)

[amd-bot]

pr-test-amd.yml — 45 failure(s)

Run started (UTC): 2026-07-11 00:26 UTC
Last scanned (UTC): 2026-07-12 18:23 UTC
Commits: sglang 80856ab

Summary

I have the cluster and test data. Let me grep the In-flight Fix Check lines to aggregate per-cluster.

---

Note: agent timed out after 300s; showing the last assistant response captured before the kill. Re-run the bot command to retry.

---

Job	Failed Steps	Started
jit-kernel-unit-test-amd (linux-mi325-1gpu-sglang)	Install dependencies	2026-07-11T18:24
multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 0)	N/A	2026-07-11T17:41
multimodal-gen-test-2-gpu-amd (linux-mi325-2gpu-sglang, 2)	Run diffusion server tests (2-GPU)	2026-07-11T18:38
multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 1)	N/A	2026-07-11T18:26
multimodal-gen-test-2-gpu-amd (linux-mi325-2gpu-sglang, 0)	Run diffusion server tests (2-GPU)	2026-07-11T16:32
multimodal-gen-test-2-gpu-amd (linux-mi325-2gpu-sglang, 1)	Run diffusion server tests (2-GPU)	2026-07-11T17:05
multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 3)	Run diffusion server tests (1-GPU)	2026-07-11T18:40
stage-b-test-large-8-gpu-mi35x-disaggregation-amd (linux-mi35x-gpu-8.fabric)	Run test_disaggregation	2026-07-11T13:12
stage-b-test-1-gpu-small-amd (linux-mi325-1gpu-sglang, 10)	Start CI container	2026-07-11T19:58
stage-b-test-2-gpu-large-amd (linux-mi325-2gpu-sglang, 0)	Run test	2026-07-11T20:48
stage-c-test-large-8-gpu-amd-mi35x (linux-mi35x-gpu-8, 1)	Run test	2026-07-11T12:18
multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 3)	Run diffusion server tests (1-GPU)	2026-07-11T02:15
multimodal-gen-test-2-gpu-amd (linux-mi325-2gpu-sglang, 2)	Run diffusion server tests (2-GPU)	2026-07-11T04:46
stage-c-test-large-8-gpu-amd-mi35x (linux-mi35x-gpu-8, 0)	Run test	2026-07-11T12:18
multimodal-gen-test-2-gpu-amd (linux-mi325-2gpu-sglang, 0)	Run diffusion server tests (2-GPU)	2026-07-11T05:25
multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 1)	N/A	2026-07-11T02:19
multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 0)	N/A	2026-07-11T02:16
stage-b-test-1-gpu-small-amd-mi35x (linux-mi35x-gpu-1)	Install dependencies	2026-07-11T00:26
multimodal-gen-test-2-gpu-amd (linux-mi325-2gpu-sglang, 1)	Run diffusion server tests (2-GPU)	2026-07-11T05:31
stage-b-test-large-8-gpu-mi35x-disaggregation-amd (linux-mi35x-gpu-8.fabric)	Run test_disaggregation	2026-07-11T00:33
stage-b-test-1-gpu-small-amd (linux-mi325-1gpu-sglang, 6)	Install dependencies	2026-07-11T02:23
stage-c-test-large-8-gpu-amd-mi35x (linux-mi35x-gpu-8, 1)	Run test	2026-07-11T00:26
stage-c-test-large-8-gpu-amd-mi35x (linux-mi35x-gpu-8, 0)	Run test	2026-07-11T00:26
stage-c-test-large-8-gpu-amd-mi35x (linux-mi35x-gpu-8, 0)	Run test	2026-07-12T00:28
stage-b-test-1-gpu-small-amd-mi35x (linux-mi35x-gpu-1)	Run test	2026-07-12T00:28
stage-c-test-large-8-gpu-amd-mi35x (linux-mi35x-gpu-8, 1)	Run test	2026-07-12T00:28
stage-b-test-large-8-gpu-mi35x-disaggregation-amd (linux-mi35x-gpu-8.fabric)	Run test_disaggregation	2026-07-12T01:24
call-pr-test-amd-extra / extra-a-test-2-gpu-large-amd (linux-mi325-2gpu-sglang)	Run test	2026-07-12T01:25
multimodal-gen-test-2-gpu-amd (linux-mi325-2gpu-sglang, 1)	Run diffusion server tests (2-GPU)	2026-07-12T01:50
multimodal-gen-test-2-gpu-amd (linux-mi325-2gpu-sglang, 2)	N/A	2026-07-12T01:47
multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 3)	Run diffusion server tests (1-GPU)	2026-07-12T02:53
multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 1)	N/A	2026-07-12T02:46
multimodal-gen-test-2-gpu-amd (linux-mi325-2gpu-sglang, 0)	Run diffusion server tests (2-GPU)	2026-07-12T01:50
multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 0)	N/A	2026-07-12T03:23
stage-c-test-large-8-gpu-amd-mi35x (linux-mi35x-gpu-8, 0)	Run test	2026-07-12T12:18
stage-b-test-1-gpu-small-amd-mi35x (linux-mi35x-gpu-1)	Run test	2026-07-12T12:18
stage-c-test-large-8-gpu-amd-mi35x (linux-mi35x-gpu-8, 1)	Run test	2026-07-12T12:18
stage-b-test-large-8-gpu-mi35x-disaggregation-amd (linux-mi35x-gpu-8.fabric)	Run test_disaggregation	2026-07-12T12:35
call-pr-test-amd-extra / extra-a-test-2-gpu-large-amd (linux-mi325-2gpu-sglang)	Run test	2026-07-12T13:26
multimodal-gen-test-2-gpu-amd (linux-mi325-2gpu-sglang, 1)	Run diffusion server tests (2-GPU)	2026-07-12T13:51
multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 3)	Run diffusion server tests (1-GPU)	2026-07-12T14:30
multimodal-gen-test-2-gpu-amd (linux-mi325-2gpu-sglang, 0)	N/A	2026-07-12T13:20
multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 1)	N/A	2026-07-12T16:22
multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 0)	N/A	2026-07-12T16:24
multimodal-gen-test-2-gpu-amd (linux-mi325-2gpu-sglang, 2)	Run diffusion server tests (2-GPU)	2026-07-12T15:22

Per-Job Analysis

jit-kernel-unit-test-amd (linux-mi325-1gpu-sglang) — failed step(s): Install dependencies

Commit Info

• sglang: 32cb89d4
• aiter: N/A (no [CI-AITER-CHECK] markers in log; this is a jit-kernel build/test job, AITER_COMMIT_OVERRIDE empty)

Failed Tests

Test File	Test Function	Error	Log
N/A — infrastructure failure (no tests ran)	N/A	git submodule clone of 3rdparty/spdk failed: curl 56 GnuTLS recv error (-54); fatal: protocol error: bad pack header → exit code 128	Log

Failure Summary

The job never reached the JIT-kernel unit tests. It failed during the scripts/ci/amd/amd_ci_install_dependency.sh step while reinstalling the MORI dependency. Cloning MORI's 3rdparty/spdk git submodule aborted with a network/transport error (curl 56 GnuTLS recv error (-54), fatal: protocol error: bad pack header), yielding exit code 128.

Regression Status

Infrastructure issue (network flake during git submodule fetch) — not a code regression.

Recent history of jit-kernel-unit-test-amd (linux-mi325-1gpu-sglang) in workflow pr-test-amd.yml (branch main, event schedule, completed runs only):

Date	Run	Job	Job Status	Note
Jul 11	29152390308	job	❌ Failed	spdk submodule clone network error (this run)
Jul 10	29060215979	job	✅ Passed	—
Jul 09	29019359571	job	✅ Passed	—
Jul 09	28985427127	job	✅ Passed	—

First observed failure: this run (2026-07-11). Last known passing: 2026-07-10 (29060215979). The 3 newer scheduled runs (29132618067, 29093581297, and this one's siblings) were still queued at snapshot time and are excluded.

Failure Cluster

Git submodule fetch network failure during dependency install (MORI/spdk clone, curl 56 GnuTLS recv error)

Facts (verified observations, no interpretation)

• Log line 2905: [MORI] Reinstalling MORI e31d426a... (MORI_GPU_ARCHS=gfx942) — MORI reinstall begins in the install-dependency step.
• Log lines 2925–2931: MORI submodules 3rdparty/msgpack-c, 3rdparty/spdk, 3rdparty/spdlog registered; msgpack-c checked out successfully.
• Log lines 2932–2934 (verbatim):

  error: RPC failed; curl 56 GnuTLS recv error (-54): Error in the pull function.
  fatal: protocol error: bad pack header
  fatal: Fetched in submodule path '3rdparty/spdk', but it did not contain 2ef883ef96e79c3cc16da02f667a7a58c2453f2f. Direct fetching of that commit failed.
  

• Log line 2936: ##[error]Process completed with exit code 128.
• The failing step is Run bash scripts/ci/amd/amd_ci_install_dependency.sh (step group starts log line 1695). No pytest/JIT-kernel test invocation ever executed.
• Earlier transient network symptom at log line 1443: [local-pull] Error response from daemon: ... net/http: request canceled ... Client.Timeout exceeded (docker registry, non-fatal, job continued).
• Same job passed in the 3 most recent completed scheduled runs (see history table).

Hypothesised Causes (with confidence, with disconfirming evidence)

• [HIGH] Transient network/transport failure fetching the MORI 3rdparty/spdk submodule from GitHub.
  • Supporting: curl 56 GnuTLS recv error (-54) and bad pack header are classic mid-transfer connection-drop errors during git fetch; a preceding docker-registry timeout (line 1443) in the same run shows the runner had network instability; the failing SHA is in a third-party submodule, unrelated to the PR/sglang commit; same job green in the previous 3 completed runs at different sglang SHAs.
  • Disconfirming: (none found) — no code path in sglang commit 32cb89d4 is involved; the error occurs before any sglang code is exercised.
  • How to verify: re-run the job; if it passes with no code change, confirms a flake.
• [LOW] spdk upstream repo transiently unavailable / rate-limited at fetch time.
  • Supporting: only the 3rdparty/spdk submodule failed; msgpack-c succeeded moments earlier.
  • Disconfirming: curl 56 is a client-side receive error, more consistent with a dropped connection than a server 4xx/5xx.
  • How to verify: check whether concurrent jobs cloning spdk at the same UTC minute also failed.

In-flight Fix Check

• In-flight fix: ❌ none found — infrastructure flake, no code fix applicable. (Searched open PRs for spdk / mori submodule: curl -s -H "Authorization: token $GH_PAT" "https://api.github.com/search/issues?q=repo:sgl-project/sglang+is:pr+is:open+spdk" — no matching CI-fix PRs.)

Suggested Triage Steps

• Re-run the failed job (29152390308 / jit-kernel-unit-test-amd); a clean pass with no code change confirms the network flake.
• If MORI/spdk submodule clone failures recur across jobs, consider a shallow/retried submodule fetch or a local git mirror/cache for 3rdparty/spdk in scripts/ci/amd/amd_ci_install_dependency.sh (harness/infra change, not a PR fix).

Status (factual, no priority assignment)

• Persistence: single occurrence on 2026-07-11; the same job passed in the previous 3 completed scheduled runs (Jul 09–10).
• Scope: this job only, at the dependency-install step (git submodule fetch). No JIT-kernel test executed.
• Blocked work: JIT-kernel unit-test signal missing for this scheduled run only.
• In-flight fix: ❌ none found — infrastructure flake, re-run recommended.

multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 0) — failed step(s): N/A

Commit Info

• sglang: 32cb89d4 — ci: prune uv cache in job teardown to bound its growth (#30813)
• aiter: N/A — the job's log blob was never uploaded (runner died mid-step), so no [CI-AITER-CHECK] markers are available.

Failed Tests

Test File	Test Function	Error	Log
N/A (infrastructure failure — no test reached a verdict)	N/A	The self-hosted runner lost communication with the server (GitHub check-run annotation)	Job

This is not a test failure. The runner died during step 9 "Run diffusion server tests (1-GPU)" (step was still in_progress at teardown), so no per-job log was uploaded — the log endpoint returns BlobNotFound (HTTP 404). No pytest results, exit code, or stack trace exist for this run.

Failure Summary

Job multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 0) was marked failure because the self-hosted AMD MI325 runner lost communication with GitHub while running the diffusion server tests step (ran 17:41→18:47 UTC, ~66 min). Because the runner process terminated, no test log was uploaded and the only surviving signal is the GitHub infrastructure annotation. No test file executed to a pass/fail verdict.

Regression Status

Infrastructure issue (today's run). Distinct from the recurring test-level failures on this shard.

Recent history of shard multimodal-gen-test-1-gpu-amd (…, 0) (test/server/test_server_1_gpu.py::TestDiffusionServerOneGpu::test_diffusion_generation) — pr-test-amd.yml, event=schedule, branch=main, completed runs only (2 later runs 29132618067, 29093581297 are queued → excluded as [IN-FLIGHT]):

Date (UTC)	Run	Shard-0 Job	Job Status	Failure cause (annotation)
Jul 11 12:17	29152390308	86543869849	❌ Failed	runner lost communication (infra) ← this run
Jul 10 00:32	29060215979	86260258192	❌ Failed	Process completed with exit code 1 (test failures)
Jul 09 12:49	29019359571	86154130455	❌ Failed	Process completed with exit code 1 (test failures)
Jul 09 00:31	28985427127	86013325878	❌ Failed	Process completed with exit code 1 (test failures)
Jul 08 12:37	28943113233	85870034116	❌ Failed	Process completed with exit code 1 (test failures)
Jul 08 00:37	28908777371	85761538620	✅ Passed	—

The shard has been red for 5 consecutive completed runs (last green Jul 8 00:37), but the cause changed today: runs Jul 8–10 died with exit code 1 (real test failures), whereas today's run died from a runner-communication loss with no test log. At the failure-cause level, today's infra event is new; the underlying recurring test failures were masked because tests never completed.

Failure Cluster

Self-hosted MI325 runner lost communication during diffusion server tests (infra / runner death). Separate cluster from the recurring "diffusion server startup / port-5555 / Cosmos3 test failures" seen in prior runs.

Facts (verified observations)

• GitHub check-run annotation (job 86543869849): "The self-hosted runner lost communication with the server. … Anything in your workflow that terminates the runner process, starves it for CPU/Memory, or blocks its network access can cause this error." [FACT]
• Job step state at teardown: step 9 Run diffusion server tests (1-GPU) = in_progress; steps 10 & 20 = pending. Steps 1–8 (setup, container start, deps, cache) all success. [FACT]
• Log endpoint /actions/jobs/86543869849/logs returns HTTP 404 BlobNotFound (retried 3×) — no log was uploaded because the runner died. [FACT]
• Job wall-clock: started 2026-07-11 17:41:28 UTC, completed 18:47:31 UTC (~66 min). [FACT]
• Prior completed shard-0 failures are genuine test failures, not infra: run 29060215979 log shows RuntimeError: Scheduler port 5555 is unavailable and --strict-ports is enabled (server startup for fast_hunyuan_video, wan2_2_ti2v_5b, fastwan2_2_ti2v_5b, lingbot_world_realtime_plastic_beach), TimeoutError: Server not ready within 1200.0s, NotImplementedError: AITer backend does not support Grouped Query Attention yet and AttributeError: module diffusers has no attribute Cosmos3OmniTransformer (cosmos3_nano_t2v). [FACT]

Hypothesised Causes (with confidence)

• [FACT] The job failed because the self-hosted MI325 runner lost communication with GitHub (per the check-run annotation). This is the observable failure; it is an infrastructure/runner event, not a code defect surfaced by a test.
• [LOW] Resource starvation (OOM / CPU) during heavy diffusion model loading on the MI325 runner caused the runner agent to be killed or stop heartbeating.
  • Supporting: the step that was running loads large video/image diffusion models; prior runs of the same shard show 20-min (1200s) server-startup timeouts, indicating high memory/time pressure on this shard.
  • Disconfirming: no log survived, so there is no OOM-killer message or memory trace to confirm; equally consistent with a plain network blip or the runner host being reclaimed.
  • How to verify: check the MI325 runner host's system logs (dmesg / OOM-killer, runner-agent service logs) for the 17:41–18:47 UTC window; re-run the shard and watch host memory.
• [SPECULATION] Transient network partition between the runner and GitHub.
  • Supporting: annotation explicitly lists "blocks its network access" as a possible cause.
  • Disconfirming: none either way — not distinguishable from starvation without host logs.

Commit-level analysis is not applicable: this is not a code regression (no test ran), and the head commit 32cb89d4 only touches CI uv-cache teardown, unrelated to runner liveness during the test step.

In-flight Fix Check

• Infra (runner-lost-communication): In-flight fix: ❌ none found — not a code-fixable issue; needs runner-fleet/infra attention, not a PR. (Searched open PRs for strict-ports diffusion → 1 unrelated result; no runner-health PR.)
• Underlying recurring test failures (port 5555 / Cosmos3 / AITer GQA), for context only: In-flight fix: ❌ none found among open PRs matching diffusion port (29 open diffusion PRs, none targeting the --strict-ports 5555 startup collision or the Cosmos3/AITer-GQA errors).

Suggested Triage Steps

• Treat this run as an infrastructure retry candidate: re-run job 86543869849 on a healthy MI325 runner to get a real test verdict.
• Inspect the MI325 runner host that served this job for the 17:41–18:47 UTC window: check runner-agent service logs and dmesg for OOM-killer / process termination to confirm or rule out the [LOW] starvation hypothesis.
• Separately triage the pre-existing shard-0 test failures (independent of today's infra death): the Scheduler port 5555 is unavailable and --strict-ports is enabled collision suggests a prior server process wasn't torn down / port not released between parametrized cases — investigate diffusion server teardown and consider dynamic port allocation in test/server/test_server_1_gpu.py. Also track NotImplementedError: AITer backend does not support GQA and module diffusers has no attribute Cosmos3OmniTransformer (cosmos3_nano_t2v) as separate defects.
• Bisect for the shard-0 test regression (not this infra event) in b363249423..cfc66e05c5 (last green Jul 8 00:37 → first red Jul 8 12:37) if the port-5555 failures need a code owner.

Status (factual)

• Persistence: Shard-0 job red for 5 consecutive completed scheduled runs (since Jul 8 12:37 UTC); last green Jul 8 00:37 UTC. Today's run is the first failing with an infra runner-loss cause rather than exit code 1.
• Scope: This report covers shard 0. Other shards of the same job in the last completed run 29060215979 also failed (shards 1, 3) — the recurring diffusion test failures are broader than shard 0, but today's infra failure is specific to this runner/job instance.
• Signal impact: Because the runner died, this run yields no diffusion test signal for shard 0 — a green re-run is required before drawing any conclusion about the underlying tests.
• In-flight fix: ❌ none (infra); ❌ none for the underlying recurring test failures.

multimodal-gen-test-2-gpu-amd (linux-mi325-2gpu-sglang, 2) — failed step(s): Run diffusion server tests (2-GPU)

Commit Info

• sglang: 32cb89d4
• aiter: 9127c94a — v0.1.17.dev110+g9127c94a1 (source: dev image detected, rebuild skipped; matches Dockerfile default AITER_COMMIT=9127c94a)

Failed Tests

Test File	Test Function	Error	Log
sglang/multimodal_gen/test/single_test_file/test_ar_models.py	TestGLMImage::test_generates_image	ERROR at setup → TimeoutError: Server ... not healthy after 600.0s → RuntimeError: AR model failed to start for glmimage → HFValidationError/OSError: Repo id must be in the form 'repo_name' or 'namespace/repo_name': 'zai-org/GLM-Image/vision_language_encoder/'	Log

1 error in 1823.25s — the single collected test in this shard errored during setUpClass.

Failure Summary

The only test in this shard, TestGLMImage::test_generates_image, errored in setUpClass while bringing up the GLM-Image AR server. The sglang serve subprocess for role glmimage never became healthy (2 launch attempts, then a 600 s health-check timeout), because it exits immediately on start-up with HFValidationError: Repo id must be in the form ... 'zai-org/GLM-Image/vision_language_encoder/'. The test harness re-runs the failure once (run-last-failure) and it fails identically.

Regression Status

Never-passed (no green run found).

The test file test_ar_models.py, the TestGLMImage class, and the offending model-path construction f"{self.model}/vision_language_encoder/" were all added on 2026-07-09 by #25381 (7aab39a1, "[Diffusion] SGLang backend for GLM Image AR. Step 1 - Separate server"). The first scheduled pr-test-amd run after that merge already shows the identical error.

Recent history of test_ar_models.py (TestGLMImage::test_generates_image) in job multimodal-gen-test-2-gpu-amd (…, 2) (branch main, event=schedule, completed runs only):

Date	Run	Job	Test File Status	Failed Function	Error
Jul 11	29152390308	multimodal-gen-test-2-gpu-amd (…, 2)	❌ Error	TestGLMImage::test_generates_image	HFValidationError: invalid repo id …/vision_language_encoder/
Jul 10	29060215979	multimodal-gen-test-2-gpu-amd (…, 2)	❌ Error	TestGLMImage::test_generates_image	same HFValidationError (verified from log)
≤ Jul 09	—	—	⬜ Test did not exist	—	added by #25381 at 2026-07-09 15:54

First observed failure: 2026-07-10 (first scheduled run after merge). No passing run exists for this test in the queryable window (2026-07-06 → 2026-07-11); it has failed every scheduled run since it was introduced.

Failure Cluster

GLM-Image AR server start-up failure — invalid HF repo id from subfolder model-path.

Facts (verified observations, no interpretation)

• The AR server is launched with --model-path zai-org/GLM-Image/vision_language_encoder/ and --tokenizer-path zai-org/GLM-Image/processor/ — see test_ar_models.py:59-62. self.model = DEFAULT_AR_MODEL_NAME_FOR_TEST = "zai-org/GLM-Image" (test_utils.py:154).
• Server exits on start-up with (verbatim): huggingface_hub.errors.HFValidationError: Repo id must be in the form 'repo_name' or 'namespace/repo_name': 'zai-org/GLM-Image/vision_language_encoder/'. Use 'repo_type' argument if needed.
• Stack trace path: server_args.py __post_init__ → _handle_cuda_graph_config → _apply_cuda_graph_compatibility → get_model_config → ModelConfig.__init__ → get_config → config.py:84 parse → config.py:63 _try_load_longcat_config → PretrainedConfig.get_config_dict → transformers cached_files → validate_repo_id.
• _try_load_longcat_config (python/sglang/srt/utils/hf_transformers/config.py:62-72) calls PretrainedConfig.get_config_dict(model, …) on the raw model string before any longcat check, so any invalid model string raises here.
• Failure reproduces at each launch attempt (attempt 1, attempt 2) and on the pytest run-last-failure rerun; overall setUpClass fails after a 600 s health-check TimeoutError (test_utils.py:289, test_ar_models.py:79).
• Same test + same error verified in the prior completed scheduled run 29060215979 (Jul 10), head cfc66e05.
• aiter version is unchanged (Dockerfile default dev image) — not a variable here.
• This is pr-test-amd.yml (not amd-aiter-scout.yml); no baseline A/B check required.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [FACT] The literal string passed as --model-path (zai-org/GLM-Image/vision_language_encoder/) is not a valid Hugging Face repo id (three /-separated segments + trailing slash), so huggingface_hub.validate_repo_id rejects it. This is directly observable in the log.
• [MEDIUM] The GLM-Image model snapshot is not present as a local directory on the AMD MI325 runner, so transformers falls back to treating zai-org/GLM-Image/vision_language_encoder/ as a remote repo id and validation fails. The test's path construction (test_ar_models.py:60) only resolves when a local snapshot exists at <cache>/zai-org/GLM-Image/vision_language_encoder/; a real HF repo cannot be addressed by a subfolder-as-repo-id.
  • Supporting: error is a repo-id validation error, not a "file not found" or network error — meaning transformers never treated it as a local path; the model dir is absent. The test is brand-new ([#25381](https://github.com/sgl-project/sglang/pull/25381), Jul 09) and has never passed on this AMD job.
  • Disconfirming: I did not confirm the exact model-cache mount / download step for AMD CI, nor whether zai-org/GLM-Image is a private/gated repo requiring auth (which could also explain a missing local snapshot). NVIDIA behavior for this test was not checked, so I cannot say whether the model-path scheme works anywhere.
  • How to verify: on the AMD rocm/sgl-dev container, check whether zai-org/GLM-Image/vision_language_encoder/ resolves to an existing directory (e.g. ls $HF_HOME/... /zai-org/GLM-Image/); if absent, this is a model-availability/test-setup gap on AMD.
• [LOW] Production-code regression in config/model-path handling.
  • Supporting: none beyond the stack trace touching config.py.
  • Disconfirming: _try_load_longcat_config / config.py were not modified by the introducing PR; the error reproduces from the very first run of a newly-added test, so no production code changed under a previously-passing test. This is a test-introduction failure, not a regression of existing code.

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs for test_ar_models (0 results), vision_language_encoder (#20359, #30683 — both GLM-Image feature PRs, neither addresses the invalid-repo-id/model-path start-up failure), and GLM-Image AR (#30683 only). No PR fixes this test.

Suggested Triage Steps

• Confirm on an AMD MI325 rocm/sgl-dev:v0.5.15-rocm700-mi30x container whether the GLM-Image snapshot exists locally at the path the test builds (zai-org/GLM-Image/vision_language_encoder/). If it does not exist, the AR test needs the model pre-downloaded/mounted on AMD runners (same way other multimodal_gen models are staged).
• If the model is expected to be fetched from HF, the model-path scheme in test_ar_models.py:60 (f"{self.model}/vision_language_encoder/") is the problem — a subfolder cannot be a repo id; the test author (#25381) should resolve to a local snapshot dir (e.g. via snapshot_download) before appending the subcomponent.
• Since the test is Never-passed and newly added, consider disabling/skipping TestGLMImage on AMD (or gating it behind model availability) until the model staging is in place, so it stops masking the multimodal-gen-test-2-gpu-amd shard-2 signal.
• Ping the #25381 author to confirm whether this test was ever intended/validated to run in AMD scheduled CI.

Status

• Persistence: Failing since the test was introduced on 2026-07-09; every completed scheduled pr-test-amd run since (Jul 10, Jul 11) shows the identical error. Never passed.
• Scope: multimodal-gen-test-2-gpu-amd (linux-mi325-2gpu-sglang, 2) shard on pr-test-amd.yml. The whole shard errors because it collects only this one test.
• Blocked work: GLM-Image AR (separate-server) CI signal on AMD 2-GPU is red; the shard cannot pass until the model-path/availability issue is resolved.
• In-flight fix: ❌ none found — needs a new PR (test fix or AMD model staging) or a skip on AMD.

multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 1) — failed step(s): N/A

Commit Info

• sglang: 32cb89d4 — ci: prune uv cache in job teardown to bound its growth (#30813) (CI-only change, does not touch multimodal_gen)
• aiter: 9127c94a v0.1.17.dev110 (source: Dockerfile default, no override — read from sibling shard 3 log 86543869876 of the same run, since this job's own log is unavailable)

Failed Tests

Test File	Test Function	Error	Log
N/A — log unavailable (BlobNotFound)	N/A	Job terminated mid-test; specific case not recoverable	Log (404)

The job ran the 1-gpu diffusion suite, partition 1 of 4, -k "not flux_2" (T2V/T2I/I2V/LoRA cases across FLUX.1 / Qwen-Image / video models — pr-test-amd.yml:773). The exact failing case cannot be pinned because the job's log blob was never uploaded (see Facts).

Failure Summary

The job is marked failure, but its log is BlobNotFound and its step state is frozen: step 9 "Run diffusion server tests (1-GPU)" is still in_progress and steps 10/20 are pending. This is the signature of an abnormal termination — the runner was lost / became unresponsive mid-test and GitHub marked the job failed without finalizing logs. The diffusion test step had been running ~55 min (started 18:35:54, job ended 19:30:43) — under the 90 min step timeout — before termination.

Regression Status

Recurring / intermittent hang on this shard (not a clean assertion failure).

Recent history of multimodal-gen-test-1-gpu-amd shard 1 (same branch main, event=schedule, completed runs only). "STEP9-STUCK" = step 9 frozen in_progress + downstream pending + log BlobNotFound (identical signature to the current run):

Date (UTC)	Run	Job	Shard-1 Status	Signature
Jul 11 12:17	29152390308	shard 1	❌ Failed	STEP9-STUCK (this job)
Jul 10 00:32	29060215979	shard 1	❌ Failed	STEP9-STUCK
Jul 09 12:49	29019359571	shard 1	❌ Failed	STEP9-STUCK
Jul 09 00:31	28985427127	shard 1	✅ Passed	—
Jul 08 12:37	28943113233	shard 1	✅ Passed	—
Jul 08 00:37	28908777371	shard 1	❌ Failed	STEP9-STUCK

Failing pattern: intermittent, with the last 3 completed shard-1 occurrences (Jul 9 12:49, Jul 10 00:32) + current all showing the hang. Caveat: the suite is LPT-partitioned by estimated runtime (run_suite.py), so "shard 1" is not a fixed set of test files across runs — but the STEP9-STUCK hang signature is consistent. In the current run, sibling shard 0 (86543869849) shows the identical STEP9-STUCK signature; shard 2 passed; shard 3 failed differently (clean assertion failure in test_update_weights_from_disk.py).

Failure Cluster

Diffusion 1-GPU test step hang → runner lost → job log never uploaded (BlobNotFound, step 9 frozen in_progress)

Facts (verified observations, no interpretation)

• Log unavailable: GET /actions/jobs/86543869858/logs → 302 to Azure blob → BlobNotFound. Same for all historical STEP9-STUCK shard-1 jobs (86260258176, 86154130475, 85761538614) — all 404 BlobNotFound.
• Frozen step state (via jobs API): step 9 "Run diffusion server tests (1-GPU)" status=in_progress, steps 10 (Upload artifacts) and 20 (Post Checkout) pending, yet job.conclusion=failure.
• Timing: job started 2026-07-11T18:26:39Z, step 9 started 18:35:54Z, job completed 19:30:43Z → step 9 ran ~54m49s (job total 1:04:04). Step timeout is timeout-minutes: 90 — not reached.
• Contrast: normally-completing shard 3 same run ran step 9 in ~17 min and finished with a real pytest failure + exit code 1.
• HEAD commit 32cb89d4 modifies only CI teardown (uv cache prune), not multimodal_gen.
• Overall run still queued at snapshot (some stage-c 8-GPU jobs pending) — this per-job verdict is final regardless.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [LOW] The diffusion server (or a T2V/T2I/I2V/LoRA case) hung on the single MI325 GPU — e.g. server never became ready, an inference request never returned, or a GPU/kernel hang — leaving the runner unresponsive until GitHub reclaimed it, so logs were never flushed.
  • Supporting: step 9 ran ~55 min (vs ~17 min for a healthy shard) before abrupt termination; frozen in_progress + BlobNotFound is the classic lost-runner signature; the same shard shows this repeatedly.
  • Disconfirming: no log means the hang location is unverified; shard-1 also passed twice (Jul 8 12:37, Jul 9 00:31) with the same code, so it is not a deterministic failure. Could equally be a runner/hardware fault rather than a test bug.
  • How to verify: re-run multimodal-gen-test-1-gpu-amd shard 1 alone on an MI325 runner with per-case timeouts + py-spy dump on hang; check the runner host for a hung HIP process / GPU in a bad state.
• [LOW] Infrastructure: MI325 runner lost / VRAM not clear / HF model download stall during the long-running diffusion cases.
  • Supporting: AMD-only, single-GPU, long model-download-heavy step; intermittent pass/fail with unchanged code points to environment variance.
  • Disconfirming: sibling shard 0 hung in the same run on a different runner, which argues against a single bad host and toward a workload-triggered hang.
  • How to verify: correlate the two stuck jobs' runner hostnames; inspect runner-side kernel/dmesg + GPU state at ~19:30 UTC.

No hypothesis reaches MEDIUM: with the log missing, there is no code-level evidence. Root cause is undetermined pending a reproduced run that retains logs.

In-flight Fix Check

• In-flight fix: ❌ none found targeting the 1-GPU diffusion test hang. Searched open PRs for multimodal_gen diffusion (92 results — all feature/kernel work, e.g. #30017, #30103) and diffusion hang (6 results, none matching this symptom). No PR addresses the AMD shard-1 hang / lost-runner behavior.

Suggested Triage Steps

• Re-run multimodal-gen-test-1-gpu-amd shard 1 in isolation on an MI325 runner with a per-case wall-clock timeout so a hang produces a captured pytest timeout + log instead of a lost runner (this is the single most valuable step — it converts BlobNotFound into actionable evidence).
• Add a step-level py-spy dump / rocm-smi capture on timeout in the Run diffusion server tests (1-GPU) step so a future hang leaves a stack trace and GPU state even when the runner is later reclaimed.
• Compare the two same-run stuck jobs' runner hosts (shard 0, shard 1); if different hosts, favor the workload-hang hypothesis over a bad-runner hypothesis.
• Once a case is identified from a captured hang, bisect the multimodal_gen diffusion server / that model's forward path; the HEAD CI-only commit is not a candidate.

Status (factual, no priority assignment)

• Persistence: Intermittent on multimodal-gen-test-1-gpu-amd shard 1 with the hang signature — failed Jul 8 00:37, passed Jul 8 12:37 & Jul 9 00:31, then failed Jul 9 12:49, Jul 10 00:32, and this run (Jul 11 12:17). Last 3 completed occurrences + current all hung.
• Scope: In this run, shards 0 and 1 of multimodal-gen-test-1-gpu-amd both hung (STEP9-STUCK); shard 2 passed; shard 3 failed with an unrelated clean assertion failure (test_update_weights_from_disk.py). Confined to the AMD 1-GPU diffusion suite.
• Blocked work: Blocks reliable AMD 1-GPU diffusion (multimodal_gen) CI signal; no log capture on hang makes root-causing impossible today.
• In-flight fix: ❌ none found in open PRs.

multimodal-gen-test-2-gpu-amd (linux-mi325-2gpu-sglang, 0) — failed step(s): Run diffusion server tests (2-GPU)

Commit Info

• sglang: 32cb89d4 (#30813 "ci: prune uv cache in job teardown")
• aiter: 9127c94a — v0.1.17.dev110+g9127c94a1 (source: dev image; Dockerfile default AITER_COMMIT=9127c94a… matched, rebuild skipped)

Failed Tests

Job: multimodal-gen-test-2-gpu-amd (linux-mi325-2gpu-sglang, 0) — pytest result: 1 failed, 6 passed, 1 skipped, 17 errors (+ a rerun session: 18 errors).

Test File	Test Function	Error	Log
python/sglang/multimodal_gen/test/server/test_server_2_gpu.py	TestDiffusionServerTwoGpu::test_diffusion_generation[flux2_modelopt_fp8_tp2_t2i]	openai.InternalServerError: Internal Server Error — server-side RuntimeError: CUDA error: HIPBLAS_STATUS_NOT_SUPPORTED in torch._scaled_mm	Log
python/sglang/multimodal_gen/test/server/test_server_2_gpu.py	test_diffusion_generation[…] × 17 (cascade, see below)	ERROR at setup: RuntimeError: Scheduler port 5555 is unavailable and --strict-ports is enabled → Server exited early (code 1)	Log

The 17 cascaded ERRORs (all setup failures on port 5555): joy_echo_t2v_2gpu, wan2_1_t2v_1.3b_cfg_parallel, wan2_1_t2v_1_3b_cache_dit_sp_only_2gpu, fsdp-inference, mova_360p_tp2, mova_360p_ring1_uly2, ltx_2_two_stage_t2v, ltx_2_3_two_stage_ti2v_2gpus, wan2_1_i2v_14b_480P_2gpu, ltx_2.3_two_stage_t2v_2gpus, wan2_1_i2v_14b_lora_2gpu, wan2_1_i2v_14b_720P_2gpu, zimage_image_t2i_2_gpus, zimage_image_t2i_2_gpus_non_square, flux_image_t2i_2_gpus, flux_2_image_t2i_2_gpus, ltx_2.3_one_stage_ti2v.

Failure Summary

The FLUX.2 modelopt-fp8 case (flux2_modelopt_fp8_tp2_t2i) is the sole real test failure: its server's fp8 GEMM raised HIPBLAS_STATUS_NOT_SUPPORTED during warmup and generation, so the generate request returned HTTP 500. After that case, GPU memory was not released ([server-test] ROCm GPU memory not fully cleared after 60s (used: 19.17 GB)) and the server process kept scheduler port 5555 bound, causing the remaining 17 cases in the shard (and 18 in the rerun) to ERROR at setup with port 5555 is unavailable … --strict-ports. So this shard has two symptom clusters: one root fp8 failure and one downstream port-leak cascade.

Regression Status

Known recurring failure — no green run found for test_diffusion_generation[flux2_modelopt_fp8_tp2_t2i] in the queryable window (schedule, branch=main, completed runs Jul 8–11).

Recent history of test_server_2_gpu.py (test_diffusion_generation[flux2_modelopt_fp8_tp2_t2i]) in multimodal-gen-test-2-gpu-amd (test is dynamically sharded, so all 3 shards were checked per run; same branch, event=schedule, completed runs only):

Date	Run	Shard/Job	Test File Status	Failed Function	Error
Jul 11	29152390308	shard 0	❌ Failed	…[flux2_modelopt_fp8_tp2_t2i]	HIPBLAS_STATUS_NOT_SUPPORTED (warmup/gen)
Jul 9	29019359571	shard 1	❌ Failed	…[flux2_modelopt_fp8_tp2_t2i]	"FAILED during server startup" (~19 min startup hang, different sub-symptom)
Jul 10	29060215979	shards 0/1/2	⚠️ not observed	—	flux2 case did not appear in any shard log (run failed/collected differently); no HIPBLAS/port-5555 seen

The port 5555 cascade is itself recurring: Jul 9 shard 0 and shard 1 logs contain 38 and 46 port 5555 is unavailable occurrences respectively. First observed failure for this test in-window: Jul 9; no last-known-passing date found in the Jul 8–11 window.

Failure Cluster

Primary: "FP8 _scaled_mm HIPBLAS_STATUS_NOT_SUPPORTED on ROCm (flux2 modelopt-fp8)". Secondary (cascade): "Scheduler port 5555 leak → subsequent diffusion server setups ERROR".

Facts (verified observations)

• Server-side stack trace (verbatim, current run): apply_fp8_linear → python/sglang/srt/layers/quantization/fp8_utils.py:1891 output = torch._scaled_mm(qinput, weight, …) → RuntimeError: CUDA error: HIPBLAS_STATUS_NOT_SUPPORTED when calling HIPBLAS_STATUS_NOT_SUPPORTED. Reached via flux_2.py:759(attn) → flux_2.py:77 _get_qkv_projections → attn.to_q → linear.py:427 → modelopt_quant.py:432.
• Confirmed fp8_utils.py:1891 is the per-tensor fused torch._scaled_mm GEMM path (read from source at the analysed commit).
• Model under test: black-forest-labs/FLUX.2-dev with transformer=lmsys/flux2-dev-modelopt-fp8-sglang-transformer, tp_size=2, scheduler_port=5555, strict_ports=true, ROCm gfx942/mi30x, attention backend aiter.
• Warmup log: server warmup req (1024x1024, 2/5 steps) processing failed … HIPBLAS_STATUS_NOT_SUPPORTED; Synthetic server warmup failed; continuing startup — server stayed up (health 200 OK), so the client generate request then also hit the same error → HTTP 500.
• Cascade evidence: ROCm GPU memory not fully cleared after 60s (used: 19.17 GB). Proceeding anyway immediately precedes each subsequent port 5555 is unavailable setup ERROR.
• Reproducibility: 1 FAILED + 17 ERROR (first session) + 18 ERROR (rerun) in this one job; port-5555 cascade also present in Jul 9 shards.

Hypothesised Causes (with confidence)

• [FACT] The flux2 case fails because torch._scaled_mm (e4m3fn fp8 GEMM) is unsupported by hipBLAS on this ROCm/gfx942 stack — the error and code path are directly in the log/stack trace.
• [MEDIUM] The 17/18 downstream ERRORs are caused by the flux2 server not being torn down (GPU memory + port 5555 leak), not by their own model code.
  • Supporting: every cascaded ERROR is ERROR at setup with port 5555 unavailable; each is preceded by the "ROCm GPU memory not fully cleared" warning; the errors are independent of which model each case uses.
  • Disconfirming: could not confirm from log whether the leaked process is specifically the flux2 server vs a stale process from an earlier case; port-5555 cascade also appeared Jul 9 where the shard-1 root symptom was a startup hang rather than HIPBLAS, so the cascade may be triggered by any hard server failure, not uniquely by the fp8 error.
  • How to verify: run test_server_2_gpu.py with only flux2_modelopt_fp8_tp2_t2i excluded and see whether the rest pass; and inspect teardown/port-release logic in test_server_utils.py when a server exits non-cleanly.
• [LOW] aiter/triton version skew (AITER_USE_SYSTEM_TRITON=1: aiter gluon kernels require triton>=3.6.0, found 3.4.0) contributes.
  • Supporting: warning present at import.
  • Disconfirming: the failing frame is torch._scaled_mm (a torch/hipBLAS call), not an aiter gluon kernel; unrelated to the triton version.

In-flight Fix Check

• flux2 fp8 HIPBLAS cluster: In-flight fix: ✅ #28889 "[AMD] Fix FP8 e4m3fn GEMM on ROCm: dequant->bf16 fallback in apply_fp8_linear (fixes flux2 modelopt-fp8 HIPBLAS_STATUS_NOT_SUPPORTED)" — open (non-draft) since 2026-06-22, mergeable_state: blocked, last updated 2026-07-11. Directly targets this root cause; chase reviewers, do NOT open a duplicate. (Found via search/issues q=…HIPBLAS_STATUS_NOT_SUPPORTED and …flux2+fp8.)
• port 5555 cascade cluster: In-flight fix: ❌ none found matching "port 5555" / "strict-ports" / server teardown — likely needs a separate issue for test-harness cleanup.

Suggested Triage Steps

• Verify #28889 resolves the fp8 path: apply it on a branch and re-run only test_diffusion_generation[flux2_modelopt_fp8_tp2_t2i] on AMD MI325 (gfx942); confirm torch._scaled_mm no longer raises HIPBLAS_STATUS_NOT_SUPPORTED.
• Separately investigate the port-5555 leak in the test harness: inspect test_server_utils.py teardown (server kill + port release) so that one hard server failure does not cascade into ERRORing the rest of the shard — the "GPU memory not fully cleared after 60s" warning suggests the prior server process survives teardown.
• Since the cascade masks the real signal for 17 sibling cases, consider whether --strict-ports should fall back to an alternate scheduler port in CI, or whether teardown should hard-kill by port owner, so a single failure doesn't poison the shard.
• If #28889 lands, re-check whether the flux2 case then passes and whether the cascade disappears; if the cascade persists behind other failures (as on Jul 9), the harness cleanup fix is still needed independently.

Status

• Persistence: test_diffusion_generation[flux2_modelopt_fp8_tp2_t2i] failing on every completed run where it was collected in the Jul 8–11 window (Jul 9 and Jul 11); no green run found. Port-5555 cascade recurring across runs.
• Scope: multimodal-gen-test-2-gpu-amd shard 0 this run (1 root failure + 17 cascade ERRORs + 18 on rerun); cascade also observed in Jul 9 shards 0 and 1. AMD/ROCm-specific (fp8 hipBLAS path).
• Blocked work: All 2-GPU diffusion-server generation cases in the affected shard are blocked once flux2 fails, since the port leak ERRORs every subsequent case.
• In-flight fix: ✅ #28889 (open since 2026-06-22, mergeable_state blocked) for the fp8 root cause; ❌ none for the port-5555 cascade.

---

Generated by amd-bot using Claude Code CLI (last updated: 2026-07-12 18:23 UTC)

[amd-bot]

pr-test-amd.yml — Per-Job Analysis (continued 2/9)

multimodal-gen-test-2-gpu-amd (linux-mi325-2gpu-sglang, 1) — failed step(s): Run diffusion server tests (2-GPU)

Commit Info

• sglang: 32cb89d4
• aiter: 9127c94a (v0.1.17.dev110+g9127c94a1, source: Dockerfile default — AITER_COMMIT_OVERRIDE empty, not a scout run)

Failed Tests

Test File	Test Function	Error	Log
python/sglang/multimodal_gen/test/single_test_file/test_disagg_server.py	TestDisaggZImage1Rank::test_generates_image	RuntimeError: Warmup request failed for zimage_1rank: 500 … DiffusionServer timeout: request not completed within 120.0s	Log
.../test_disagg_server.py	TestDisaggZImage2RankDenoiser::test_generates_image_with_sp2_denoiser	RuntimeError: Warmup request failed for zimage_sp2: 500 … DiffusionServer timeout … 120.0s	Log
.../test_disagg_server.py	TestDisaggZImageTracing::test_disagg_spans_share_trace_id	RuntimeError: decoder failed to start for zimage_trace … zmq.error.ZMQError: Address already in use (addr='tcp://0.0.0.0:38809')	Log

Failure Summary

The 2-gpu partition 2/3 (standalone) ran test_disagg_server.py and all 3 disaggregated Z-Image test classes errored at setUpClass. In two cases the warmup image-generation request returned HTTP 500 because the denoiser never completed the request within the 120 s DiffusionServer timeout. In the third (Tracing), the decoder process crashed at startup failing to bind its pool_work_endpoint ZMQ socket (Address already in use on port 38809 after 5 same-port retries). Total suite runtime was 2515 s (0:41:55).

Regression Status

New regression that began ~2026-07-08, now recurring in every completed run since.

Recent history of test_disagg_server.py (all 3 Z-Image disagg classes) in the multimodal-gen-test-2-gpu-amd standalone partition (branch main, event=schedule, completed runs only; shard index varies per run because partitioning is reshuffled):

Date (UTC)	Run	Job	Test File Status	Failed Functions	Error
Jul 11	29152390308	2gpu shard 1	❌ 3 errors	1Rank / sp2 / Tracing	warmup timeout + decoder ZMQ bind
Jul 10	29060215979	2gpu shard 1	❌ 3 errors	1Rank / sp2 / Tracing	same
Jul 09	29019359571	2gpu shard 0	❌ 3 errors	1Rank / sp2 / Tracing	same
Jul 08 12:37	28943113233	2gpu shard 2	❌ 3 errors (first red)	1Rank / sp2 / Tracing	same
Jul 08 00:37	28908777371	2gpu shard	✅ 3 passed (last green)	—	—
Jul 07 12:52	28867568883	2gpu shard	✅ 3 passed	—	—

First observed failure: run 28943113233 (created 2026-07-08 12:37 UTC, head 108a183f6b). Last known pass: run 28908777371 (created 2026-07-08 00:37 UTC, head b363249423). Lookback window searched: Jul 07–11 scheduled runs.

Failure Cluster

DiffusionServer generation timeout during Z-Image disagg warmup (denoiser never completes within 120 s; Tracing case cascades to a decoder ZMQ EADDRINUSE).

Facts (verified observations)

• All three failures are at setUpClass/warmup, not in assertions. The Z-Image model loads successfully in every case (Loaded vae, Loaded scheduler, Pipeline instantiated, Worker 0: Initialized device, model, and distributed environment) — so this is not a model-load/registry failure.
• Server-side error (verbatim): DiffusionServer timeout: request <id> not completed within 120.0s → RuntimeError: Model generation returned no output at entrypoints/openai/utils.py:355.
• Tracing decoder crash (verbatim): Failed to bind ZMQ socket after 5 attempts. Original endpoint: tcp://0.0.0.0:38809 → zmq.error.ZMQError: Address already in use at disaggregation/scheduler_mixin.py:410 → utils/common.py:291.
• pool_work_endpoint is derived as tcp://0.0.0.0:<scheduler_port> (server_args/disagg.py:62), and the test assigns a distinct find_free_port per role (test_disagg_server.py:119-125). In the healthy denoiser its port was 44847 — roles do not collide by design, so the 38809 EADDRINUSE is consistent with an orphaned process from the two preceding failed test classes still holding the port (cleanup cascade), not a port-derivation bug.
• Config facts (from server_args in log): attention_backend=aiter, dit_cpu_offload=true, enable_breakable_cuda_graph=false, warmup_mode=request, warmup_steps=1.
• Base docker image: green runs used rocm/sgl-dev:v0.5.14-rocm700-mi30x; today's red uses v0.5.15. However the first red run [28943113233] pulled v0.5.14-rocm700-mi30x-20260709 — the same v0.5.14 family as the greens — so the image bump is NOT the trigger.
• Reproducibility: all 2-gpu shards fail in every completed run since Jul 08; the 1-gpu multimodal-gen jobs also fail this run (broad multimodal_gen breakage).

Hypothesised Causes (with confidence + disconfirming evidence)

Regression window (verified via GitHub compare API) is 27 commits: b363249423...108a183f6b. The timeout produces no stack trace of the hung frame, so no candidate reaches MEDIUM. Candidates, restricted to commits that touch the exercised Z-Image denoiser/warmup path:

• [LOW] fa185ed84d — #29742 "[diffusion] fix: fix z-Image accuracy"
  • Supporting: rewrites runtime/models/dits/zimage.py (+517/-54) and runtime/layers/attention/layer.py — the exact failing model (Z-Image-Turbo), and this path definitely executes (BCG is off). A numeric/perf regression here could push generation past the 120 s warmup timeout.
  • Disconfirming: "accuracy fix" is about numerics, not an obvious hang; no log evidence pinpoints the denoiser forward as the hung frame.
  • How to verify: bisect; if implicated, revert on a branch and re-run test_disagg_server.py on AMD MI325 2-GPU.
• [LOW] 33c3dfd7e0 — #27436 "[diffusion] Enable breakable CUDA graph (BCG) for diffusion DiTs"
  • Supporting: modifies pipelines_core/stages/denoising.py (+132/-9), dits/zimage.py (+108/-6), warmup_request_builder.py, attention/layer.py in-place — all on the warmup/denoise path.
  • Disconfirming (strong): the headline feature is disabled in the failing run (enable_breakable_cuda_graph=false), so the new CUDA-graph capture/runner code does not run. Only the in-place edits to denoising/zimage could matter.
  • How to verify: bisect; check whether the in-place denoising.py/zimage.py edits change behavior with BCG off.
• [LOW] 68901ba387 — #29777 "[diffusion] Support SP for Krea-2"
  • Supporting: edits runtime/layers/attention/layer.py (shared by the denoiser); the sp2 test exercises sequence-parallel denoising.
  • Disconfirming: primarily Krea-2 specific; the 1-rank (non-SP) test also times out, which SP changes wouldn't explain.
  • How to verify: bisect.
• Ruled out: 3d96bb97 (#30518) is a 1-line registry.py rename + docs, and the model loads fine. Docker image bump (v0.5.14→v0.5.15) ruled out (first red ran on v0.5.14).

No hypothesis above LOW confidence; needs git bisect across b363249423..108a183f6b.

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs for disagg+diffusion and multimodal_gen+disagg; results are feature PRs (#24200, #23078, #24347), not a fix for this warmup timeout. No revert/fix PR for the Z-Image disagg regression exists.

Suggested Triage Steps

• Bisect the 27-commit window b363249423..108a183f6b by running test_disagg_server.py on an AMD MI325 2-GPU runner; prioritize the diffusion commits fa185ed84d (#29742), 33c3dfd7e0 (#27436), 68901ba387 (#29777).
• Since the failure is a silent 120 s timeout, capture the denoiser server log during a repro (or attach py-spy to the sglang-denoiser process) to identify the hung frame in the denoising forward pass — this will turn the LOW hypotheses into an actionable pin.
• Verify the port cascade: confirm whether the two failed warmup tests leave orphan role processes holding pool_work_endpoint ports (which would explain the Tracing decoder EADDRINUSE); if so, the decoder crash is a symptom, not an independent bug, and will clear once the warmup timeout is fixed.
• If bisect points to fa185ed84d, revert on a branch and re-run; if generation completes under 120 s, that confirms a Z-Image perf/hang regression on AMD.

Status

• Persistence: Failing since 2026-07-08 12:37 UTC (first red run 28943113233); every completed scheduled run since is red (~4 days).
• Scope: All 2-gpu multimodal-gen-test-2-gpu-amd shards this run, plus the 1-gpu multimodal-gen shards — broad Z-Image/diffusion CI breakage on AMD.
• Blocked work: Disaggregated diffusion (Z-Image) CI signal on AMD MI325.
• In-flight fix: ❌ none found — needs new PR or investigation.

multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 3) — failed step(s): Run diffusion server tests (1-GPU)

Commit Info

• sglang: 32cb89d4
• aiter: 9127c94a — v0.1.17.dev110+g9127c94a1 (source: dev image preserved; [CI-AITER-CHECK] Dev/patched version detected → skipping rebuild, log line 4335). Not attackable here — this is a Python/torch-semantics failure, not a kernel.

Failed Tests

Job: multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 3) · test file sglang/multimodal_gen/test/single_test_file/test_update_weights_from_disk.py

Test Function	Error	Log
TestUpdateWeightsFromDisk::test_update_weights_from_disk_default[Qwen-Image]	assert 500 == 200 (server returned 500)	Log
TestUpdateWeightsFromDisk::test_update_weights_specific_modules[Qwen-Image]	AssertionError: Update failed: … 'Inplace update to inference tensor outside InferenceMode is not allowed.'	Log
TestUpdateWeightsFromDisk::test_corrupted_weights_rollback[Qwen-Image]	assert (500 == 200)	Log
TestUpdateWeightsFromDiskWithOffload::test_update_weights_with_offload_enabled[FLUX.2-klein-base-4B]	AssertionError: Expected 200, got 500	Log
TestUpdateWeightsFromDiskWithOffload::test_update_weights_with_offload_enabled[Qwen-Image]	AssertionError: Expected 200, got 500	Log
TestUpdateWeightsFromDisk::{default, specific_modules, rejects_invalid_requests, corrupted_weights_rollback}[FLUX.2-klein-base-4B] (4 × setup ERROR)	AssertionError: No weights dir for transformer in /tmp/sglang_perturbed_vae_* (fixture diffusion_server_no_offload, test file line 410 → _compute_checksum_from_disk line 220)	Log

Result line: 5 failed, 1 passed, 11 warnings, 4 errors in 969.28s (job log line 13025), then ##[error]Process completed with exit code 1 (line 13050).

Failure Summary

The multimodal_gen update_weights_from_disk feature is broken on this AMD job. Server-side, every weight-update request returns HTTP 500 because param_data.copy_(loaded_weight) in multimodal_gen/runtime/layers/linear.py (lines 412 and 575) raises RuntimeError: Inplace update to inference tensor outside InferenceMode is not allowed. A separate AssertionError: param_data.shape == loaded_weight.shape (linear.py:411) also fires on some modules and, on the resulting rollback path, hits the same inplace-copy error. The 4 [FLUX.2-klein-base-4B] cases are fixture setup ERRORs (perturbed-model dir has no transformer/ subdir), a distinct test-harness precondition failure in the same file.

Regression Status

Known recurring failure — no green run found in the queryable window (persistent ≥4 days).

Recent history of test_update_weights_from_disk.py in job multimodal-gen-test-1-gpu-amd (…, 3) (branch main, event=schedule, completed runs only):

Date (UTC)	Run	Test File Status	Signature
Jul 11	29152390308	❌ Failed	5 failed, 1 passed, 4 errors; 18× inplace-tensor RuntimeError
Jul 10	29060215979	❌ Failed	identical (5 failed / 1 passed / 4 errors; 18× inplace)
Jul 09	29019359571	❌ Failed	identical
Jul 09	28985427127	❌ Failed	identical (18× inplace)
Jul 08	28943113233	❌ Failed	identical
Jul 08	28923624498	❌ Failed	identical

First observed failure: at/before the earliest queryable run (Jul 08 06:53 UTC). No passing run for this test file exists in the window searched (Jul 08–11, 6 completed runs). Note: the parent run 29152390308 shows workflow-level queued (sibling jobs still pending), but this job itself completed with failure.

Failure Cluster

"Inplace update to inference tensor during multimodal_gen weight update"

Facts (verified observations)

• RuntimeError: Inplace update to inference tensor outside InferenceMode is not allowed occurs 18× in the current log; server-side stack (log lines 5978–6018): weights_updater.py:200 load_weights_into_model → linear.py:533 → linear.py:575 param_data.copy_(loaded_weight). [FACT]
• A shape-mismatch AssertionError at linear.py:411 (assert param_data.shape == loaded_weight.shape) occurs 14× and is the initial failure that triggers _apply_weights → _rollback (weights_updater.py:350→381); the rollback then re-enters load_weights_into_model and hits the same inplace-copy RuntimeError. [FACT]
• POST /update_weights_from_disk HTTP/1.1 500 Internal Server Error logged for every attempt (e.g. lines 6021, 7214). [FACT]
• 4 [FLUX.2-klein-base-4B] cases are pytest setup ERRORs, not the 500 path: fixture diffusion_server_no_offload (test line 410) → _compute_checksum_from_disk (line 220) → AssertionError: No weights dir for transformer in /tmp/sglang_perturbed_vae_*. A related ValueError: Could not find model at /sgl-data/hf-cache/hub/models--black-forest-labs--FLUX.2-klein-base-4B/... appears in _rollback → maybe_download_model (hf_diffusers_utils.py:874, log line 12825). [FACT]
• Identical failure signature (5 failed / 1 passed / 4 errors; 18× inplace RuntimeError) in all 6 completed scheduled runs Jul 08–11. [FACT]
• weight_loader code path (linear.py:200 load_weights_into_model → weight_loader(param, loaded_weight.to(param.dtype))) calls param_data.copy_() with no torch.inference_mode()/no_grad guard (source verified). [FACT]

Hypothesised Causes (with confidence)

• [MEDIUM] The pipeline's parameters are inference tensors (model loaded/run under torch.inference_mode()), and multimodal_gen/runtime/post_training/weights_updater.py::load_weights_into_model performs in-place param_data.copy_() (via linear.py weight_loader lines 412/575) without an inference-mode/_unsafe_preserve_version_counter/clone guard, so any weight update or rollback throws RuntimeError and returns 500.
  • Supporting: the RuntimeError message itself names the exact cause; the failing frame is param_data.copy_(loaded_weight); the copy path in load_weights_into_model has no inference-mode guard (source read). The feature is fundamentally unable to complete any update, matching the 100%-fail, never-green history.
  • Disconfirming: I did not confirm from the log that the model is actually loaded under inference_mode (inferred from the error semantics, not directly observed); the separate linear.py:411 shape-mismatch AssertionError is an independent trigger that would fail even if the inplace-copy issue were fixed, so this hypothesis does not fully explain the Qwen-Image default/corrupted_weights_rollback shape-mismatch path.
  • How to verify: run test_update_weights_from_disk.py::TestUpdateWeightsFromDisk::test_update_weights_specific_modules[Qwen-Image] locally on AMD; wrap the param_data.copy_() calls in with torch.inference_mode(): (or clone the param before copy) on a branch and re-run to confirm the 500 clears.
• [LOW] The linear.py:411 shape mismatch is a second, independent bug (loaded shard shape ≠ param shape for certain Qwen-Image/FLUX modules), separate from the inference-tensor issue.
  • Supporting: 14× distinct AssertionError at line 411 before any copy.
  • Disconfirming: could be a downstream symptom of param-name mapping selecting the wrong param; not root-caused.
  • How to verify: log name, param_data.shape, loaded_weight.shape at linear.py:411 for the first failing module.
• [LOW] The 4 FLUX.2 setup ERRORs are a distinct test-fixture/data problem (perturbed model dir missing transformer/; FLUX.2 model path resolution), not the server 500.
  • Supporting: they fail in fixture setup, never reaching the server request; note CI comment "All 1-GPU tests except FLUX.2 (FLUX.1 covers same code path)" (log line 4454) suggests FLUX.2 handling is already special-cased.
  • Disconfirming: none found — these are clearly pre-request setup failures.
  • How to verify: inspect _perturb/checksum fixture for FLUX.2 module layout.

In-flight Fix Check

• In-flight fix: ❌ none found addressing the inplace-tensor weight-update bug in multimodal_gen. Searched open PRs for update_weights_from_disk, weights_updater, InferenceMode/inference tensor. Closest matches are unrelated: #30761/#30749 ("[RL] update_weights_from_disk: load quantized checkpoints") target the RL engine, not the diffusion multimodal_gen path; #30759 ("dspark target-verify buffers … inference-tensor conflict") is speculative-decoding, not this code. Needs a new PR/investigation.

Suggested Triage Steps

• Reproduce on AMD MI325 in the diffusion CI container: pytest sglang/multimodal_gen/test/single_test_file/test_update_weights_from_disk.py::TestUpdateWeightsFromDisk::test_update_weights_specific_modules -k Qwen-Image.
• On a branch, guard the in-place copy in multimodal_gen/runtime/post_training/weights_updater.py::load_weights_into_model (and the linear.py weight_loader copy_ at 412/575) with torch.inference_mode() (or clone the param to a normal tensor before copy); re-run the test to confirm the 500 clears.
• Separately instrument linear.py:411 to dump name / param_data.shape / loaded_weight.shape and identify which Qwen-Image/FLUX modules shape-mismatch.
• File the 4 FLUX.2 fixture setup ERRORs as a distinct issue (perturbed-model dir missing transformer/, FLUX.2 model-path resolution) — do not conflate with the server-500 bug.

Status

• Persistence: Failing every completed scheduled run since at least Jul 08 06:53 UTC (6/6 completed runs Jul 08–11), identical signature — persistent ≥4 days, no green run in the queryable window.
• Scope: Confirmed on pr-test-amd.yml job multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 3). Blocks the entire update_weights_from_disk diffusion post-training test suite (5 test failures + 4 fixture errors; only 1 test passes). NVIDIA/other-backend scope not checked in this analysis.
• In-flight fix: ❌ none found — needs new PR or investigation.

stage-b-test-large-8-gpu-mi35x-disaggregation-amd (linux-mi35x-gpu-8.fabric) — failed step(s): Run test_disaggregation

Commit Info

• sglang: 32cb89d4
• aiter: 9127c94a (v0.1.17.dev110+g9127c94a1, source: dev image preserved; matches Dockerfile default)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/disaggregation/test_disaggregation_basic.py	TestDisaggregationAccuracy.test_structured_output	HW Exception … reason :GPU Hang → Fatal Python error: Aborted (exit 134 / SIGABRT)	Log

Failure Summary

The Run test_disaggregation step ran test_disaggregation_basic.py against Qwen/Qwen3-8B in prefill+decode disaggregation (aiter attention backend, mooncake transfer) on the MI35x (gfx950) 8-GPU runner. test_first_token_finish, test_gsm8k, and test_logprob (accuracy 0.955) passed. During test_structured_output the runtime emitted HW Exception by GPU node-3 … reason :GPU Hang on all 8 GPU agents, followed by Fatal Python error: Aborted. The process aborted with exit 134; a single automatic retry also aborted with exit 134, and the step reported Process completed with exit code 134.

Regression Status

Known recurring / flaky failure (intermittent GPU Hang). Pass and fail runs interleave across adjacent, overlapping head SHAs — not a clean deterministic regression.

Recent history of test_disaggregation_basic.py (GPU-Hang abort) in job stage-b-test-large-8-gpu-mi35x-disaggregation-amd (branch main, event schedule, completed runs only):

Date (UTC)	Run	Test File Status	Aborting Function	Error
Jul 11 12:17	29152390308 (current)	❌ Aborted (GPU Hang)	test_structured_output	GPU Hang → exit 134
Jul 11 00:26	29132618067	❌ Aborted (GPU Hang)	later method (TestDisaggregationMooncakeFailure.test_gsm8k)	GPU Hang → exit 134
Jul 10 12:44	29093581297	✅ Passed (basic.py)	—	job failed later on a different file (exit 255)
Jul 10 00:32	29060215979	❌ Aborted (GPU Hang)	pre-method (crashed before method log)	GPU Hang → exit 134
Jul 9 12:49	29019359571	✅ Passed (basic.py)	—	job exit 255 (different file)
Jul 9 00:31	28985427127	✅ Passed (job success)	—	—
Jul 8 12:37	28943113233	✅ Passed (basic.py)	—	job exit 255 (different file)

First GPU-Hang occurrence in the lookback window: Jul 10 00:32. test_structured_output last passed Jul 10 12:44 (run 29093581297) and Jul 9 00:31 (full green run 28985427127). The job's separate exit-255 failures (Jul 8–10) are a distinct failure mode on other disaggregation test files and are out of scope here.

Failure Cluster

GPU Hang HW exception → process abort (exit 134) during PD-disaggregation on MI35x runner.

Facts (verified observations)

• Log (verbatim, current run): 8 lines HW Exception by GPU node-3 (Agent handle: 0x…) reason :GPU Hang, immediately followed by Fatal Python error: Aborted. The last preceding health probe GET /health → 200 succeeded ~1s earlier.
• Last test method logged before the hang: [CI Test Method] TestDisaggregationAccuracy.test_structured_output. Prior methods (test_first_token_finish, test_gsm8k, test_logprob) all passed; test_logprob reported Accuracy: 0.955.
• Exit sequence: First attempt failed with exit code 134 → retry → ##[error]Process completed with exit code 134.
• Reproducibility: reason :GPU Hang + exit 134 present in 3 of the last 7 completed runs of this job (29152390308, 29132618067, 29060215979); absent in the other 4.
• Environment: Qwen/Qwen3-8B, --disaggregation-mode prefill/decode, --attention-backend aiter, --disaggregation-transfer-backend mooncake, --disaggregation-ib-device rdma0, runner linux-mi35x-gpu-8.fabric (gfx950); aiter v0.1.17.dev110+g9127c94a1.
• The faulthandler dump shows normal idle threads (zmq recv, watchdog, all_reduce in sampler); no Python-level exception preceded the HW GPU Hang — the abort was signalled by the ROCm runtime, not Python.

Hypothesised Causes (with confidence)

• [LOW] MI35x (gfx950) disaggregation runner GPU/driver-level instability (hardware GPU Hang), not a code regression.
  • Supporting: HW-level HW Exception … reason :GPU Hang on all 8 agents (runtime-signalled, no Python exception); intermittent across runs with overlapping/adjacent head SHAs; two open PRs actively debug this exact runner class (see In-flight Fix Check).
  • Disconfirming: could not fully rule out a workload trigger — the hang consistently appears late in test_disaggregation_basic.py under sustained decode load; a genuinely random hardware fault would be expected to land in earlier test files too.
  • How to verify: re-run this job on the same runner ×3–5 with no code change; if hang recurs on unrelated SHAs, hardware/driver is confirmed. Cross-check dmesg/GPU reset logs on linux-mi35x-gpu-8.fabric.
• [LOW] Workload interaction in the aiter attention / grammar-constrained (test_structured_output) path under PD-disaggregation on gfx950.
  • Supporting: current-run hang occurred within test_structured_output after test_logprob passed.
  • Disconfirming: in run 29132618067 test_structured_output passed and the GPU Hang hit a later TestDisaggregationMooncakeFailure method — so the hang is not specific to test_structured_output, weakening any structured-output-specific cause.
  • How to verify: run test_disaggregation_basic.py in isolation on MI35x looping the accuracy suite; watch which method (if any) reliably precedes the hang.

No hypothesis reaches MEDIUM: pass/fail runs interleave across overlapping SHAs, so a git-bisect regression window cannot be cleanly derived.

In-flight Fix Check

• In-flight fix: ⚠️ partial — no dedicated fix for a GPU-Hang-during-structured_output, but two open PRs target this MI35x disaggregation-runner instability class:
  • #30431 [AMD] ci: self-heal driver-level VRAM leaks + graceful GPU cleanup on shutdown (open since 2026-07-07) — GPU cleanup/self-heal on AMD CI.
  • #29127 [DEBUG] Isolate faulty RDMA NIC (rdma5) on MI35x disaggregation runner (open since 2026-06-24) — active hardware isolation on this exact runner.
  • Queries run: search/issues q=repo:sgl-project/sglang is:pr is:open GPU Hang disaggregation / mi35x disaggregation / disaggregation structured output.

Suggested Triage Steps

• Re-run this job ×3–5 on linux-mi35x-gpu-8.fabric at unrelated head SHAs to confirm intermittency and rule in/out hardware (GPU Hang is runtime-signalled, so treat hardware/driver as the leading candidate).
• Check runner-side dmesg / ROCm GPU-reset / RDMA NIC logs around 2026-07-11 13:23:46 UTC for the node-3 hang and correlate with the faulty-NIC investigation in #29127.
• Track #30431 (GPU cleanup/self-heal) — verify whether it reduces GPU-Hang recurrence on this runner after merge; chase reviewers rather than opening a duplicate.
• If hardware is exonerated, loop test_disaggregation_basic.py in isolation on MI35x to find whether a specific decode-load method reliably precedes the hang before attributing to any sglang/aiter code path.

Status

• Persistence: Intermittent GPU Hang → exit 134 in 3 of the last 7 completed runs (first seen 2026-07-10 00:32 UTC; recurred 07-11 00:26 and 07-11 12:17). test_structured_output last passed 2026-07-10 12:44 UTC.
• Scope: stage-b-test-large-8-gpu-mi35x-disaggregation-amd on the linux-mi35x-gpu-8.fabric runner; blocks reliable PD-disaggregation CI signal on MI35x.
• In-flight fix: ⚠️ partial — #30431 (GPU cleanup) and #29127 (MI35x runner NIC isolation) address the runner-instability class; no test-specific fix.

stage-b-test-1-gpu-small-amd (linux-mi325-1gpu-sglang, 10) — failed step(s): Start CI container

Commit Info

• sglang: 32cb89d4 — ci: prune uv cache in job teardown to bound its growth (#30813) (verified in local repo)
• aiter: N/A — the Install dependencies step was skipped and the log blob is missing, so no [CI-AITER-CHECK] markers are available.

Failed Tests

Test File	Test Function	Error	Log
N/A	N/A	Infrastructure failure — Start CI container step cancelled; no tests executed	Job

This is not a test failure. The job never reached any test.

Failure Summary

The job stage-b-test-1-gpu-small-amd (linux-mi325-1gpu-sglang, 10) failed because its Start CI container step was cancelled at 20:05:39 UTC (after starting at 20:00:04 UTC, ~5.5 min). As a direct consequence, Install dependencies and Run test were both skipped, so no test code ran. The job's log blob returns BlobNotFound, consistent with a job that was terminated during container startup before any log was uploaded.

Regression Status

Infrastructure issue (isolated, not recurring for this shard).

Recent history of the Start CI container step for job stage-b-test-1-gpu-small-amd (linux-mi325-1gpu-sglang, 10) (branch main, event schedule, completed runs only):

Date	Run	Job	Container Step	Overall Job	Note
Jul 11	29152390308	shard 10	❌ Cancelled	❌ Failure	this run
Jul 10	29060215979	shard 10	✅ Success	✅ Success	container OK
Jul 9	29019359571	shard 10	✅ Success	❌ Failure	failed at Run test (different, actual test failure — not this container issue)
Jul 9	28985427127	shard 10	✅ Success	✅ Success	container OK

The Start CI container step succeeded in all three prior completed runs, so the container-start cancellation is a one-off, not a persistent regression for this shard.

Failure Cluster

Container-start cancelled on AMD MI325 runner (no tests executed)

Facts (verified observations, no interpretation)

• Step timeline (from GitHub API job steps): Start CI container — start 2026-07-11T20:00:04Z, end 2026-07-11T20:05:39Z, conclusion cancelled. Install dependencies — skipped. Run test — skipped. Post Checkout code / Complete job — success.
• The job log blob is unavailable: fetching the job logs returns <Error><Code>BlobNotFound</Code> (no log was uploaded).
• Runner: linux-mi325-1gpu-sglang-57dnj-runner-9js5q, label linux-mi325-1gpu-sglang, group Default.
• The failure is isolated to this job among sibling jobs in the run: of 51 jobs, 32 succeeded, 3 skipped, 12 failed, 4 still queued. This is the only failed job whose bad step is Start CI container; the other 11 failures are genuine test-step failures (e.g. Run test, Run diffusion server tests, Run test_disaggregation). This rules out a global workflow cancellation.
• Run 29152390308 is still overall status=queued (4 jobs queued) → [IN-FLIGHT, partial data] at snapshot time; this specific job is completed / failure.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [MEDIUM] Runner pod for this shard was preempted / lost / evicted during container startup (autoscaled MI325 runner terminated mid-Start CI container).
  • Supporting: the step ran 5.5 min then flipped to cancelled (not failure); log blob was never uploaded (BlobNotFound) — the classic signature of a runner killed before it could flush logs; only this one job is affected while its siblings on the same label completed normally.
  • Disconfirming: no runner-level log is available to directly confirm eviction; a per-step timeout on image pull could produce a similar cancelled state.
  • How to verify: check the AMD self-hosted runner controller / k8s events for pod linux-mi325-1gpu-sglang-57dnj-runner-9js5q around 20:00–20:06 UTC on 2026-07-11 for an eviction/OOM/termination event.
• [LOW] Container image pull/start exceeded a timeout on this runner.
  • Supporting: 5.5 min elapsed before the step ended.
  • Disconfirming: the same image starts in seconds-to-minutes on the same runner label in adjacent runs; a hard timeout usually reports failure, not cancelled.
  • How to verify: not actionable without runner-side logs; secondary to the eviction hypothesis.
• [SPECULATION] Related to sglang HEAD commit 32cb89d4.
  • Supporting: none.
  • Disconfirming: the commit only prunes the uv cache in job teardown (CI hygiene) and does not touch container startup; the failure is at container start, before any repo code runs; 31 sibling jobs at the same SHA started their containers fine. This is not a code-level failure.

In-flight Fix Check

Not applicable — this is a runner/container-infra cancellation, not a code defect, and it is not recurring. No fix PR is warranted.

• In-flight fix: ❌ none needed — transient infra (single occurrence, container-start cancelled).

Suggested Triage Steps

• Re-run the failed job (shard 10) — a clean re-run is the fastest confirmation that this was transient infra.
• Check the AMD self-hosted runner controller / k8s events for runner linux-mi325-1gpu-sglang-57dnj-runner-9js5q between 20:00–20:06 UTC (2026-07-11) for pod eviction / termination.
• If shard-10 container-start cancellations recur across future runs, escalate to AMD CI infra to investigate MI325 runner pool stability / autoscaling preemption.
• No sglang code investigation is warranted — no test executed and the container never came up.

Status (factual, no priority assignment)

• Persistence: Single occurrence. Start CI container for this shard succeeded in the 3 prior completed scheduled runs (Jul 9–10).
• Scope: 1 job in run 29152390308. It is the only failure in that run attributable to container-start cancellation (the other 11 failures are unrelated genuine test-step failures).
• Blocked work: The linux-mi325-1gpu-sglang shard-10 test subset produced no signal for this run; a re-run is needed to obtain coverage. Note the run is still [IN-FLIGHT] (4 jobs queued) at snapshot time.
• In-flight fix: ❌ none needed — transient infra.

stage-b-test-2-gpu-large-amd (linux-mi325-2gpu-sglang, 0) — failed step(s): Run test

Commit Info

• sglang: 32cb89d4 (ci: prune uv cache in job teardown #30813)
• aiter: 9127c94a (source: Dockerfile default — AITER_COMMIT_OVERRIDE empty)

Failed Tests

Test File	Test Function	Error	Log
test/registered/rl/test_update_weights_from_distributed.py	TestUpdateWeightsFromDistributed.test_update_weights_from_distributed	Hang → ##[error]The action 'Run test' has timed out after 45 minutes (stuck at custom NCCL process-group rendezvous)	Log

Failure Summary

The job did not fail on an assertion — the Run test step hit the 45-minute GitHub Action timeout (line ##[error]…timed out after 45 minutes). The suite stage-b-test-2-gpu-large-amd (partition 0/2) ran 5 files; test_bench_serving_2gpu.py (1255 s) and test_load_snapshot_server.py (234 s) passed, then test_update_weights_from_distributed.py hung at custom process-group initialization. The last log line at 21:24:53 is rank 1 (SGL server) printing init custom process group: master_address=localhost, master_port=60000, rank_offset=1, rank=1, world_size=2, group_name=test_parameter_update_group, backend=nccl, followed by ~19 min of silence until the action-level timeout at 21:44:04.

Regression Status

Flaky test — intermittent distributed-rendezvous hang. On healthy runs this test completes in ~90 s; when it fails, one of the two ranks never joins the TCP rendezvous store, and the other blocks (either producing DistStoreError: Timed out … 1/2 clients joined after 1801 s, or — as here — being cut off first by the 45-min action timeout).

Recent history of test/registered/rl/test_update_weights_from_distributed.py in job stage-b-test-2-gpu-large-amd (…2gpu…, 0) (branch main, event=schedule, partition 0, completed runs only):

Date (UTC)	Run	Test File Status	Failed Function	Error
Jul 11 12:17	29152390308	❌ Hung (action timeout)	test_update_weights_from_distributed	rendezvous hang → 45-min timeout
Jul 10 00:32	29060215979	✅ Passed	—	elapsed=90s
Jul 9 12:49	29019359571	❌ Failed	test_update_weights_from_distributed	DistStoreError: Timed out after 1801 seconds waiting for clients. 1/2 clients joined
Jul 9 00:31	28985427127	✅ Passed	—	elapsed=89s

First observed failure (this test): Jul 9 12:49 UTC. Last known pass: Jul 10 00:32 UTC. Runs Jul 10 12:44 (29093581297) and Jul 11 00:26 (29132618067) are still queued/in-flight and excluded.

Failure Cluster

Distributed rendezvous hang in init_custom_process_group (weight-update NCCL group)

Facts (verified observations)

• ##[error]The action 'Run test' has timed out after 45 minutes — action-level timeout, not a per-file timeout. Action started 20:58:59, timeout at 21:44:04 (~45 min).
• Suite invoked with --timeout-per-file 2700 --continue-on-error; the 2700 s per-file timeout never fired because the cumulative action clock reached 45 min first (1255 s + 234 s + hang).
• Current run: only rank 1 (SGL) reached init custom process group … backend=nccl (21:24:53); rank 0 (HF trainer subprocess) never printed its [hf] rank=0 … init custom process group line before the timeout.
• Jul 9 sister failure (same test, same job/partition) produced the explicit torch.distributed.DistStoreError: Timed out after 1801 seconds waiting for clients. 1/2 clients joined, with stack: init_custom_process_group (python/sglang/srt/utils/common.py:2434) ← init_process_hf (test line 229) ← init_process (test line 140) ← test line 581/839.
• The stalled side alternates: Jul 9 = rank 1 didn't join (rank 0 waited); Jul 11 = rank 0 didn't join (rank 1 waited). Consistent with a flaky rendezvous, not a deterministic code break.
• No sglang commit in the window touches the test or init_custom_process_group; common.py:2434 is a plain import of torch's rendezvous inside the helper.

Hypothesised Causes (with confidence)

• [MEDIUM] Flaky TCP-store rendezvous between the HF trainer process (rank 0) and the SGL inference server (rank 1) on tcp://localhost:60000. One rank fails to reach the rendezvous within the window, so the other blocks on _create_c10d_store.
  • Supporting: identical stack + 1/2 clients joined message in the Jul 9 failure; current run shows exactly one rank reaching the rendezvous; test passes in ~90 s on healthy runs, so the code path is normally fine.
  • Disconfirming: which rank stalls is not stable across runs; no code change in the regression window touches the rendezvous path — argues environmental/timing (slow server bring-up, GPU contention on the shared 2-GPU MI325 runner, or port reuse on localhost:60000) rather than a sglang regression.
  • How to verify: re-run test_update_weights_from_distributed.py on MI325 in isolation ~10× to measure flake rate; add rank-0/rank-1 arrival logging around init_custom_process_group; check for a lingering listener on port 60000 from a prior test in the same suite.
• [LOW] In-flight PR #24284 (NCCL comm not bound to worker gpu_id in init_weights_update_group) is adjacent but likely not this bug.
  • Supporting: same weight-update NCCL group; multi-GPU (tp_size=2) inference at non-zero rank_offset.
  • Disconfirming: #24284 affects the first NCCL collective (device binding), whereas this hang is upstream at the c10d TCP store rendezvous — the group never reaches the collective stage. PR is stale (opened 2026-05-03, no updates since).
  • How to verify: not actionable for this hang until the rendezvous-arrival hypothesis is ruled out.

In-flight Fix Check

• Searched open PRs for update_weights_from_distributed and init_custom_process_group.
• In-flight fix: ❌ none found targeting the rendezvous hang. Adjacent PR #24284 (device-binding of the weight-update NCCL comm) exists but is stale and addresses a later stage; not a fix for this rendezvous timeout.

Suggested Triage Steps

• Reproduce flake rate: run test/registered/rl/test_update_weights_from_distributed.py ~10× standalone on an MI325 2-GPU runner to confirm intermittency and capture a hanging repro.
• Add arrival logging on both ranks immediately before init_custom_process_group (and check for a stale listener on localhost:60000) to identify why one rank never joins.
• Consider lowering the store rendezvous timeout for this test so a stuck rank fails fast (well under the 45-min action budget) instead of consuming the whole job — this prevents one flaky rendezvous from masking the other 2 files' signal in the partition.
• If confirmed environmental (runner/GPU contention, port reuse), quarantine/retry the test on AMD rather than treating it as a sglang code regression; there is no implicated commit in the pass..fail window.

Status

• Persistence: Intermittent — failed Jul 9 12:49, passed Jul 10 00:32, hung Jul 11 12:17 (this run). Not every completed run.
• Scope: stage-b-test-2-gpu-large-amd partition 0 (this run). The 45-min timeout also aborts the two remaining files in the partition (test_gemma4_swa_triton_oob_regression.py, test_patch_torch.py), which never ran.
• Blocked work: RL update_weights_from_distributed CI signal on AMD 2-GPU; a single flaky rendezvous consumes the whole 45-min job budget.
• In-flight fix: ❌ none found for the rendezvous hang; adjacent #24284 is stale and not a match.

stage-c-test-large-8-gpu-amd-mi35x (linux-mi35x-gpu-8, 1) — failed step(s): Run test

Commit Info

• sglang: 32cb89d4
• aiter: 9127c94a (source: Dockerfile default — AITER_COMMIT_OVERRIDE empty; image rocm/sgl-dev:v0.5.15-rocm700-mi35x-20260711)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/test_deepseek_r1_mxfp4_8gpu.py	N/A (crash during server startup / weight load, before test body)	HW Exception by GPU node-3 ... reason: GPU Hang → Fatal Python error: Aborted → exit code 134	Log

The other two files in this partition (test_kimi_k25_mxfp4.py, test_qwen3_coder_next_8gpu.py) never ran — the process aborted on the first file (Begin (0/2)) and the partition exited.

Failure Summary

The 8-GPU amd/DeepSeek-R1-MXFP4-Preview server process on the MI35x runner (linux-mi35x-gpu-8-zf775-runner-47pvl, gfx950) aborted with a GPU Hang HW Exception on node-3 roughly 3m44s into startup (during HF weight download/load), producing Fatal Python error: Aborted and exit code 134. The automatic retry crashed again ~13s after launch on node-5, before even printing server_args. Both test_kimi_k25_mxfp4.py and test_qwen3_coder_next_8gpu.py were never reached.

Regression Status

Known recurring failure (GPU Hang / exit 134 cluster).

Recent history of test_deepseek_r1_mxfp4_8gpu.py in job stage-c-test-large-8-gpu-amd-mi35x (linux-mi35x-gpu-8, 1) (branch=main, event=schedule, completed runs only):

Date	Run	Test File Status	Mode	Error
Jul 11 12:17	29152390308 (this)	❌ Failed	GPU Hang → exit 134	GPU Hang node-3, retry node-5
Jul 10 00:32	29060215979	❌ Failed	GPU Hang → exit 134	GPU Hang node-3, retry node-5 (identical)
Jul 09 12:49	29019359571	❌ Failed	exit 1 (different cluster)	all 3 files failed, no GPU Hang
Jul 09 00:31	28985427127	✅ Passed	—	passed=true, elapsed=1023s

• GPU-Hang/exit-134 cluster first observed 2026-07-10 00:32 UTC, recurring identically now.
• Last known green for this test file: 2026-07-09 00:31 UTC. The intervening Jul 09 12:49 failure is a different symptom (exit 1, no hang) and is not part of this cluster.
• Runs Jul 10 12:44, Jul 11 00:26, and this run's siblings are still queued/in-flight and excluded.

Failure Cluster

GPU Hang HW Exception during 8-GPU MXFP4 server startup on MI35x (exit 134)

Facts (verified observations, no interpretation)

• Log (this run): HW Exception by GPU node-3 ... reason :GPU Hang at 12:31:55 (line 4729+), First attempt failed with exit code 134 (line 5068), retry HW Exception by GPU node-5 ... reason :GPU Hang at 12:32:40 (line 5115), Process completed with exit code 134 (line 5118).
• Startup reached weight load: Init torch distributed ends (all 8 TP ranks, ~10 GB each), Load weight begin. avail mem=277 GB, [CI Download] ... ACQUIRED lock for amd/DeepSeek-R1-MXFP4-Preview at 12:28:11 — then silence until the hang at 12:31:55 (~3m44s later).
• The Python threads captured in the Fatal Python error dump are in download_weights_from_hf (weight_utils.py:571) / hf_hub_download / filelock _poll_until_acquired and the watchdog.py thread — i.e. the interpreter was mid weight-download when the GPU-level SIGABRT arrived; these frames are where threads were parked, not a Python-level fault.
• Retry crashed ~13s after launch on a different node (node-5), before server_args printed.
• The Jul 10 00:32 run (86260258876) shows the byte-for-byte same pattern: GPU Hang node-3 during Begin (0/2) deepseek startup, retry GPU Hang node-5, exit 134.
• aiter version is unchanged (Dockerfile default 9127c94a) across the passing (Jul 09) and failing (Jul 10, Jul 11) runs — this is not a scout/override run.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [LOW] GPU/driver-level hang on the MI35x (gfx950) 8-GPU runner — a hardware/runner-state issue rather than a test-logic bug.
  • Supporting: symptom is a GPU Hang HW Exception (ROCm driver level), not a Python assertion/accuracy failure; the retry crashes almost immediately on a different GPU node, consistent with the runner being left in a wedged state after the first hang; identical symptom on the same runner type two runs in a row; failure occurs during idle weight download (no heavy compute launched yet).
  • Disconfirming: the test did pass on Jul 09 00:31 on the same runner type, and the Jul 09 12:49 failure was a different mode (exit 1, no hang) — so a plain "runner is permanently broken" story is incomplete; a sglang/model regression in the Jul 09 00:31 → Jul 10 00:32 window cannot be excluded from this evidence alone.
  • How to verify: check rocm-smi/dmesg on linux-mi35x-gpu-8-zf775-runner-47pvl around 12:31 UTC; re-run this single file in isolation on a freshly-reset MI35x 8-GPU node.
• [LOW] sglang regression landed between the last green (Jul 09 00:31, 074bb928) and the first GPU-Hang run (Jul 10 00:32, cfc66e05) affecting DeepSeek-R1-MXFP4 8-GPU init.
  • Supporting: the GPU-Hang cluster starts precisely in that window and reproduces deterministically across two runs.
  • Disconfirming: no code-level evidence tying any commit to the hang; the hang manifests during HF weight download (not a compute kernel), which argues against a model/kernel code change; a mid-window unrelated failure mode (exit 1 on Jul 09 12:49) muddies the timeline.
  • How to verify: bisect 074bb928..cfc66e05, re-running test_deepseek_r1_mxfp4_8gpu.py on a clean MI35x 8-GPU node per candidate.

No hypothesis above LOW confidence — the symptom is a driver-level GPU hang with no Python/kernel stack pinpointing sglang code; needs a runner-state check and/or bisect between 074bb928..cfc66e05.

In-flight Fix Check

• In-flight fix: ⚠️ [#30431](https://github.com/sgl-project/sglang/pull/30431) "[AMD] ci: self-heal driver-level VRAM leaks + graceful GPU cleanup on shutdown" — open, directly targets driver-level GPU wedging/cleanup on AMD runners; likely mitigates the "retry crashes immediately on a hung node" behavior. Worth chasing, but it is a CI self-heal mitigation, not necessarily a root-cause fix. (Search: is:pr is:open mi35x hang.)
• In-flight fix: ❌ none found specifically for DeepSeek-R1-MXFP4 8-GPU GPU-Hang (search: is:pr is:open DeepSeek-R1-MXFP4 mi35x).

Suggested Triage Steps

• Inspect runner linux-mi35x-gpu-8-zf775-runner-47pvl health (rocm-smi, dmesg, kernel/GCD reset state) around 2026-07-10 00:42 and 2026-07-11 12:31 UTC — confirm whether the hang is node-local and whether the retry-on-hung-node pattern matches #30431.
• Re-run test_deepseek_r1_mxfp4_8gpu.py in isolation on a freshly-reset MI35x 8-GPU node to separate "runner wedged" from "test regression".
• If it reproduces on a clean node, bisect 074bb928..cfc66e05 (last green → first GPU-Hang window).
• Verify whether #30431 merges cleanly and re-run this job with it applied to see if the cluster clears.

Status

• Persistence: GPU-Hang/exit-134 cluster failing since 2026-07-10 00:32 UTC — every completed run since (2/2 completed scheduled runs). Test file last green 2026-07-09 00:31 UTC.
• Scope: stage-c-test-large-8-gpu-amd-mi35x (linux-mi35x-gpu-8, 1) on pr-test-amd.yml; blocks all 3 MXFP4 large-model files in this partition (kimi + qwen3-coder never run because deepseek aborts first).
• Blocked work: DeepSeek-R1 / Kimi-K2.5 / Qwen3-Coder MXFP4 8-GPU CI signal on MI35x.
• In-flight fix: ⚠️ #30431 (AMD driver-level GPU cleanup) is a candidate mitigation; no deepseek-specific fix PR found.

---

Generated by amd-bot using Claude Code CLI (last updated: 2026-07-12 18:23 UTC)

[amd-bot]

pr-test-amd.yml — Per-Job Analysis (continued 3/9)

multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 3) — failed step(s): Run diffusion server tests (1-GPU)

Commit Info

• sglang: 7de33ce8
• aiter: 9127c94a (v0.1.17.dev110+g9127c94a1, source: dev image, matches Dockerfile default; AITER_COMMIT_OVERRIDE empty — not a scout run)

Failed Tests

Job: multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 3) — standalone file test_update_weights_from_disk.py. Result: 5 failed, 1 passed, 4 errors.

Test File	Test Function	Error	Log
python/sglang/multimodal_gen/test/single_test_file/test_update_weights_from_disk.py	TestUpdateWeightsFromDisk::test_update_weights_from_disk_default[Qwen-Image]	RuntimeError: Inplace update to inference tensor outside InferenceMode (HTTP 500)	Log
same	TestUpdateWeightsFromDisk::test_update_weights_specific_modules[Qwen-Image]	same RuntimeError (HTTP 500)	Log
same	TestUpdateWeightsFromDisk::test_corrupted_weights_rollback[Qwen-Image]	same RuntimeError (HTTP 500)	Log
same	TestUpdateWeightsFromDiskWithOffload::test_update_weights_with_offload_enabled[Qwen-Image]	weight-update failure (HTTP 500)	Log
same	TestUpdateWeightsFromDiskWithOffload::test_update_weights_with_offload_enabled[FLUX.2-klein-base-4B]	weight-update failure (HTTP 500)	Log
same	TestUpdateWeightsFromDisk::test_update_weights_from_disk_default[FLUX.2-klein-base-4B]	ERROR — AssertionError at linear.py:411 (shape mismatch)	Log
same	TestUpdateWeightsFromDisk::test_update_weights_specific_modules[FLUX.2-klein-base-4B]	ERROR	Log
same	TestUpdateWeightsFromDisk::test_update_weights_rejects_invalid_requests[FLUX.2-klein-base-4B]	ERROR	Log
same	TestUpdateWeightsFromDisk::test_corrupted_weights_rollback[FLUX.2-klein-base-4B]	ERROR	Log

Passed: TestUpdateWeightsFromDisk::test_update_weights_rejects_invalid_requests[Qwen-Image] (the negative-path test that never performs a real weight load).

Failure Summary

The standalone file test_update_weights_from_disk.py (partition 4/4 of the 1-gpu diffusion suite) fails when exercising the /update_weights_from_disk endpoint of the multimodal_gen runtime. Qwen-Image weight-update calls return HTTP 500 with RuntimeError: Inplace update to inference tensor outside InferenceMode is not allowed raised at weights_updater.py:200 → linear.py:575 (param_data.copy_(loaded_weight)). The FLUX.2 variant fails earlier in the same path with an AssertionError shape mismatch at linear.py:411 plus name-mapping warnings (img_in.bias → x_embedder.bias not found), then errors during rollback.

Regression Status

Never-passed (no green run found in the queryable window).

Recent history of test_update_weights_from_disk.py in job multimodal-gen-test-1-gpu-amd (…, 3) (branch main, event schedule, completed runs only):

Date (UTC)	Run	Job	Test File Status	Failed Functions	Error
Jul 11 00:26	29132618067	job	❌ Failed	5 failed, 4 errors (same set)	InferenceMode / shape mismatch
Jul 10 00:32	29060215979	job	❌ Failed	identical 5 failed, 4 errors	identical
Jul 09 12:49	29019359571	job	❌ Failed	identical 5 failed, 4 errors	identical
Jul 09 00:31	28985427127	job	❌ Failed	identical 5 failed, 4 errors	identical

Lookback range: Jul 09 00:31 → Jul 11 00:26 UTC (4 completed scheduled runs). No passing run for this test file in that window; the identical 5 failed / 1 passed / 4 errors signature appears in every run. Runs newer than this job (29152390308, 29173750641) are still queued — excluded [IN-FLIGHT].

Failure Cluster

"update_weights_from_disk weight-load failure in multimodal_gen (InferenceMode inplace copy_ + FLUX weight-shape/name mismatch)"

Facts (verified observations)

• RuntimeError: Inplace update to inference tensor outside InferenceMode is not allowed raised at python/sglang/multimodal_gen/runtime/post_training/weights_updater.py:200 → python/sglang/multimodal_gen/runtime/layers/linear.py:575 param_data.copy_(loaded_weight) (log lines ~6010, ~5977). Endpoint returns 500 Internal Server Error for POST /update_weights_from_disk.
• FLUX.2 default path fails first with AssertionError at linear.py:411 (assert param_data.shape == loaded_weight.shape, log line ~5567), preceded by warnings Skipping weight update for Flux2Transformer2DModel: parameter 'img_in.bias' not found after mapping to 'x_embedder.bias' and transformer_blocks.10.attn.add_v_proj.bias not found; rollback then re-raises the InferenceMode error.
• Secondary symptom: a PytestUnhandledThreadExceptionWarning from the test's own _compute_checksum_from_disk thread — AssertionError: No weights dir for transformer in …models--Qwen--Qwen-Image/…/transformer (test_update_weights_from_disk.py:220). This is a checksum helper reacting to the failed update, not the primary cause.
• aiter is the Dockerfile default (9127c94a, dev image, no override) — the failure is not aiter-related and reproduces on stock aiter.
• Reproducibility: identical failure set across all 4 completed scheduled runs in the window; NVIDIA-side comparison not in scope for this AMD job.
• Involved source files each have a single recent commit at their current path — e703f9e5 (#[30492], Jul 09) — a large refactor/relocation of the multimodal_gen tree, so git --follow dates are rename artifacts, not true feature-introduction dates.

Hypothesised Causes (with confidence)

• [HIGH] The multimodal_gen disk weight-update path performs an in-place param_data.copy_() on model parameters that are inference tensors (loaded under torch.inference_mode()), which PyTorch forbids outside InferenceMode.
  • Supporting: exact RuntimeError at linear.py:575 param_data.copy_(loaded_weight) in the failing stack trace; reproduces every run; message explicitly names inference-tensor inplace update.
  • Disconfirming: the negative-path Qwen-Image test_update_weights_rejects_invalid_requests passes (it never reaches copy_), consistent with the hypothesis rather than against it — no strong disconfirming evidence found.
  • How to verify: on an AMD MI325 runner, clone the param before copy (param_data.copy_ under torch.inference_mode(False) / .clone() path) or wrap the loader in a non-inference context, then re-run the Qwen-Image cases.
• [MEDIUM] The FLUX.2 (Flux2Transformer2DModel) branch has a distinct weight name-mapping / shape bug (img_in.bias→x_embedder.bias missing, add_v_proj.bias missing) causing the linear.py:411 shape assertion before the InferenceMode error is even reached.
  • Supporting: FLUX default fails at linear.py:411, a different line than the Qwen path; mapping warnings printed immediately before.
  • Disconfirming: could be a downstream effect of the same broken update flow rather than an independent bug — not conclusively separable from the log alone.
  • How to verify: run only test_update_weights_from_disk_default[FLUX.2-klein-base-4B] and inspect the param-name map for Flux2Transformer2DModel.
• [LOW] Introduced/exposed by the multimodal_gen relocation refactor e703f9e5 (#[30492]).
  • Supporting: it is the only commit touching the current paths of weights_updater.py / linear.py / the test.
  • Disconfirming: the earliest failing scheduled run (28985427127, Jul 09 00:31 UTC, head 074bb928) predates that commit's timestamp (Jul 09 09:09 UTC), so #[30492] cannot be the origin — the --follow date is a rename artifact and the feature was already failing before it.
  • How to verify: locate the pre-refactor path of this test and bisect the first scheduled run where it appeared.

In-flight Fix Check

• In-flight fix: ❌ none found — no open PR addresses the InferenceMode inplace weight-update error. Query is:pr is:open update_weights_from_disk returned 13 PRs, but the closest (#30749, #30761) concern loading quantized checkpoints for RL, not the diffusion InferenceMode/shape failures. is:pr is:open InferenceMode diffusion returned 0.

Suggested Triage Steps

• Reproduce locally on AMD MI325: pytest python/sglang/multimodal_gen/test/single_test_file/test_update_weights_from_disk.py::TestUpdateWeightsFromDisk::test_update_weights_from_disk_default -k Qwen-Image and confirm the 500 at linear.py:575.
• Inspect weights_updater.py:load_weights_into_model / _load_weights_into_module: ensure target params are cloned out of inference-mode (or the loader runs outside torch.inference_mode()) before param_data.copy_().
• Separately debug the Flux2Transformer2DModel weight name map (img_in.bias→x_embedder.bias, add_v_proj.bias) driving the linear.py:411 shape assertion.
• Identify the pre-refactor path of this test to find its true first-failing scheduled run and confirm whether it ever passed on AMD; if it is a never-passed AMD port, consider quarantining on AMD until fixed.

Status

• Persistence: Failing in every completed scheduled run in the queryable window (Jul 09 00:31 → Jul 11 00:26 UTC, 4/4 runs), identical 5 failed / 1 passed / 4 errors signature; no green run found.
• Scope: Partition 3 (standalone test_update_weights_from_disk.py) of multimodal-gen-test-1-gpu-amd; blocks the diffusion /update_weights_from_disk (RL/post-training) CI signal on AMD for both Qwen-Image and FLUX.2 models.
• In-flight fix: ❌ none found — needs new PR or investigation.

multimodal-gen-test-2-gpu-amd (linux-mi325-2gpu-sglang, 2) — failed step(s): Run diffusion server tests (2-GPU)

Commit Info

• sglang: 7de33ce8
• aiter: 9127c94a (v0.1.17.dev110+g9127c94a1; source: Dockerfile default, dev image preserved — AITER_COMMIT_OVERRIDE empty, not a scout run)

Failed Tests

Test File	Test Function	Error	Log
python/sglang/multimodal_gen/test/single_test_file/test_ar_models.py	TestGLMImage.test_generates_image (fails in setUpClass)	RuntimeError: AR model failed to start for glmimage → huggingface_hub.errors.HFValidationError: Repo id must be in the form 'repo_name' or 'namespace/repo_name': 'zai-org/GLM-Image/vision_language_encoder/'	Log

The job itself was terminated by ##[error]The action 'Run diffusion server tests (2-GPU)' has timed out after 150 minutes.

Failure Summary

The single test in this shard, TestGLMImage.test_generates_image, errored during setUpClass because the AR-stage server never became healthy. The sglang serve process for the glmimage cluster crashes at startup: it is launched with --model-path zai-org/GLM-Image/vision_language_encoder/, which HuggingFace rejects as an invalid repo id (a repo id may contain at most one /). wait_for_server_health then times out after 600 s (ConnectError: [Errno 111] Connection refused). The harness retries the cluster 3× and re-invokes pytest repeatedly, so the ~10-min startup timeout compounds until the whole step hits the 150-minute action limit and the job is killed.

Regression Status

Never-passed (since the test was introduced). Lookback: scheduled pr-test-amd.yml runs on main, shard (linux-mi325-2gpu-sglang, 2), completed runs only.

Recent history of test_ar_models.py (TestGLMImage.test_generates_image) in job multimodal-gen-test-2-gpu-amd (…, 2):

Date	Run	Job	Test File Status	Failed Function	Error
Jul 11	29132618067	job	❌ Failed	TestGLMImage.test_generates_image	HFValidationError → 150-min timeout
Jul 10	29060215979	job	❌ Failed	TestGLMImage.test_generates_image	HFValidationError → 150-min timeout (identical)
Jul 09	29019359571	job	— Not present	—	Predates PR #25381 merge (2026-07-09 15:54); GLM-Image test not yet in tree

The test was added by PR #25381 (7aab39a1, merged 2026-07-09). It has failed in every scheduled run in which it has executed (first ran 2026-07-10). No green run exists.

Failure Cluster

GLM-Image AR server startup crash — invalid HF repo id zai-org/GLM-Image/vision_language_encoder/ (subpath appended to repo id), leading to 150-min diffusion-step timeout.

Facts (verified observations)

• DEFAULT_AR_MODEL_NAME_FOR_TEST = "zai-org/GLM-Image" (python/sglang/multimodal_gen/test/test_utils.py:154) — an HF repo id, not a local snapshot path.
• The AR launch command builds --model-path f"{self.model}/vision_language_encoder/" and --tokenizer-path f"{self.model}/processor/" (test_ar_models.py:58 / :60). With the repo-id model, --model-path becomes zai-org/GLM-Image/vision_language_encoder/.
• The cache-check step (=== Checking for cached diffusion models (2-GPU tests) ===, log line ~4431) lists only Wan2.2, Wan2.1, Qwen-Image, FLUX.1-dev (all ✗ NOT CACHED). GLM-Image is not checked or pre-downloaded, so the path is never resolved to a local directory and is handed to HF as a repo id.
• Stack trace (verbatim, log ~line 4757): HFValidationError … → OSError raised from transformers/utils/hub.py:488 ← hf_transformers/config.py:63 (_try_load_longcat_config) ← config.py:84 (parse) ← model_config.py:277 ← server_args.py cuda-graph-compat path.
• wait_for_server_health(...) (test_utils.py:289) raises TimeoutError: Server at http://127.0.0.1:44421/v1/models not healthy after 600.0s. last_status=None last_err=ConnectError('[Errno 111] Connection refused') — i.e. the server process died, not a slow warmup.
• Reproducibility: identical error + identical 150-min timeout in the 2026-07-10 shard-2 job 86260258170.
• aiter is the Dockerfile default dev build; unchanged across the two failing runs — not implicated.

Hypothesised Causes (with confidence)

• [HIGH] The test model reference zai-org/GLM-Image is an HF repo id, but the AR launch code appends a subdirectory (/vision_language_encoder/, /processor/) that is only valid for a local model snapshot. On this AMD runner GLM-Image is not pre-cached, so sglang serve passes zai-org/GLM-Image/vision_language_encoder/ to HuggingFace, which rejects any repo id containing more than one /.
  • Supporting: HF error names the exact composed path; DEFAULT_AR_MODEL_NAME_FOR_TEST is a bare repo id (test_utils.py:154); model-path is built by string concat at test_ar_models.py:58; GLM-Image absent from the runner's cached-model list.
  • Disconfirming: (none found) — the composed path and the error string match exactly, and the failure reproduces every run.
  • How to verify: on an AMD runner, huggingface-cli download zai-org/GLM-Image --local-dir /models/GLM-Image, then re-run with the test pointed at the local dir (or set --model-path to the resolved snapshot subdir). Server should start.
• [LOW] The _try_load_longcat_config code path (e703f9e5) is where the exception surfaces (unconditional PretrainedConfig.get_config_dict(model) before the arch check, config.py:61-63).
  • Supporting: it is the top sglang frame in the trace.
  • Disconfirming: it merely reports the invalid repo id first; the fallback AutoConfig.from_pretrained would raise the same HFValidationError. Not the root cause — the invalid model path is.
  • How to verify: not actionable independently; fixing the model path removes the error regardless of this frame.

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs for GLM-Image + vision_language_encoder, GLM Image AR, multimodal_gen + glmimage. Matches are feature PRs (#30683, #20359, #20447), none addressing this startup/repo-id failure.

Suggested Triage Steps

• Confirm whether the GLM-Image weights are expected to be pre-cached on the MI325 runners; if not, add GLM-Image to the diffusion model cache/prefetch step (the same step that pre-caches Wan/Qwen/FLUX) so --model-path resolves to a local snapshot directory.
• If caching is out of scope, fix the test to resolve the model to a local path first (e.g. snapshot_download("zai-org/GLM-Image") and build --model-path {local_dir}/vision_language_encoder/), since HF repo ids cannot carry a subpath — see test_ar_models.py:58,60.
• Reduce blast radius: the 3× cluster retry × repeated pytest invocation turns a fast, deterministic startup crash into a 150-min timeout that consumes a 2-GPU runner. Consider failing fast when the AR server exits with a config/repo-id error rather than waiting the full 600 s health timeout and retrying.
• Ping the author of PR #25381 — the test has never passed on AMD 2-GPU CI since it merged.

Status

• Persistence: Failing every completed scheduled run since the test began executing on 2026-07-10 (2/2 runs); never passed.
• Scope: All three multimodal-gen-test-2-gpu-amd shards fail in each run, but this test-file failure is isolated to shard (…, 2). Each failing run burns the full 150-minute action budget on a 2-GPU runner.
• Blocked work: Blocks GLM-Image AR (diffusion) 2-GPU CI signal on AMD; also a CI-time sink due to the compounding retry/timeout.
• In-flight fix: ❌ none found — needs a new PR (model prefetch/caching or test model-path fix).

stage-c-test-large-8-gpu-amd-mi35x (linux-mi35x-gpu-8, 0) — failed step(s): Run test

Commit Info

• sglang: 32cb89d4
• aiter: 9127c94a (v0.1.17.dev110+g9127c94a1, source: dev image preserved — matches Dockerfile default AITER_COMMIT=9127c94a…; AITER_COMMIT_OVERRIDE empty)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/test_kimi_k25_mxfp4_bcg_mi35x.py	TestKimiK25MXFP4BcgMI35x.test_bcg_gsm8k	HW Exception … reason :GPU Hang → Fatal Python error: Aborted → exit 134	Log

(The other test in this partition, test/registered/cuda_graph/breakable/test_breakable_cuda_graph.py, never ran — the job aborted during the Kimi test.)

Failure Summary

The test_bcg_gsm8k test for amd/Kimi-K2.5-MXFP4 (DeepseekV3 arch, tp=4, aiter attention + aiter all-reduce fusion, piecewise/breakable CUDA graph) crashed on both attempts with a ROCm HW Exception … reason :GPU Hang, aborting via the CI soft-watchdog with exit code 134. Attempt 1 hung during model weight loading; the offline retry (HF_HUB_OFFLINE=1) hung again seconds after model init. No accuracy assertion was reached.

Regression Status

New regression (GPU-Hang symptom, started 2026-07-10 00:32 UTC).

Recent history of test_kimi_k25_mxfp4_bcg_mi35x.py (test_bcg_gsm8k) in job stage-c-test-large-8-gpu-amd-mi35x (branch=main, event=schedule, completed runs only; both stage-c partitions checked — the Kimi test lands in partition 0):

Date (UTC)	Run	Job	Test File Status	Detail
Jul 11 12:17	29152390308	stage-c …mi35x (0)	❌ Failed	GPU Hang → exit 134 (this run)
Jul 10 00:32	29060215979	stage-c …mi35x (0)	❌ Failed	GPU Hang → exit 134 (same symptom)
Jul 09 12:49	29019359571	stage-c …mi35x (0)	❌ Failed (different)	ImportError: cannot import name 'get_attention_cp_size'; model loaded fine, ran 1877s
Jul 09 00:31	28985427127	stage-c …mi35x (0)	⚠️ Infra	job died early (No docker containers found on host, exit 1) — Kimi test not reached
Jul 08 12:37	28943113233	stage-c …mi35x (0)	✅ Passed	—

First GPU-Hang failure: 2026-07-10 00:32 UTC. Last clean pass: 2026-07-08 12:37 UTC. In between, Jul 09 12:49 shows a different failure (an ImportError from the parallel-accessor refactor, now superseded).

Failure Cluster

GPU Hang (HW Exception) during Kimi-K2.5-MXFP4 model init/warmup on MI35x

Facts (verified observations)

• Attempt 1 (12:29:10 UTC): HW Exception by GPU node-3 … reason :GPU Hang (×many) → Fatal Python error: Aborted raised from python/sglang/srt/utils/watchdog.py:127/147; main-thread stack was inside weight loading (model_loader/loader.py:829 load_weights_and_postprocess, models/kimi_k25.py stream_language_weights, models/deepseek_v2.py:2888 load_weights). → First attempt failed with exit code 134.
• Attempt 2 (HF_HUB_OFFLINE=1, 12:29:57 UTC): crashed right after Piecewise CUDA graph is enabled, use MLA for prefill + Enable Aiter AllReduce Fusion for DeepseekV3ForCausalLM → HW Exception by GPU node-5 … reason :GPU Hang → ##[error]Process completed with exit code 134.
• Server config (from log): model=amd/Kimi-K2.5-MXFP4, tp_size=4, attention_backend=aiter, enable_aiter_allreduce_fusion=True, breakable/piecewise CUDA graph, kv_cache_dtype=fp8_e4m3.
• aiter is the unmodified Dockerfile-default dev image (v0.1.17.dev110+g9127c94a1); no AITER_COMMIT_OVERRIDE. This is pr-test-amd.yml (schedule), not an AITER Scout run.
• GPU node id differs across occurrences (node-3 on Jul 10 attempt 1; node-5 on both retries) — not obviously a single bad physical node.
• Reproducibility: 2/2 consecutive completed scheduled runs (Jul 10, Jul 11); both retries within each run.

Hypothesised Causes (with confidence)

• [LOW] Regression from the Jul 09 DeepSeek-V2 / parallel-refactor commit series (window 6ab7a65d…cfc66e05, 23 commits). Leading code-level candidate: 87992eee "[DeepSeek V2] Reorder dual-stream MoE to main-first to avoid CUDA graph stream explosion" (#30460) — Kimi-K2.5 loads via deepseek_v2.py/DeepseekV3ForCausalLM.
  • Supporting: lands in the hang window; touches DeepSeek-V2 MoE + CUDA-graph stream handling.
  • Disconfirming: attempt-1 hang stack is in weight loading, before any graph capture; the change is not AMD-specific and affects forward/graph, not load.
  • How to verify: bisect the window; revert on a branch and rerun the Kimi test on MI35x.
• [LOW] aiter fused all-reduce + piecewise CUDA graph interaction on gfx950.
  • Supporting: attempt 2 crashed immediately after Enable Aiter AllReduce Fusion + PCG; open PR #27650 "[AMD] Fix PCG capture fail with aiter fused all-reduce+RMSNorm" targets this exact path.
  • Disconfirming: attempt 1 hung during weight load, well before capture; symptom is a HW hang, not a capture error.
  • How to verify: rerun with --disable-cuda-graph and/or enable_aiter_allreduce_fusion off.
• [LOW] MI35x runner/driver instability (HW GPU Hang during H2D weight copies is atypical of a pure Python regression; open PR #30431 addresses MI35x driver-level VRAM leaks / GPU cleanup).
  • Disconfirming: reproduces on two consecutive scheduled runs and the test passed cleanly Jul 08 — points to a code change rather than pure flake.

No hypothesis above LOW confidence; needs git bisect between 6ab7a65d…cfc66e05.

In-flight Fix Check

• In-flight fix: ⚠️ candidate #27650 (open) — "[AMD] Fix PCG capture fail with aiter fused all-reduce+RMSNorm"; targets the aiter-allreduce-fusion + PCG path this test uses (not confirmed to address a HW hang).
• In-flight fix: ⚠️ related #30431 (open) — "[AMD] ci: self-heal driver-level VRAM leaks + graceful GPU cleanup" (MI35x GPU instability).
• In-flight fix: ❌ none found specifically for "Kimi-K2.5-MXFP4 GPU hang on MI35x". (Searched sglang open PRs for kimi+mxfp4 and mi35x.)

Suggested Triage Steps

• Bisect 6ab7a65d…cfc66e05 and rerun test/registered/amd/test_kimi_k25_mxfp4_bcg_mi35x.py on an MI35x (gfx950) 8-GPU runner; check 87992eee (#30460) first.
• Because attempt 1 hangs during weight loading, capture a py-spy/rocgdb dump at the hang to localize host-vs-kernel; check whether MXFP4 online-requant kernels execute during load.
• Rerun with enable_aiter_allreduce_fusion disabled and/or --disable-cuda-graph to test hypothesis 2 (relevance of #27650).
• Re-run on a different MI35x node to rule out hardware; if it still hangs, hardware is unlikely.

Status

• Persistence: GPU-Hang symptom on 2/2 consecutive completed scheduled runs since 2026-07-10 00:32 UTC (both attempts each run); test file red in some form since 2026-07-09.
• Scope: stage-c-test-large-8-gpu-amd-mi35x partition 0, pr-test-amd.yml. (Partition 1 of the Jul 10 run also showed GPU Hangs on other tests — possibly the same MI35x instability, but different test files.)
• Blocked work: blocks the Kimi-K2.5-MXFP4 BCG accuracy signal on MI35x.
• In-flight fix: candidate #27650; related #30431; none confirmed to fix this hang.

multimodal-gen-test-2-gpu-amd (linux-mi325-2gpu-sglang, 0) — failed step(s): Run diffusion server tests (2-GPU)

Commit Info

• sglang: 7de33ce8
• aiter: 9127c94a — v0.1.17.dev110+g9127c94a1 (source: dev image; [CI-AITER-CHECK] reports "Dev/patched version detected → skipping rebuild")

Failed Tests

All in python/sglang/multimodal_gen/test/server/test_server_2_gpu.py::TestDiffusionServerTwoGpu::test_diffusion_generation[...]. Final summary: 7 deselected, 18 errors — 1 real FAILED + 17 cascade ERRORs (summary line).

Test Param	Result	Error	Log
flux2_modelopt_fp8_tp2_t2i	❌ FAILED	RuntimeError: CUDA error: HIPBLAS_STATUS_NOT_SUPPORTED in torch._scaled_mm	Log
joy_echo_t2v_2gpu	❌ ERROR	FAILED during server startup (model download stalled ~19.5 min) → held scheduler port 5555	Log
16 subsequent cases (wan2_1_t2v_1.3b_cfg_parallel, fsdp-inference, mova_360p_*, ltx_2*, wan2_1_i2v_14b_*, zimage_*, flux_image_*, flux_2_image_*, …)	❌ ERROR (cascade)	RuntimeError: Scheduler port 5555 is unavailable and --strict-ports is enabled	Log

Many cases ran fine before the cascade: ideogram4_fp8_tp2_t2i, wan2_2_i2v_a14b_2gpu, wan2_2_t2v_a14b_2gpu, wan2_2_t2v_a14b_teacache_2gpu, wan2_2_t2v_a14b_lora_2gpu all PASSED.

Failure Summary

Two independent problems in one job. First, flux2_modelopt_fp8_tp2_t2i fails deterministically inside apply_fp8_linear → torch._scaled_mm with HIPBLAS_STATUS_NOT_SUPPORTED on the flux_2 modelopt-FP8 transformer. Separately, joy_echo_t2v_2gpu fails during server startup (its HF model download stalls for ~19.5 min then aborts) and leaves a process holding scheduler port 5555; every one of the 16 subsequent testcases then ERRORs with "port 5555 unavailable" under --strict-ports.

Regression Status

Both are Known recurring failures for this test file; test_server_2_gpu.py has not had a clean run in the queryable window.

Recent history of test_server_2_gpu.py (shard 0, multimodal-gen-test-2-gpu-amd (…, 0), branch main, event=schedule, completed runs only):

Date	Run	Test File Status	Detail
Jul 11	29132618067	❌ Failed	flux2 HIPBLAS (new) + port-5555 cascade from joy_echo
Jul 9 12:49	29019359571	❌ Failed	port-5555 cascade (38×) from a "Server exited early (code 1)" first failure; joy_echo also failed startup. No flux2 test present
Jul 10 / Jul 9 00:31	29060215979 / 28985427127	⚠️ logs expired (BlobNotFound)	job marked failure; log retention gone, could not compare

Note: flux2_modelopt_fp8_tp2_t2i is a newly-added case (absent from the Jul-9 shard-0 matrix), so its HIPBLAS failure is new-this-week; the port-5555 cascade pattern is pre-existing. Lookback limited to ~3 days by log retention.

Failure Cluster

• Cluster A: flux2 modelopt-FP8 HIPBLAS_STATUS_NOT_SUPPORTED in torch._scaled_mm
• Cluster B: Scheduler port 5555 cascade after a server-startup failure (--strict-ports)

Facts (verified observations)

• Cluster A stack trace (line 4722-4765): flux_2.py:77 _get_qkv_projections → linear.py:427 → modelopt_quant.py:432 apply → fp8_utils.py:1810 apply_fp8_linear → output = torch._scaled_mm(...) → RuntimeError: HIPBLAS_STATUS_NOT_SUPPORTED. 29 HIPBLAS occurrences (retries of the same request).
• Disconfirming for a blanket FP8 break: ideogram4_fp8_tp2_t2i (also FP8, tp2) PASSED at line 5804 immediately after flux2 — the FP8 scaled-mm path works for other models on this MI325; the failure is flux_2 modelopt-FP8-transformer specific.
• Cluster B trigger (line 8124): joy_echo logged "Downloading model snapshot from HF Hub for jdopensource/JoyAI-Echo (attempt 1/5)…" at 06:20:13 and "FAILED during server startup" at 06:39:48 (~19.5 min later). 68 subsequent "port 5555 unavailable" lines; first at line 8165 (server_args.py:937 _require_port).
• Cluster B is not caused by Cluster A: flux2 failed at 05:42 but the suite continued and multiple later cases passed; the cascade only began at joy_echo (06:39).

Hypothesised Causes (with confidence)

Cluster A — flux2 FP8 HIPBLAS_STATUS_NOT_SUPPORTED

• [HIGH] hipBLAS does not support the FP8 (e4m3fn) torch._scaled_mm configuration used by flux_2's modelopt-FP8 transformer on MI325/gfx942.
  • Supporting: verbatim HIPBLAS_STATUS_NOT_SUPPORTED at the torch._scaled_mm call in apply_fp8_linear; deterministic across all retries; the matching open fix PR describes exactly this (dequant→bf16 fallback in apply_fp8_linear).
  • Disconfirming: ideogram4_fp8_tp2_t2i passes on the same FP8 path — so it is flux_2-weight-shape/scale-specific, not all FP8. (This is consistent with the cutlass_compatible_b/shape-gated branch selection in fp8_utils.py.)
  • How to verify: run only flux2_modelopt_fp8_tp2_t2i on MI325 with PR #28889 applied.

Cluster B — port 5555 cascade

• [MEDIUM] joy_echo server startup aborts after a stalled HF model download and its process is not fully reaped, so scheduler port 5555 stays bound; under --strict-ports every later case fails _require_port.
  • Supporting: ~19.5 min gap between download-start and startup-failure; the cascade begins exactly at the first case after joy_echo; identical port-5555 cascade recurred Jul 9 seeded by a different first failure ("Server exited early").
  • Disconfirming: the log does not show the exact zombie PID still holding 5555, so "leaked process holds the port" is inferred, not directly observed; the joy_echo abort itself may be an HF-Hub/network flake rather than an sglang bug.
  • How to verify: reproduce a joy_echo startup abort, then ss -ltnp | grep 5555 / check for orphaned sglang serve PIDs; confirm the test harness does not force-free scheduler_port between cases.

In-flight Fix Check

• Cluster A: ✅ #28889 "[AMD] Fix FP8 e4m3fn GEMM on ROCm: dequant→bf16 fallback in apply_fp8_linear (fixes flux2 modelopt-fp8 HIPBLAS_STATUS_NOT_SUPPORTED)" — open since 2026-06-22, not draft, base main. Chase reviewers; do NOT open a duplicate. (query: search/issues q=repo:sgl-project/sglang is:pr is:open HIPBLAS_STATUS_NOT_SUPPORTED)
• Cluster B: ❌ none found — searched open PRs for strict-ports scheduler port and port 5555 unavailable (only unrelated PR #24227 returned). Needs new investigation.

Suggested Triage Steps

• Cluster A: verify PR #28889 merges cleanly and re-run test_diffusion_generation[flux2_modelopt_fp8_tp2_t2i] on MI325 (container rocm/sgl-dev:v0.5.15-rocm700-mi30x-20260710); if green, this closes Cluster A.
• Cluster B: make the diffusion server test harness reclaim scheduler_port between cases (force-kill the previous sglang serve process tree / wait for port release), OR relax --strict-ports for these sequential cases, so a single startup failure stops cascading into 16 ERRORs.
• Cluster B (secondary): investigate why jdopensource/JoyAI-Echo download stalls ~19.5 min — check startup/download timeout and whether the model should be pre-cached on the runner; joy_echo also failed startup on Jul 9, so it is a repeat offender.
• To isolate signal, treat the 16 port-5555 ERRORs as one cascade (root = joy_echo), not 16 independent failures.

Status

• Persistence: Cluster A (flux2 HIPBLAS) — new this week (test newly added); deterministic every run since. Cluster B (port-5555 cascade) — recurring, also seen Jul 9 29019359571.
• Scope: job multimodal-gen-test-2-gpu-amd (linux-mi325-2gpu-sglang, 0), test file test_server_2_gpu.py; the cascade blocks CI signal for ~16 diffusion 2-GPU cases whenever any earlier case's server startup fails.
• Blocked work: AMD 2-GPU multimodal-gen diffusion server coverage.
• In-flight fix: Cluster A ✅ #28889; Cluster B ❌ none.

multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 1) — failed step(s): N/A

Commit Info

• sglang: 7de33ce8
• aiter: 9127c94a (v0.1.17.dev110+g9127c94a1, source: dev image preserved — read from a same-run sibling shard log since the target job log is purged)

[!IMPORTANT]
The target job's log is unavailable. The log blob for job 86490781937 (1-gpu shard 1) returns BlobNotFound, and the run-level archive returns 404. The run metadata is also inconsistent (API reports the run status=queued, frozen at 2026-07-11T00:26:42Z, while this job actually completed failure at 2026-07-11T03:27:06Z). The exact failing test in shard 1 cannot be recovered from logs — the analysis below is reconstructed from retrievable sibling shard logs in the same run.

Failed Tests

Shard 1's own failing test(s) are not recoverable (log purged; the Jul-10 shard-1 log 86260258176 is likewise purged). Because pytest-split distributes items by test file, 1-gpu shard 1 ran a subset of {test_ar_models, test_diffusion_bcg_zimage_turbo, test_disagg_server, test_generate_i2i, test_pi05_e2e} (shard 2 ran test_generate_zimage_turbo_cli and passed; shard 3 ran test_update_weights_from_disk) — which subset landed on shard 1 is not reconstructable.

Failures observed in retrievable sibling shards of the same run 29132618067 (proxy evidence only — not necessarily what shard 1 hit):

Job	Test File	Test Function	Error	Log
1gpu shard 3	python/sglang/multimodal_gen/test/single_test_file/test_update_weights_from_disk.py	TestUpdateWeightsFromDisk::{test_update_weights_from_disk_default, test_update_weights_specific_modules, test_corrupted_weights_rollback}, TestUpdateWeightsFromDiskWithOffload::test_update_weights_with_offload_enabled	RuntimeError: Inplace update to inference tensor outside InferenceMode is not allowed → HTTP 500 (5 failed, 4 errors)	Log
2gpu shard 0	python/sglang/multimodal_gen/test/server/test_server_2_gpu.py	TestDiffusionServerTwoGpu::test_diffusion_generation	test failed	Log
2gpu shard 1	python/sglang/multimodal_gen/test/single_test_file/test_disagg_server.py	TestDisaggZImage1Rank, TestDisaggZImage2RankDenoiser, TestDisaggZImageTracing (3 setup errors)	ERROR at setup	Log
2gpu shard 2	python/sglang/multimodal_gen/test/single_test_file/test_ar_models.py	TestGLMImage::test_generates_image	AR model failed to start for glmimage (1 error)	Log

Failure Summary

The multimodal-gen test suite on AMD MI325 is broadly red this run: 6 of 7 shards failed (1-gpu shards 0/1/3 and 2-gpu shards 0/1/2; only 1-gpu shard 2 and the unit-test job passed). The failures are heterogeneous — at least four distinct symptoms across shards, so there is no single shared root cause. The target job (1-gpu shard 1) is one of the failing shards, but its specific failing test is unrecoverable because the log blob was purged.

Regression Status

Known recurring failure (job level); test-file level unverifiable for shard 1.

• Job-level: 1-gpu shard 1 also failed in the previous completed scheduled run 29060215979 (Jul 10), and the whole multimodal-gen suite has been failing in every completed scheduled pr-test-amd run Jul 8–10 (conclusions all failure).
• Test-file level: cannot be established for shard 1 — its log is purged in both this run and the Jul-10 run, and shard→file assignment is non-deterministic across runs.
• The best-documented cluster (update_weights InferenceMode) is confirmed recurring: present 18× in the Jul-10 shard-3 log 86260258188 and again this run.

Recent completed scheduled runs (branch=main, event=schedule) — note the 4 most recent runs (Jul 10–12) show a spurious queued status via the API despite jobs having run:

Date	Run	Suite status
Jul 11	29132618067	❌ 6/7 multimodal-gen shards failed (this run)
Jul 10	29060215979	❌ shard 1 + 5 others failed
Jul 9	29019359571	❌ failure
Jul 9	28985427127	❌ failure
Jul 8	28943113233	❌ failure

Failure Cluster

Multiple clusters in this run (shard 1's exact cluster unknown):

1. "Inplace update to inference tensor outside InferenceMode" during weight update (update_weights suite)
2. 2-GPU diffusion generation failure (test_server_2_gpu)
3. Disagg ZImage server setup error (test_disagg_server)
4. glmimage AR model fails to start (test_ar_models)

Facts (verified observations, no interpretation)

• Target job 86490781937: conclusion=failure, started 02:19:03Z, completed 03:27:06Z; steps 9 (Run diffusion server tests), 10, 20 have conclusion=None; log blob → BlobNotFound.
• Sibling 1-gpu shard 3 86490781912: ======== 5 failed, 1 passed, 11 warnings, 4 errors in 870.65s ========.
• Verbatim server error (shard 3, both models FLUX.2-klein-base-4B & Qwen-Image): RuntimeError: Inplace update to inference tensor outside InferenceMode is not allowed.You can make a clone to get a normal tensor before doing inplace update.
• Server-side stack (shard 3): runtime/managers/scheduler.py:229 → post_training/scheduler_post_training_mixin.py:11 → post_training/gpu_worker_post_training_mixin.py:38 → post_training/weights_updater.py:286 (_apply_weights). In weights_updater.py, _apply_weights performs param.data.copy_(...) / dtensor_param._local_tensor.copy_(...) (lines 214/222) — the in-place op that trips the inference-tensor guard.
• Environment warning present in all shards (non-fatal): [aiter] AITER_USE_SYSTEM_TRITON=1: aiter gluon kernels require triton>=3.6.0, found 3.4.0.
• aiter unchanged from Dockerfile default (dev image v0.1.17.dev110+g9127c94a1 preserved, AITER_COMMIT_OVERRIDE empty) — read from same-run sibling shard.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [LOW] The target shard-1 failure is the same suite-wide AMD breakage affecting the other shards.
  • Supporting: 6/7 shards fail this run and shard 1 has failed in consecutive completed runs (Jul 10 + Jul 11); the whole suite is persistently red on AMD.
  • Disconfirming: the retrievable shards fail with four different symptoms, so "one cause" is not supported — shard 1 could match any of them or a fifth. No shard-1 log exists to confirm.
  • How to verify: re-run job 86490781937 (or the whole suite) so a fresh, non-purged log is captured, then read shard 1's actual failing test.
• [MEDIUM] The update_weights cluster (shard 3, not confirmed as shard 1's failure) is caused by pipeline weights being created as torch inference tensors (loaded under torch.inference_mode()), so _apply_weights' param.data.copy_() at weights_updater.py:214/222 raises outside InferenceMode.
  • Supporting: exact stack trace lands on weights_updater.py:286→_apply_weights; error text is torch's inference-tensor in-place guard; reproduces for both models and with/without offload; recurring across Jul 10 + Jul 11.
  • Disconfirming: this is a shard-3 symptom — there is no evidence it is what shard 1 failed on. This code was not modified in the queryable (shallow) git window, so the trigger may be an environment/torch-version change rather than a code change.
  • How to verify: git bisect the multimodal_gen post_training path against a torch/inference_mode change; locally hit POST /update_weights_from_disk on MI325 and check whether pipeline params are inference tensors at load time.

No hypothesis about shard 1 specifically exceeds LOW confidence — its log is required.

In-flight Fix Check

• In-flight fix: ❌ none found for the multimodal_gen update_weights InferenceMode error. Searched is:pr is:open inference tensor update weights (102 loose matches, none on-topic) and is:pr is:open multimodal_gen update_weights (1 match, #19152 "Support offload and wake up of SGLang Diffusion" — unrelated feature). #30749 touches update_weights_from_disk but is RL/srt, not multimodal_gen.

Suggested Triage Steps

• First, recover shard-1 signal: re-run job 86490781937 so a non-purged log is captured, then identify shard 1's actual failing test file/function. All shard-1 analysis is blocked until then.
• Investigate log/artifact retention for AMD self-hosted runners — shard-1 logs are purged within ~1 day in both the current and previous run, defeating post-hoc triage (the spurious queued run status may be related).
• For the update_weights cluster: locally reproduce POST /update_weights_from_disk on MI325 for Qwen-Image; if params are torch inference tensors, weights_updater._apply_weights likely needs the copy performed under torch.inference_mode(False) / on a cloned tensor.
• For the other clusters: triage test_server_2_gpu, test_disagg_server (ZImage), and test_ar_models (glmimage startup) separately — they are distinct symptoms, not the same bug.

Status

• Persistence: multimodal-gen suite failing every completed scheduled pr-test-amd run since at least Jul 8; 1-gpu shard 1 failed in the last two completed runs (Jul 10, Jul 11).
• Scope: 6 of 7 multimodal-gen shards this run (1-gpu shards 0/1/3, 2-gpu shards 0/1/2); ≥4 distinct symptoms; AMD MI325 only (this analysis covers pr-test-amd).
• Evidence gap: target job log purged — shard 1's exact failing test is unknown; findings above are from sibling shards.
• In-flight fix: ❌ none found for the documented update_weights InferenceMode cluster.

---

Generated by amd-bot using Claude Code CLI (last updated: 2026-07-12 18:23 UTC)