fix(cco): guard hipMemAllocationTypeUncached for older ROCm#480
Merged
Conversation
hipMemAllocationTypeUncached was added to HIP on 2025-09-11 (HIP_VERSION > 70051831). CCO used the symbol unguarded in CcoWindowAllocType(), so it fails to compile on older ROCm such as 7.0.0 (the reason sglang had to revert the mori pin). Mirror the existing shmem guard in symmetric_memory.cpp: use the symbol when available, the raw value 0x40000000 on the 70051831 build (symbol may be missing; exclude the buggy 7c9236b16 build), and fall back to Pinned memory with a warning on anything older. The CCO_UNCACHED_WINDOW=0 opt-out is preserved. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Merged
5 tasks
The dispatch/combine cross-device (xdb) barrier matches per-call flags across ranks; if one rank gets a full call ahead it overwrites the flag a slower rank is still waiting on -> deadlock. The graph/eager warmup fired the kernel back-to-back with no per-iteration cross-rank sync, so at small token counts (short kernels, e.g. ct=128) rank drift intermittently hung the EP8 smoke. Add sync()+comm.barrier() inside the warmup loops: sync() waits this rank's kernel to finish, comm.barrier() holds until all ranks finished, so no rank starts call N+1 before every rank completed N. Warmup is untimed, so measured replay bandwidth is unchanged. Verified on MI355X (8 ranks, MODE=graph SWEEP=128,512,4096,8192): 8/8 runs clean (was ~1-in-2 hang); perf identical to baseline. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
CcoWindowAllocType()usedhipMemAllocationTypeUncachedunguarded. That symbol was only added to HIP on 2025-09-11 (HIP_VERSION > 70051831), so CCO fails to compile on older ROCm such as 7.0.0 — the reason sgl-project/sglang#31406 had to revert the mori pin.HIP_VERSIONguard already used by the shmem path insrc/application/memory/symmetric_memory.cpp:> 70051831→ use the symbol== 70051831→ use raw0x40000000(symbol may be missing in that build; exclude the buggy7c9236b16build)< 70051831(incl. ROCm 7.0.0) → warn and fall back to Pinned memoryCCO_UNCACHED_WINDOW=0opt-out is preserved; both call sites already route through this single function, so no other changes are needed.Test plan
HIP_VERSION < 70051831) — compiles, falls back to Pinned with a warning.HIP_VERSION > 70051831) — still uses uncached windows.ci_cco.yml) on MI355X-AINIC passes unchanged.🤖 Generated with Claude Code