[P2P] Add Cambricon MLU (CNRT) support for cross-node P2P RDMA#1011
Open
Iwandy777 wants to merge 4 commits into
Open
[P2P] Add Cambricon MLU (CNRT) support for cross-node P2P RDMA#1011Iwandy777 wants to merge 4 commits into
Iwandy777 wants to merge 4 commits into
Conversation
Map the gpu* runtime abstraction onto CNRT/CNDRV under the __CAMBRICON_PLATFORM_MLU__ guard, leaving CUDA/HIP builds unchanged: - gpu_rt.h: CNRT macro mappings + inline wrappers where signatures differ - util.h: MLU vendor id (0xcabc), CNRT-based BDF enumeration, get_dev_idx - engine.cc: open IPC handles in the owner GPU context and copy across cards via cnrtMemcpyPeerAsync; use the exact allocation base for cnrtAcquireMemHandle - nccl_types.h: alias cudaStream_t to cnrtQueue_t - add Makefile.cnrt and a torch-free one-sided IPC test Verified on MLU370-X8: MLU0<->MLU1 one-sided IPC (write/read/writev/readv and async variants) all pass. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…bench - probe_peermem_regmr.py: verify ibv_reg_mr on MLU device memory via cambricon_peer_mem (the cross-node MLU P2P gating capability) - xnode_cnrt.py: inter-node RDMA-WRITE between two MLU nodes with built-in TCP rendezvous; correctness + bandwidth sweep - bench_ipc_cnrt.py: intra-node IPC P2P bandwidth/latency Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- engine: transparent bounded peer-able staging for write/read so plain cnrtMalloc buffers (non peer-able, e.g. torch_mlu tensors) work over RDMA; plain device memory can't be NIC-DMA'd directly on Cambricon. - engine: cache the IPC mem handle per allocation (cnrtAcquireMemHandle only succeeds once per allocation) so repeated advertise_ipc is idempotent; fixes a hang in bench_ipc_cnrt.py at the second advertised size. - Makefile.cnrt: auto-generate header deps (-MMD -MP) to avoid ABI mismatch when engine.h changes. - tests/xnode_cnrt.py: single-node self-test (spawn two local MLUs, RDMA over NIC loopback, write+read) plus manual real inter-node mode. - benchmarks/benchmark_nixl.py: use the resolved device string in the padding block allocation. All changes are guarded by __CAMBRICON_PLATFORM_MLU__ and do not affect CUDA/HIP/MUSA. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Iwandy777
requested review from
MaoZiming,
YangZhou1997,
derekwin,
monopodium,
praveingk and
zhongjiechen
as code owners
Contributor
Author
|
Config: 2x Cambricon MLU370-X8, Neuware, ConnectX-5 (mlx5, RoCEv2), built with Intra-node IPC (bench_ipc_cnrt.py):
Cross-node GDR (128 MB, 4 chunks): WRITE 6.89 GB/s, READ 6.31 GB/s. |
Reformat probe_peermem_regmr.py and bench_ipc_cnrt.py with black to satisfy the python-format-check CI. Formatting only, no behavior change. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Adds Cambricon MLU (CNRT) cross-node P2P RDMA (write + read), extending the
existing intra-node IPC support. Plain cnrtMalloc / torch_mlu tensors are not
peer-able, so the NIC cannot DMA them directly; uccl relays user buffers through
a bounded peer-able staging buffer in chunks (as CNCL does). Also caches the IPC
handle per allocation (cnrtAcquireMemHandle succeeds only once per allocation),
and adds header dependencies to Makefile.cnrt. All changes are guarded by
__CAMBRICON_PLATFORM_MLU__; non-Cambricon builds (CUDA/HIP/MUSA) arebyte-identical to main.
Type of Change
How Has This Been Tested?
Checklist
format.shto follow the style guidelines.make -f Makefile.cnrt).