Skip to content

feat(cco): add fabric handle support for gfx1250#463

Open
kawhil-amd wants to merge 1 commit into
mainfrom
cco_fabric_support
Open

feat(cco): add fabric handle support for gfx1250#463
kawhil-amd wants to merge 1 commit into
mainfrom
cco_fabric_support

Conversation

@kawhil-amd

Copy link
Copy Markdown
Contributor

Summary

  • Add fabric handle support to CCO P2P memory sharing, following RCCL's pattern
  • When the HIP runtime supports fabric handles (hipMemHandleTypeFabric, 0x8), CCO uses 64-byte opaque tokens exchanged via Allgather instead of dma-buf
    FDs passed through Unix sockets + SCM_RIGHTS
  • Falls back transparently to the existing FD-based LocalBootstrapNetwork path when fabric is unavailable (runtime probe at ccoCommCreate)

Motivation

The current dma-buf FD exchange requires per-pair Unix domain socket setup (socket path construction, leader disambiguation, SCM_RIGHTS ancillary data).
Fabric handles reduce this to a single bootNet->Allgather of 64-byte tokens — simpler, faster, and required for upcoming gfx1250 / UALoE interconnect.

Changes

  • include/mori/utils/hip_compat.hpp: Add hipMemFabricHandle_compat_t and hipMemHandleTypeFabricCompat compat shim (uses native types when HIP_FABRIC_API
    is defined)
  • include/mori/cco/cco.hpp: Add ccoFabricHandle_t, change AllocMeta to union of shareFd/fabricHandle with isFabric flag, add handleType to ccoComm
  • src/cco/cco_init.cpp:
    • ccoCommCreate: probe fabric support (alloc + export test), set handleType
    • ccoMemAlloc: use probed handle type for allocation and export
    • ccoWindowRegister: fabric path uses Allgather + hipMemImportFromShareableHandle; FD path unchanged as fallback. RDMA MR uses
      RegisterRdmaMemoryRegionAuto (ibv_reg_mr) in fabric mode
    • ccoMemFree/ccoCommDestroy: skip close(fd) when using fabric handles

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant