Skip to content

Unclear GPUDirect RDMA behavior when GPU and RoCE v2 NIC are on different NUMA nodes #11121

Open
Open
Unclear GPUDirect RDMA behavior when GPU and RoCE v2 NIC are on different NUMA nodes#11121
@hahahannes

Description

@hahahannes
hahahannes
opened on Jan 15, 2026

Describe the bug

We are running a setup with two nodes using BroadCom RoCEv2 NICs (100 Gb) and NVIDIA GPUs. Initially we have two NUMA nodes configured on both nods.
Running ib_write_bw seems to indicate that GPUDirect RDMA works between all GPUs, independent of which NUMA node is used. We get an expected bandwith of around 85 Gb/s.
Though with UCX it does not work if the GPU is on a different NUMA node than the NIC. In this case the bandwith drops to 65 Gb/s and we see staging through
If we configure the node to use one NUMA node, GPUDirect RDMA will work again, independent of which NUMA node is used.

We are assuming that UCX has some logic to decide if it will stage through host memory or use RDMA based on the topology.
But given, that RDMA seems to work across NUMA nodes, we were wondering if this can be either configured or if there is another problem.

Steps to Reproduce

ib_write_bw

On the server node:

ib_write_bw -d bnxt_re0 --use_cuda=0 -a -x 3 --report_gbits -F --use_cuda_dmabuf --ipv6 -v

On the client node:

nsys profile --trace=cuda,ucx --output=$OUTDIR --force-overwrite=trueib_write_bw -d bnxt_re0 --use_cuda=4 -a -x 3 --report_gbits -F --use_cuda_dmabuf --ipv6 -v $IB_HOST
nsys stats --report cuda_api_sum --force-export=true $OUTDIR.nsys-rep

This should report expected bandwith and no cuMemCopy calls from CUDA.

UCX

export OUT_DIR=$(pwd) && export BIN=./main																						
mpirun --mca pml ucx -x UCX_IB_GPU_DIRECT_RDMA=yes -x UCX_TLS=rc,cuda -x CUDA_VISIBLE_DEVICES=GPU_IDX -x TMPDIR=$OUT_DIR -x BIN=$BIN -np 1 sh -c 'nvidia-smi && ucx_info -v && echo "host $(hostname): $(which mpirun)" && $BIN DtoD' : -x UCX_IB_GPU_DIRECT_RDMA=no -x UCX_TLS=rc,cuda -x CUDA_VISIBLE_DEVICES=NODE-1 -x TMPDIR=$OUT_DIR -x BIN=$BIN -np 1 sh -c 'nvidia-smi && ucx_info -v && echo "host $(hostname): $(which mpirun)" && nsys profile --output=$TMPDIR --force-overwrite=true $BIN DtoD'																						
nsys stats --report cuda_api_sum --force-export=true $OUT_DIR.nsys-rep

This should report a lower perfomance if the GPU and the NIC are on different NUMA nodes and cuMemCopy from CUDA.

  • UCX version used (from github branch XX or release YY) + UCX configure flags (can be checked by ucx_info -v)
    UCX version from release 1.20.1
ucx_info -v
# Library version: 1.20.0
# Library path: /usr/local/ucx/lib/libucs.so.0
# API headers version: 1.20.0
# Git branch '', revision 4b7a6ca
# Configured with: --prefix=/usr/local/ucx --with-cuda=/usr/local/cuda-12.8 --with-gdrcopy=/usr/local/gdrcopy --with-go=no --with-verbs --without-java --disable-doxygen-doc --enable-optimizations --enable-mt --enable-debug --disable-assertions --disable-params-check --disable-dependency-tracking --enable-cma --with-rc --with-ud --with-dc --with-mlx5-dv --with-ib-hw-tm --with-dm --with-avx --without-cm --with-rdmacm
  • Any UCX environment variables used

Setup and versions

  • OS version (e.g Linux distro) + CPU architecture (x86_64/aarch64/ppc64le/...)
    • cat /etc/redhat-release AlmaLinux release 9.6 (Sage Margay) + uname -a: Linux ngt-003-h100-nvl--rnmcnohij2v7-node-2 6.8.4-200.fc39.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Apr 4 20:45:21 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
  • For RDMA/IB/RoCE related issues:
    • Driver version:
      • rpm -q rdma-core: rdma-core-54.0-1.el9.x86_64 or rpm -q libibverbs: libibverbs-57.0-2.el9.x86_64
    • HW information from ibstat:
ibstat
CA 'bnxt_re0'
	CA type: Broadcom NetXtreme-C/E RoCE Driver HCA
	Number of ports: 1
	Firmware version: 231.0.154.0
	Hardware version: 0x14e4
	Node GUID: 0x8e8474fffe1f2550
	System image GUID: 0x8e8474fffe1f2550
	Port 1:
		State: Active
		Physical state: LinkUp
		Rate: 100
		Base lid: 0
		LMC: 0
		SM lid: 0
		Capability mask: 0x001d0000
		Port GUID: 0x8e8474fffe1f2550
		Link layer: Ethernet
CA 'bnxt_re1'
	CA type: Broadcom NetXtreme-C/E RoCE Driver HCA
	Number of ports: 1
	Firmware version: 231.0.154.0
	Hardware version: 0x14e4
	Node GUID: 0x8e8474fffe1f2551
	System image GUID: 0x8e8474fffe1f2551
	Port 1:
		State: Down
		Physical state: Disabled
		Rate: 2.5
		Base lid: 0
		LMC: 0
		SM lid: 0
		Capability mask: 0x001d0000
		Port GUID: 0x8e8474fffe1f2551
		Link layer: Ethernet
  • For GPU related issues:
    • GPU type: NVIDIA H100 NVL and NVIDIA L40S
    • Cuda:
      • Drivers version: 570.124.06
      • Check if peer-direct is loaded: lsmod|grep nv_peer_mem dmabuf is used instead
      lsmod|grep gdrdrv
gdrdrv                 45056  0
nvidia              13099008  115 nvidia_uvm,nvidia_fs,gdrdrv,nvidia_modeset

lsmod | grep dma
rdma_ucm               36864  0
rdma_cm               155648  1 rdma_ucm
iw_cm                  65536  1 rdma_cm
ib_cm                 155648  1 rdma_cm
ib_uverbs             200704  2 bnxt_re,rdma_ucm
ib_core               540672  7 rdma_cm,iw_cm,bnxt_re,ib_umad,rdma_ucm,ib_uverbs,ib_cm

Additional information (depending on the issue)

  • OpenMPI version
ompi_info   
                 Package: Open MPI
                          Distribution
                Open MPI: 4.1.1
  Open MPI repo revision: v4.1.1
   Open MPI release date: Apr 24, 2021
                Open RTE: 4.1.1
  Open RTE repo revision: v4.1.1
   Open RTE release date: Apr 24, 2021
                    OPAL: 4.1.1
      OPAL repo revision: v4.1.1
       OPAL release date: Apr 24, 2021
                 MPI API: 3.1.0
            Ident string: 4.1.1
                  Prefix: /usr/local/mpi
 Configured architecture: x86_64-pc-linux-gnu
          Configure host: ngt-003-h100-nvl--rnmcnohij2v7-node-2
           Configured by: hannesja
           Configured on: Tue Jan 13 14:51:48 UTC 2026
          Configure host: ngt-003-h100-nvl--rnmcnohij2v7-node-2
  Configure command line: '--prefix=/usr/local/mpi'
                          '--with-ucx=/usr/local/ucx'
                          '--with-gdrcopy=/usr/local/gdrcopy'
                          '--with-cuda=/usr/local/cuda-12.8'
                          '--enable-mca-no-build=btl-uct'
                          '--enable-mca-no-build=btl-openib'
                          '--without-verbs'
                Built by: root
                Built on: Tue Jan 13 15:02:40 UTC 2026
              Built host: ngt-003-h100-nvl--rnmcnohij2v7-node-2
              C bindings: yes
            C++ bindings: no
             Fort mpif.h: yes (all)
            Fort use mpi: yes (full: ignore TKR)
       Fort use mpi size: deprecated-ompi-info-value
        Fort use mpi_f08: yes
 Fort mpi_f08 compliance: The mpi_f08 module is available, but due to
                          limitations in the gfortran compiler and/or Open
                          MPI, does not support the following: array
                          subsections, direct passthru (where possible) to
                          underlying Open MPI's C functionality
  Fort mpi_f08 subarrays: no
           Java bindings: no
  Wrapper compiler rpath: runpath
              C compiler: gcc
     C compiler absolute: /usr/bin/gcc
  C compiler family name: GNU
      C compiler version: 11.5.0
            C++ compiler: g++
   C++ compiler absolute: /usr/bin/g++
           Fort compiler: gfortran
       Fort compiler abs: /usr/bin/gfortran
         Fort ignore TKR: yes (!GCC$ ATTRIBUTES NO_ARG_CHECK ::)
   Fort 08 assumed shape: yes
      Fort optional args: yes
          Fort INTERFACE: yes
    Fort ISO_FORTRAN_ENV: yes
       Fort STORAGE_SIZE: yes
      Fort BIND(C) (all): yes
      Fort ISO_C_BINDING: yes
 Fort SUBROUTINE BIND(C): yes
       Fort TYPE,BIND(C): yes
 Fort T,BIND(C,name="a"): yes
            Fort PRIVATE: yes
          Fort PROTECTED: yes
           Fort ABSTRACT: yes
       Fort ASYNCHRONOUS: yes
          Fort PROCEDURE: yes
         Fort USE...ONLY: yes
           Fort C_FUNLOC: yes
 Fort f08 using wrappers: yes
         Fort MPI_SIZEOF: yes
             C profiling: yes
           C++ profiling: no
   Fort mpif.h profiling: yes
  Fort use mpi profiling: yes
   Fort use mpi_f08 prof: yes
          C++ exceptions: no
          Thread support: posix (MPI_THREAD_MULTIPLE: yes, OPAL support: yes,
                          OMPI progress: no, ORTE progress: yes, Event lib:
                          yes)
           Sparse Groups: no
  Internal debug support: no
  MPI interface warnings: yes
     MPI parameter check: runtime
Memory profiling support: no
Memory debugging support: no
              dl support: yes
   Heterogeneous support: no
 mpirun default --prefix: no
       MPI_WTIME support: native
     Symbol vis. support: yes
   Host topology support: yes
            IPv6 support: no
      MPI1 compatibility: no
          MPI extensions: affinity, cuda, pcollreq
   FT Checkpoint support: no (checkpoint thread: no)
   C/R Enabled Debugging: no
  MPI_MAX_PROCESSOR_NAME: 256
    MPI_MAX_ERROR_STRING: 256
     MPI_MAX_OBJECT_NAME: 64
        MPI_MAX_INFO_KEY: 36
        MPI_MAX_INFO_VAL: 256
       MPI_MAX_PORT_NAME: 1024
  MPI_MAX_DATAREP_STRING: 128
           MCA allocator: basic (MCA v2.1.0, API v2.0.0, Component v4.1.1)
           MCA allocator: bucket (MCA v2.1.0, API v2.0.0, Component v4.1.1)
           MCA backtrace: execinfo (MCA v2.1.0, API v2.0.0, Component v4.1.1)
                 MCA btl: self (MCA v2.1.0, API v3.1.0, Component v4.1.1)
                 MCA btl: smcuda (MCA v2.1.0, API v3.1.0, Component v4.1.1)
                 MCA btl: tcp (MCA v2.1.0, API v3.1.0, Component v4.1.1)
                 MCA btl: vader (MCA v2.1.0, API v3.1.0, Component v4.1.1)
            MCA compress: bzip (MCA v2.1.0, API v2.0.0, Component v4.1.1)
            MCA compress: gzip (MCA v2.1.0, API v2.0.0, Component v4.1.1)
                 MCA crs: none (MCA v2.1.0, API v2.0.0, Component v4.1.1)
                  MCA dl: dlopen (MCA v2.1.0, API v1.0.0, Component v4.1.1)
               MCA event: libevent2022 (MCA v2.1.0, API v2.0.0, Component
                          v4.1.1)
               MCA hwloc: hwloc201 (MCA v2.1.0, API v2.0.0, Component v4.1.1)
                  MCA if: linux_ipv6 (MCA v2.1.0, API v2.0.0, Component
                          v4.1.1)
                  MCA if: posix_ipv4 (MCA v2.1.0, API v2.0.0, Component
                          v4.1.1)
         MCA installdirs: env (MCA v2.1.0, API v2.0.0, Component v4.1.1)
         MCA installdirs: config (MCA v2.1.0, API v2.0.0, Component v4.1.1)
              MCA memory: patcher (MCA v2.1.0, API v2.0.0, Component v4.1.1)
               MCA mpool: hugepage (MCA v2.1.0, API v3.0.0, Component v4.1.1)
             MCA patcher: overwrite (MCA v2.1.0, API v1.0.0, Component
                          v4.1.1)
                MCA pmix: isolated (MCA v2.1.0, API v2.0.0, Component v4.1.1)
                MCA pmix: flux (MCA v2.1.0, API v2.0.0, Component v4.1.1)
                MCA pmix: pmix3x (MCA v2.1.0, API v2.0.0, Component v4.1.1)
               MCA pstat: linux (MCA v2.1.0, API v2.0.0, Component v4.1.1)
              MCA rcache: grdma (MCA v2.1.0, API v3.3.0, Component v4.1.1)
              MCA rcache: gpusm (MCA v2.1.0, API v3.3.0, Component v4.1.1)
              MCA rcache: rgpusm (MCA v2.1.0, API v3.3.0, Component v4.1.1)
           MCA reachable: weighted (MCA v2.1.0, API v2.0.0, Component v4.1.1)
               MCA shmem: mmap (MCA v2.1.0, API v2.0.0, Component v4.1.1)
               MCA shmem: posix (MCA v2.1.0, API v2.0.0, Component v4.1.1)
               MCA shmem: sysv (MCA v2.1.0, API v2.0.0, Component v4.1.1)
               MCA timer: linux (MCA v2.1.0, API v2.0.0, Component v4.1.1)
              MCA errmgr: default_app (MCA v2.1.0, API v3.0.0, Component
                          v4.1.1)
              MCA errmgr: default_hnp (MCA v2.1.0, API v3.0.0, Component
                          v4.1.1)
              MCA errmgr: default_orted (MCA v2.1.0, API v3.0.0, Component
                          v4.1.1)
              MCA errmgr: default_tool (MCA v2.1.0, API v3.0.0, Component
                          v4.1.1)
                 MCA ess: env (MCA v2.1.0, API v3.0.0, Component v4.1.1)
                 MCA ess: hnp (MCA v2.1.0, API v3.0.0, Component v4.1.1)
                 MCA ess: pmi (MCA v2.1.0, API v3.0.0, Component v4.1.1)
                 MCA ess: singleton (MCA v2.1.0, API v3.0.0, Component
                          v4.1.1)
                 MCA ess: tool (MCA v2.1.0, API v3.0.0, Component v4.1.1)
                 MCA ess: slurm (MCA v2.1.0, API v3.0.0, Component v4.1.1)
               MCA filem: raw (MCA v2.1.0, API v2.0.0, Component v4.1.1)
             MCA grpcomm: direct (MCA v2.1.0, API v3.0.0, Component v4.1.1)
                 MCA iof: hnp (MCA v2.1.0, API v2.0.0, Component v4.1.1)
                 MCA iof: orted (MCA v2.1.0, API v2.0.0, Component v4.1.1)
                 MCA iof: tool (MCA v2.1.0, API v2.0.0, Component v4.1.1)
                MCA odls: default (MCA v2.1.0, API v2.0.0, Component v4.1.1)
                MCA odls: pspawn (MCA v2.1.0, API v2.0.0, Component v4.1.1)
                 MCA oob: tcp (MCA v2.1.0, API v2.0.0, Component v4.1.1)
                 MCA plm: isolated (MCA v2.1.0, API v2.0.0, Component v4.1.1)
                 MCA plm: rsh (MCA v2.1.0, API v2.0.0, Component v4.1.1)
                 MCA plm: slurm (MCA v2.1.0, API v2.0.0, Component v4.1.1)
                 MCA ras: simulator (MCA v2.1.0, API v2.0.0, Component
                          v4.1.1)
                 MCA ras: slurm (MCA v2.1.0, API v2.0.0, Component v4.1.1)
                MCA regx: fwd (MCA v2.1.0, API v1.0.0, Component v4.1.1)
                MCA regx: naive (MCA v2.1.0, API v1.0.0, Component v4.1.1)
                MCA regx: reverse (MCA v2.1.0, API v1.0.0, Component v4.1.1)
               MCA rmaps: mindist (MCA v2.1.0, API v2.0.0, Component v4.1.1)
               MCA rmaps: ppr (MCA v2.1.0, API v2.0.0, Component v4.1.1)
               MCA rmaps: rank_file (MCA v2.1.0, API v2.0.0, Component
                          v4.1.1)
               MCA rmaps: resilient (MCA v2.1.0, API v2.0.0, Component
                          v4.1.1)
               MCA rmaps: round_robin (MCA v2.1.0, API v2.0.0, Component
                          v4.1.1)
               MCA rmaps: seq (MCA v2.1.0, API v2.0.0, Component v4.1.1)
                 MCA rml: oob (MCA v2.1.0, API v3.0.0, Component v4.1.1)
              MCA routed: binomial (MCA v2.1.0, API v3.0.0, Component v4.1.1)
              MCA routed: direct (MCA v2.1.0, API v3.0.0, Component v4.1.1)
              MCA routed: radix (MCA v2.1.0, API v3.0.0, Component v4.1.1)
                 MCA rtc: hwloc (MCA v2.1.0, API v1.0.0, Component v4.1.1)
              MCA schizo: flux (MCA v2.1.0, API v1.0.0, Component v4.1.1)
              MCA schizo: ompi (MCA v2.1.0, API v1.0.0, Component v4.1.1)
              MCA schizo: orte (MCA v2.1.0, API v1.0.0, Component v4.1.1)
              MCA schizo: jsm (MCA v2.1.0, API v1.0.0, Component v4.1.1)
              MCA schizo: singularity (MCA v2.1.0, API v1.0.0, Component
                          v4.1.1)
              MCA schizo: slurm (MCA v2.1.0, API v1.0.0, Component v4.1.1)
               MCA state: app (MCA v2.1.0, API v1.0.0, Component v4.1.1)
               MCA state: hnp (MCA v2.1.0, API v1.0.0, Component v4.1.1)
               MCA state: novm (MCA v2.1.0, API v1.0.0, Component v4.1.1)
               MCA state: orted (MCA v2.1.0, API v1.0.0, Component v4.1.1)
               MCA state: tool (MCA v2.1.0, API v1.0.0, Component v4.1.1)
                 MCA bml: r2 (MCA v2.1.0, API v2.0.0, Component v4.1.1)
                MCA coll: adapt (MCA v2.1.0, API v2.0.0, Component v4.1.1)
                MCA coll: basic (MCA v2.1.0, API v2.0.0, Component v4.1.1)
                MCA coll: han (MCA v2.1.0, API v2.0.0, Component v4.1.1)
                MCA coll: inter (MCA v2.1.0, API v2.0.0, Component v4.1.1)
                MCA coll: libnbc (MCA v2.1.0, API v2.0.0, Component v4.1.1)
                MCA coll: self (MCA v2.1.0, API v2.0.0, Component v4.1.1)
                MCA coll: sm (MCA v2.1.0, API v2.0.0, Component v4.1.1)
                MCA coll: sync (MCA v2.1.0, API v2.0.0, Component v4.1.1)
                MCA coll: tuned (MCA v2.1.0, API v2.0.0, Component v4.1.1)
                MCA coll: cuda (MCA v2.1.0, API v2.0.0, Component v4.1.1)
                MCA coll: monitoring (MCA v2.1.0, API v2.0.0, Component
                          v4.1.1)
                MCA fbtl: posix (MCA v2.1.0, API v2.0.0, Component v4.1.1)
               MCA fcoll: dynamic (MCA v2.1.0, API v2.0.0, Component v4.1.1)
               MCA fcoll: dynamic_gen2 (MCA v2.1.0, API v2.0.0, Component
                          v4.1.1)
               MCA fcoll: individual (MCA v2.1.0, API v2.0.0, Component
                          v4.1.1)
               MCA fcoll: two_phase (MCA v2.1.0, API v2.0.0, Component
                          v4.1.1)
               MCA fcoll: vulcan (MCA v2.1.0, API v2.0.0, Component v4.1.1)
                  MCA fs: ufs (MCA v2.1.0, API v2.0.0, Component v4.1.1)
                  MCA io: ompio (MCA v2.1.0, API v2.0.0, Component v4.1.1)
                  MCA io: romio321 (MCA v2.1.0, API v2.0.0, Component v4.1.1)
                  MCA op: avx (MCA v2.1.0, API v1.0.0, Component v4.1.1)
                 MCA osc: sm (MCA v2.1.0, API v3.0.0, Component v4.1.1)
                 MCA osc: monitoring (MCA v2.1.0, API v3.0.0, Component
                          v4.1.1)
                 MCA osc: pt2pt (MCA v2.1.0, API v3.0.0, Component v4.1.1)
                 MCA osc: rdma (MCA v2.1.0, API v3.0.0, Component v4.1.1)
                 MCA osc: ucx (MCA v2.1.0, API v3.0.0, Component v4.1.1)
                 MCA pml: v (MCA v2.1.0, API v2.0.0, Component v4.1.1)
                 MCA pml: cm (MCA v2.1.0, API v2.0.0, Component v4.1.1)
                 MCA pml: monitoring (MCA v2.1.0, API v2.0.0, Component
                          v4.1.1)
                 MCA pml: ob1 (MCA v2.1.0, API v2.0.0, Component v4.1.1)
                 MCA pml: ucx (MCA v2.1.0, API v2.0.0, Component v4.1.1)
                 MCA rte: orte (MCA v2.1.0, API v2.0.0, Component v4.1.1)
            MCA sharedfp: individual (MCA v2.1.0, API v2.0.0, Component
                          v4.1.1)
            MCA sharedfp: lockedfile (MCA v2.1.0, API v2.0.0, Component
                          v4.1.1)
            MCA sharedfp: sm (MCA v2.1.0, API v2.0.0, Component v4.1.1)
                MCA topo: basic (MCA v2.1.0, API v2.2.0, Component v4.1.1)
                MCA topo: treematch (MCA v2.1.0, API v2.2.0, Component
                          v4.1.1)
           MCA vprotocol: pessimist (MCA v2.1.0, API v2.0.0, Component
                          v4.1.1)
  • Output of ucx_info -d to show transports and devices recognized by UCX
ucx_info -d
#
# Memory domain: self
#     Component: self
#             register: unlimited, cost: 0 nsec
#           remote key: 0 bytes
#           rkey_ptr is supported
#         memory types: host (access,reg_nonblock,reg,cache)
#
#      Transport: self
#         Device: memory
#           Type: loopback
#  System device: <unknown>
#
#      capabilities:
#            bandwidth: 0.00/ppn + 19360.00 MB/sec
#              latency: 0 nsec
#             overhead: 10 nsec
#            put_short: <= 4294967295
#            put_bcopy: unlimited
#            get_bcopy: unlimited
#             am_short: <= 8K
#             am_bcopy: <= 8K
#               domain: cpu
#           atomic_add: 32, 64 bit
#           atomic_and: 32, 64 bit
#            atomic_or: 32, 64 bit
#           atomic_xor: 32, 64 bit
#          atomic_fadd: 32, 64 bit
#          atomic_fand: 32, 64 bit
#           atomic_for: 32, 64 bit
#          atomic_fxor: 32, 64 bit
#          atomic_swap: 32, 64 bit
#         atomic_cswap: 32, 64 bit
#           connection: to iface
#      device priority: 0
#     device num paths: 1
#              max eps: inf
#       device address: 0 bytes
#        iface address: 8 bytes
#       error handling: ep_check
#   device mem_element: 0 bytes
#
#
# Memory domain: tcp
#     Component: tcp
#         memory types: 
#
#      Transport: tcp
#         Device: enp194s0f0np0
#           Type: network
#  System device: enp194s0f0np0 (0)
#
#      capabilities:
#            bandwidth: 2200.00/ppn + 0.00 MB/sec
#              latency: 5206 nsec
#             overhead: 50000 nsec
#            put_zcopy: <= 18446744073709551590, up to 6 iov
#  put_opt_zcopy_align: <= 1
#        put_align_mtu: <= 0
#             am_short: <= 8K
#             am_bcopy: <= 8K
#             am_zcopy: <= 64K, up to 6 iov
#   am_opt_zcopy_align: <= 1
#         am_align_mtu: <= 0
#            am header: <= 8037
#           connection: to ep, to iface
#      device priority: 0
#     device num paths: 1
#              max eps: 256
#       device address: 6 bytes
#        iface address: 2 bytes
#           ep address: 10 bytes
#       error handling: peer failure, ep_check, keepalive
#   device mem_element: 0 bytes
#
#      Transport: tcp
#         Device: enp3s0f4u1u2c2
#           Type: network
#  System device: <unknown>
#
#      capabilities:
#            bandwidth: 11.32/ppn + 0.00 MB/sec
#              latency: 10960 nsec
#             overhead: 50000 nsec
#            put_zcopy: <= 18446744073709551590, up to 6 iov
#  put_opt_zcopy_align: <= 1
#        put_align_mtu: <= 0
#             am_short: <= 8K
#             am_bcopy: <= 8K
#             am_zcopy: <= 64K, up to 6 iov
#   am_opt_zcopy_align: <= 1
#         am_align_mtu: <= 0
#            am header: <= 8037
#           connection: to ep, to iface
#      device priority: 1
#     device num paths: 1
#              max eps: 256
#       device address: 6 bytes
#        iface address: 2 bytes
#           ep address: 10 bytes
#       error handling: peer failure, ep_check, keepalive
#   device mem_element: 0 bytes
#
#      Transport: tcp
#         Device: lo
#           Type: network
#  System device: <unknown>
#
#      capabilities:
#            bandwidth: 11.91/ppn + 0.00 MB/sec
#              latency: 10960 nsec
#             overhead: 50000 nsec
#            put_zcopy: <= 18446744073709551590, up to 6 iov
#  put_opt_zcopy_align: <= 1
#        put_align_mtu: <= 0
#             am_short: <= 8K
#             am_bcopy: <= 8K
#             am_zcopy: <= 64K, up to 6 iov
#   am_opt_zcopy_align: <= 1
#         am_align_mtu: <= 0
#            am header: <= 8037
#           connection: to ep, to iface
#      device priority: 1
#     device num paths: 1
#              max eps: 256
#       device address: 18 bytes
#        iface address: 2 bytes
#           ep address: 10 bytes
#       error handling: peer failure, ep_check, keepalive
#   device mem_element: 0 bytes
#
#      Transport: tcp
#         Device: tunl0
#           Type: network
#  System device: <unknown>
#
#      capabilities:
#            bandwidth: 11.60/ppn + 0.00 MB/sec
#              latency: 10960 nsec
#             overhead: 50000 nsec
#            put_zcopy: <= 18446744073709551590, up to 6 iov
#  put_opt_zcopy_align: <= 1
#        put_align_mtu: <= 0
#             am_short: <= 8K
#             am_bcopy: <= 8K
#             am_zcopy: <= 64K, up to 6 iov
#   am_opt_zcopy_align: <= 1
#         am_align_mtu: <= 0
#            am header: <= 8037
#           connection: to ep, to iface
#      device priority: 1
#     device num paths: 1
#              max eps: 256
#       device address: 6 bytes
#        iface address: 2 bytes
#           ep address: 10 bytes
#       error handling: peer failure, ep_check, keepalive
#   device mem_element: 0 bytes
#
#
# Connection manager: tcp
#      max_conn_priv: 2064 bytes
#
# Memory domain: sysv
#     Component: sysv
#             allocate: unlimited
#           remote key: 12 bytes
#           rkey_ptr is supported
#         memory types: host (access,alloc,cache)
#
#      Transport: sysv
#         Device: memory
#           Type: intra-node
#  System device: <unknown>
#
#      capabilities:
#            bandwidth: 0.00/ppn + 15360.00 MB/sec
#              latency: 80 nsec
#             overhead: 10 nsec
#            put_short: <= 4294967295
#            put_bcopy: unlimited
#            get_bcopy: unlimited
#             am_short: <= 100
#             am_bcopy: <= 8256
#               domain: cpu
#           atomic_add: 32, 64 bit
#           atomic_and: 32, 64 bit
#            atomic_or: 32, 64 bit
#           atomic_xor: 32, 64 bit
#          atomic_fadd: 32, 64 bit
#          atomic_fand: 32, 64 bit
#           atomic_for: 32, 64 bit
#          atomic_fxor: 32, 64 bit
#          atomic_swap: 32, 64 bit
#         atomic_cswap: 32, 64 bit
#           connection: to iface
#      device priority: 0
#     device num paths: 1
#              max eps: inf
#       device address: 16 bytes
#        iface address: 8 bytes
#       error handling: ep_check
#   device mem_element: 0 bytes
#
#
# Memory domain: posix
#     Component: posix
#             allocate: <= 1406352400K
#           remote key: 32 bytes
#           rkey_ptr is supported
#         memory types: host (access,alloc,cache)
#
#      Transport: posix
#         Device: memory
#           Type: intra-node
#  System device: <unknown>
#
#      capabilities:
#            bandwidth: 0.00/ppn + 15360.00 MB/sec
#              latency: 80 nsec
#             overhead: 10 nsec
#            put_short: <= 4294967295
#            put_bcopy: unlimited
#            get_bcopy: unlimited
#             am_short: <= 100
#             am_bcopy: <= 8256
#               domain: cpu
#           atomic_add: 32, 64 bit
#           atomic_and: 32, 64 bit
#            atomic_or: 32, 64 bit
#           atomic_xor: 32, 64 bit
#          atomic_fadd: 32, 64 bit
#          atomic_fand: 32, 64 bit
#           atomic_for: 32, 64 bit
#          atomic_fxor: 32, 64 bit
#          atomic_swap: 32, 64 bit
#         atomic_cswap: 32, 64 bit
#           connection: to iface
#      device priority: 0
#     device num paths: 1
#              max eps: inf
#       device address: 16 bytes
#        iface address: 16 bytes
#       error handling: ep_check
#   device mem_element: 0 bytes
#
#
# Memory domain: cuda_cpy
#     Component: cuda_cpy
#             allocate: unlimited
#             register: unlimited, cost: 0 nsec
#         memory types: host (access,reg), cuda (access,alloc,reg,detect,dmabuf), cuda-managed (access,alloc,reg,cache,detect)
#
#      Transport: cuda_copy
#         Device: cuda
#           Type: accelerator
#  System device: <unknown>
#
#      capabilities:
#            bandwidth: 10000.00/ppn + 0.00 MB/sec
#              latency: 8000 nsec
#             overhead: 0 nsec
#            put_short: <= 4294967295
#            put_zcopy: unlimited, up to 1 iov
#  put_opt_zcopy_align: <= 1
#        put_align_mtu: <= 1
#            get_short: <= 4294967295
#            get_zcopy: unlimited, up to 1 iov
#  get_opt_zcopy_align: <= 1
#        get_align_mtu: <= 1
#           connection: to iface
#      device priority: 0
#     device num paths: 1
#              max eps: inf
#       device address: 0 bytes
#        iface address: 8 bytes
#       error handling: none
#   device mem_element: 0 bytes
#
#
# Memory domain: cuda_ipc
#     Component: cuda_ipc
#             register: unlimited, cost: 0 nsec
#           remote key: 192 bytes
#           memory invalidation is supported
#         memory types: cuda (access,reg,cache)
#
#      Transport: cuda_ipc
#         Device: cuda
#           Type: intra-node
#  System device: <unknown>
#
#      capabilities:
#            bandwidth: 400000.00/ppn + 0.00 MB/sec
#              latency: 1000 nsec
#             overhead: 7000 nsec
#            put_zcopy: unlimited, up to 1 iov
#  put_opt_zcopy_align: <= 1
#        put_align_mtu: <= 1
#            get_zcopy: <= 0, up to 1 iov
#  get_opt_zcopy_align: <= 1
#        get_align_mtu: <= 1
#           connection: to iface
#      device priority: 0
#     device num paths: 1
#              max eps: inf
#       device address: 8 bytes
#        iface address: 4 bytes
#       error handling: peer failure
#   device mem_element: 8 bytes
#
#
# Memory domain: gdr_copy
#     Component: gdr_copy
#             register: unlimited, cost: 50 nsec
#           remote key: 24 bytes
#         memory types: cuda (access,reg)
#
#      Transport: gdr_copy
#         Device: cuda
#           Type: accelerator
#  System device: <unknown>
#
#      capabilities:
#            bandwidth: 6911.00/ppn + 250.00 MB/sec
#              latency: 400 nsec
#             overhead: 0 nsec
#            put_short: <= 4294967295
#            get_short: <= 4294967295
#           connection: to iface
#      device priority: 0
#     device num paths: 1
#              max eps: inf
#       device address: 0 bytes
#        iface address: 8 bytes
#       error handling: none
#   device mem_element: 0 bytes
#
#
# Memory domain: bnxt_re0
#     Component: ib
#             register: unlimited, dmabuf, cost: 16000 + 0.060 * N nsec
#           remote key: 8 bytes
#           local memory handle is required for zcopy
#         memory types: host (access,reg,cache)
#
#      Transport: rc_verbs
#         Device: bnxt_re0:1
#           Type: network
#  System device: bnxt_re0 (0)
#
#      capabilities:
#            bandwidth: 10957.84/ppn + 0.00 MB/sec
#              latency: 800 + 1.000 * N nsec
#             overhead: 75 nsec
#            put_short: <= 96
#            put_bcopy: <= 8256
#            put_zcopy: <= 1G, up to 5 iov
#  put_opt_zcopy_align: <= 512
#        put_align_mtu: <= 1K
#            get_bcopy: <= 8256
#            get_zcopy: 65..1G, up to 5 iov
#  get_opt_zcopy_align: <= 512
#        get_align_mtu: <= 1K
#             am_short: <= 95
#             am_bcopy: <= 8255
#             am_zcopy: <= 8255, up to 4 iov
#   am_opt_zcopy_align: <= 512
#         am_align_mtu: <= 1K
#            am header: <= 127
#               domain: cpu
#           atomic_add: 64 bit
#          atomic_fadd: 64 bit
#         atomic_cswap: 64 bit
#           connection: to ep
#      device priority: 0
#     device num paths: 1
#              max eps: 256
#       device address: 18 bytes
#           ep address: 4 bytes
#       error handling: peer failure, ep_check
#   device mem_element: 0 bytes
#
#
#      Transport: ud_verbs
#         Device: bnxt_re0:1
#           Type: network
#  System device: bnxt_re0 (0)
#
#      capabilities:
#            bandwidth: 10957.84/ppn + 0.00 MB/sec
#              latency: 830 nsec
#             overhead: 105 nsec
#             am_short: <= 88
#             am_bcopy: <= 1016
#             am_zcopy: <= 1016, up to 5 iov
#   am_opt_zcopy_align: <= 512
#         am_align_mtu: <= 1K
#            am header: <= 920
#           connection: to ep, to iface
#      device priority: 0
#     device num paths: 1
#              max eps: inf
#       device address: 18 bytes
#        iface address: 3 bytes
#           ep address: 6 bytes
#       error handling: peer failure, ep_check
#   device mem_element: 0 bytes
#
#
# Memory domain: bnxt_re1
#     Component: ib
#             register: unlimited, dmabuf, cost: 16000 + 0.060 * N nsec
#           remote key: 8 bytes
#           local memory handle is required for zcopy
#         memory types: host (access,reg,cache)
#   < no supported devices found >
#
# Connection manager: rdmacm
#      max_conn_priv: 54 bytes
#
# Memory domain: cma
#     Component: cma
#         memory types: 
#
#      Transport: cma
#         Device: memory
#           Type: intra-node
#  System device: <unknown>
#
#      capabilities:
#            bandwidth: 0.00/ppn + 11145.00 MB/sec
#              latency: 80 nsec
#             overhead: 2000 nsec
#            put_zcopy: unlimited, up to 16 iov
#  put_opt_zcopy_align: <= 1
#        put_align_mtu: <= 1
#            get_zcopy: unlimited, up to 16 iov
#  get_opt_zcopy_align: <= 1
#        get_align_mtu: <= 1
#           connection: to iface
#      device priority: 0
#     device num paths: 1
#              max eps: inf
#       device address: 16 bytes
#        iface address: 16 bytes
#       error handling: peer failure, ep_check
#   device mem_element: 0 bytes
#
  • Configure result - config.log
  • Log file - configure UCX with "--enable-logging" - and run with "UCX_LOG_LEVEL=data"

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions