fix(docker): set LD_LIBRARY_PATH for AWS EFA OFI plugin discovery

[ko3n1g]

What

Adds /opt/amazon/ofi-nccl/lib and /opt/amazon/efa/lib to LD_LIBRARY_PATH in the release stage of both docker/Dockerfile and docker/Dockerfile.ngc_pytorch, and registers them in /etc/ld.so.conf.d/aws-efa.conf so a runtime ldconfig refreshes the cache once host paths are mounted.

Why

Without these on the dynamic-linker search path, NCCL silently falls back to its socket transport on EFA-enabled AWS hosts (p4d / p5 / p5en) instead of routing through the SRD fabric. Same intent as #2410, but with the correct install path for the current aws-efa-installer layout.

Why a new PR rather than amending #2410

#2410 prepends /opt/amazon/aws-ofi-nccl/lib — that directory does not exist on a stock current aws-efa-installer install. The OFI plugin actually lives at /opt/amazon/ofi-nccl/lib:

ls /opt/amazon
bin  efa  efa_installed_packages  modules  ofi-nccl  openmpi  openmpi5  pmix  prrte

ls /opt/amazon/ofi-nccl/lib
libnccl-net.so  libnccl-net-ofi.so  libnccl-ofi-tuner.so  libnccl-tuner-ofi.so

So #2410 as-is is inert on these hosts.

Verification

Built an image carrying #2410's change (nvcr.io/nvidian/nemo-rl:testing-2410-cuda) and ran it on a p5.48xlarge EFA node with /opt/amazon hostPath-mounted:

Test	Result
echo $LD_LIBRARY_PATH (image-baked)	/opt/amazon/aws-ofi-nccl/lib:/opt/amazon/efa/lib:...
ls /opt/amazon/aws-ofi-nccl/lib	No such file or directory
ldconfig -p | grep libnccl-net	only HPCX libs, no libnccl-net-ofi.so

Repeated with the corrected path (LD_LIBRARY_PATH=/opt/amazon/ofi-nccl/lib:... plus a /etc/ld.so.conf.d entry):

Test	Result
ld.so --library-path "$LD_LIBRARY_PATH" --list .../libnccl-net-ofi.so	All deps resolve
After ldconfig: ldconfig -p | grep libnccl-net	libnccl-net-ofi.so → /opt/amazon/ofi-nccl/lib/libnccl-net-ofi.so

So the path correction lets NCCL's runtime dlopen("libnccl-net-ofi.so") resolve, which is what the original PR aimed for.

Note on the test methodology in #2410

ldconfig -p reads /etc/ld.so.cache, not LD_LIBRARY_PATH — so the validation step in #2410's description (exec pod0 -- ldconfig -p | grep libnccl-net) wouldn't reflect any ENV LD_LIBRARY_PATH change, with either the original or corrected path. The fix on this PR side: write the paths to /etc/ld.so.conf.d/aws-efa.conf too — then a ldconfig && ldconfig -p at runtime (after host mounts are in place) does show the plugin in the cache.

Related

• Refs deps: re-export LD_LIBRARY_PATH for AWS EFA OFI discovery #2410 — replaces it with the correct directory name and a proper test path.
• Refs NGC v0.5.0 container LD_LIBRARY_PATH missing /opt/amazon/aws-ofi-nccl/lib (NCCL falls to Socket on EFA) #1973 — same underlying issue.

[copy-pr-bot]

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

[copy-pr-bot]

Auto-sync is disabled for ready for review pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

[ko3n1g]

/ok to test