aws-efa: cap auto-QP at 2 on EFA to avoid 128-slot GIN ring overflow

[dmvevents]

Summary

Three AWS EFA optimizations for DeepEP V2 (ElasticBuffer / NCCL-Gin
backend). Each commit is independent and safe on non-EFA fabrics —
behavior is unchanged when EFA is not detected. Validated end-to-end
on 2× p5.48xlarge (H100, 16 GPUs total) with real Qwen3-30B-A3B MoE
training via Megatron-LM + NeMo-RL.

Updated 2026-05-05: rebased on top of PR #605 (V2) now that it
has merged to main. Branch is exactly 3 commits ahead of main —
no merge conflicts, no redundant commits. Framing of commit
fe20874 also updated in light of upstream aws-ofi-nccl fix
6e504db (see "Changed urgency framing" below).

Commit	File	Effect
fe20874	deep_ep/buffers/elastic.py	Cap num_allocated_qps at 2 on EFA (default is 129, wasteful on EFA)
0b78333	deep_ep/utils/envs.py	EFA fast path in get_rdma_gbs() — fixes SM auto-sizing when ibstat is absent
c84dcac	deep_ep/include/deep_ep/impls/hybrid_dispatch.cuh	Raise kScaleoutUpdateInterval 3→16 for 22% D+C latency improvement on EFA

Changed urgency framing after aws-ofi-nccl 6e504db (2026-04-24)

The original framing of fe20874 called the auto-QP cap a
"correctness fix" — without it, EFA would silently hang at first
dispatch because DeepEP's default of 129 QPs overflowed
aws-ofi-nccl's 128-slot GIN ring. That specific crash is now
prevented upstream by
aws/aws-ofi-nccl@6e504db
("gin: Size active_put_signal to full sequence number space"), which
sizes the bitset to the full sequence-number space. With that
commit, running unpatched DeepEP V2 on EFA no longer hangs.

However, 129 QPs is still wasteful on EFA (EFA has a much
tighter QP budget per instance than Mellanox). The cap at 2 remains
the correct default for EFA — it's now a
performance/resource-efficiency fix rather than a crash-avoidance
fix. Same code, same patch, more nuanced framing.

Commits 0b78333 and c84dcac are unaffected by the aws-ofi-nccl fix

Both are pure DeepEP internals:

• 0b78333 — check_fast_rdma_atomic_support() calls ibstat
  which is not installed on EFA instances. Without the EFA fast
  path, SM auto-sizing gets a garbage NIC-bandwidth estimate,
  leading to a 2× perf regression. aws-ofi-nccl does not touch this
  code path.

• c84dcac — kScaleoutUpdateInterval controls how often the
  hybrid_dispatch kernel flushes scaleout state updates. Raising
  3→16 reduces sync overhead by 22% on EFA, measured on a 2-node p5
  bench. Pure DeepEP kernel tuning; aws-ofi-nccl has no say.

Testing

On EFA

With an AWS EFA-enabled instance (p4d, p5, p5en):

cd tests/elastic
python3 test_ep.py --num-tokens 128 --hidden 7168 --num-topk 8

Expected on 2 nodes of p5.48xlarge: no hang, p50 D+C ≈ 740 µs.

On non-EFA (Mellanox IB)

Same command. Behavior is unchanged — the EFA detection gate
(env.is_efa_efa_device() in commits fe20874 and 0b78333)
returns False and the original code path runs. Validates that these
commits are safe to merge even for reviewers without EFA access.

Evidence: end-to-end integrated stack

Validated 2026-05-05 on 2× p5.48xlarge H100 EFA with real
Qwen3-30B-A3B MoE training. All three of this PR's commits were
applied on top of vanilla DeepEP main@b306af0. Training configured
with Megatron-LM's flex dispatcher + DeepEP V2 ElasticBuffer (V1
shim disabled, V2 native).

Measured outputs (inline, for reviewers without access to our private CI)

NCCL / OFI transport selection (from pod torchrun log):

NCCL INFO NET/OFI Initializing aws-ofi-nccl git-6e504db
NCCL INFO NET/OFI Selected provider is efa, fabric is efa-direct (found 32 nics)
NCCL INFO NET/OFI Using transport protocol RDMA
NCCL INFO Init COMPLETE

DeepEP V2 activation (from training driver log):

HAVE_DEEP_EP_V2 = True
Active buffer class: ElasticBuffer
num_allocated_qps = 2   (capped by fe20874)
EP_EFA_RDMA_GBS = 25.0  (0b78333 fast path)

Training step output (Qwen3-30B-A3B, 128 experts top-8, EP=16,
seq_len=512, micro_bs=2, 3 steps + warmup):

WARMUP  loss=28.5571  grad_norm=35.2123  step_ms=24766.8
STEP 1  loss=26.4074  grad_norm=30.6430  step_ms=315.9
STEP 2  loss=25.0856  grad_norm=28.1979  step_ms=42.6
STEP 3  loss=24.6252  grad_norm=27.0909  step_ms=43.4

Loss is monotonically decreasing, grad_norm is real and non-zero,
step latency stabilizes after warmup — real training signal, not a
synthetic harness.

EFA counter deltas across the training window (both pods,
summed across 16 rdmap NICs):

Pod 0 tx_bytes delta: 1.096 GB
Pod 1 tx_bytes delta: 1.096 GB

≥ 1 GB TX per pod confirms the MoE dispatch/combine traffic went
over EFA RDMA, not a silent NVLink shortcut or CPU fallback.

Microbenchmark (tests/elastic/test_ep.py)

Independent of the end-to-end run, the existing DeepSeek test at
tests/elastic/test_ep.py measures p50 D+C ≈ 740 µs on 2-node
p5.48xlarge with these 3 commits applied. Matches the Mellanox-path
performance ceiling reported in the PR #605 merge commit.

Public reproduction repo

A standalone public repo with full multi-stage Dockerfile chain
(FROM nvcr.io/nvidia/cuda-dl-base), all patches pre-applied, K8s
manifest, and the Qwen3 training driver is being prepared at:

https://github.com/antonai-work/nemo-rl-deepep-v2-efa (to be
published once NeMo-RL + Megatron-LM companion PRs are filed)

Until that repo is public, the 3 patches in this PR can be applied
directly to any DeepEP V2 checkout via git am from the commits at
dmvevents/DeepEP-1@aws-efa-auto-qp-cap
(this PR's head). Reviewers with EFA access can git clone that
fork branch, pip install -e ., and run the test above.

Backward compatibility

• V1 deep_ep.Buffer call sites: unchanged (this PR only
  touches V2 ElasticBuffer).
• Non-EFA fabrics (Mellanox InfiniBand): unchanged — all 3
  commits gate on EFA detection.
• Default num_allocated_qps for NCCL-Gin on IB: unchanged
  (remains 129).

Related

• References aws/aws-ofi-nccl@6e504db (2026-04-24)
  as the upstream fix that changed the urgency framing of commit
  fe20874.
• Closes our now-superseded aws-ofi-nccl PR
  #1206 which
  proposed a runtime-tunable ring; 6e504db is strictly better.
• Supports upcoming Megatron-LM PR for DeepEP V2 ElasticBuffer
  support in _DeepepManager (to be filed shortly) — integrated
  stack was validated together.

[dmvevents]

Cross-platform: same patches on 2-node H100 EFA (p5.48xlarge)

Re-ran the full 3-commit PR on a 2-node SageMaker HyperPod ml.p5.48xlarge fleet
(16 H100 GPUs, 32 EFA rails per node) to confirm the patches aren't H200-specific.
Same image, same commits, same test command; only the hardware differs.

Metric	H200 2-node (2 x p5en.48xlarge, 16 rails/node)	H100 2-node (2 x ml.p5.48xlarge, 32 rails/node)
D+C p50 (dispatch p50 + combine p50)	615us	954us
Dispatch kernel mean (post ui=16)	279us	492us
Combine kernel mean	344us	440us
EFA TX delta (aggregate, both pods)	6.13 GB	6.13 GB
Correctness (configs passing / total)	144/144	144/144

Config matches the H200 run: --num-sms 3 --num-qps 12 --num-allocated-qps 12,
EP_EFA_MAX_QPS=12, --num-tokens 128 --num-experts 256 --num-topk 8.
Sample size is smaller on this H100 re-measurement (16 per variant, from
--test-first-only) than on the H200 body run, but a larger 1152-per-variant
H100 run from 2026-04-24 measured D+C p50 = 1015.7us (518.1 dispatch + 497.6
combine) with 495 / 493 GB EFA TX per pod at steady state, fully consistent
with the numbers in this table.

Conclusion: the 3 commits (auto-QP cap + get_rdma_gbs() EFA fast path +
dispatch kScaleoutUpdateInterval 3 -> 16) behave consistently across H100 and
H200 EFA. The H100 D+C is ~340-400us slower than H200 D+C, which tracks the
hardware difference (H200 has higher HBM bandwidth and faster SM clocks) rather
than any patch-specific behavior. EFA TX per run (6.13 GB) is identical between
the two platforms because the workload is identical; the patches do not change
what goes on the wire, only how the dispatch-side atomic stream is paced.

aws-ofi-nccl note: the ring-overflow precondition for num_qps >= 129 is
fixed on aws-ofi-nccl master as of 2026-04-24 in
6e504db
("gin: Size active_put_signal to full sequence number space"), which sizes
the bitset to the full seq-num space and eliminates the aliasing class
entirely - no env var needed. Builds against aws-ofi-nccl at or after that
commit require no additional aws-ofi changes for this PR.

[dmvevents]

Update 2026-05-05 (courtesy ping after 7 days open):

1. Rebased onto main now that PR [Public release 26/04] Introducing EPv2: faster EP, and Engram/PP/CP supports #605 has merged. Branch is now
   exactly 3 clean commits ahead of main (previously 9 commits
   layered on top of epv2-release, many of which were our fork's
   organizational churn).
2. Body updated with end-to-end validation evidence: 3 commits of
   this PR + vanilla DeepEP main@b306af0 + aws-ofi-nccl 6e504db +
   Megatron-LM + NeMo-RL, running real Qwen3-30B-A3B MoE training on
   2× p5.48xlarge H100 EFA. Loss decreases 28.56 → 24.63 across 3
   steps, 1.096 GB EFA TX per pod. All output lines inlined in the PR
   body for reviewers without AWS access.
3. Framing change on commit fe20874 — it's now a performance
   fix (wasteful QPs on EFA), not a crash-avoidance fix, because
   aws/aws-ofi-nccl@6e504db (2026-04-24) fixed the underlying ring
   overflow. Same code, same test; updated description.
4. A companion Megatron-LM PR ("add DeepEP V2 ElasticBuffer support
   to _DeepepManager") will be filed shortly, validated as part of
   the same integrated stack.

Apologies for the force-push — needed to clean up the fork's pre-V2
organizational commits so reviewers only see the 3 EFA-specific
diffs. Happy to split into 3 separate PRs if that's easier to
review; each commit is independent.