[Performance] support muon comm opt with ep >8 by AlAuAu · Pull Request #79448 · PaddlePaddle/Paddle

AlAuAu · 2026-07-09T12:13:43Z

PR Category

Performance Optimization

PR Types

Performance

Description

support muon comm opt with ep >8

是否引起精度变化

否

risemeup1111

发现了两个会影响 comm_group_call_opt 正确性的阻塞问题，具体证据和建议已放在行级评论中。请优先修复通信组构造相关问题后再继续评估。

^{Powered by Nyanpasu with gpt-5.5 xhigh, please check the suggestions carefully.}

risemeup1111 · 2026-07-09T12:24:27Z

+            assert (
+                ep_degree >= 8 and moe_sharding_degree > 1 and tp_degree == 1
+            ), (
+                "comm_group_call_opt should be enabled when ep_degree is larger then 8 and moe_sharding_degree is larger then 1"
            )


优先级：P1

问题： 这里放宽成 ep_degree >= 8 后，ep_degree=12 这类不是 8 的倍数的配置也会通过，但下面新建 trainer_same_card_comms 时使用 range(expert_parallel_degree // 8)，并且通信时用 card_idx = rank % 8。这些 rank 中超过 floor(ep/8)*8 的 EP rank 不会被加入任何 same-card group，后续 _coalescing_manager(trainer_same_card_group, ...) 会在非成员 group 上执行，容易直接报错或卡住。

影响： comm_group_call_opt 声明支持 ep > 8，但当前实际只支持每 8 卡一个 trainer 的整数倍 EP。

处理要求：请针对该评论修复并提交新的 commit。

建议把校验收紧到 8 的倍数，并同步更新错误信息：

Suggested change

assert (

ep_degree >= 8 and moe_sharding_degree > 1 and tp_degree == 1

), (

"comm_group_call_opt should be enabled when ep_degree is larger then 8 and moe_sharding_degree is larger then 1"

)

assert (

ep_degree >= 8

and ep_degree % 8 == 0

and moe_sharding_degree > 1

and tp_degree == 1

), (

"comm_group_call_opt should be enabled when ep_degree is at "

"least 8 and divisible by 8, moe_sharding_degree is larger "

"than 1, and tensor_parallel_degree is 1"

)

PaddlePaddle-bot · 2026-07-09T13:35:17Z

🤖 Paddle-CI-Agent | ci_status_monitor | 2026-07-16 08:33:41 UTC+08:00

CI报告基于以下代码生成（30分钟更新一次）:
PR commit: 50f1173 | Merge base: a842085 (branch: develop)

1 Required任务 : 47/48 通过

总执行（rerun次数）	总任务	✅ 通过	❌ 失败	⏳ 运行中	⏸️ 等待中	跳过
92(11)	81	80	1	0	0	0

任务	错误类型	置信度	日志
`Coverage test`	PR问题	高	Job

2 失败详情

🔴 Coverage test — PR问题（置信度: 高）

失败用例: 覆盖率门禁

用例	错误摘要
`python/paddle/distributed/fleet/meta_optimizers/muon_sharding_optimizer.py`	Python Diff Coverage 仅 6.7%，低于 90.0% 门禁

关键日志:

Assert Python Diff Coverage
expected >= 90.0 %, actual 6.7 %, failed
Coverage check failed, unit tests have all passed, please do not rerun,
check whether the newly added code lines are fully covered by unit tests.

根因摘要: 新增 muon 通信优化分支缺少单测覆盖

PR 本次只修改 python/paddle/distributed/fleet/meta_optimizers/muon_sharding_optimizer.py，Coverage 日志显示单测均已通过，但新增 Python diff 覆盖率只有 6.7%。新增逻辑集中在 get_gpus_per_node、动态 gpus_per_node 的 rank 计算、trainer_same_card_comms 构造，以及 reduce_gradients / _sharding_sync_parameters 的 group_call_opt reduce/broadcast 分支，当前测试没有覆盖这些新增路径，因此触发覆盖率门禁。

修复建议:

在 test/collective/fleet/test_parallel_dygraph_muon.py 或 test/collective/fleet/hybrid_parallel_sharding_muon_model.py 增加覆盖 comm_group_call_opt=True 的用例，至少覆盖 get_gpus_per_node、get_same_card_rank(..., gpus_per_node) 以及 trainer_same_card_comms 初始化逻辑。
对新增的 reduce_gradients 和 _sharding_sync_parameters 两段 group_call_opt 路径补充分布式/Mock 测试，覆盖 inner_dst、inner_src、同卡 group reduce/broadcast 等关键分支。
修复后重新触发 Coverage；日志明确提示该失败不是环境抖动，不建议单纯 rerun。

关联变更: muon_sharding_optimizer.py 行 72、96、213、450、849、1137 附近新增/修改了覆盖率未达标的主要逻辑。

codecov-commenter · 2026-07-09T15:39:35Z

Codecov Report

❌ Patch coverage is 6.66667% with 70 lines in your changes missing coverage. Please review.
⚠️ Please upload report for BASE (develop@a842085). Learn more about missing BASE report.

Files with missing lines	Patch %	Lines
...d/fleet/meta_optimizers/muon_sharding_optimizer.py	6.66%	70 Missing ⚠️

❌ Your patch status has failed because the patch coverage (6.66%) is below the target coverage (90.00%). You can increase the patch coverage or adjust the target coverage.

Additional details and impacted files

@@            Coverage Diff             @@
##             develop   #79448   +/-   ##
==========================================
  Coverage           ?    6.66%           
==========================================
  Files              ?        1           
  Lines              ?       75           
  Branches           ?        0           
==========================================
  Hits               ?        5           
  Misses             ?       70           
  Partials           ?        0

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

AlAuAu · 2026-07-10T08:24:44Z

/re-run all-failed

risemeup1111

已复查新提交。此前关于非 8 倍数 EP 的校验问题已修复；通信组仍按手写 rank 公式构造，和可配置的 MoE topology 顺序不兼容，当前仍是阻塞点。我已在原行级线程补充了当前 head 的复查证据。

CI 目前还有检查在运行中，后续仍需以最终 CI 结果为准。

^{Powered by Nyanpasu with gpt-5.5 xhigh, please check the suggestions carefully.}

risemeup1111

已复查这轮提交。前两轮的阻塞点这次都已落地修复，当前没有新的阻塞问题。这轮未新增 inline 评论，细节仍在前面的 inline review comments 里。

^{Powered by Nyanpasu with gpt-5.5 xhigh, please check the suggestions carefully.}

AlAuAu · 2026-07-14T06:24:39Z

/re-run all-failed

AlAuAu · 2026-07-14T09:04:14Z

/re-run all-failed

AlAuAu · 2026-07-14T13:37:57Z

/re-run all-failed

paddle-bot · 2026-07-14T14:57:21Z

你的PR提交成功，感谢你对开源项目的贡献!
请关注后续CI自动化测试结果，详情请参考Paddle-CI手册。
Your PR has been submitted. Thanks for your contribution!
Please wait for the result of CI firstly. See Paddle CI Manual for details.

risemeup1111

已复查最新提交。此前两个 P1 阻塞点仍保持已修复状态；这轮新增的本地卡数推导把未启用 comm_group_call_opt 的默认路径也绑定到了 CUDA/launcher 环境，已在行级评论中给出修复建议。请修复后再更新。

CI 当前仍有多项检查在运行，且 review-bot approval 相关检查尚未通过，后续以最终 CI 结果为准。

^{Powered by Nyanpasu with gpt-5.5 xhigh, please check the suggestions carefully.}

risemeup1111 · 2026-07-15T08:01:11Z

+        self._gpus_per_node = get_gpus_per_node()
        if self.use_group_call_opt:
-            assert ep_degree == 8 and moe_sharding_degree != 1, (
-                "comm_group_call_opt should be enabled when ep_degree is 8 and moe_sharding_degree is not 1"
+            assert (


优先级：P1

问题： get_gpus_per_node() 现在在 self.use_group_call_opt 判断前执行。comm_group_call_opt 默认是 false，且当前文件里 _gpus_per_node 只在该开关为 true 的分支使用；但 get_gpus_per_node() 在 PADDLE_LOCAL_SIZE 缺失且 CUDA 设备数为 0 时会触发 assert。这样即使用户没有启用该优化，CPU/XPU 或自定义 launcher 场景也会在构造 MuonShardingOptimizer 时失败。

影响： 这把一个可选通信优化变成了初始化期的 CUDA/launcher 环境依赖，属于默认配置下的兼容性回归。

处理要求：请针对该评论修复并提交新的 commit。

建议把本地卡数的解析延迟到开启 comm_group_call_opt 后：

Suggested change

self._gpus_per_node = get_gpus_per_node()

if self.use_group_call_opt:

assert ep_degree == 8 and moe_sharding_degree != 1, (

"comm_group_call_opt should be enabled when ep_degree is 8 and moe_sharding_degree is not 1"

assert (

self._gpus_per_node = None

if self.use_group_call_opt:

self._gpus_per_node = get_gpus_per_node()

assert (

risemeup1111

已复查最新提交，上轮 Nyanpasu 提出的 P1 兼容性问题已通过后续 commit 修复；此前两个 P1 阻塞点也仍保持已修复状态。当前未发现新的阻塞问题。

CI 仍有 clone/coverage clone 等检查在运行，且 review-bot approval 检查当前未通过，最终以 CI 和仓库必需检查结果为准。

^{Powered by Nyanpasu with gpt-5.5 xhigh, please check the suggestions carefully.}

AlAuAu · 2026-07-15T08:38:13Z

/re-run all-failed

PaddlePaddle-bot

🤖 Paddle-CI-Agent | pr_review | 2026-07-15 16:44:55

📋 Review 摘要

PR 概述：支持 MuonShardingOptimizer 的 comm_group_call_opt 按每节点 GPU 数拆分 ep>8 的机内/跨机同卡通信。
变更范围：python/paddle/distributed/fleet/meta_optimizers/muon_sharding_optimizer.py
影响面 Tag：[Distributed Strategy] [Communication Library] [Performance Optimization]

问题

未发现阻塞性问题。PR 规范问题在下面章节报，不要在这里重复。

历史 Findings 修复情况

Finding	问题	状态
F1	`get_gpus_per_node()` 曾在判断 `self.use_group_call_opt` 前无条件调用	✅ 已修复
F2	`gpus_per_node == 1` 曾仍可创建单 rank trainer group 并进入通信路径	✅ 已修复

📝 PR 规范检查

符合规范。

总体评价

本轮按分布式通信路径审查了新增 GPU-per-node 推导、trainer/same-card 进程组构造，以及 reduce/broadcast 分段逻辑；未发现需要拦截合入的问题。

AlAuAu · 2026-07-15T11:25:44Z

/re-run all-failed

sneaxiy

LGTM for coverage due to lack of 8 GPUs.

AlAuAu · 2026-07-16T03:04:35Z

/re-run all-failed

risemeup1111 suggested changes Jul 9, 2026

View reviewed changes

This comment was marked as outdated.

Sign in to view

AlAuAu force-pushed the support_comm_opt_epg8 branch from 4cbed80 to 586caf9 Compare July 13, 2026 14:40

risemeup1111 suggested changes Jul 13, 2026

View reviewed changes

This comment was marked as outdated.

Sign in to view

AlAuAu force-pushed the support_comm_opt_epg8 branch from 586caf9 to 2b1cc44 Compare July 14, 2026 03:38

risemeup1111 approved these changes Jul 14, 2026

View reviewed changes

AlAuAu force-pushed the support_comm_opt_epg8 branch from 2b1cc44 to fc408d3 Compare July 14, 2026 06:28

This comment was marked as outdated.

Sign in to view

AlAuAu force-pushed the support_comm_opt_epg8 branch from fc408d3 to 83dc876 Compare July 15, 2026 07:55

risemeup1111 suggested changes Jul 15, 2026

View reviewed changes

This comment was marked as outdated.

Sign in to view

[Performance] support muon comm opt with ep >8

50f1173

AlAuAu force-pushed the support_comm_opt_epg8 branch from 83dc876 to 50f1173 Compare July 15, 2026 08:32

risemeup1111 approved these changes Jul 15, 2026

View reviewed changes

PaddlePaddle-bot reviewed Jul 15, 2026

View reviewed changes

sneaxiy approved these changes Jul 16, 2026

View reviewed changes

swgu98 added the skip-ci: coverage label Jul 16, 2026

LiYuRio merged commit a35b002 into PaddlePaddle:develop Jul 16, 2026
104 of 109 checks passed

Uh oh!

Conversation

AlAuAu commented Jul 9, 2026

PR Category

PR Types

Description

是否引起精度变化

Uh oh!

risemeup1111 left a comment

Choose a reason for hiding this comment

Uh oh!

risemeup1111 Jul 9, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

This comment was marked as outdated.

Uh oh!

PaddlePaddle-bot commented Jul 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

1 Required任务 : 47/48 通过

2 失败详情

Uh oh!

codecov-commenter commented Jul 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

AlAuAu commented Jul 10, 2026

Uh oh!

risemeup1111 left a comment

Choose a reason for hiding this comment

Uh oh!

This comment was marked as outdated.

Uh oh!

risemeup1111 left a comment

Choose a reason for hiding this comment

Uh oh!

AlAuAu commented Jul 14, 2026

Uh oh!

This comment was marked as outdated.

Uh oh!

AlAuAu commented Jul 14, 2026

Uh oh!

AlAuAu commented Jul 14, 2026

Uh oh!

paddle-bot Bot commented Jul 14, 2026

Uh oh!

risemeup1111 left a comment

Choose a reason for hiding this comment

Uh oh!

risemeup1111 Jul 15, 2026

Choose a reason for hiding this comment

Uh oh!

This comment was marked as outdated.

Uh oh!

risemeup1111 left a comment

Choose a reason for hiding this comment

Uh oh!

AlAuAu commented Jul 15, 2026

Uh oh!

PaddlePaddle-bot left a comment

Choose a reason for hiding this comment

📋 Review 摘要

问题

历史 Findings 修复情况

📝 PR 规范检查

总体评价

Uh oh!

AlAuAu commented Jul 15, 2026

Uh oh!

sneaxiy left a comment

Choose a reason for hiding this comment

Uh oh!

AlAuAu commented Jul 16, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

PaddlePaddle-bot commented Jul 9, 2026 •

edited

Loading

codecov-commenter commented Jul 9, 2026 •

edited

Loading