test(ep): add configurable CLI knobs to dispatch/combine benchmark#459
Open
amd-arozanov wants to merge 5 commits into
Open
test(ep): add configurable CLI knobs to dispatch/combine benchmark#459amd-arozanov wants to merge 5 commits into
amd-arozanov wants to merge 5 commits into
Conversation
Expose the previously hardcoded warmup/iteration counts as CLI flags for --cmd bench and --cmd profile, defaulting to None so each command keeps its existing default when the flag is unset.
Let --cmd bench skip the expensive host-side correctness check during warmup when only performance numbers are needed (default: 1, same as before).
Let --cmd bench pick a routing pattern via --routing (default: random, same as before), including a new remote_round_robin pattern that always routes tokens to a remote rank.
Adds a --kernel-type CLI flag to select the dispatch/combine kernel implementation to benchmark (IntraNode, IntraNodeLL, InterNode, InterNodeV1, InterNodeV1LL, AsyncLL), threaded through bench_dispatch_combine() and _bench_dispatch_combine() into EpDispatchCombineConfig. Defaults to "IntraNode", matching the existing EpDispatchCombineConfig default, so legacy behavior is unchanged when the flag is omitted.
Adds a --graph-replay-iters CLI flag controlling how many times each captured CUDA graph is replayed per --iters sample in run() (bench command only), to amortize host-side timing overhead. Defaults to None/unset so the existing hardcoded default of 10 is preserved when the flag is omitted.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
The dispatch/combine benchmark script only supported a fixed set of scenarios. This makes it hard to explore different kernel implementations, routing patterns, or tune benchmark timing without editing the script. This PR adds CLI flags to cover these cases.
Technical Details
Adds the following flags to
--cmd bench(all optional, defaulting to prior behavior):--warmup/--iters– number of warmup and timing iteration.--routing– select the token-to-expert routing pattern; adds a newremote_round_robinpattern that always routes to a remote rank.--kernel-type– select the dispatch/combine kernel implementation--verify– skip the host-side correctness check during warmup.--graph-replay-iters– control how many times each captured CUDA graph is replayed per timed sample.Test Plan
Run bench_dispatch_combine.py with new command line options
Test Result
bench_dispatch_combine.py works, default values were not changed, new command line arguments work as expected
Submission Checklist