Add MiniMax-M3 NVFP4 B300 single-node vLLM benchmark (EAGLE3 spec decode)#1929
Add MiniMax-M3 NVFP4 B300 single-node vLLM benchmark (EAGLE3 spec decode)#1929Ankur-singh wants to merge 5 commits into
Conversation
… decode New minimaxm3-fp4-b300-vllm-mtp config (fp4 vLLM aggregated on b300 with EAGLE3 speculative decoding, 3 draft tokens via Inferact/MiniMax-M3-EAGLE3). The benchmark script overlays vllm-project/vllm PR #46380 (MiniMax-M3 modelopt NVFP4 support, commit 6c08558) before serve and routes prompts through the chat template. Target weights are pre-staged read-only at /scratch/models/MiniMax-M3-NVFP4 (added MiniMax-M3-NVFP4 to launch_b300-nv.sh STAGED_MODELS); the EAGLE3 draft downloads to the writable /data/models.
|
Thanks for the contribution! For vLLM & SGLang, please ensure that your recipes is similar to the official vLLM recipes and/or the SGLang cookbook If it is not, please create a PR first before we can merge your single node PR into the master branch. Let's ensure that the documentation is first class such that the entire ML community can benefit from your hard work! Thank you PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. If re-running failed jobs is attempted, PR authors are responsible for ensuring it passes. See GitHub's docs on re-running failed jobs: https://docs.github.com/en/actions/how-tos/manage-workflow-runs/re-run-workflows-and-jobs#re-running-failed-jobs-in-a-workflow As a rule of thumb, generally, PR authors should request a review & get a PR approval from the respective companies' CODEOWNERS before requesting a review from core maintainers. If additional help is needed, PR authors can reach out to core maintainers over Slack. 感谢你的贡献！对于 vLLM 与 SGLang，请确保你的 recipe 与官方 vLLM recipes 和/或 SGLang cookbook 保持一致 如果不一致，请先创建一个 PR，之后我们才能将你的单节点 PR 合并到 master 分支。让我们确保文档保持一流水准，使整个 ML 社区都能从你的辛勤工作中受益！谢谢 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。如果选择重新运行失败的任务，PR 作者有责任确保其最终通过。参见 GitHub 关于重新运行失败任务的文档：https://docs.github.com/en/actions/how-tos/manage-workflow-runs/re-run-workflows-and-jobs#re-running-failed-jobs-in-a-workflow 一般而言，PR 作者应先向相应公司的 CODEOWNERS 请求审阅并获得 PR 批准，然后再请求核心维护者审阅。 如需更多帮助，PR 作者可通过 Slack 联系核心维护者。 |
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
| # recognise the NVFP4 quant config and falls back to an unsupported path. | ||
| VLLM_DIR=$(python3 -c "import vllm, os; print(os.path.dirname(vllm.__file__))") | ||
| for f in \ | ||
| model_executor/layers/fused_moe/experts/trtllm_nvfp4_moe.py \ | ||
| model_executor/layers/quantization/modelopt.py \ | ||
| model_executor/layers/quantization/utils/flashinfer_utils.py | ||
| do | ||
| curl -fsSL "https://raw.githubusercontent.com/vllm-project/vllm/6c08558/vllm/${f}" -o "${VLLM_DIR}/${f}" |
There was a problem hiding this comment.
🟡 The 3-file vLLM patch overlay loop (lines 25-32) runs curl -fsSL ... -o without set -e and without || exit 1, so a transient 5xx / rate-limit / commit-reachability failure on file #2 (modelopt.py) or file #3 (flashinfer_utils.py) returns non-zero and the loop continues silently. The post-patch python3 -c only imports TrtLlmNvFp4ExpertsModular from file #1, so a partial patch is undetected and vllm serve boots on the comment's "unsupported path". Match the sister recipe minimaxm3_fp8_b300_mtp.sh (which wraps its patch step with || { echo ...; exit 1; }) — either add || exit 1 to the curl, or assert all three modules import in the verification.
Extended reasoning...
The defect. Lines 25-32 fetch three vLLM source files from a pinned vllm-project/vllm commit and overwrite them in the installed package:
for f in \
model_executor/layers/fused_moe/experts/trtllm_nvfp4_moe.py \
model_executor/layers/quantization/modelopt.py \
model_executor/layers/quantization/utils/flashinfer_utils.py
do
curl -fsSL "https://raw.githubusercontent.com/vllm-project/vllm/6c08558/vllm/${f}" -o "${VLLM_DIR}/${f}"
done
python3 -c "from vllm.model_executor.layers.fused_moe.experts.trtllm_nvfp4_moe import TrtLlmNvFp4ExpertsModular; print('[nvfp4-patch] OK')"The script has no set -e (set -x later is just command tracing) and no || exit 1 inside the loop. With curl -f, an HTTP 4xx/5xx exits curl with status 22 and leaves the target file untouched — the original installed copy stays in place.
Why the validation doesn't catch it. The post-loop python3 -c imports only TrtLlmNvFp4ExpertsModular from trtllm_nvfp4_moe.py (file #1). If file #2 (modelopt.py) or file #3 (flashinfer_utils.py) fails to download, that import still succeeds, the [nvfp4-patch] OK line prints, and the script proceeds to vllm serve with two unpatched modules. Per the script's own preamble: "without it vLLM does not recognise the NVFP4 quant config and falls back to an unsupported path" — exactly the failure modeled by files #2/#3 being unpatched.
Step-by-step proof.
curl -fsSLfortrtllm_nvfp4_moe.pysucceeds → file [NVIDIA] Add TRT-LLM 70B FP8 via slurm #1 overwritten.curl -fsSLformodelopt.pyhits a transient 503 fromraw.githubusercontent.com→ curl exits 22, no write to${VLLM_DIR}/.../modelopt.py, original vLLM copy retained.- Bash loop sees no
set -e, no|| exit 1→ continues to file [NVIDIA] update vllm b200 image. TODO: add logic for docker runner. #3. curl -fsSLforflashinfer_utils.pysucceeds → file [NVIDIA] update vllm b200 image. TODO: add logic for docker runner. #3 overwritten.python3 -cimports fromtrtllm_nvfp4_moe(the file that WAS patched). Import succeeds, prints[nvfp4-patch] OK.vllm serveboots with the newtrtllm_nvfp4_moe.pycalling into an unpatchedmodelopt.py→ NVFP4 quant config not recognized, fallback path triggers, benchmark fails opaquely at serve/inference time instead of at the patch step.
Why this is real. Verified that the script contains no set -[eE] or set -o errexit; the only set -e in benchmark_lib.sh is scoped to run_agentic_replay_and_write_outputs and doesn't propagate to callers. Verified curl 8.x behavior: -f -o file against a 404/5xx exits non-zero without writing the file. Pinned commit hashes on a stable CDN make this uncommon, but not zero — raw.githubusercontent.com does have transient 5xx windows.
Repo convention. The sister script benchmarks/single_node/fixed_seq_len/minimaxm3_fp8_b300_mtp.sh:33 already gates its patch step with python3 - <<PYEOF || { echo ...; exit 1; } — explicit fail-fast. This recipe should match.
Fix. Minimal:
curl -fsSL "https://raw.githubusercontent.com/vllm-project/vllm/6c08558/vllm/${f}" -o "${VLLM_DIR}/${f}" || exit 1Or assert the other two modules in the verification:
python3 -c "from vllm.model_executor.layers.fused_moe.experts.trtllm_nvfp4_moe import TrtLlmNvFp4ExpertsModular; from vllm.model_executor.layers.quantization import modelopt; from vllm.model_executor.layers.quantization.utils import flashinfer_utils; print('[nvfp4-patch] OK')"(The second form only catches import-breaking download failures; the first is the strictly safer fix and matches the fp8 twin.)
| - "Image vllm/vllm-openai:vllm-minimax-m3-perf-x86_64-13.0.1-7a67223; benchmark script overlays vllm-project/vllm PR #46380 (MiniMax-M3 modelopt NVFP4 support, commit 6c08558) before serve; prompts routed through the chat template" | ||
| - "Target weights pre-staged read-only at /scratch/models/MiniMax-M3-NVFP4 (added MiniMax-M3-NVFP4 to launch_b300-nv.sh STAGED_MODELS); EAGLE3 draft downloaded to the writable /data/models; --block-size 128 (MSA), --language-model-only" | ||
| - "Sweeps tp 4/8 with and without EP and dp-attn at 1k1k and 8k1k, conc 1-512" | ||
| pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/XXX |
There was a problem hiding this comment.
🟡 The perf-changelog entry for minimaxm3-fp4-b300-vllm-mtp ends with pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/XXX — the literal XXX placeholder was never filled in. Every other recent entry links to a real PR number (e.g. /pull/1927 directly above); this should be /pull/1929 to match this PR.
Extended reasoning...
What the bug is. perf-changelog.yaml line 4194 was added as part of this PR with the value pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/XXX. The XXX is a literal placeholder string that the author clearly intended to replace with the actual PR number before pushing, but didn't. Every other recent entry in the file links to a real PR — the immediately preceding entry uses /pull/1927, and earlier ones use /pull/1762, /pull/1865, etc. This is a documentation/metadata oversight.\n\nHow it manifests. Anyone clicking the changelog link for the minimaxm3-fp4-b300-vllm-mtp entry after merge will hit GitHub's 404 page for /pull/XXX rather than landing on the PR that introduced the config. Tooling that walks the changelog looking up PRs (e.g. cross-referencing config additions back to their PR for release notes) will fail to resolve the placeholder.\n\nThe specific code path. In the diff applied to perf-changelog.yaml, the new YAML block ending at line 4194 contains:\n\nyaml\n- config-keys:\n - minimaxm3-fp4-b300-vllm-mtp\n description:\n - "Add MiniMax-M3 NVFP4 ..."\n ...\n pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/XXX\n\n\nThe XXX is verbatim text, not a template variable that any downstream tool would substitute.\n\nWhy existing code doesn't prevent it. The changelog is hand-edited YAML; there is no schema validator or CI step that pattern-matches pr-link against /pull/\d+ to catch placeholders. The PR number isn't knowable at commit time (you only learn it after opening the PR), so it has to be filled in as an amendment — exactly the step that was skipped here.\n\nStep-by-step proof.\n1. Open perf-changelog.yaml after this PR merges.\n2. Scroll to the last entry — config minimaxm3-fp4-b300-vllm-mtp.\n3. Read line 4194: pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/XXX.\n4. Click / curl the link → GitHub returns 404 (/pull/XXX is not a valid PR id).\n5. Compare with line 4185 directly above: pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/1927 — resolves correctly.\n\nHow to fix. Replace XXX with 1929 on line 4194 so the entry reads pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/1929, matching this PR's number.
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=28145405377 |
|
As a PR reviewer and CODEOWNER, I have reviewed this and have:
Additional detail section:vLLM Recipe PR: vllm-project/recipes#577 Signed: |
|
PASS — all four checks satisfied; the sign-off now carries the required recipe link (supersedes the earlier Check 3 fail).
|
| # Apply vllm-project/vllm PR #46380 (Add MiniMax-M3 modelopt NVFP4 support, commit 6c08558). | ||
| # This patch is required for nvidia/MiniMax-M3-NVFP4: without it vLLM does not | ||
| # recognise the NVFP4 quant config and falls back to an unsupported path. | ||
| VLLM_DIR=$(python3 -c "import vllm, os; print(os.path.dirname(vllm.__file__))") |
There was a problem hiding this comment.
Hmmm, I get new model and we trying to go SOL but can we at least wait til this is merged and it makes the nightly?
Or some other official image at least.
…lm-mtp # Conflicts: # perf-changelog.yaml
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=28189445361 |
functionstackx
left a comment
functionstackx
left a comment
There was a problem hiding this comment.
- fix patchwork as discussed in slack
- missing vllm recipes
The vllm/vllm-openai:vllm-minimax-m3-perf-x86_64-13.0.1-8b00f41 image bakes in MiniMax-M3 modelopt NVFP4 support (vllm-project/vllm PR #46380), so the EAGLE3 benchmark script no longer overwrites vLLM files at runtime.
…lm-mtp # Conflicts: # perf-changelog.yaml
6 participants
Adds the
minimaxm3-fp4-b300-vllm-mtpconfig: MiniMax-M3 NVFP4 (nvidia/MiniMax-M3-NVFP4) single-node aggregated vLLM on B300 with EAGLE3 speculative decoding (spec-decoding: mtp, 3 draft tokens viaInferact/MiniMax-M3-EAGLE3).nvidia-master.yamlentry (fp4 / vllm / runnerb300), every search-space rowspec-decoding: mtp; sweeps tp 4/8 with and without EP and dp-attn at 1k1k and 8k1k, conc 1-512.benchmarks/single_node/fixed_seq_len/minimaxm3_fp4_b300_mtp.sh— overlays vllm-project/vllm PR #46380 (MiniMax-M3 modelopt NVFP4 support, commit 6c08558) before serve;--speculative-configEAGLE3;--block-size 128(MSA),--language-model-only; prompts routed through the chat template./scratch/models/MiniMax-M3-NVFP4— addedMiniMax-M3-NVFP4to theSTAGED_MODELSallow-list inlaunch_b300-nv.sh. The EAGLE3 draft (not staged) is downloaded to the writable/data/modelsrather than next to the read-only target.