Fix rope-bwd cudagraph crash and tighten gdn_fwd_h accuracy gate#2467
Merged
Conversation
Two perf-dashboard fixes: 1. rope-bwd was failing every shape on B200 + H100 with "Offset increment outside graph capture" — tritonbench's torch_compile rope-bwd recompiles during CUDA graph capture. Disable cudagraph for rope-bwd, mirroring the existing grouped_gemm / layer_norm-bwd / rms_norm-bwd configs. 2. gdn_fwd_h accuracy was failing for batch=16 on B200 (seq>=2048) and H100 (all seqs) for *both* Triton and Helion. PR pytorch#2455 replaced the rtol=0.5/atol=2.0 tolerance with precision=tf32, but tf32 doesn't address bf16-accumulation drift in the kernels. Restore a shape-conditional accuracy patch: keep the strict default for batch=1 (always passes), and only widen to rtol=0.5/atol=2.0 for batch>=16 where the 16x longer reductions diverge from eager.
meta-cla
Bot
added
the
CLA Signed
jansel
approved these changes
May 16, 2026
Helion autotune at full effort consistently fails on the (64, 64, 1, 8192, 256, 64, 128) chunk_state shape on H100 with "No working config found": all 100 initial-population configs error out at runtime in ~56s. The same kernel + shape works fine on a freshly-cleared GPU (manual configs all run; full-effort autotune succeeds well into generation 4 with ~98% of sampled configs OK). The CI failure correlates with carryover from the 5 prior chunk_state shapes (~1.5h of autotune leaves behind cached buffers / JIT state / tritonbench-retained tensors), pushing the largest 4.3 GB-input shape past whatever resource limit the autotune subprocesses are hitting. Match the existing chunk_scan gating: skip the shape on devices with < 100 GB free. B200 (192 GB) keeps running it; H100 (96 GB total) skips and the rest of the chunk_state shapes still report. Rename the constants to MAMBA2_LARGE_SHAPE since they now apply to both chunk_scan and chunk_state.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two perf-dashboard fixes
Was failing every shape with:
Same cause as the existing grouped_gemm / layer_norm-bwd / rms_norm-bwd configs: tritonbench's torch_compile backward recompiles during CUDA graph capture. Remove CG capture.
At batch=16, bf16-reduction-order drift in the kernels, so both Triton and Helion started failing accuracy