[ROADMAP][2026 Q3] Megatron Core Roadmap

[sbhavani]

This roadmap outlines the key Megatron Core features, enhancements, and improvements planned for Q3 2026, with a focus on the August 2026 / 26.08 release. This is a tentative roadmap and subject to change.

For detailed information on past releases, see the Megatron Core release notes. For the previous quarterly roadmap, see Q2 2026 Roadmap #4997.

---

Q3 Roadmap (Target: 26.08)

MoE

• DeepSeek-V4 architecture support - Integrate CSA/HCA hybrid attention, DSA, and mHC; support Hash MoE and ClampedSwiGLU; add MTP with mHC, fused kernels, Muon/AdamW training recipes, packed sequence handling, and long-context validation (#4468, #4866, #4867, #4869).
• Hash MoE and ClampedSwiGLU - Implement Megatron Core support for DeepSeek-V4-specific routing and activation functions.
• MoE communication overlap - Add latent shared-expert overlap, THD dispatcher support, and DeepEP V2 ElasticBuffer support (#5135, #5008, #4632).
• Optimized MoE kernels - Optimize Blackwell performance by integrating GEMM + SwiGLU fused MLP, mHC, GDN, and batchedGEMM fusion techniques.
• MegaMoE / ultra-sparse MoE - Research expanded expert counts, sparsity-driven activations, predictive routing mechanisms, and adaptable recomputation strategies.

Model Architecture and Support

• DeepSeek-V4 recipes - Complete recipes with end-to-end convergence validation across Megatron-LM and Megatron Bridge.
• Long-context validation - Verify convergence for 4K/16K dense and 64K/1M sparse training setups.
• HybridModel - Introduce a heterogeneous-layer model definition for DeepSeek-V4 and future hybrid configurations, replacing GPTModel (#4537, #5030, #5031, #4538, #4539).
• Nemotron 3 on HybridModel - Make Nemotron 3 native to HybridModel.
• * DeepSeek-V4 transition to HybridModel - Move DeepSeek-V4 development from GPTModel to the new HybridModel (#5042).
• Qwen 3.5 / Qwen3-Next support - Add model architecture logic, packed sequence handling, Qwen3.5-VL validation, and bidirectional checkpoint conversion between Hugging Face and Megatron Core (#4495, #4753, #4754, #4755, #4756).
• SFTDataset class - Add a new pre-tokenized, packed implementation (#5017).

Parallelism

• Megatron FSDP - Provide an FSDP2-compatible API, communication-compute overlap, activation recompute support, and pooled memory allocation.
• NCCL EP support - Add an expert-parallel transport path and dispatch-manager support (#5129, #4644).
• Fuse per-sequence AlltoAll - Create a unified AlltoAll operation (#4913).
• Next-generation dynamic DP - Support multimodal data and improve performance.
• Communication enhancements / DeepEPv2 - Integrate the DeepEPv2 token dispatcher and enable NCCL EP support.

Performance and Memory

• HybridModel 1F1B overlap - Add grouped HybridStack support and extend EP-overlap scheduling and checkpoint compatibility to grouped hybrid layers (#4941, #4942, #4943, #4944).
• Checkpoint load performance - Reduce cross-rank reads for torch_dist load (#4628).
• Reduce MoE peak memory - Avoid full intermediate list/cat buffers and chunk the return all-to-all/unpermute path (#4746).
• Full-model CUDA Graph with paged stashing - Advance paged stashing to enable full-iteration CUDA Graph capture and optimize memory for dropless MoE configurations.
• Enhanced attention mechanisms - Advance kernel development for GDN, NSA, and associated Transformer Engine or cuDNN Frontend integrations.
• GDN memory optimization - Implement fine-grained activation offloading and selective recompute strategies for in-projection, conv1d, and gated delta rule logic.

Precision and Optimizers

• MXFP8 and NVFP4 training paths - Continue improving MXFP8/NVFP4 parameter gathering and add support for low-precision GroupedGEMM.
• Low-precision guide - Document practical low-precision training recipes and optimizer configuration recipes.
• Advanced Muon optimizer features - Implement Newton-Schulz orthogonalization coefficients, FP8 primary weights, and precision-aware behavior for DeepSeek-V4.
• MLA support for Muon - Extend the existing metadata-tagging scheme for splitting MLA up-projections (#5015).
• Model-Optimizer PTQ and QAD - Surface Model-Optimizer-based FP8/NVFP4 post-training quantization (PTQ) and quantization-aware distillation (QAD).
• Move quantize.py - Move quantize.py to the Megatron-LM root folder.
• PTQ and QAD documentation - Surface PTQ and QAD documentation using Nemotron 3 as examples.

Inference

• Sliding-window attention in dynamic batching - Bring dynamic inference to parity with static Transformer Engine attention for SWA models (#5138).
• ETP - Add support for expert tensor parallelism (#4743).
• Async scheduling - Introduce asynchronous decode scheduling for decode-only inference (#4604).

Multimodal

• MIMO (Multimodal In, Multimodal Out) extensions - Expand the TP/DP heterogeneous primitive and enable autonomous nD parallelism for submodules.
• Pipeline parallel support for the language model - Allow colocated layouts where the destination language grid has PP > 1 while the source encoder grid remains PP = 1 (#4784).
• VLM CUDA Graph support - Capture the language model submodule (#4519).

* : Feature is being developed on the dev branch first and may not be part of the 26.08 main release.

---

How to Provide Feedback

We welcome community input on prioritization. Please:

1. React to items you would like prioritized.
2. Comment on this issue with use cases, constraints, and hardware / model configurations.
3. Open focused feature requests with the enhancement label.
4. Contribute pull requests for roadmap items where possible.

---

Credits

This roadmap reflects the collective efforts of NVIDIA, external contributors, and the Megatron Core community.