English | 简体中文
A modular, scalable, high-performance training framework for LLMs, VLMs, diffusion, and embodied models.
📖 Quick Start · 📊 Performance · 🤖 Supported Models · 🚀 Roadmap
🐉 LoongForge is named after the traditional Chinese loong boat (龙舟), a symbol of coordinated power and forward momentum.
LoongForge is a unified training framework for LLMs, VLMs, diffusion, and embodied models, covering pre-training, continued pre-training, and SFT. Its primary goal is to provide broad coverage of mainstream open-source models while delivering efficient, high-throughput training.
Since training requirements vary across model scenarios, LoongForge builds on multiple distributed backends. LLM/VLM/diffusion run on Megatron-LM, while embodied models use a torch-native DDP/FSDP stack. Each is deeply optimized to outperform mainstream open-source baselines.
Before going open-source, LoongForge was developed as AIAK-Training-LLM, Baidu Baige's training acceleration stack. It has supported production training for enterprise customers across Education, Computer Vision, and Embodied AI, typically delivering 30%~50% speedup over customer baselines, with the largest production runs reaching 5,000+ XPUs.
- [2026/07] 🤖 Released the LoongForge-Embodied submodule, delivering training support for mainstream VLA and WAM models.
- [2026/07] ✨ Added training support for DeepSeek v4 flash / DeepSeek v4 pro.
- [2026/05] ⚡ Accelerated Wan 2.2 training by 116%, and added CP and data packing support.
- [2026/05] ✨ Added training support for Kimi K2.5 / K2.6, and introduced INT4 / NVFP4 PTQ.
- [2026/05] 🎉 v0.1.0 — first official tagged release of LoongForge.
- [2026/05] 🌟 Powered the training and public release of LLaVA-OneVision-2.0.
- [2026/05] 🤖 Expanded VLA coverage with GR00T N1.6; 60%+ speedup on Pi0.5 and GR00T training. [blog]
- [2026/04] 🧩 Added training support for MiniMax-M2.7 on both NVIDIA GPU and Kunlun XPU.
- [2026/04] 🚀 LoongForge source code publicly available on GitHub. [blog]
- [2025/10] 🌟 Powered the training and public release of LLaVA-OneVision-1.5 under AIAK-Training-LLM, the predecessor of LoongForge. [blog]
See the full documentation for installation, tutorials, and advanced usage — English · 中文.
1. Install — using prebuilt Docker images or source build:
- NVIDIA GPU: Installation Guide
- Kunlun XPU: Installation Guide
2. Launch your first training run — follow a tutorial for your target hardware and modality:
- NVIDIA GPU: LLM · VLM · VLA & WAM (docs being updated) · Diffusion
- Kunlun XPU: Kunlun XPU Tutorials
3. Explore — browse configs/models/ and examples/ / examples_xpu/ for ready-to-run scripts.
- 🧩 Flexible Multi-Modal Composition — Configuration-driven assembly of VLMs from interchangeable ViT and LLM components.
- ⚡ Heterogeneous Parallelism — Independent TP / DP / recompute per model component (e.g., ViT vs. LLM) for optimal throughput and memory. [blog]
- 🔀 Decoupled Encoder-Decoder Training — Separates ViT and LLM into independent tasks, eliminating encoder-induced pipeline bubbles.
- ⚖️ DP Load Balancing — Load-aware data redistribution mitigates sequence-packing imbalance, improving multi-node scaling efficiency. [blog]
- 🚀 MoE-Native Optimization — Overlapped All2All / activation offload / compute, with further memory reduction beyond upstream Megatron-LM on DeepSeek-V3, Qwen3-MoE, etc.
- 🚦 MoE Expert Load Balancing — Dynamically replicates hot experts using a topology-aware algorithm to balance Expert Parallel (EP) workloads and improve training efficiency.
- 🔬 Adaptive FP8 Training — End-to-end FP8 for LLMs and VLMs with standard blockwise FP8; optional adaptive mode picks per-operator precision by GEMM shape and efficiency.
- 🔧 Custom Fused Operators — Fused kernels like FusedDSA for DSA-style models — TileLang version open-sourced, high-performance CUDA version available on Baidu Baige platform.
- 🔁 Flexible Checkpointing — Offline bidirectional Megatron ↔ HuggingFace conversion plus native online HF load/save — no format barriers across your workflow.
- 🧰 Versatile Pipelines & Data Tools — Out-of-the-box Pretrain / MidTrain / SFT / LoRA, with built-in dataset format conversion and sequence packing.
- 🤖 Embodied Model Training — A dedicated torch-native DDP/FSDP subsystem for VLA and world-action (WAM) models (e.g. Pi0.5, GR00T N1.6, FastWAM), decoupled from the Megatron core, with flexible DDP / ZeRO-1 / FSDP / HSDP strategies.
- 🌐 Heterogeneous Hardware — Native support for NVIDIA GPUs and Kunlun XPUs via a minimally-intrusive plugin design.
📖 Deep-dive: LLM features · VLM features
Training speedups over mainstream open-source baselines. Each row is tagged with the version it was measured on, and refreshed per-model as the stack evolves:
📋 Details
| Model | Type | Baseline | Speedup | Measured |
|---|---|---|---|---|
| DreamZero (DROID Wan2.2-5B Full) | WAM | DreamZero | 2.67× | master · 2026-07 |
| GR00T N1.6 | VLA | LeRobot | 2.31× | master · 2026-07 |
| Pi0.5 | VLA | OpenPI | 2.23× | master · 2026-07 |
| LingBot VA | WAM | LingBot-VA | 1.80× | master · 2026-07 |
| X-VLA | VLA | X-VLA | 1.6× | master · 2026-07 |
| DeepSeek-V3.2 Lite § | MoE + DSA | Megatron-LM | 5.04× | v0.1.1 |
| Qwen3-VL-30B-A3B | VLM | VeOmni | 1.45× | v0.1.1 |
| Qwen3-30B-A3B | MoE | Megatron-LM | 1.16× | v0.1.1 |
§ Due to test-bed scale limits, DeepSeek-V3.2 was validated separately on a reduced-layer configuration — LoongForge's DSA CUDA kernel optimizations still deliver ~5× speedup over Megatron-LM and reach 64K sequence (baseline OOMs beyond 8K).
Numbers reflect baseline and LoongForge versions at the time of measurement (see the Measured column), and may evolve as implementations change.
Open-source models trained with LoongForge or its predecessor AIAK-Training-LLM:
- LLaVA-OneVision-2.0 — Next-generation multimodal model, with new VideoCaption and Spatial datasets.
- Innovator-VL — Scientific Multimodal Large Language Model for Advanced Reasoning.
- LLaVA-OneVision-1.5 — Fully open framework for democratized multimodal training.
- Qianfan-VL — Domain-Enhanced Vision-Language Models for Enterprise, 3B to 70B parameters.
LoongForge supports a broad range of state-of-the-art models across LLM, VLM, diffusion, and embodied.
| Category | Architectures | Models |
|---|---|---|
| LLM | DeepSeek-V2 | deepseek-v2-lite, deepseek-v2 |
| DeepSeek-V3 | deepseek-v3, deepseek-v32 | |
| DeepSeek-V4 | deepseek-v4-flash, deepseek-v4-pro | |
| LLaMA2 | llama2-7b, llama2-13b, llama2-70b | |
| LLaMA3 | llama3-8b, llama3-70b | |
| LLaMA3.1 | llama3.1-8b, llama3.1-70b, llama3.1-405b | |
| Qwen | qwen-1.8b → qwen-72b | |
| Qwen1.5 | qwen1.5-0.5b → qwen1.5-72b | |
| Qwen2 | qwen2-0.5b → qwen2-72b | |
| Qwen2.5 | qwen2.5-0.5b → qwen2.5-72b | |
| Qwen3 | qwen3-0.6b → qwen3-480b-a35b, qwen3-coder-30b-a3b | |
| Qwen3-Next | qwen3-next-80b-a3b | |
| MiniMax | minimax-m2.1, minimax-m2.5, minimax-m2.7 | |
| MIMO | mimo-7b | |
| GLM | glm5 | |
| VLM | Qwen2.5-VL | qwen2.5-vl-3b → qwen2.5-vl-72b |
| Qwen3-VL | qwen3-vl-30b-a3b, qwen3-vl-235b-a22b | |
| Qwen3.5 | qwen3.5-0.8b → qwen3.5-397b-a17b | |
| Qwen3.6 | qwen3.6-27b, qwen3.6-35b-a3b | |
| Kimi-K2.5 | kimi-k2.5, kimi-k2.6 | |
| ERNIE4.5-VL | ernie4.5vl-28b-a3b | |
| LLaVA-OneVision-1.5 | llava-onevision-1.5-4b | |
| InternVL2.5 | internvl2.5-8b → internvl2.5-78b | |
| InternVL3.5 | internvl3.5-8b → internvl3.5-241b-a28b | |
| CustomCombinedModel | Flexible ViT + LLM backbone configuration (example) | |
| Diffusion | Wan2.1 | wan2-1-i2v |
| Wan2.2 | wan2-2-i2v | |
| Qwen-Image | qwen-image-edit-2511 | |
| Embodied | Pi | pi0.5 |
| GR00T | groot_n1_6, groot_n1_7 | |
| XVLA | xvla | |
| FastWAM | fastwam | |
| LingBot-VA | lingbot_va | |
| Cosmos3 | cosmos3_nano | |
| DreamZero | dreamzero_lora_wan22_5b, dreamzero_full_wan22_5b, dreamzero_lora_wan21_14b, dreamzero_full_wan21_14b, dreamzero_libero_wan22_5b, dreamzero_agibot_wan21_14b, dreamzero_yam_wan21_14b |
📁 Directory tree
LoongForge/
├── loongforge/ # Core training framework
│ ├── train/ # Training entry points & trainers
│ │ ├── pretrain/ # Pretrain (LLM, VLM)
│ │ ├── sft/ # SFT (LLM, VLM, InternVL, ERNIE)
│ │ └── diffusion/ # Diffusion (WAN, Qwen-Image)
│ ├── models/ # Unified model abstractions
│ │ ├── foundation/ # LLM backbones (LLaMA, Qwen, DeepSeek, ...)
│ │ ├── encoder/ # Vision encoders (ViT, Qwen-VL, InternVL, ...)
│ │ ├── omni_models/ # Multi-modal composition
│ │ ├── diffusion/ # Diffusion models (WAN, Qwen-Image)
│ │ └── common/ # Shared layers and utilities
│ ├── embodied/ # LoongForge-Embodied: standalone torch-native (DDP/FSDP)
│ │ # embodied (VLA + world-action) subsystem — see loongforge/embodied/README.md
│ ├── data/ # Data pipelines (multi-modal, video, DP balance)
│ ├── tokenizer/ # Tokenizers
│ └── utils/ # Config map, constants, etc.
├── third_party/Loong-Megatron/ # Patched Megatron-LM (git submodule)
├── configs/ # Hydra YAML configs (models, data)
├── examples/ # GPU launch scripts
├── examples_xpu/ # Kunlun XPU launch scripts
├── tools/ # Checkpoint conversion, data preprocessing
├── ops/ # Custom fused operators (incl. open-sourced TileLang)
├── patches/ # TransformerEngine patches
├── docker/ # Dockerfiles (GPU & XPU)
├── tests/ # E2E test suite (YAML-driven)
└── docs/ # Documentation
We warmly welcome community contributions — bug reports, feature proposals, and PRs alike. Please read our Contributing Guidelines before submitting.
LoongForge is released under the Apache License 2.0. Some files are derived from third-party open-source projects; please refer to the specific file headers for their respective copyright and attribution.
@software{LoongForge2026,
title = {LoongForge: A modular, scalable, high-performance training framework for LLMs, VLMs, diffusion, and embodied models},
author = {{The LoongForge Authors}},
year = {2026},
url = {https://github.com/baidu-baige/LoongForge}
}LoongForge builds on NVIDIA's Megatron-LM and draws inspiration from many excellent open-source projects, including HuggingFace Transformers, LLaMA-Factory, Megatron-Bridge, and LeRobot, as well as the official implementations of the models it supports (e.g. OpenPI, NVIDIA Isaac GR00T). We sincerely thank these communities for their outstanding contributions.
Open a GitHub issue for questions, feedback, or feature requests. You can also join our developer community:
- WeChat — Scan QR code to join
- Slack — Join here