LoongForge is an open-source training framework developed by the Baidu AI Cloud Baige team, built to deliver faster training for mainstream LLMs, VLMs, diffusion, and embodied models.
- High Performance — Built on Megatron-LM and torch-native backends, with deep optimizations for training speed that keep training loss aligned with the baseline.
- Easy to Use — Ready-to-run configs and launch scripts for 40+ open-source models, plus checkpoint conversion and data preprocessing tools. Supports pre-training, mid-training, SFT, and LoRA.
- Production-Proven — Open-sourced from AIAK-Training-LLM, which powers enterprise customers' proprietary models and several open-source projects. Production runs reach 5,000+ XPUs.
Example: training the embodied model DreamZero on LoongForge reaches 4.38× the open-source baseline's throughput, with loss curves staying aligned
🏗️ Architecture
Since optimal training strategies differ across model families and scales, LoongForge adopts a multi-backend architecture.
- Megatron Backend — For LLM, VLM, and diffusion models, extended with MoE parallelism, per-component heterogeneous parallelism, long-sequence optimizations, etc.
- torch-native Backend — For embodied models (VLA and WAM). Built on native PyTorch with DDP / ZeRO-1 / FSDP / HSDP, with deep optimizations for representative models across I/O, communication strategy, kernel efficiency, etc.
🔥 Latest News
- [2026/10] 🤖 Added VLA training support for LingBot-VLA-v2, built on the new tensor_wise distributed optimizer with Muon.
--optimizer muon also runs under plain DDP for any model.
- [2026/10] 🤖 Introducing LoongForge-VLDA, a Vision-Language-Delta-Action model trained with LoongForge: 98.5% on LIBERO, 84.5% zero-shot on LIBERO-Plus, and +17–18 points over π0.5 on real robots. [project page] [paper]
- [2026/10] 🦾 Open-sourced ego2robot — a pipeline that converts first-person videos of human manipulation into dual-arm robot training data, outputting LeRobot v3.0 format across 16 robot morphologies.
- [2026/09] ✨ Added training support for GLM-5.3-flash.
- [2026/09] ✨ Added Kimi-K3 BF16 training support for both LLMs and VLMs.
- [2026/09] ⚡ Added an optimized DreamZero Wan2.2-5B FSDP recipe with cache-aware data loading, compiled attention blocks, frozen-module handling, and FSDP2 Delta-FP8 Param AllGather.
- [2026/08] 🤖 Added VLA training support for Wall-OSS-0.5, with custom fused operators for higher training throughput.
- [2026/07] 🐳 Unified the prebuilt Docker images — all model families (LLM / VLM / Embodied / Diffusion) now share a single image.
- [2026/07] 🤖 Released LoongForge-Embodied, a torch-native DDP/FSDP training subsystem for embodied models (Pi0.5, GR00T-N1.6/N1.7, xVLA, LingBot-VA, FastWAM, DreamZero, and Cosmos3), with up to 4.38× speedup.
- [2026/07] ✨ Added training support for DeepSeek-V4-Flash / DeepSeek-V4-Pro.
- [2026/07] ✨ Added training support for Qwen-Image-Edit-2511.
- [2026/06] 🤖 Expanded VLA coverage with GR00T N1.6; 2.3× speedup on GR00T training. [blog]
- [2026/05] ⚡ Accelerated Wan 2.2 training by 116%, and added CP and data packing support.
- [2026/05] ✨ Added training support for Kimi K2.5 / K2.6, and introduced INT4 / NVFP4 PTQ.
- [2026/05] 🎉 v0.1.0 — first official tagged release of LoongForge.
- [2026/05] 🌟 Powered the training and public release of LLaVA-OneVision-2.0.
- [2026/04] 🧩 Added training support for MiniMax-M2.7 on both NVIDIA GPU and Kunlun XPU.
- [2026/04] 🚀 LoongForge source code publicly available on GitHub. [blog]
- [2025/10] 🌟 Powered the training and public release of LLaVA-OneVision-1.5 under AIAK-Training-LLM, the predecessor of LoongForge. [blog]
🤖 Embodied Models
Data processing, training, and evaluation for Vision-Language-Action (VLA) and World-Action (WAM) models, with 1.79×–4.38× training throughput over open-source baselines across benchmarked models. [Docs] (Paper coming soon)
Data
- Unified Data Loading — A unified data-loading interface supporting mainstream formats like LeRobot (v2.0 / v2.1 / v3.0); each model's data pipeline matches its original open-source implementation.
- Feature Cache Precomputation (optional) — Precompute frozen-encoder outputs for DreamZero (VAE latents, text embeddings) and FastWAM (text embeddings) ahead of training, read directly during training to avoid re-encoding every step. [DreamZero] [FastWAM]
- Ego2Robot — Converts first-person videos of human manipulation into LeRobot v3.0 training data across 16 robot morphologies, each adapted to its own joint layout and gripper. [README]
Training
- Per-Model Deep Optimization — Custom optimizations targeting each model's bottlenecks (I/O, communication strategy,
torch.compile, CUDA Graph, custom fused operators, etc.), reaching 1.79×–4.38× measured throughput over open-source baselines (see Performance).
- Precision Alignment — Training loss aligns with open-source baselines; comparison curves are already available for several models and will keep expanding alongside new model support. [Precision Comparisons]
- Multiple Distributed Strategies — Supports DDP / DDP + ZeRO-1 / FSDP / HSDP, selectable by model scale and cluster topology. [Tutorial]
- FP8 Communication Optimization (optional) — On supported NVIDIA GPUs, compresses cross-rank traffic to FP8: blockwise FP8 delta compression for FSDP2 parameter AllGather, and FP8 compression for DDP gradient all-reduce. [Usage]
Evaluation
- Unified Evaluation — Closed-loop evaluation of trained policies in the LIBERO / CALVIN / SimplerEnv / RoboTwin / ManiSkill simulators; coverage is expanding continuously. [README]
[!NOTE]
Coming soon: evaluation results for models trained with LoongForge.
🚀 LLM · VLM · Diffusion
Built on the Megatron backend, supporting large-scale pre-training and SFT for LLMs and VLMs, as well as diffusion model training. Key optimizations for each scenario:
LLM
- MoE EP Communication Optimization — Overlapped All2All / activation offload / compute, with further memory reduction beyond upstream Megatron-LM. [Usage]
- MoE Expert Load Balancing — Topology-aware dynamic replication of hot experts to balance EP workloads, with up to 74% lower overhead than industry solutions. [TAOT Paper]
- Adaptive FP8 Training — End-to-end FP8 for LLMs and VLMs with standard blockwise FP8; an optional adaptive mode picks per-operator precision by GEMM shape and efficiency. [Usage]
- Custom Fused Operators — Fused kernels like FusedDSA for DSA-style models — TileLang version open-sourced, high-performance CUDA version available on Baidu Baige platform.
- Long-Sequence Training — Scales LLM training to long sequences via Context Parallel (CP) and chunked-pipeline scheduling.
VLM
- Flexible Composition — Assemble VLMs from interchangeable ViT and LLM components (e.g. GLM-5.2 + MoonViT) straight from config — no custom model code. [Usage]
- Heterogeneous Parallelism — Independent TP / DP / recompute / freeze per model component (e.g. ViT vs. LLM) for optimal throughput and memory. [blog] [Usage]
- Decoupled Encoder-Decoder Training — Eliminates encoder-induced pipeline bubbles by separating ViT and LLM into independent tasks. [Usage]
- DP Load Balancing — Improves multi-node scaling efficiency with load-aware data redistribution that mitigates sequence-packing imbalance. [blog] [Usage]
- Flexible Data Pipeline — Energon WebDataset input for multimodal data, with both online and offline sequence packing. [Usage]
Diffusion
- Context Parallelism — Supports Ring, Ulysses, and hybrid context parallelism (CP) to split long video sequences across GPUs.
- Variable-Length Sample Packing — Packs multiple variable-length video samples into a single sequence to reduce padding waste, compatible with context parallelism. [Usage]
Compatibility
- Mcore Bridge — Supports both offline bidirectional Megatron ↔ HuggingFace conversion and online native HF load/save. [Usage]
- Heterogeneous Hardware — Native support for NVIDIA GPUs and Kunlun XPUs via a minimally-intrusive plugin design.
📖 Deep-dive: LLM · VLM
📊 Performance
Training throughput speedups over mainstream open-source baselines — each model and its baseline were benchmarked on the same machine type with the same training hyperparameters:
DeepSeek-V3.2 Lite reflects DSA operator-level optimizations and was validated on a reduced-layer configuration due to test-bed scale limits.
Numbers were measured at a point in time and may evolve as implementations change on both sides.
⚡ Quick Start
1. Install
Using NVIDIA GPU as an example, pull the prebuilt image and start a container (requires the NVIDIA Container Toolkit):
docker pull loongforge/loongforge:latest
mkdir -p workspace
docker run --gpus all --ipc=host -it --rm \
-v "$(pwd)/workspace:/workspace/data" \
-w /workspace/LoongForge \
loongforge/loongforge:latest bash
The example above mounts ./workspace to /workspace/data inside the container, for storing weights, datasets, and training outputs — these persist after the container exits.
For more installation options including source builds, see: Full GPU installation guide · Kunlun XPU installation guide
2. Pick a tutorial — by hardware and modality
3. Launch example
Launch scripts for every supported model are already included in this repo: NVIDIA GPU scripts live under examples/, Kunlun XPU scripts under examples_xpu/ — see Supported Models for the full list.
The example below walks through DDP fine-tuning of Pi0.5 on LIBERO (single node, 8 GPUs); run all commands inside the container above.
① Prepare weights, tokenizer, and dataset (hf is the Hugging Face CLI):
hf download lerobot/pi05_base --local-dir /workspace/data/pi05_base
hf download google/paligemma-3b-pt-224 --local-dir /workspace/data/paligemma-3b-pt-224
hf download lerobot/libero_10 --repo-type dataset --local-dir /workspace/data/libero_10
② Launch training (paths are set via environment variables; defaults to 20 steps to validate the workflow):
export CHECKPOINT_PATH=/workspace/data/pi05_base
export TOKENIZER_PATH=/workspace/data/paligemma-3b-pt-224
export DATA_PATH=/workspace/data/libero_10
export OUTPUT_DIR=/workspace/data/outputs/pi05
bash examples/embodied/pi05/run_pi05_ddp_finetune.sh
Logs and TensorBoard data are saved under OUTPUT_DIR. For FSDP / ZeRO-1 launch options and LIBERO evaluation, see the Pi0.5 tutorial.
🏛️ Supported Models
Click any model for its training examples; see configs/models/ for the matching parameter configs. See the model support matrix for all variants.
[!TIP]
For VLMs, also see the custom composition example, which assembles a new model from a ViT and an LLM purely through configuration.
🌟 Awesome Projects Built with LoongForge
See more on the Projects page, and if you've built something with LoongForge (or its predecessor, AIAK-Training-LLM), let us know — we'd love to feature it.
From the Community
- LLaVA-OneVision-2.0 — Next-generation multimodal model, with new VideoCaption and Spatial datasets
- Innovator-VL — Scientific multimodal LLM for advanced reasoning
- LLaVA-OneVision-1.5 — Fully open framework for democratized multimodal training
- Qianfan-VL — Domain-enhanced vision-language models for enterprise, 3B–70B parameters
From the LoongForge Team
- LoongForge-VLDA — Vision-Language-Delta-Action model for robotic manipulation: 98.5% on LIBERO, 84.5% zero-shot on LIBERO-Plus, and 17–18 points ahead of π₀.₅ on real robots.
https://github.com/user-attachments/assets/8563afca-f79a-4a45-bb4f-36e06c0ccbcb
📝 Community Blogs
📂 Repository Layout
LoongForge/
├── loongforge/ # Core training framework
│ ├── train/ # Training entry points and trainers
│ │ ├── pretrain/ # Pretraining
│ │ ├── sft/ # Supervised fine-tuning
│ │ └── diffusion/ # Diffusion training
│ ├── models/ # Unified model abstractions
│ │ ├── foundation/ # LLM backbones
│ │ ├── encoder/ # Vision encoders
│ │ ├── omni_models/ # Multimodal composition (encoder + projector + decoder)
│ │ ├── diffusion/ # Diffusion models
│ │ └── common/ # Shared base configs, layers and utilities
│ ├── embodied/ # Embodied (VLA + world-action) training: models, data, trainers, distributed, eval
│ ├── data/ # Data loading and processing pipelines
│ ├── tokenizer/ # Tokenizers
│ └── utils/ # Config registry, constants and shared helpers
├── third_party/ # Third-party dependencies and patches
│ ├── Loong-Megatron/ # Patched Megatron-LM (git submodule)
│ └── patches/ # TransformerEngine patches
├── configs/ # Hydra YAML configs (models, data)
├── examples/ # Launch scripts for NVIDIA GPUs
├── examples_xpu/ # Launch scripts for Kunlun XPUs
├── cookbook/ # End-to-end project recipes built on LoongForge
├── tools/ # Standalone dev tools: checkpoint conversion, data preprocessing, etc. (see tools/README.md)
├── ops/ # Custom fused operators (incl. open-sourced TileLang)
├── docker/ # Dockerfiles (GPU & XPU)
├── tests/ # Test suites (E2E, regression, unit)
└── docs/ # Documentation
📝 Citation
If you find LoongForge helpful, please consider citing this project:
@software{LoongForge2026,
title = {LoongForge: A high-performance framework for training LLMs, VLMs, diffusion, and embodied models},
author = {{The LoongForge Authors}},
year = {2026},
url = {https://github.com/baidu-baige/LoongForge}
}
If you find TAOT helpful, please consider citing our paper:
@article{zhang2026taot,
title = {{TAOT}: Topology-Aware Optimal Transport for Dynamic Expert Replica Placement in {MoE} Training},
author = {Zhang, Lingyun and Zhang, Henghua and Gu, Shilei and Mo, Kai and Han, Shuai and Li, Shiyong and Wang, Yanpeng and Shen, Dou},
journal = {arXiv preprint arXiv:2608.03676},
year = {2026},
url = {https://arxiv.org/abs/2608.03676}
}
If you find LoongForge-VLDA helpful, please consider citing our paper:
@article{zhang2026vlda,
title = {{Vision-Language-Delta-Action}: Learning Compact Deltas as Actionable Transition Interfaces for Robotic Manipulation},
author = {Zhang, Lingsen and Li, Wei and He, Jie and Zhang, Henghua and Mo, Kai and Li, Shiyong and Wang, Yanpeng and Shao, Rui},
journal = {arXiv preprint arXiv:2610.25874},
year = {2026},
url = {https://arxiv.org/abs/2610.25874}
}
🤝 Contributing
We warmly welcome community contributions — bug reports, feature proposals, and PRs alike. Please read our Contributing Guidelines before submitting.
Thanks to all our contributors:
💬 Contact Us
| Channel | What it's for |
|---|
| GitHub Issues | Bug reports, usage questions, and feature requests |
| Developer Communities | WeChat group, Xiaohongshu, and more |
| Email | Enterprise adoption, large-scale deployment, partnership, or any other topic |
🙏 Acknowledgments
LoongForge stands on the shoulders of the open-source community. Its Megatron backend builds on NVIDIA's Megatron-LM, and the project also draws on HuggingFace Transformers, LLaMA-Factory, Megatron-Bridge, LeRobot, and the official implementations of the models we support (e.g. OpenPI, NVIDIA Isaac GR00T).
We also thank the LINUX DO community for providing a welcoming space for technical discussion, and AtomGit for hosting LoongForge's official mirror.
📄 License
LoongForge is released under the Apache License 2.0; some files derive from third-party projects — see their file headers.
