Megatron-LM and Megatron Core

About

This repository contains two components: Megatron-LM and Megatron Core.

Megatron-LM is a reference example that includes Megatron Core plus pre-configured training scripts, ideal for research teams, learning distributed training, and quick experimentation.

Megatron Core is a composable library with GPU-optimized building blocks for custom training frameworks. It provides transformer building blocks, advanced parallelism strategies (TP, PP, DP, EP, and CP), mixed precision support (FP16, BF16, FP8, and FP4), and model architectures, ideal for framework developers and ML engineers building custom training pipelines.

Megatron Bridge provides bidirectional Hugging Face ↔ Megatron checkpoint conversion with production-ready recipes.

Getting Started

Compatibility: Python 3.12+ and PyTorch 2.6.0+ recommended. For source development, the tested NGC PyTorch base images are pinned in docker/.ngc_version.dev and docker/.ngc_version.lts.

Install from PyPI:

uv pip install megatron-core

Or clone and install from source:

git clone https://github.com/NVIDIA/Megatron-LM.git
cd Megatron-LM
uv pip install -e .

Note: Building from source can use a lot of memory. If the build runs out of memory, limit parallel compilation jobs by setting MAX_JOBS (for example, MAX_JOBS=4 uv pip install -e .).

For NVIDIA GPU Cloud (NGC) container setup and all installation options, review the Installation Guide.

• Your First Training Run - End-to-end training examples with data preparation
• Parallelism Strategies - Scale training across GPUs with TP, PP, DP, EP, and CP
• Contribution Guide - How to contribute to Megatron Core
• Style Guide - Python style overrides

Latest News

• [2026/10] Scale Bitwise-Deterministic Pretraining with NVIDIA Megatron Core - A guide to reproducing training runs, validating checkpoint resume, and reducing determinism overhead at scale with Megatron Core.
• [2026/08] Megatron Core 0.19 - Introduces HybridModel, quantile-based MoE router balancing, packed-sequence MoE dispatch, and CUDA Graph-compatible activation offloading.
• [2026/07] DeepSeek-V3 pretraining on GB300 NVL72 - Megatron Core achieves 1,648 TFLOPs/s per GPU for DeepSeek-V3 671B on 256 GPUs.
• [2026/05] DeepSeek-V4 initial support - Megatron Core's dev branch includes the initial DeepSeek-V4 implementation; Megatron Bridge provides conversion, inference, and pretraining recipes.
• [2026/04] Advancing Emerging Optimizers for Accelerated LLM Training with NVIDIA Megatron - Muon and other emerging optimizers are now supported in Megatron Core via the new Emerging-Optimizers library.

Previous News

Project Structure

Megatron-LM/
├── megatron/
│   ├── core/                    # Megatron Core (kernels, parallelism, building blocks)
│   │   ├── models/              # Transformer, hybrid, and multimodal models
│   │   ├── transformer/         # Transformer building blocks
│   │   ├── tensor_parallel/     # Tensor parallelism
│   │   ├── pipeline_parallel/   # Pipeline parallelism
│   │   ├── context_parallel/    # Context parallelism
│   │   ├── distributed/         # Distributed training (Megatron FSDP, DDP)
│   │   ├── dist_checkpointing/  # Distributed checkpoint saving, loading, and resharding
│   │   ├── optimizer/           # Optimizers
│   │   ├── datasets/            # Dataset loaders
│   │   ├── inference/           # Inference engines and server
│   │   └── export/              # Model export (example: TensorRT-LLM)
│   ├── training/                # Training scripts
│   └── post_training/           # Post-training (quantization, distillation, pruning, etc.)
└── examples/                    # Ready-to-use training examples

Performance Benchmarking

The NVIDIA Megatron Bridge Performance Summary publishes training throughput, system configurations, parallelism settings, and links to performance recipes and reproduction instructions. Use the container and matching code version specified by those instructions when reproducing a result.

Selected results from the 26.08.01 NeMo container (August 2026):

This is the beginning of the README. Read the rest at the original. Read the full README ↗