A PyTorch-native platform for pretraining and post-training generative AI models at scale
Quick start · Models · Features · RL · Docs · Contributing
torchtitan is a clean-room, PyTorch-native implementation of large-scale
training. One readable codebase takes a model from pretraining through
supervised fine-tuning to reinforcement learning, composing FSDP, tensor,
pipeline, context, and expert parallelism without rewriting the model.
- Composable parallelism. Models declare how they shard; FSDP2, TP, PP, CP,
and EP stack into N-D layouts through
DeviceMesh and DTensor.
- Built for research. Minimal, hackable code; every run is a Python
recipe you can read, copy, and modify.
- Pretraining to RL in one stack. TitanRL runs the
same model definitions and kernels in the trainer and in vLLM generation, with
a bitwise-reproducible batch-invariant mode.
- Frontier models and techniques. DeepSeek, Qwen, Kimi, gpt-oss, Llama, and
FLUX, with MXFP8/NVFP4 low precision, Dist-MoE, CUDA graphs, and more.
[!NOTE]
torchtitan is under active development. To use the latest features, use a
recent PyTorch nightly.
Latest news
- [2026/10] Zephon, a data loader by DatologyAI, released its TorchTitan integration. Zephon keeps the same training batches in the same order even when your GPU count changes (elastic determinism). Check out the blog post on pytorch.org.
- [2026/08] TitanRL is a hackable RL stack for scaling and debugging. It reuses TorchTitan model definitions and kernels across training and vLLM generation and supports batch-invariant mode.
- [2025/11] AMD released an optimized fork of
torchtitan for AMD GPUs.
- [2025/10] We released
torchtitan v0.2.0.
- [2025/10] SkyPilot now supports
torchtitan! See the tutorial.
- [2025/07] We published instructions on how to add a model to
torchtitan.
- [2025/04] Our paper was accepted by ICLR 2025.
- [2024/12] GPU MODE lecture on torchtitan.
- [2024/07] Presentation at PyTorch Conference 2024.
Quick start
1. Install with a PyTorch nightly (Python 3.11+). Replace cu132 with your CUDA or ROCm build.
git clone https://github.com/pytorch/torchtitan && cd torchtitan
pip install --pre torch --index-url https://download.pytorch.org/whl/nightly/cu132
pip install -r requirements.txt
2. Train a debug model. It uses the tokenizer bundled with the tests, so no download is needed.
NGPU=1 ./run_train.sh # Llama 3 debug model, 10 steps
3. Train a real model. Download the tokenizer (Llama weights require
access on Hugging Face), then
select a recipe by module and function name:
python scripts/download_hf_assets.py --repo_id meta-llama/Llama-3.1-8B --assets tokenizer --hf_token=...
MODULE=torchtitan_recipes.models.llama3 CONFIG=llama3_8b ./run_train.sh # 8 GPUs
4. Make it yours. A recipe is a Python function returning a complete config,
so changing a run means writing a function, not passing flags:
# my_recipes.py
from torchtitan_recipes.models.llama3 import llama3_8b
def llama3_8b_tp2():
config = llama3_8b()
config.parallelism.tensor_parallel_degree = 2
return config
MODULE=my_recipes CONFIG=llama3_8b_tp2 ./run_train.sh
See the configuration guide for the full model,
and torchtitan_recipes for the verified recipes.
This is the beginning of the README. Read the rest at the original. Read the full README ↗
