🚀 NeMo AutoModel
📖 Documentation • 🔥 Ready-to-Use Recipes • 💡 Examples • Model Coverage • Performance • 🤝 Contributing
📣 News and Discussions
-
[10/29/2026]Mistral Large 4 Fine-tune Mistral AI's 1T-A52B multimodal MoE model with expert, pipeline, and context parallelism. See the model card. Explore the MedPix SFT and 128K long-context packed SFT recipes.
-
[10/25/2026]Nanbeige4.3-33B Fine-tune Nanbeige's dense 33B agentic model on 128K packed sequences with context parallelism. See the model card. Explore the 128K SFT CP16 and 128K LoRA CP8 recipes.
-
[10/23/2026]NVIDIA Nemotron 3.5 Bolt Fine-tune NVIDIA-Nemotron-3.5-Bolt-E4B-4GB-BF16 with the MedPix full-parameter SFT and MedPix LoRA recipes, and NVIDIA-Nemotron-3.5-Bolt-74B-A9B-BF16 with the CORD-v2 LoRA recipe. See the E4B and 74B-A9B model coverage pages.
-
[10/23/2026]MiniMax-M3.1-Flash We now support fine-tuning MiniMax AI's MoE vision-language model, loading its NVFP4 and MXFP8 checkpoint directly. Check out the MedPix SFT EP128 recipe and the model coverage page.
-
[10/21/2026]Flint Fine-tune Recursive's 39.8B-A2.5B MoE model with expert and context parallelism. See the model card. Explore the SFT, LoRA, 128K packed TE, and 128K packed MagiAttention (H100) recipes.
-
[10/21/2026]EAGLE-4 We now support training EAGLE-4 drafters: the EAGLE-3.1 draft with a sliding-window layer and a low-rank candidate head over the full vocabulary, trained in two stages and exported to the LlamaForCausalLMEagle4 serving layout, many thanks to @Khazic. Check out the Qwen3-8B draft and candidate-head recipes and the speculative decoding overview.
-
[10/16/2026]Beam-1.0-501B-A23B We now support LoRA and full-parameter fine-tuning for Reflection AI's 501B-A23B sparse MoE with mixed sliding-window and full attention, using FSDP2, expert parallelism, and packed sequences. Check out the Tulu3 LoRA EP8 and Tulu3 SFT EP64 8-node recipes and the model coverage page.
-
[10/15/2026]Muse Spark Open Fine-tune Meta's MoE model with expert, pipeline, and context parallelism. See the model card. Explore the image-text SFT, LoRA, and 128K long-context packed SFT recipes.
-
[10/15/2026]NVIDIA Nemotron 3.5 Super VL Fine-tune the 120B-A12B vision-language model with CORD-v2 SFT, CORD-v2 PEFT (LoRA), 256K packed SFT (GB200). See the CORD-v2 tutorial for data preparation, training, and evaluation.
-
[10/14/2026]Step-5-Preview-BF16 We now support StepFun's 600B-A27B sliding-window and sparse-attention MoE with FSDP2, expert, pipeline and context parallelism. Check out the model coverage page and the Tulu3 SFT, LoRA, and 128K long-context CP32 recipes.
-
[10/12/2026]FLUX 3 Image Klein Base We added text-to-image and image-editing LoRA fine-tuning for Black Forest Labs' base image model. See the T2I LoRA recipe and model guide for checkpoint setup and the 100-step validation scope.
-
[10/04/2026]HunyuanImage-3.0 We now support text-to-image full fine-tuning and LoRA for Tencent's 80B-A13B MoE multimodal decoder with FSDP2 and expert parallelism, many thanks to @Khazic. Check out the full fine-tuning and LoRA recipes and the model coverage page.
-
[09/28/2026]Qwen-Image-2.1 We now support text-to-image full fine-tuning and LoRA for Qwen's 7B single-stream DiT conditioned on Qwen3-VL. Check out the full fine-tuning and LoRA recipes and the model coverage page.
-
[09/26/2026]MiMo-V2.5-Pro We now support full-parameter fine-tuning for Xiaomi's hybrid full and sliding-window attention Mixture-of-Experts model from its FP8 checkpoint. Check out the HellaSwag PP18/EP8 recipe and model coverage page.
-
[09/22/2026]MiMo-V2.6-Flash-RL We now support full-parameter fine-tuning for Xiaomi's multimodal Mixture-of-Experts model with expert, pipeline, and context parallelism. Check out the packed Tulu3 EP64/CP2 recipe, non-packed MedPix EP64 recipe, and model coverage page.
-
[09/11/2026]DeepSeek-V4.1-Flash We now support SFT and LoRA for DeepSeek's causal encoder/decoder backbone with CSA2, single-pass mHC, and Engram, many thanks to @Khazic. Check out the HellaSwag EP64 16-node recipe and model coverage page.
-
[08/29/2026]GLM-5.3 We now support full-parameter fine-tuning of Z.ai's GLM-5.3 with cuDNN DSA and HybridEP. Check out the EP64/PP4 recipe and model coverage page.
-
[08/28/2026]Ox Alpha / GLM-5.3-Flash We now support fine-tuning Z.ai's 320B-A18B hybrid-attention MoE VLM with packed CP and EP. Check out the MedPix EP72/CP2 recipe and model coverage page.
-
[08/26/2026]Qwen3.8-Flash-Next We now support full-parameter fine-tuning for the 180B hybrid-attention Engram MoE with FlexAttention sparse QSA, packed (THD) sequences, and model-owned context parallelism. Check out the HellaSwag EP64 recipe and model coverage page.
-
[08/21/2026]DFlash 2 We now support training DFlash 2 drafters: the DFlash block-diffusion backbone plus a two-tap dynamic convolution and a pairwise path selector for higher acceptance lengths. Check out the Qwen3-8B recipe and the speculative decoding overview.
-
[08/14/2026]Qwen3.8-27B We now support full-parameter SFT and LoRA fine-tuning for Qwen's dense 27B vision-language model. Check out the SFT and LoRA recipes, fine-tuning guide, and model coverage page.
-
[08/12/2026]Qwen3.8-2.4T-A95B We now support full-parameter fine-tuning for Qwen/Qwen3.8-2.4T-A95B checkpoints. Check out the HellaSwag EP32/PP8 recipe and model coverage page.
-
[08/12/2026]North Micro Vision We now support LoRA fine-tuning for Cohere Labs' 2.4B-parameter native-resolution vision-language model. Check out the RDR recipe and model coverage page.
-
[08/10/2026]MuseGlimmer We now support SFT/LoRA the dense 30B MuseGlimmer vision-language model, including TP/CP packed sequence recipes. Check out the recipes and model coverage page.
-
[08/08/2026]Nemotron 3.5 Lightning We now support LoRA fine-tuning for NVIDIA's 30B-A3B hybrid MoE model with multi-token prediction. Check out the HellaSwag recipe and model coverage page.
-
[07/30/2026]Inkling-Small We now support full-parameter fine-tuning for the 276B-parameter, 12B-active Inkling-Small model on 64 H100 GPUs. Check out the MedPix EP64 recipe and model coverage page.
-
[07/29/2026]Kimi K3 We now support full-parameter fine-tuning for Moonshot AI's 2.8T-parameter MoE model on NVIDIA GB200. Check out the HellaSwag EP32/PP8 recipe on 256 GB200.
-
[07/21/2026]Laguna S 2.1 We now support finetuning Poolside's 118B-A8B Laguna S 2.1 MoE model. Check out our HellaSwag EP16 recipe.
-
[07/18/2026]Inkling VLM MoE We now support finetuning Inkling. Check out our MedPix recipe.
-
[06/27/2026] Speculative decoding in NeMo AutoModel Train target-aligned drafters end to end: EAGLE-1/2/3, P-EAGLE, DFlash, and DeepSeek's newly released DSpark. Huge thanks to @Khazic for the speculative decoding stack and @kashif for DSpark support.
-
[06/21/2026]GLM-5.2 We now support finetuning zai-org/GLM-5.2 with IndexShare DSA, optional TileLang sparse kernels, and long-context CP recipes. Check out our 32K long-context recipe and model coverage page.
-
[06/15/2026]DiffusionGemma. We now support fine-tuning the google/diffusiongemma-26B-A4B-it model. Check out our recipe and guide.
-
[06/12/2026]MiniMax M3 We now support finetuning MiniMax's MiniMax-M3. Check out our recipe and guide.
-
[06/04/2026]Nemotron-3 Ultra We now support finetuning NVIDIA's Nemotron 3 Ultra 550B A55B. Check out our recipe and guide.
-
[06/03/2026]Gemma 4 12B We now support finetuning the dense google/gemma-4-12B model. Check out our recipe.
-
[05/27/2026]Step-3.7-Flash We added model coverage for Stepfun AI's 198B-A13B MoE vision-language model, targeting image/video agentic developer workflows with a 256k context language backbone and 1.8B ViT vision tower. See the model coverage page.
-
[05/19/2026]Ling 2.0 We now support finetuning the inclusionAI Ling 2.0 MoE family (inclusionAI/Ling-mini-2.0, inclusionAI/Ling-flash-2.0, and inclusionAI/Ling-1T), thanks to @Hayden727. Check out our recipes.
-
[05/17/2026]ERNIE 4.5 and MiMo-V2-Flash We now support finetuning baidu/ERNIE-4.5-0.3B-PT, baidu/ERNIE-4.5-21B-A3B-PT, and XiaomiMiMo/MiMo-V2-Flash. Check out our ERNIE dense recipe, ERNIE MoE recipe, and MiMo recipe.
-
[04/29/2026]Mistral Medium 3.5 We now support finetuning Mistral AI's 128B FP8-native VLM Mistral Medium 3.5. Check out our recipe and guide.
-
[04/28/2026]Nemotron-3-Nano-Omni We now support finetuning nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16, NVIDIA's 30B-A3B omnimodal MoE (text · image · audio) with NemotronH hybrid Mamba+Attention backbone. Check out our SFT recipe, LoRA recipe, and guide.
-
[04/28/2026]Hy3-preview We now support finetuning tencent/Hy3-preview, thanks to @Khazic. Check out our recipe.
-
[04/25/2026]DeepSeek V4 Flash We now support finetuning deepseek-ai/DeepSeek-V4-Flash, thanks to @Khazic. Check out our recipe and guide.
-
[04/22/2026]Qwen3.6-27B We now support finetuning Qwen/Qwen3.6-27B. Check out our recipe.
-
[04/20/2026]Qwen-Image We now support finetuning Qwen/Qwen-Image, thanks to @harshareddy832. Check out our recipe.
-
[04/16/2026]Qwen3.6 MoE We now support finetuning Qwen/Qwen3.6-35B-A3B. Check out our recipe.
-
[04/16/2026]LLaVA-OneVision-1.5 We now support finetuning lmms-lab/LLaVA-OneVision-1.5-4B-Instruct, thanks to @vgauraha62. Check out our recipe.
-
[04/12/2026]MiniMax-M2.7 We now support finetuning MiniMaxAI/MiniMax-M2.7. Check out our recipe.
-
[04/07/2026]GLM-5.1 We now support finetuning zai-org/GLM-5.1. GLM-5.1 is Zhipu AI's latest open-source MoE model featuring MLA + DeepSeek Sparse Attention. Check out our recipe and discussion.
-
[04/02/2026]Gemma 4 We support fine-tuning for Gemma4 (2B, 4B, 31B, 26BA4B)! Check out our recipes.
-
[03/30/2026]NeMo AutoModel ships with agent-friendly skills in skills/ to help you with common development tasks (e.g., running a recipe, model onboarding, development) across the repo. We welcome PRs that improve existing skills or add new ones.
-
[03/16/2026]Mistral Small 4 We support fine-tuning for Mistral4 119B! Check out our recipe.
-
[03/11/2026]Nemotron Super v3 We support fine-tuning for nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16. Check out our recipe.
-
[03/11/2026]GLM-5 We now support finetuning zai-org/GLM-5. Check out our recipe.
-
[03/02/2026]Qwen3.5 small models We support finetuning for Qwen3.5 small models 0.8B, 2B, 4B (recipe) and 9B (recipe)
-
[02/16/2026]Qwen3.5 MoE We support finetuning for Qwen/Qwen3.5-397B-A17B (recipe) and Qwen/Qwen3.5-35B-A3B (recipe)
Overview
Nemo AutoModel is a Pytorch DTensor‑native SPMD open-source training library under NVIDIA NeMo Framework, designed to streamline and scale training and finetuning for LLMs, VLMs, diffusion models, and retrieval models. Designed for flexibility, reproducibility, and scale, NeMo AutoModel enables both small-scale experiments and massive multi-GPU, multi-node deployments for fast experimentation in research and production environments.
What you can expect:
- Hackable with a modular design that allows easy integration, customization, and quick research prototypes.
- Minimal ceremony: YAML-driven recipes; override any field using CLI.
- High performance and flexibility with custom kernels and DTensor support.
- Seamless integration with Hugging Face for day-0 model support, ease of use, and wide range of supported models.
- Efficient resource management using Kubernetes and Slurm, enabling scalable and flexible deployment across configurations.
- Documentation with step-by-step guides and runnable examples.
Why PyTorch Distributed and SPMD
- One program, any scale: The same training script runs on 1 GPU or 1000+ by changing the mesh.
- PyTorch Distributed native: Partition model/optimizer states with
DeviceMesh + placements (Shard, Replicate).
- SPMD first: Parallelism is configuration. No model rewrites when scaling up or changing strategy.
- Decoupled concerns: Model code stays pure PyTorch; parallel strategy lives in config.
- Composability: Mix tensor, sequence, and data parallel by editing placements.
- Portability: Fewer bespoke abstractions; easier to reason about failure modes and restarts.
Table of Contents
TL;DR: SPMD turns “how to parallelize” into a runtime layout choice, not a code fork.
Feature Roadmap
✅ Implemented on main | 🔜 Planned
For released features, see the release notes. Docker examples target the 26.10 container.
High-throughput scalable training
- ✅ PyTorch DTensor-native SPMD training Same training script can scale from 1 GPU to large multi-node jobs by changing the device mesh/config.
- ✅ Composable Parallelism - PyTorch native FSDP2, HSDP, TP, CP, SP and PP for distributed training.
- ✅ Optimized kernels - Uses NVIDIA-oriented kernel paths such as Transformer Engine, DeepEP, FlexAttn, TorchSDPA, fused attention, rotary embeddings, Triton, and optional kernel patches.
- ✅ MoE acceleration - Includes MoE routing and DeepEP integration, plus expert-parallel configurations used in DeepSeek, Qwen MoE, GPT-OSS, and Nemotron MoE benchmarks.
- ✅ FP8 and mixed precision - FP8 support with torchao and Transformer Engine.
- ✅ MXFP8 MoE training - Transformer Engine and torchao MXFP8 grouped-expert training on GB200.
- ✅ Activation checkpointing - Trades recomputation for lower activation memory, especially useful with FSDP and memory-efficient losses.
- ✅ Memory-efficient loss - Linear-Cut / fused linear cross entropy avoids materializing full logits for the loss, reducing output-layer memory pressure.
- ✅ Sequence packing - Packs variable-length examples together to reduce padding compute and improve GPU utilization.
- ✅ FlashAttention packed-sequence support - Packed masks can feed variable-length FlashAttention paths using per-document cu_seqlens.
- ✅ DCP - Supports PyTorch DCP and SafeTensors, sharded and consolidated layouts, merge/reshard utilities, and Hugging Face-compatible outputs.
- ✅ Async checkpointing - Can write checkpoints in the background to reduce training stalls caused by I/O.
- ✅ Dion and Muon optimizers - Distributed optimizer integrations with typed recipe configuration.
- ✅ Environment Support - SLURM, interactive, SkyPilot, and Kubernetes (via SkyPilot) launchers.
SOTA algorithms
- ✅ Pre-training - Support for model pre-training, including DeepSeekV3.
- ✅ Learning Algorithms - SFT (Supervised Fine-Tuning), PEFT (LoRA, QLoRA), and QAT (Quantization-Aware Training).
- ✅ Knowledge distillation - Support for knowledge distillation with LLMs.
- ✅ VLM knowledge distillation - Chunked KD loss and a Qwen3.5-VL teacher-student recipe.
- ✅ Speculative decoding training - Train target-aligned EAGLE-1/2/3/4, P-EAGLE, DFlash, DFlash 2, and DSpark drafters for faster verified generation.
Model Coverage and 🤗 Ecosystem compatibility
- ✅ Transformers v5 🤗 - Built on latest transformers with device-mesh driven parallelism.
- ✅ 🤗 HuggingFace Integration - Works with dense models (e.g., Qwen, Llama3, etc) and large MoEs (e.g., DSv3, DSv4).
- ✅ VLM - Finetuning for VLMs (Qwen2.5/3/3.5/3.6 VL, Gemma-3/3n/4 VL, Mistral 3.5/4, LLaVA-OneVision-1.5, Kimi-VL, etc.).
- ✅ Omnimodal - Finetuning for omnimodal MoE models (Nemotron-3-Nano-Omni, Qwen3-Omni).
- ✅ Diffusion - Pretraining and LoRA finetuning for image/video diffusion models (Qwen-Image, Qwen-Image-2.1, FLUX, FLUX 3 Image Klein Base, Wan2.1, Wan2.2-T2V-A14B, Hunyuan).
- ✅ dLLM - Discrete diffusion LM finetuning (LLaDA, LLaDA2, Nemotron-Labs-Diffusion, DiffusionGemma).
- ✅ Retrieval - Bi-encoder and cross-encoder training with in-batch negative sampling.
- ✅ Extended MoE support - GPT-OSS, Kimi K3, Qwen3 / Qwen3.5 / Qwen3.6 MoE, Qwen-next, MiniMax-M2.x, GLM-4.7 / GLM-5 / GLM-5.1 / GLM-5.2 / GLM-5.3 / GLM-5.3-Flash, DeepSeek V3.2 / V4 / V4-Flash, ERNIE 4.5, MiMo-V2-Flash / MiMo-V2.5-Pro, Ling 2.0, Hy3-preview.
Agentic Development and UX
- ✅ Agent-friendly skills - Curated
skills/ for common dev tasks (recipe runs, model onboarding, CI).
Planned
- 🔜 Unified Engine API and recipes - Introduce a common engine and consolidate the LLM and VLM recipe paths.
- 🔜 Composable component configuration - Complete the typed config and
.build() refactor across data and remaining components.
- 🔜 Packed long-context training with CP - Combine THD sequence packing with context parallelism, including DeepSeek V4 coverage.
- 🔜 Kernel and runtime upgrades - Add partial CUDA graphs, evaluate FlashAttention 3/4, and upgrade to DeepEP v2.
- 🔜 Multimodal retrieval - Expand vision-language retrieval datasets and models and improve retriever performance.
- 🔜 Convergence and regression testing - Add automated loss, evaluation, and performance gates to NeMo CI.
- 🔜 Transformers 5.12 alignment - Upgrade the Transformers integration and restore affected model parity coverage.
- 🔜 Recipe consistency - Align diffusion recipes with the shared AutoModel recipe and configuration patterns.
Getting Started
Run a 10-step Qwen3-0.6B fine-tuning example on two GPUs with the container:
mkdir -p checkpoints
docker run --gpus all -it --rm --shm-size=8g \
-v "$PWD/checkpoints:/opt/Automodel/checkpoints" \
nvcr.io/nvidia/nemo-automodel:26.10.00 \
automodel /opt/Automodel/examples/llm_finetune/qwen/qwen3_0p6b_hellaswag.yaml \
--nproc-per-node 2 \
--step_scheduler.max_steps 10
The model and HellaSwag dataset are public; no Hugging Face token is required. Checkpoints are saved to the mounted checkpoints directory.
For source installation, use uv for reproducible Python environments.
# Setup environment before running any recipes
uv venv
# Choose ONE:
uv sync --frozen # LLM recipes (default)
# uv sync --frozen --extra vlm --extra vlm-media # VLM recipes (Qwen/Mistral/Omni need vlm-media for video/vision; fixes: ImportError: qwen_vl_utils is not installed)
# uv sync --frozen --extra cuda # Optional CUDA deps (e.g., Transformer Engine, Mamba SSM)
# uv sync --frozen --extra cuda_source # Optional bitsandbytes dependency
# uv sync --frozen --extra all # Most optional deps (includes `vlm` and `cuda`; NOTE: excludes media — add --extra media for video/image decode)
# uv sync --frozen --all-extras # Everything (includes `fa`, `moe`, `media`, etc.)
# One-off runs (examples):
# uv run --extra vlm <command>
# uv run --extra cuda <command>
uv run python -c "import nemo_automodel; print('NeMo AutoModel ready')"
Run a Recipe
All recipes are launched via the automodel CLI (or its short alias am). Each YAML config specifies the recipe class and all training parameters:
# LLM example: multi-GPU fine-tuning of Qwen3-0.6B with FSDP2 (no HF token required)
automodel examples/llm_finetune/qwen/qwen3_0p6b_hellaswag.yaml --nproc-per-node 8
# VLM example: single-GPU fine-tuning (Gemma-3-VL) with LoRA
automodel examples/vlm_finetune/gemma3/gemma3_vl_4b_cord_v2_peft.yaml
# Both commands also work with uv run:
uv run automodel examples/llm_finetune/qwen/qwen3_0p6b_hellaswag.yaml --nproc-per-node 8
[!TIP]
NeMo-Run submission: The cli extra adds NeMo Run to the base package: uv pip install "nemo-automodel[cli]". It is additive; the base package still installs its core training dependencies, including PyTorch.
LLM Pre-training
LLM Pre-training Single Node
We provide an example SFT experiment using the FineWeb dataset with a nano-GPT model, ideal for quick experimentation on a single node.
automodel examples/llm_pretrain/nanogpt_pretrain.yaml --nproc-per-node 8
LLM Supervised Fine-Tuning (SFT)
We provide an example SFT experiment using the HellaSwag dataset.
LLM SFT Single Node
The default SFT configuration is set to run on a single GPU. To start the experiment:
automodel examples/llm_finetune/qwen/qwen3_0p6b_hellaswag.yaml
This fine-tunes Qwen3-0.6B on HellaSwag using a single GPU. The model and dataset are public; no Hugging Face token is required.
To use multiple GPUs on a single node, add the --nproc-per-node argument:
automodel examples/llm_finetune/qwen/qwen3_0p6b_hellaswag.yaml --nproc-per-node 8
LLM SFT Multi Node
To launch on a SLURM cluster, copy the reference sbatch script and adapt it to your cluster:
cp slurm.sub my_cluster.sub
# Edit my_cluster.sub — change CONFIG, #SBATCH directives, container, mounts, etc.
sbatch my_cluster.sub
All cluster-specific settings (nodes, GPUs, partition, container, mounts) live in your sbatch script.
NeMo-Run (nemo_run:) sections are also supported -- see our
cluster guide for details.
LLM Parameter-Efficient Fine-Tuning (PEFT)
We provide a PEFT example using the HellaSwag dataset.
LLM PEFT Single Node
# Memory-efficient SFT with LoRA
automodel examples/llm_finetune/qwen/qwen3_0p6b_hellaswag_peft.yaml
# Override any YAML parameter via the command line:
automodel examples/llm_finetune/qwen/qwen3_0p6b_hellaswag_peft.yaml \
--step_scheduler.local_batch_size 16
[!NOTE]
Launching a multi-node PEFT example uses the same sbatch slurm.sub workflow as the SFT case above.
VLM Supervised Fine-Tuning (SFT)
We provide a VLM SFT example using Qwen2.5-VL for end-to-end fine-tuning on image-text data.
VLM SFT Single Node
# Qwen2.5-VL on 8 GPUs
automodel examples/vlm_finetune/qwen2_5/qwen2_5_vl_3b_rdr.yaml --nproc-per-node 8
VLM Parameter-Efficient Fine-Tuning (PEFT)
We provide a VLM PEFT (LoRA) example for memory-efficient adaptation with Gemma3 VLM.
VLM PEFT Single Node
# Gemma-3-VL PEFT on 8 GPUs
automodel examples/vlm_finetune/gemma3/gemma3_vl_4b_medpix_peft.yaml --nproc-per-node 8
Supported Models
NeMo AutoModel provides native support for a wide range of models available on the Hugging Face Hub, enabling efficient fine-tuning for various domains. Below is a small sample of ready-to-use families (train as-is or swap any compatible 🤗 causal LM), you can specify nearly any LLM/VLM model available on 🤗 hub:
[!NOTE]
Check out more LLM and VLM examples. Any causal LM on Hugging Face Hub can be used with the base recipe template, just overwrite --model.pretrained_model_name_or_path <model-id> in the CLI or in the YAML config.
Performance
NeMo AutoModel achieves great training performance on NVIDIA GPUs. Below are highlights from our benchmark results:
| Model | #GPUs | Seq Length | Model TFLOPs/sec/GPU | Tokens/sec/GPU | Kernel Optimizations |
|---|
| DeepSeek V3 671B | 256 | 4096 | 250 | 1,002 | TE + DeepEP |
| GPT-OSS 20B | 8 | 4096 | 279 | 13,058 | TE + DeepEP + FlexAttn |
| Qwen3 MoE 30B | 8 | 4096 | 212 | 11,842 | TE + DeepEP |
For complete benchmark results including configuration details, see the Performance Summary.
🔌 Interoperability
- NeMo RL: Use AutoModel checkpoints directly as starting points for DPO/RM/GRPO pipelines.
- Hugging Face: Train any LLM/VLM from 🤗 without format conversion.
- Megatron Bridge: Optional conversions to/from Megatron formats for specific workflows.
🗂️ Project Structure
NeMo-Automodel/
├── cli/ # `automodel` / `am` CLI entry-point
│ └── app.py
├── docker/ # Container build files
├── docs/ # Documentation and guides
├── examples/
│ ├── convergence/ # Convergence test configs
│ ├── diffusion/ # Diffusion pretrain/finetune configs
│ ├── dllm_sft/ # Discrete diffusion LM SFT configs
│ ├── dllm_generate/ # Discrete diffusion LM generation
│ ├── llm_benchmark/ # LLM benchmarking configs
│ ├── llm_finetune/ # LLM finetune YAML configs
│ ├── llm_kd/ # LLM knowledge-distillation configs
│ ├── llm_pretrain/ # LLM pretrain configs
│ ├── llm_seq_cls/ # LLM sequence classification configs
│ ├── retrieval/ # Bi-encoder / cross-encoder configs
│ ├── vlm_benchmark/ # VLM benchmarking configs
│ ├── vlm_finetune/ # VLM finetune configs
│ └── vlm_generate/ # VLM generation configs
├── nemo_automodel/
│ ├── _diffusers/ # HF Diffusers integration (NeMoAutoDiffusionPipeline)
│ ├── _transformers/ # HF Transformers integration
│ ├── components/ # Core library
│ │ ├── _peft/ # PEFT implementations (LoRA, QLoRA)
│ │ ├── attention/ # Attention implementations
│ │ ├── checkpoint/ # Distributed checkpointing
│ │ ├── config/
│ │ ├── datasets/ # LLM, VLM, diffusion, retrieval datasets
│ │ ├── distributed/ # FSDP2, Megatron FSDP, pipelining, CP, etc.
│ │ ├── launcher/ # Launcher backends (SLURM, NeMo-Run, SkyPilot)
│ │ ├── loggers/ # Loggers
│ │ ├── loss/ # Optimized loss functions
│ │ ├── models/ # User-defined model examples
│ │ ├── moe/ # Optimized kernels for MoE models
│ │ ├── optim/ # Optimizer/LR scheduler components (incl. Dion)
│ │ ├── quantization/ # FP8, QAT, QLoRA
│ │ ├── training/ # Train utils
│ │ └── utils/ # Misc utils
│ ├── recipes/
│ │ ├── llm/ # Main LLM train loop
│ │ ├── vlm/ # Main VLM train loop
│ │ ├── diffusion/ # Diffusion training loop
│ │ ├── dllm/ # Discrete diffusion LM training loop
│ │ └── retrieval/ # Retrieval / biencoder training loop
│ └── shared/
├── tools/ # Developer tooling
└── tests/ # Comprehensive test suite
Citation
If you use NeMo AutoModel in your research, please cite it using the following BibTeX entry:
@misc{nemo-automodel,
title = {NeMo AutoModel: DTensor-native SPMD library for scalable and efficient training},
howpublished = {\url{https://github.com/NVIDIA-NeMo/Automodel}},
year = {2025--2026},
note = {GitHub repository},
}
🤝 Contributing
We welcome contributions! Please see our Contributing Guide for details.
📄 License
NVIDIA NeMo AutoModel is licensed under the Apache License 2.0.
