axolotl

🎉 Latest Updates

• 2026/10:
  • MistralAI's Mistral Large 4 is now supported in Axolotl, with merge-aware NVFP4 LoRA and expert-parallel training.
  • Reflection's Beam is now supported in Axolotl, with QLoRA and expert-parallel LoRA configs including NVFP4.
  • Distributed training improvements keep EP groups node-local when the topology permits, add DeepSpeed Ulysses support through context_parallel_size, and fix expert-LoRA resume and FSDP2 activation checkpointing with LoRA.
• 2026/09:
  • New model support has been added in Axolotl for MiniCPM5, Nex-N2.5-mini, Qwen3.6, Qwen3.8-Flash-Next, and K2 Horizon.
  • CPU-staged FSDP2 QLoRA loading lets you load models larger than a single GPU's memory without materializing a full quantized replica on each GPU.
  • GGUF export is now available via axolotl export, with optional quantization for deployment with llama.cpp and Ollama.
  • NVFP4 LoRA now supports native TorchAO dense weights and merge-aware training, optimizing against the quantized merged weights for deployment.
  • New training options: MixLoRA adds routed LoRA experts to dense models, and Dynamic Fine-Tuning (DFT) weights the SFT loss by token probability.
  • Ringmaster context parallelism adds Ulysses and combined Ulysses/Ring (USP) support, sample packing, and CP fixes for GDN, KDA, and Mamba/Mamba2 models.
  • Torch all-to-all Expert Parallelism enables MoE training over NVLink or PCIe without requiring DeepEP. Expanded N-D parallelism supports EP with FSDP2, HSDP, and context parallelism, plus FSDP2 + tensor parallelism.
  • Selective activation checkpointing adds rules to save specific projections or large matrix multiplications, with optional CPU offload to tune the memory/recomputation tradeoff.
  • Improved plugin integration includes new cloud provider interfaces for launching training jobs, with support for Nebius Serverless Jobs.
• 2026/08:
  • New model support has been added in Axolotl for Ling 3.0, Muse Glimmer, North Micro Vision Instruct and Shieldstral.
• 2026/07:
  • NVFP4 (4-bit) MoE LoRA training is now supported via ScatterMoE (W4A16) and SonicMoE (W4A4), including adapter merge back into a plain NVFP4 checkpoint.
• 2026/06:
  • Expert Parallelism (EP) for distributed MoE training via DeepEP, remote training through Tinker-compatible APIs, Context Parallelism for hybrid SSM models (Nemotron-H, Falcon-H1, Bamba), BitNet 1.58-bit fine-tuning, and a multimodal assistant-only loss-masking fix.

This is the beginning of the README. Read the rest at the original. Read the full README ↗