Office Hours

DeepSpeed hosts regular office hours on the last Tuesday of each month at 12:00 America/New_York to discuss development plans, features, etc. This meeting is public for anyone to join and ask questions. The meeting is hosted on Zoom and can be joined here.

Latest News

• [2026/05] Using Muon Optimizer with DeepSpeed

• [2026/05] System DMA (SDMA) for ZeRO-3: offload collectives off compute units on AMD GPUs for better overlap

• [2026/03] DeepSpeed Team gave a tutorial at ASPLOS 2026 titled "Building Efficient Large-Scale Model Systems with DeepSpeed: From Open-Source Foundations to Emerging Research"

• [2026/03] Our SuperOffload work received an Honorable Mention for the ASPLOS 2026 Best Paper Award

• [2025/12] DeepSpeed Core API updates: PyTorch-style backward and low-precision master states

• [2025/11] DeepSpeed ZeRO++ powers large-scale distillation training of LLMs for Recommendation Systems at LinkedIn

• [2025/10] We hosted the Ray x DeepSpeed Meetup at Anyscale. We shared our most recent work on SuperOffload, ZenFlow, Muon Optimizer Support, Arctic Long Sequence Training and DeepCompile. Please find the meetup slides here.

• [2025/10] SuperOffload: Unleashing the Power of Large-Scale LLM Training on Superchips

• [2025/10] Study of ZenFlow and ZeRO offload performance with DeepSpeed CPU core binding

• [2025/08] ZenFlow: Stall-Free Offloading Engine for LLM Training

• [2025/06] Arctic Long Sequence Training (ALST) with DeepSpeed: Scalable And Efficient Training For Multi-Million Token Sequences

• [2025/06] DeepNVMe: Affordable I/O scaling for Deep Learning Applications

---

Extreme Speed and Scale for DL Training

DeepSpeed enabled the world's most powerful language models (at the time of this writing) such as MT-530B and BLOOM. DeepSpeed offers a confluence of system innovations, that has made large scale DL training effective, and efficient, greatly improved ease of use, and redefined the DL training landscape in terms of scale that is possible. These innovations include ZeRO, ZeRO-Infinity, 3D-Parallelism, Ulysses Sequence Parallelism, DeepSpeed-MoE, etc.

---

DeepSpeed Adoption

DeepSpeed was an important part of Microsoft’s AI at Scale initiative to enable next-generation AI capabilities at scale, where you can find more information here.

DeepSpeed has been used to train many different large-scale models, below is a list of several examples that we are aware of (if you'd like to include your model please submit a PR):

This is the beginning of the README. Read the rest at the original. Read the full README ↗