prime-rl

---

---

Overview

prime-rl is a framework for large-scale reinforcement learning. It is designed to be easy to use and hackable, yet capable of scaling to 1000+ GPUs. Here is what we think sets it apart:

1. Fully asynchronous RL for high-throughput agentic training at scale.
2. Performant: built to train 1T+ MoE models on 1000+ GPUs with FSDP2 for training and vLLM for inference, with FP8 inference, PD disaggregation, EP and CP parallelism, and more.
3. Native integration with verifiers environments through the Environments Hub, including built-in support for SWE and agentic environments.
4. End-to-end post-training: SFT, RL training, and evals.
5. Multi-node deployment with Slurm support.
6. Multimodal support for Qwen3.5 and Qwen3.8 VLMs, with image RL and SFT examples.
7. Hackable, modular, and extensible by design.
8. One-line SLURM deployment for frontier models — e.g. GLM-5 FP8 with P/D disaggregation, the llm-d router, and Mooncake KV offload.

Models support

The trainer runs the PrimeRL model implementations under src/prime_rl/trainer/models/: dense Llama, Qwen3 and Qwen3.5, plus the families in the table below, with expert parallelism (EP) for MoE layers, context parallelism (CP) for long sequences, and additional kernels like quack-kernels. Other architectures fail at trainer setup.

Family	Example IDs	MoE	EP	CP
GLM-5 (glm_moe_dsa)	zai-org/GLM-5, zai-org/GLM-5-FP8	yes	✅	✅
GLM-5.3 Flash, text only (glm5_next)	zai-org/GLM-5.3-Flash-BF16	yes	✅	✅
Qwen3 MoE (qwen3_moe)	Qwen/Qwen3-30B-A3B, …	yes	✅	✅
Qwen3.5 MoE (qwen3_5_moe)	Qwen/Qwen3.5-35B-A3B, …	yes	✅	✅
Qwen3.5 VLMs	see advanced.md (qwen3_5, qwen3_5_moe)	MoE only on MoE VLMs	MoE only	✅
Qwen3.8 Flash Next VLM (qwen4_exp)	Qwen/Qwen3.8-Flash-Next	yes	✅	✅
Poolside Laguna (laguna)	poolside/Laguna-XS.2	yes	✅	✅
MiniMax M2 (minimax_m2)	MiniMax/MiniMax-M2	yes	✅	✅
Muse Glimmer, text only (muse_glimmer)	meta-models/Muse-Glimmer-30B	no	—	✅
Nemotron H (nemotron_h)	nvidia/Nemotron-3-Nano-30B-A3B, nvidia/Nemotron-3-Super-120B-A12B, …	yes	✅	✅
Nemotron H Omni (nemotron_h_omni, text only)	nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-BF16	yes	✅	✅
Trinity (afmoe)	arcee-ai/Trinity-Mini, …	yes	✅	✅
GLM-4 · GLM-4.5 MoE · INTELLECT-3 (glm4_moe)

Setup

We develop and test on NVIDIA RTX 3090/4090/5090, A100, H100, H200, and B200. If your setup fails, please create an issue.

Prerequisites

Currently, you need at least one NVIDIA GPU to use prime-rl. If you don't already have access to one, we recommend our compute platform for everything from renting on-demand single GPUs for developing, debugging and small ablations, to reserving 1000+ GPU clusters for production-scale training.

Quick Setup

Set up prime-rl in a single command.

curl -sSL https://raw.githubusercontent.com/PrimeIntellect-ai/prime-rl/main/scripts/install.sh | bash

1. Clone the repository

git clone https://github.com/PrimeIntellect-ai/prime-rl.git
cd prime-rl

2. Initialize submodules

git submodule update --init -- deps/verifiers deps/renderers deps/prime-envs deps/pydantic-config

3. Install uv

curl -LsSf https://astral.sh/uv/install.sh | sh
source $HOME/.local/bin/env

4. Install dependencies from the lock file

uv sync --all-extras

NOTE: Environments are opt-in uv workspace members — uv sync --all-extras does not install them. To train on them, install all with uv sync --all-extras --all-packages, or a subset with uv sync --package prime-rl --package <env>.

1. Check that the environment uses Python 3.12

uv run python -V

2. Check that flash-attn is installed

uv run python -c "import flash_attn"

3. Check that you can run SFT trainer (this requires 1 GPU)

uv run sft @ configs/debug/fake/sft.toml

4. Check that you can run the inference server (this requires 1 GPU)

uv run inference --vllm.model Qwen/Qwen3-0.6B

5. Check that the full RL stack (inference + orchestrator + trainer) runs end-to-end (this requires 2 GPUs)

uv run rl @ examples/basic/reverse-text/rl.toml

Additional Setup

1. If you want to log your runs to W&B, log in

uv run wandb login
# Or set `export WANDB_API_KEY=...`

2. If you require gated/ private models or datasets from HuggingFace, log in

uv run hf auth login
# Or set `export HF_TOKEN=...`

Training Examples

We provide end-to-end training examples in the examples directory to highlight features of the framework and guide you through the process of training your own models.

Basic Training: 1 to 8 GPUs

Follow this guide to learn the basics of prime-rl. You can train your own models on 1 to 8 GPUs. Ideal for getting started and exploring the capabilities of the framework. These guides cover most use cases -- single-turn, multi-turn, tool calling, etc. -- on toy environments and small models.

This is the beginning of the README. Read the rest at the original. Read the full README ↗