EasyR1: An Efficient, Scalable, Multi-Modality RL Training Framework
Check our new open-source project —🐧 PenguinHarness: Local-first multi-agent app development platform - create, fine-tune, deploy AI apps automatically!
Follow us and give us a star ⭐: https://github.com/Prism-Shadow/penguin-harness
https://github.com/user-attachments/assets/9b7033e8-f08a-4c3f-bd33-547896664e6e
This project is a clean fork of the original veRL project to support vision language models, we thank all the authors for providing such a high-performance RL training framework.
EasyR1 is efficient and scalable due to the design of HybirdEngine and the latest release of vLLM's SPMD mode.
Features
-
Supported models
- Llama3/Qwen2/Qwen2.5/Qwen3 language models
- Qwen2-VL/Qwen2.5-VL/Qwen3-VL vision language models
- DeepSeek-R1 distill models
-
Supported algorithms
- GRPO
- DAPO
- Reinforce++
- ReMax
- RLOO
- GSPO
- CISPO
-
Supported datasets
-
Supported tricks
- Padding-free training
- LoRA training
- Resuming from the latest/best checkpoint
- Wandb & SwanLab & Mlflow & Tensorboard tracking
Requirements
Software Requirements
- Python 3.9+
- transformers>=4.54.0
- flash-attn>=2.4.3
- vllm>=0.8.3
We provide a Dockerfile to easily build environments.
We recommend using the pre-built docker image in EasyR1.
docker pull hiyouga/verl:ngc-th2.8.0-cu12.9-vllm0.11.0
docker run -it --ipc=host --gpus=all hiyouga/verl:ngc-th2.8.0-cu12.9-vllm0.11.0
If your environment does not support Docker, you can consider using Apptainer:
apptainer pull easyr1.sif docker://hiyouga/verl:ngc-th2.8.0-cu12.9-vllm0.11.0
apptainer shell --nv --cleanenv --bind /mnt/your_dir:/mnt/your_dir easyr1.sif
Use USE_MODELSCOPE_HUB=1 to download models from the ModelScope hub.
Hardware Requirements
* estimated
| Method | Bits | 1.5B | 3B | 7B | 32B | 72B |
|---|
| GRPO Full Fine-Tuning | AMP | 2*24GB | 4*40GB | 8*40GB | 16*80GB | 32*80GB |
| GRPO Full Fine-Tuning | BF16 | 1*24GB | 1*40GB | 4*40GB | 8*80GB | 16*80GB |
| GRPO LoRA Fine-Tuning | AMP | 1*12GB | 1*24GB | 2*32GB | 2*80GB | 4*80GB |
[!NOTE]
Use worker.actor.fsdp.torch_dtype=bf16 and worker.actor.optim.strategy=adamw_bf16 to enable bf16 training.
Tutorial: Run Qwen2.5-VL GRPO on Geometry3K Dataset in Just 3 Steps
Installation
git clone https://github.com/hiyouga/EasyR1.git
cd EasyR1
pip install -e .
GRPO Full Training
bash examples/qwen2_5_vl_7b_geo3k_grpo.sh
GRPO LoRA Training
bash examples/qwen3_vl_4b_geo3k_grpo_lora.sh
Merge Checkpoint in Hugging Face Format
python3 scripts/model_merger.py --local_dir checkpoints/easy_r1/exp_name/global_step_1/actor
[!TIP]
If you encounter issues with connecting to Hugging Face, consider using export HF_ENDPOINT=https://hf-mirror.com.
If you want to use SwanLab logger, consider using bash examples/qwen2_5_vl_7b_geo3k_swanlab.sh.
Custom Dataset
Please refer to the example datasets to prepare your own dataset.
How to Understand GRPO in EasyR1
How to Run 70B+ Model in Multi-node Environment
- Start the Ray head node.
ray start --head --port=6379 --dashboard-host=0.0.0.0
- Start the Ray worker node and connect to the head node.
ray start --address=<head_node_ip>:6379
- Check the Ray resource pool.
ray status
- Run training script on the Ray head node only.
bash examples/qwen2_5_vl_7b_geo3k_grpo.sh
See the veRL's official doc for more details about multi-node training and Ray debugger.
Other Baselines
We also reproduced the following two baselines of the R1-V project.
- CLEVR-70k-Counting: Train the Qwen2.5-VL-3B-Instruct model on counting problem.
- GeoQA-8k: Train the Qwen2.5-VL-3B-Instruct model on GeoQA problem.
Performance Baselines
See baselines.md.
This is the beginning of the README. Read the rest at the original. Read the full README ↗
