Training Open Instruction-Following Language Models

This repo serves as an open effort on instruction-tuning and post-training popular pretrained language models on publicly available datasets. We release this repo and will keep updating it with:

1. Code for finetuning language models with latest techniques and instruction datasets in a unified format.
2. Code for DPO, preference finetuning and reinforcement learning with verifiable rewards (RLVR).
3. Checkpoints or other useful artifacts that we build in our exploration.

Olmo 3 results: The live Papers with Code badges below track Olmo 3 Think 32B on BBH and Olmo 3.1 Think 32B on MATH. See Table 14 of the Olmo 3 paper for the reported scores.

We also support some evaluations natively in the codebase, but these are now unmaintained and instead we suggest using OLMES, which we used for TÜLU 3.

Setup

Our setup follows our Dockerfile. Note that Open Instruct is a research codebase and does not guarantee backward compatibility.

Installation with uv

We use uv for installation and running code. You can install with uv sync.

• Git LFS (for running tests): Install Git LFS and run git lfs install before cloning. See CONTRIBUTING.md for details.

• Docker installation: You can also use the Dockerfile to build a Docker image. You can build the image with the following command:

docker build . \
    --build-arg GIT_COMMIT=$(git rev-parse --short HEAD) \
	--build-arg GIT_BRANCH=$(git rev-parse --abbrev-ref HEAD) \
	-t open_instruct_dev

Training

After having setup the environment, you are ready to launch some experiments. We provide a few examples below. To learn more about how to reproduce the Tulu 3 models, please refer to the Tulu 3 README. The instructions and documentations for Tulu 1 and Tulu 2 are in Tulu 1 and 2 README.

The launch commands below run on Ai2's Beaker clusters. scripts/train/build_image_and_launch.sh builds an image from your current commit and passes it to the chosen script. Most of these scripts pass mason's --preemptible, so on clusters that enforce priority they run only as backfill and can wait indefinitely. Scripts that launch evals (--try_launch_beaker_eval_jobs_on_weka) also need oe-eval-internal cloned at the repository root.

Finetuning

OLMo-core SFT: For supported models (OLMo, OLMoE, Qwen3), we recommend the more GPU-efficient OLMo-core SFT implementation. See open_instruct/olmo_core_utils.py for the list of supported models. Our entry point is open_instruct/olmo_core_finetune.py:

# Single-GPU OLMo-core SFT smoke test on Beaker (Qwen3-0.6B).
./scripts/train/build_image_and_launch.sh scripts/train/debug/oc_sft.sh

# Two-node 8xGPU OLMo-core SFT run.
./scripts/train/build_image_and_launch.sh scripts/train/debug/oc_sft_multinode.sh

Preference Tuning

We train DPO with open_instruct/dpo.py:

# Single-GPU DPO smoke test on Beaker (OLMo-2 1B).
./scripts/train/build_image_and_launch.sh scripts/train/debug/dpo/single_gpu.sh

# Two-node 8xGPU DPO run.
./scripts/train/build_image_and_launch.sh scripts/train/debug/dpo/multi_node.sh

Reinforcement Learning with Verifiable Rewards (RLVR)

For supported Olmo MoE models, use the MILES + OLMo-core integration. The existing Core/vLLM and DeepSpeed/vLLM GRPO workflows remain available for dense-model training; the commands below use DeepSpeed/vLLM.

We train with open_instruct/grpo_fast.py:

# Single-GPU smoke test on Beaker (small model, fast).
./scripts/train/build_image_and_launch.sh scripts/train/debug/single_gpu_on_beaker.sh

# Two-node 8xGPU run (Qwen2.5-7B on code RLVR).
./scripts/train/build_image_and_launch.sh scripts/train/debug/large_test_script.sh

Contamination checks

We release our scripts for measuring the overlap between instruction tuning datasets and evaluation datasets in ./decontamination. See the README for more details.

Developing

When submitting a PR to this repo, we check the core code in open_instruct/ for style with the following:

make style
make quality

Run the tests with uv run pytest.

Pre-commit hooks

To automatically run linting and formatting on each commit:

uv run pre-commit install

To run on all files (recommended after initial setup):

uv run pre-commit run --all-files

Repo structure

├── assets/                     <- Images, licenses, etc.
├── configs/
|     ├── beaker_configs/       <- AI2 Beaker configs
|     ├── ds_configs/           <- DeepSpeed configs
|     └── train_configs/        <- Training configs
├── decontamination/            <- Scripts for measuring train-eval overlap
├── docs/                       <- Documentation (MkDocs)
├── human_eval/                 <- Human evaluation interface (not maintained)
├── open_instruct/              <- Source code (training scripts and utilities)
├── scripts/
|     ├── train/                <- Training launch scripts; debug/ has smoke tests
|     ├── eval/                 <- Evaluation scripts
|     └── data/                 <- Dataset preparation scripts
├── tests/                      <- Additional tests (most live next to the code)
├── mason.py                    <- Beaker job launcher
└── Dockerfile                  <- Dockerfile

Licensing

This codebase is licensed under Apache 2.0 as given in LICENSE.

The license we use for V1 models released (along with the base model licenses) can be found in assets/model_licenses/tulu_license.txt - just replace <MODELNAME> with the actual model name (i.e., the name on HuggingFace).

V2 models are licensed under the low-risk AI2 ImpACT license. See here for more details.

Acknowledgements

Open Instruct is a project that benefited from many open-source projects and libraries. We would like to particularly thank the following projects:

• HuggingFace Transformers: We adapted Hugging Face's Trainer for our finetuning scripts.
• HuggingFace TRL and eric-mitchell/direct-preference-optimization: our preference tuning code is adapted from TRL and from Eric Mitchell's DPO code.
• OpenAI's lm-human-preferences, summarize-from-feedback, and vwxyzjn/summarize_from_feedback_details: Our core PPO code is adapted from OpenAI's original RLHF code and Huang et al (2024)'s reproduction work of OpenAI's summarize from feedback work.
• OpenRLHF: We adapted OpenRLHF's Ray + vLLM distributed code for scaling up PPO RLVR training into the 70B scale.

Citation

If you used this repository or our models, please cite our work:

Tulu 1:

@misc{wang2023far,
   title={How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resources},
   author={Yizhong Wang and Hamish Ivison and Pradeep Dasigi and Jack Hessel and Tushar Khot and Khyathi Raghavi Chandu and David Wadden and Kelsey MacMillan and Noah A. Smith and Iz Beltagy and Hannaneh Hajishirzi},
   year={2023},
   eprint={2306.04751},
   archivePrefix={arXiv},
   primaryClass={cs.CL}
}

Tulu 2:

@misc{ivison2023camels,
      title={Camels in a Changing Climate: Enhancing LM Adaptation with Tulu 2},
      author={Hamish Ivison and Yizhong Wang and Valentina Pyatkin and Nathan Lambert and Matthew Peters and Pradeep Dasigi and Joel Jang and David Wadden and Noah A. Smith and Iz Beltagy and Hannaneh Hajishirzi},
      year={2023},
      eprint={2311.10702},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}

Tulu 2.5:

@misc{ivison2024unpacking,
      title={Unpacking DPO and PPO: Disentangling Best Practices for Learning from Preference Feedback},
      author={Hamish Ivison and Yizhong Wang and Jiacheng Liu and Zeqiu Wu and Valentina Pyatkin and Nathan Lambert and Noah A. Smith and Yejin Choi and Hannaneh Hajishirzi},
      year={2024},
      eprint={2406.09279},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
}

Tulu 3:

@article{lambert2024tulu3,
  title = {Tülu 3: Pushing Frontiers in Open Language Model Post-Training},
  author = {
    Nathan Lambert and Jacob Morrison and Valentina Pyatkin and Shengyi Huang and Hamish Ivison and Faeze Brahman and Lester James V. Miranda and Alisa Liu and Nouha Dziri and Shane Lyu and Yuling Gu and Saumya Malik and Victoria Graf and Jena D. Hwang and Jiangjiang Yang and Ronan Le Bras and Oyvind Tafjord and Chris Wilhelm and Luca Soldaini and Noah A. Smith and Yizhong Wang and Pradeep Dasigi and Hannaneh Hajishirzi
  },
  year = {2024},
  email = {[email protected]}
}

OLMo 3:

@misc{olmo2025olmo3,
      title={OLMo 3},
      author={Team OLMo and Allyson Ettinger and Amanda Bertsch and Bailey Kuehl and David Graham and David Heineman and Dirk Groeneveld and Faeze Brahman and Finbarr Timbers and Hamish Ivison and Jacob Morrison and Jake Poznanski and Kyle Lo and Luca Soldaini and Matt Jordan and Mayee Chen and Michael Noukhovitch and Nathan Lambert and Pete Walsh and Pradeep Dasigi and Robert Berry and Saumya Malik and Saurabh Shah and Scott Geng and Shane Arora and Shashank Gupta and Taira Anderson and Teng Xiao and Tyler Murray and Tyler Romero and Victoria Graf and Akari Asai and Akshita Bhagia and Alexander Wettig and Alisa Liu and Aman Rangapur and Chloe Anastasiades and Costa Huang and Dustin Schwenk and Harsh Trivedi and Ian Magnusson and Jaron Lochner and Jiacheng Liu and Lester James V. Miranda and Maarten Sap and Malia Morgan and Michael Schmitz and Michal Guerquin and Michael Wilson and Regan Huff and Ronan Le Bras and Rui Xin and Rulin Shao and Sam Skjonsberg and Shannon Zejiang Shen and Shuyue Stella Li and Tucker Wilde and Valentina Pyatkin and Will Merrill and Yapei Chang and Yuling Gu and Zhiyuan Zeng and Ashish Sabharwal and Luke Zettlemoyer and Pang Wei Koh and Ali Farhadi and Noah A. Smith and Hannaneh Hajishirzi},
      year={2025},
      eprint={2512.13961},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2512.13961},
}