Training Open Instruction-Following Language Models

This repo serves as an open effort on instruction-tuning and post-training popular pretrained language models on publicly available datasets. We release this repo and will keep updating it with:

1. Code for finetuning language models with latest techniques and instruction datasets in a unified format.
2. Code for DPO, preference finetuning and reinforcement learning with verifiable rewards (RLVR).
3. Checkpoints or other useful artifacts that we build in our exploration.

Olmo 3 results: The live Papers with Code badges above track Olmo 3 Think 32B on BBH and Olmo 3.1 Think 32B on MATH. See Table 14 of the Olmo 3 paper for the reported scores.

We also support some evaluations natively in the codebase, but these are now unmaintained and instead we suggest using OLMES, which we used for TÜLU 3.

Setup

Our setup follows our Dockerfile. Note that Open Instruct is a research codebase and does not guarantee backward compatibility.

Installation with uv

We use uv for installation and running code. You can install with uv sync.

• Git LFS (for running tests): Install Git LFS and run git lfs install before cloning. See CONTRIBUTING.md for details.

• Docker installation: You can also use the Dockerfile to build a Docker image. You can build the image with the following command:

docker build . \
    --build-arg GIT_COMMIT=$(git rev-parse --short HEAD) \
	--build-arg GIT_BRANCH=$(git rev-parse --abbrev-ref HEAD) \
	-t open_instruct_dev

Training

After having setup the environment, you are ready to launch some experiments. We provide a few examples below. To learn more about how to reproduce the Tulu 3 models, please refer to the Tulu 3 README. The instructions and documentations for Tulu 1 and Tulu 2 are in Tulu 1 and 2 README.

The launch commands below run on Ai2's Beaker clusters. scripts/train/build_image_and_launch.sh builds an image from your current commit and passes it to the chosen script. Most of these scripts pass mason's --preemptible, so on clusters that enforce priority they run only as backfill and can wait indefinitely. Scripts that launch evals (--try_launch_beaker_eval_jobs_on_weka) also need oe-eval-internal cloned at the repository root.

Finetuning

OLMo-core SFT: For supported models (OLMo, OLMoE, Qwen3), we recommend the more GPU-efficient OLMo-core SFT implementation. See open_instruct/olmo_core_utils.py for the list of supported models. Our entry point is open_instruct/olmo_core_finetune.py:

# Single-GPU OLMo-core SFT smoke test on Beaker (Qwen3-0.6B).
./scripts/train/build_image_and_launch.sh scripts/train/debug/oc_sft.sh

# Two-node 8xGPU OLMo-core SFT run.
./scripts/train/build_image_and_launch.sh scripts/train/debug/oc_sft_multinode.sh

Preference Tuning

We train DPO with open_instruct/dpo.py:

# Single-GPU DPO smoke test on Beaker (OLMo-2 1B).
./scripts/train/build_image_and_launch.sh scripts/train/debug/dpo/single_gpu.sh

# Two-node 8xGPU DPO run.
./scripts/train/build_image_and_launch.sh scripts/train/debug/dpo/multi_node.sh

Reinforcement Learning with Verifiable Rewards (RLVR)

For supported Olmo MoE models, use the MILES + OLMo-core integration. The existing Core/vLLM and DeepSpeed/vLLM GRPO workflows remain available for dense-model training; the commands below use DeepSpeed/vLLM.

We train with open_instruct/grpo_fast.py:

# Single-GPU smoke test on Beaker (small model, fast).
./scripts/train/build_image_and_launch.sh scripts/train/debug/single_gpu_on_beaker.sh

# Two-node 8xGPU run (Qwen2.5-7B on code RLVR).
./scripts/train/build_image_and_launch.sh scripts/train/debug/large_test_script.sh

Contamination checks

We release our scripts for measuring the overlap between instruction tuning datasets and evaluation datasets in ./decontamination. See the README for more details.

Developing

When submitting a PR to this repo, we check the core code in open_instruct/ for style with the following:

make style
make quality

Run the tests with uv run pytest.

Pre-commit hooks

To automatically run linting and formatting on each commit:

uv run pre-commit install

To run on all files (recommended after initial setup):

uv run pre-commit run --all-files

This is the beginning of the README. Read the rest at the original. Read the full README ↗