RLHF Book

A comprehensive guide to Reinforcement Learning from Human Feedback (and a broad introduction to post-training language models).

Read online | Order print on Manning or Amazon | Join Discord Community

This book is my attempt to open-source all the knowledge I've gained working at the frontier of open models in the post-ChatGPT takeoff of language models. When I started, many established methods like rejection sampling had no canonical reference. On the other side, industry practices to make the models more personable -- colloquially called Character Training -- had no open research. It was obvious to me that there would be payoff to documenting, learning the fundamentals, carefully curating the references (in an era of AI slop), and everything in between would be a wonderful starting point for people.

Today, the repository includes code and serves as a home base for people who want to learn. You should use coding assistants to ask questions. You should buy the physical book because the real world matters. You should read the specific AI outputs tailored to you.

I also want to continue building educational resources, such as open-source slide decks and more ways to learn. In the end, with how impossible it is to measure human preferences, RLHF will never be a solved problem.

Thank you for reading. Thank you for contributing any feedback or engaging with the community.

-- Nathan Lambert, @natolambert

Repository Structure

rlhf-book/
├── book/                   # Book source and build files
│   ├── chapters/           # Markdown source (01-introduction.md, etc.)
│   ├── images/             # Figures referenced in chapters
│   ├── assets/             # Brand assets (covers, logos)
│   ├── templates/          # Pandoc templates (HTML, PDF, EPUB)
│   ├── scripts/            # Build utilities
│   └── data/               # Library data
├── code/                   # Reference implementations
│   ├── instruction_tuning/ # SFT a base model with chat templates
│   ├── policy_gradients/   # PPO, REINFORCE, GRPO, RLOO
│   ├── reward_models/      # Preference RM, ORM, PRM training
│   ├── direct_alignment/   # DPO and variants
│   ├── rejection_sampling/ # Best-of-N rejection sampling
│   └── distillation/       # On-policy distillation (SDPO)
├── diagrams/               # Diagram source files
│   ├── scripts/            # Python generation scripts
│   ├── tikz/               # LaTeX/TikZ sources
│   └── specs/              # YAML specifications
├── teach/                  # Teaching materials (courses, slides)
├── build/                  # Generated output (git-ignored)
└── Makefile                # Build system

Code Library

Reference implementations for RLHF algorithms in code/:

• Instruction tuning (SFT a base model with chat templates)
• Policy gradient methods (PPO, REINFORCE, GRPO, RLOO, etc.)
• Reward model training (preference RM, ORM, PRM)
• Direct alignment methods (DPO and variants)
• Rejection sampling (best-of-N)
• On-policy distillation (SDPO)

See code/README.md for setup and usage.

Book Source

Book source files are in book/. Build locally:

make html   # Build HTML site
make pdf    # Build PDF (requires LaTeX)

See book/README.md for detailed build instructions.

Diagrams

The diagrams/ directory contains source files for figures used in the book. These are designed to be reusable for presentations, blog posts, or your own learning materials. Generate them with:

cd diagrams && make all

Citation

For the web and arXiv version, use the citation exported by arXiv:

@misc{lambert2025reinforcementlearninghumanfeedback,
  title         = {Reinforcement Learning from Human Feedback},
  author        = {Nathan Lambert},
  year          = {2025},
  eprint        = {2504.12501},
  archivePrefix = {arXiv},
  primaryClass  = {cs.LG},
  url           = {https://arxiv.org/abs/2504.12501},
}

For the Manning edition:

@book{lambert2026reinforcement,
  author       = {Nathan Lambert},
  title        = {Reinforcement Learning from Human Feedback: Alignment and post-training of {LLMs}},
  year         = {2026},
  publisher    = {Manning Publications},
  isbn         = {9781633434301},
  url          = {https://www.manning.com/books/reinforcement-learning-from-human-feedback},
}

License

• book/chapters: CC-BY-NC-SA-4.0
• everything else (code/, diagrams/, scripts/, etc.): MIT
• note that some images in book/images/ are unlicensed photos or screenshots.

Contributors

While I get the credit as the sole "author" and creator of this project, I've been super lucky to have many contributions from early readers. These have massively accelerated the editing progress and flat-out added meaningful content to the book. I'm happy to send substantive contributors free copies of the book and expect the internet goodwill to pay them back in unexpected ways.

See all contributors.

Note: because I made the mistake of associating my commits with my Ai2 email, which I no longer have access to, the commit history lost most of my tracking, RIP!

Translations

Readers maintain unofficial translations of the book in their own repositories. These are community projects — independent of the official print editions and their professional translations — released under the same CC-BY-NC-SA license with attribution back to this book:

• 简体中文 (Simplified Chinese): jweihe/RLHF-book-Chinese
• 简体中文 (Simplified Chinese): around133/rlhf-book-zh

This is the beginning of the README. Read the rest at the original. Read the full README ↗