🦋 Molt

A Hugging Face-native, minimal, and scalable framework for agentic RL research.

Ray · vLLM · NeMo AutoModel — the smallest Hugging Face-native stack for 1T-class fully-async, multimodal, multi-turn agentic RL.

Paper · Video · News · Architecture · Why Molt · Installation · Quick Start · Scope · Agent Contract · Recipes · Scaling · Deep dives · Contributing

Package	SFT	RL	Runtime
molt	molt.cli.train_sft	molt.cli.train_rl_ray	vLLM

Molt is agentic-first, Hugging Face-native, minimal, and scalable. The agent is the program; the trainer is a single actor; reward is any Python you write inside an Env or ChatAgent — graders, multi-turn tools, VLM environments, LLM-as-judge. Three components carry the rest — Ray for placement and async queues, vLLM for rollout, NeMo AutoModel + FSDP2 for training. The model stays an ordinary PyTorch module in the Hugging Face layout: one checkpoint directory feeds both vLLM and the trainer, with no bridge to write and no weights to convert. That is the whole stack: ~9.2K lines of RL code that scale to 1T-class MoE on vLLM with TP / EP / CP — the same command runs a 4B model on one node and GLM-5.2 (753B) at --fsdp.ep_size 256, with Adam CPU offload for the largest actors. One agent API, one trainable actor, clean enough to read end-to-end.

📰 News

• 2026-11 · Molt powers REINFORCE Zero, combining trainer–inference invariance (TII) with unclipped token-level importance sampling for stable asynchronous RL through 20,000 policy updates — see the deep dive and the kernel-alignment agent skill.

• 2026-09 · Molt now supports FlashREINFORCE, critic-free single-rollout RL with stable training beyond 6,000 steps — see the quick start.

🧩 Architecture

Three boxes. One async loop.

Ray owns placement and the async queue between the three boxes — that is the entire runtime. The contract is token-first: token ids, logprobs, action ranges, rewards, and multimodal tensors stay aligned from rollout to training. Anything you can compute in Python is a valid reward, including LLM-as-judge calls back through the same vLLM engines that drive rollout.

✨ Why Molt

	What you get	Why it matters for research
🤖 Agentic-first	One Gymnasium-aligned API — Env.step() or ChatAgent.run() — covers graders, multi-turn tools, VLM environments, and OpenAI/Anthropic-compatible servers	The agent is the program — iterate on environments in plain Python, the trainer stays untouched
⚙️ Fully-async runtime	Ray placement, async rollout queues, vLLM engines, partial rollout, weight sync	Rollout, training, and weight sync overlap — a DeepSeek-V3-class actor stays fed without bespoke infra
🔥 Hugging Face-native, PyTorch end-to-end	The model is an ordinary PyTorch module in the Hugging Face layout (NeMo AutoModel + FSDP2); one checkpoint directory feeds both vLLM and the trainer	Models load as released — no bridge to write, no checkpoint to convert — and you change one by editing plain PyTorch
🎯 Single-actor simplicity	One actor, optional KL reference — the whole RL graph fits on a page	Every gradient is explicit; every loss term is one file away
🚀 Frontier-scale MoE	AutoModel + FSDP2 + TP / EP / CP + Adam CPU offload, MoE-native — e.g. GLM-5.2 (753B) with --fsdp.ep_size 256	The 4B and 753B recipes share the entry point, the agent, and the algorithm — no rewrite between scales
🔗 Token-first contract	Aligned token ids, logprobs, action ranges, rewards, multimodal tensors	Multi-turn, VLM, and tool-call traces share one format end-to-end
🪶 Small, hackable surface	~9.2K LOC of RL code across 3 thin layers; each mechanism (advantage, loss, token capture, training loop) is one file	The function you edit is the function that runs — read it in an afternoon, or hand it whole to a coding agent

This is the beginning of the README. Read the rest at the original. Read the full README ↗