Forge: teaching a 1.5B model to reason with RL on one 8GB GPU
Forge trains Qwen2.5-1.5B to solve math by reasoning, using GRPO (reinforcement
learning with verifiable rewards, the technique behind DeepSeek-R1) instead of
supervised fine-tuning, on a single RTX 5060. It lifts GSM8K pass@1 from 58.8%
to 70.0% with no measured loss of general ability, then quantizes and serves the
result behind one interface with a hosted fallback.
The playground runs real inference on a
live GPU: type any grade-school math problem and watch the stock base model and the GRPO-tuned
adapter answer it side by side, streaming. Both are served from a single vLLM process via
multi-LoRA, using the same 73MB adapter the training run produced rather than a merged copy.
It scales to zero between visits, so the first request spends about 60-90s waking the
container before tokens start.
| page | what's there |
|---|
/ | the headline result and a side-by-side sample |
/playground | live base-vs-tuned inference on a problem you type |
/method | GRPO in plain terms, the reward stack, and the cold-start bug |
/results | every figure with the committed file it came from |
/traces | full reasoning traces, including the problem both models miss |
10-second results (all measured, seed 3407)
| base | Forge (GRPO) |
|---|
| GSM8K pass@1 (1,319 held-out) | 58.8% | 70.0% |
| ARC-Challenge (forgetting check, first 200 test items) | 69.5% | 68.5% |
Trained in 86 min, 3.64 GiB peak VRAM. Reward went from 1.23 to 2.80 out of 3.25. (Training-run figures are from the training log on the
training machine, which is not committed; the plot is docs/reward_curve.png.
Eval numbers are committed as JSON under eval/results/.)
Baseline notes: what 58.8% actually is
The 58.8% vs 70.0% comparison is deliberately apples-to-apples at the training
configuration, not a leaderboard reproduction:
- Both columns ran the same 4-bit weights. The base and tuned evals load the
identical NF4-quantized checkpoint (
unsloth/qwen2.5-1.5b-instruct-unsloth-bnb-4bit)
via vLLM (the same weights training used), with the same one-shot prompt, greedy
decoding, and seed 3407, over all 1,319 held-out test problems
(eval/eval_gsm8k.py, results in
eval/results/gsm8k.json). The only difference between
the columns is the LoRA adapter, so the delta measures GRPO, not a precision,
prompt, or harness change.
- 58.8% is the lenient base score (any final number in the output counts).
Scored strictly, where the answer must sit inside the trained
<answer> tags, the base
model gets 54.4%, because it doesn't reliably follow the format. The tuned model
scores 70.0% under both rules; identical strict/lenient means 100% format
compliance, nothing lost to parsing.
- Don't compare 58.8% to Qwen's published GSM8K figure. Published numbers for
Qwen2.5-1.5B-Instruct come from a different protocol (full-precision weights and
few-shot CoT eval harnesses) and land higher than this quantized, fixed-format,
exact-match baseline. That's expected: the baseline here exists to isolate the
training effect under the training configuration, not to reproduce the
tech-report number.
- The gain survives leaving 4-bit. The adapter merged to fp16 scores 0.76
pass@1 on the first 100 held-out problems (vLLM/GPU), and the f16 GGUF agrees
exactly at 0.76
(
eval/results/served_fp16_merged.json,
export/QUANT_DELTA.md). That is consistent with the 70.0%
full-set figure (n=100 means roughly ±9 pts at 95% CI), so the improvement is not an
artifact of the 4-bit eval path.
Why GRPO
Supervised fine-tuning teaches a model to imitate answers. GRPO teaches it to
search: it samples 8 completions per problem, scores each with a verifiable
reward (is the final number correct? is it formatted?), and pushes probability
toward the better-than-average ones. No human labels, no reward model, no LLM judge.
Just a math checker. That's what makes it a genuine reasoning-RL result rather
than a commodity fine-tune.
Reward design
The reward functions are the training signal (train/rewards.py, max 3.25):
correctness +2.0, format +0.5, numeric +0.25, tag_presence +0.5 (graded).
The bug worth reading about: the first smoke run trained cleanly but every reward
was 0.0. Qwen-1.5B ignored the format instruction, so no completion earned the
correctness or format reward, and with identical rewards in every group, GRPO's
advantage is zero and nothing learns. The model was getting the math right; it just
wouldn't use the tags. Two fixes: a one-shot example in the prompt, and a
graded tag_presence reward (+0.125 per tag) so partial compliance still creates
gradient. Rewards fired immediately after. This is the kind of failure GRPO is prone
to, and catching it is the point.
This is the beginning of the README. Read the rest at the original. Read the full README ↗
