RL-Kernel is high-performance infrastructure for RL post-training. It provides
deterministic operators for consistent numerical computation across rollout and training
engines, together with hardware-specific kernels for faster execution and lower memory
use in GRPO, PPO, and related workloads.
Today, the end-to-end path covers Qwen3-8B Dense with vime, vLLM, and Megatron-LM.
Work on DeepSeek-V4 Flash MoE, Miles, and AReaL is ongoing.
Why RL-Kernel?
Rollout and training engines can produce different log probabilities for the same tokens
and model weights because their kernels, batching, and reduction orders differ. Those
differences enter the policy ratios and KL terms used by RL algorithms.
- Exact train–inference consistency: deterministic operators keep rollout and training
computations aligned. The published experiment records exact runtime LogP agreement
across all 200 training steps.
- RL operators: deterministic attention, dense FFN, LogP, GRPO and PPO objectives,
and collectives cover the numerical boundaries in RL post-training.
- Performance: fused computation and hardware-specific kernels reduce rollout time,
memory use, and synchronization costs.
- vime integration: vime orchestrates vLLM rollout and Megatron-LM training, with
RL-Kernel supplying the operators used by both engines.
- Hardware: NVIDIA SM90 and AMD gfx942 are supported. Ascend dav_c220 has partial
operator coverage. Support for other hardware is in progress.
Repository layout
The directory and ownership guide describes
models, contracts, shared operators, hardware backends, rollout/train engines, tests,
CI and benchmark locations. See the Dense refactor acceptance checklist
for CUDA/ROCm validation. Legacy import and launcher paths remain supported.
Architecture
RL-Kernel sits between execution engines and accelerator backends. Its runtime adapters
select the operator implementation for each backend while keeping the same numerical
contract across rollout and training.
The architecture below shows how orchestration frameworks, execution engines, RL-Kernel
operators, and hardware backends fit together.
Current Scope and Roadmap
The current end-to-end path uses Qwen3-8B Dense with vime.
| Area | Current | Next |
|---|
| Model | Qwen3-8B Dense | DeepSeek-V4-Flash-0731 MoE |
| Orchestration | vime | Miles and AReaL |
| Engines | vLLM rollout and Megatron-LM training | More rollout and training engines |
Benchmark Highlights
CUDA H100
ROCm MI300X
Hardware Support
RL-Kernel currently supports the following hardware targets.
| Hardware | Architecture | Software | Status |
|---|
| NVIDIA H100, H200, GH200 | SM90 | CUDA | Supported |
| AMD Instinct MI300A, MI300X, MI325X | gfx942 | ROCm | Supported |
| Huawei Ascend dav_c220 | dav-2201 | CANN 9.1.0 and Ascend C | Partial |
| Moore Threads | In development | MUSA | In progress |
The published end-to-end benchmark was run on H100. The ROCm extension and backend
checks were verified on MI300X. Ascend support is limited to dav_c220. Support for other
hardware models is in progress.
Quick Start
Use a compatible vime environment on an eight-GPU H100 or MI300X node.
Clone the project and follow the installation guide
for your CUDA or ROCm build:
git clone https://github.com/RL-Align/RL-Kernel.git
cd RL-Kernel
Complete the one-time CUDA setup
or ROCm setup.
Save the backend, Python, framework, model and data paths in .rlk-profile.json
or select a profile with RLK_REPRO_PROFILE. No launcher edits are needed.
Both backends then use the same command:
./rlk run --tp 4 --rollout-tp 4 --temperature 0.7 --top-p 0.95 \
--lr 5e-7 --kl-coef 0.01 --max-response-len 6912 --max-tokens-per-gpu 4096 --steps 200
Training TP and rollout TP are independent: choose 1, 2, 4 or 8. Training CP
defaults to 8 / TP; set --cp explicitly if needed. run waits, validates
train/rollout LogP, and defaults to consistency mode without rollout-logprob reuse.
Add --mode native for a native comparison, or replace run with plan to
inspect the command without launching a job.
On ROCm, this command selects Triton chunked Attention and sparse top-p
logprob/monitoring-entropy scoring. Training logprobs are independently
recomputed and validation requires bitwise agreement with rollout. Apply the
updated ROCm companion patches and rebuild the extension when updating.
With the supplied ROCm profile, a three-step check measured 58.29 s/step
versus native's 57.40 s (+1.55%), with zero raw-bit logprob mismatches.
Use --steps 3 for that short check; timing depends on generated lengths and
the environment, and the native execution-record limitation is documented below.
See ROCm performance reproduction
for the full comparison command, measurements and validation limits.
On CUDA, rollout CP and top-k are configurable too; this short check performs
two real updates and validates their artifacts:
./rlk verify --tp 1 --rollout-tp 1 --rollout-cp 8 --temperature 0.7 --top-p 0.95 --top-k -1
Rollout TP × CP must divide eight. CUDA accepts top-k -1 (disabled) or a
positive integer, and temperature 0 for greedy sampling. ROCm currently
requires top-k -1 and positive temperature. ROCm rollout CP > 1 remains
unvalidated, and verify is CUDA-only. See the
CUDA/ROCm support table
and measured H100 results for exact coverage.
Community and Contributions
Join us on Slack
or WeChat, and
open an issue for bugs and feature requests.
Contributions to kernels, framework integrations, hardware adaptation, and benchmarks
are welcome. See the contributing guide.
Acknowledgments
RL-Kernel builds on the work of the open-source AI infrastructure community, including
vime, vLLM,
Megatron-LM, and
FlashInfer.
We thank their contributors and everyone helping bring RL-Kernel to new accelerators.
Licensed under the Apache License 2.0.
