KernelBench
Frontier coding agents write GPU kernels. Each session is one autonomous
agent, graded against a roofline (or ms/speedup) ceiling, then reward-hack
audited before anything is published. Live: kernelbench.com.
This monorepo is the website and the evals. GPU sessions launch to
Lambda / Brev (or another remote worker). Operator rules: AGENTS.md, which
points at a specialized AGENTS.md in each directory. Methodology and
history: each bench's SPEC.md and DEVLOG.md.
Benches
| bench | path | what | site |
|---|
| hard | benchmarks/hard/ | per-op kernels (CUDA or Triton), roofline-graded | /hard |
| mega | benchmarks/mega/ | full fused megakernels | /mega |
| cuda | benchmarks/cuda/ | CUDA-only writing deck (Triton/DSL fail) | /cuda |
| mini | benchmarks/mini/ | small-model (<200B) deck, capped + 5-repeat (WIP) | homepage scroll category on / when debuted (not /mini) |
| multi | benchmarks/multi/ | 4×H100 NVLink multi-GPU (WIP, frontier roster) | unpublished |
Hard / mega / cuda share harness machinery and run unlimited wall-clock
(BUDGET_SECONDS=0). Mini is capped; multi is sequential on a 4-GPU node.
Website (local)
Next.js 16 + Tailwind. Package manager is bun (bun.lock):
bun install
bun dev # http://localhost:3000
bun run build
Site data is baked at build time from benchmarks/*/results/
(app/_lib/data.ts). Publish/deploy: kb publish then kb deploy — see
app/AGENTS.md.
Layout
app/ public/ the website (app/AGENTS.md); app/_lib/data.ts bakes benchmark data at build time
benchmarks/<bench>/ AGENTS.md, SPEC.md, DEVLOG.md, problems, src, scripts, results/, outputs/runs/ (gitignored archives)
scripts/ shared runner (lib/run_harness.sh), workers, publish helpers
kbtool/ the `kb` CLI (uv package); `uv tool install -e ./kbtool` puts `kb` on PATH
media/ tracked chart generators (media/AGENTS.md); PNGs gitignored
runs/ gitignored HF staging from `kb push-runs` (not a third archive)
This list is closed. A new thing goes in the directory that owns it. kbtool/tests fails on a new top-level dir.
Bench virtualenvs exist only on the GPU boxes that run the benchmark; the
Mac checkout carries kbtool/.venv alone.
