rocm-bake
Reproducible builds of PyTorch, Triton, AITER, CuPy, NIXL, vLLM and
LMCache against ROCm, on a three-node BuildKit fleet, with every
artifact published to a local OCI registry.
Full documentation: https://sbates130272.github.io/rocm-bake/
Live Reports
Two lanes publish measurements continuously, to the gh-pages branch.
They are not comparable with each other.
| What it measures | Published | Older runs |
|---|
| Fleet runs | Every wheel across all three SNOC nodes at once, instrumented — CPU, compiler-cache and registry traffic sampled throughout | on demand, from a build node | interactive chart, selector over every run |
| Nightly builds | One warm bake per architecture on AWS CodeBuild, published to ECR — phase wall clock and cache errors | 02:17 UTC, from CI | table of every run |
A fleet run is a parallel build across three nodes sharing a compiler
cache; a nightly is sequential bakes on one rented machine. Both keep
every run published, each behind a <stamp>/ permalink — but only the
fleet page charts, because a rented machine has nothing to sample.
Details: Live reports.
For fleet runs, make perf-serve is the same chart on :8090 for a
run that is still going, regenerating every 30 s; make perf-publish PUSH=1 is what puts it on the web. It binds every interface by
default — PERF_BIND= narrows it, make perf-serve-stop ends it.
Versions
That table is generated from rocm-bake.hcl by make version-scrub, and
CI fails if it drifts. Triton is pinned to a commit (TRITON_REF) as well
as a version; RUST_VERSION and SCCACHE_VERSION are toolchain pins and
live in the bake variables reference. Built for gfx950 (default),
gfx1201 and gfx1250.
rocm-bake.hcl is the single source for every value a build reads.
make list-vars prints what is actually in effect, and make version-check
compares it against what upstream has released and what we have published.
Quick start
make help # every target, grouped
make list-nodes # which fleet this is, and what it has
make zot-check # is the registry up?
make fleet-apply # bring every node to the manifest's state
make builder-setup # a buildx builder per node
make push-all BUILDER=snoc-pool # build and publish everything
make build-watch # follow it from a second terminal
The step-by-step version is in the
quick start guide.
The fleet
| Node | GPU | Role |
|---|
snoc-strix | gfx1151 | Default builder; hosts Zot (:8080) and the sccache store (:8081) |
snoc-think | gfx1201 | Build capacity, joined over SSH to snoc-thinkstation |
snoc-gaming | gfx1201 | Build capacity, joined over SSH |
No node has a gfx950 card, which is the default build target — the
fleet cross-compiles. make gpu-test therefore needs GPU_HOST and
PYTORCH_ROCM_ARCH to agree with the hardware it lands on.
BUILDER=snoc-pool is a sentinel, not a builder: it routes through
.builders/balance.sh, which schedules one bake per target across the
fleet by current load and hands results between stages through the
registry.
That table is not written down twice. It comes from
manifests/snoc.yml, which is the only place a node name, address,
role or link speed appears — the Makefile, the balancer, the sampler,
the report and the Ansible inventory all read it through
.builders/fleet.py.
make list-nodes # the manifest in effect
make fleet-apply # make every node match it (Ansible)
make fleet-check # ask each node whether it can do its job
Details, including what each capability means and why arch is not the
same claim as the gpu capability:
The fleet.
The fleet is driven by hand and registers no CI runner. The nightly
builds on AWS CodeBuild instead, publishing to ECR and sharing a
compiler cache on S3 — one measured night is $13.94. See
Building on AWS.
This is the beginning of the README. Read the rest at the original. Read the full README ↗
