ML Systems Lab

A reproducible benchmarking framework for ML inference across heterogeneous hardware: laptops, single-board computers, and servers, from one config file and one command.

pip install ml-systems-lab
mlsys probe                     # what this machine can measure: power, thermals, GPU, clocks
mlsys membw                     # its sustained DRAM read ceiling, in about a minute

Both work straight after the install, with no config and no model. The example configs and the shipped datasets live in the repository rather than in the package, so for a first real run:

git clone https://github.com/manunicholasjacob/ml-systems-lab && cd ml-systems-lab
mlsys report results/pi5-campaign              # 43 measured points from a Raspberry Pi 5
mlsys run configs/example-smoke.yaml           # your own sweep, after editing the paths
mlsys report runs/smoke-test --full

Every run produces a self-describing JSON record carrying the hardware, OS, kernel, compiler, backend version, model, quantization, the measured metrics, and the physical state of the machine while they were measured (power, temperature, clocks, throttle flags, CPU utilization). The analysis layer turns a directory of records into publication-quality tables (text, Markdown, LaTeX/booktabs) and figures.

Built and used for a real research program: the records under results/paper12/ are the measurements behind a manuscript submitted to the IEEE Internet of Things Journal in September 2026, and the framework reproduces that manuscript's roofline fits exactly (Pi 5: 10.7 GB/s effective, R^2 = 0.980; i7-12700H: 35.7 GB/s, R^2 = 0.980). Two campaigns run natively by this framework then re-measured the same quantities independently and agreed within 1.7%.

Statement of need

Comparing inference performance across machines is mostly a bookkeeping problem, and most of the tooling solves the wrong half of it. llama-bench and onnxruntime_perf_test each measure one backend well and report a table with no hardware, no thermal state and no power in it. MLPerf Inference specifies a rigorous closed-division protocol, but its submission machinery assumes a datacenter and a submitter, not a Raspberry Pi on a desk. Neither will tell you six weeks later whether a number was taken on a throttling board, which compiler produced the binary, or how much of the DRAM ceiling that run reached.

Edge inference is where that gap hurts most, because the answers move with conditions the usual tools do not record. The same model on the same board decodes differently hot and cold, at two threads and at four, with the page cache warm and cold, and on the same silicon under a different quantization format. A result without that context is not reproducible, and reviewers cannot check it.

ml-systems-lab closes the gap by making the context part of the measurement. One YAML file describes machines, models and a sweep; one command executes it across a mixed fleet over SSH; and every point comes back as a single self-describing JSON record carrying the hardware, OS, kernel, backend build commit, model, quantization, the metrics, and the physical state of the machine while they were taken: per-rail power, temperature, clocks, throttle flags, memory pressure. Runs are addressed by a deterministic hash of their specification, so an interrupted campaign resumes and a corrected point re-executes exactly. Failures are records too, with their raw output kept, so a parser bug is fixed by reparsing rather than by repeating an overnight campaign.

It was written because five research campaigns had each grown their own copy of the same harness and the copies had drifted. The framework that replaced them reproduces those papers' reported fits from their own data, and the datasets it has produced since ship in results/ for anyone who wants to check a claim without owning the hardware.

Every point is a different model or quantization; every line is one device's effective memory bandwidth. Two independent Pi 5 campaigns (weeks apart, different harnesses) and two laptop campaigns land on top of each other: decode throughput is model bytes divided by one number per device.

What it measures

Metric	How
TTFT (time to first token)	streaming request against llama-server, first-chunk timing
Prefill / decode throughput	llama-bench, parsed from its JSON output
End-to-end request latency	same streaming path, submit to last token
Single-inference latency (mean, p50/p95/p99)	ONNX Runtime, timed in-process on the device
Memory	peak RSS, plus free-memory and swap state around every run
CPU utilization	/proc/stat deltas (Linux), GetSystemTimes + NtQuerySystemInformation (Windows), per core when it pools to the aggregate
Temperature and throttling	sysfs thermal zones, Pi throttle bitmask, in the record not a side file
Power and energy	Raspberry Pi PMIC per-rail (core vs DRAM split), Intel RAPL where present, NVIDIA GPUs through whichever of three instruments the part supports, named in the record
Derived	decode bandwidth, roofline utilization %, energy per token

Anything a platform cannot measure is reported as absent, never as zero.

This is the beginning of the README. Read the rest at the original. Read the full README ↗