llama-roofline

Why is your llama.cpp generation that fast and not faster? One command answers it.

When a local LLM generates text one token at a time, it has to read every weight in the model, from memory, for every single token. Nothing is reused. That makes token generation a memory-bandwidth problem, not a compute problem, and it means your tokens per second are governed by a single equation:

decode tok/s  =  BW_eff / bytes_per_token

llama-roofline measures both sides of that equation on your machine and tells you where you sit, in words rather than in a table of numbers.

pip install llama-roofline
llama-roofline diagnose

No flags, no config. It looks for your GGUF files, finds llama-bench, measures what your memory can actually sustain, benchmarks across a thread sweep, and prints this:

========================================================================
  llama-roofline v0.2.3  --  diagnosis
========================================================================
  Machine   : 12th Gen Intel(R) Core(TM) i7-12700H
              20 logical / 14 physical cores, 34.0 GB RAM, Windows AMD64
  Memory    : 54.3 GB/s sustained read (measured: max kernel, 20 threads)
  llama.cpp : build 10154 (0e4a03622), backends: CPU

  THE ANSWER
------------------------------------------------------------------------
  Yes. Decode is memory-bandwidth-bound, at 78% of what this machine
  sustains.

  Every token reads the whole model out of RAM, and at this utilisation
  the cores are mostly waiting for it. A faster CPU changes nothing
  here. Fewer bytes does.

  For qwen3b-q4km the ceiling on this machine is about 28 tok/s. You
  measured 21.9 at 8 threads.

  WHAT TO CHANGE
------------------------------------------------------------------------
  * Threads: use 8. Running 20 cost 25% of decode throughput, because
    decode is waiting on memory and more waiters do not make memory
    faster.
  * Size is the only dial that moves this. Halving the bytes read per
    token roughly doubles tok/s, which is why a smaller model or a lower
    quantization buys speed almost exactly in proportion to the bytes it
    removes.
  * Not every format of the same size costs the same, though. Run
    `llama-roofline advise` to see what has been measured.
  * Measured here: qwen0.5b-q4km is 4.91x smaller than qwen3b-q4km and
    decodes 4.28x faster.
  * Prompt processing is the opposite case: it scaled 4.9x with threads.
    If your workload is long prompts and short answers, you do want the
    cores.

  WHAT WAS MEASURED
------------------------------------------------------------------------
  model                     quant         size    decode   prefill  thr    GB/s  %ceil
  ------------------------------------------------------------------------------------
  qwen0.5b-q4km             Q4_K_M      392 MB     93.7t      277t    8    36.7    68%
  qwen1.5b-q4km             Q4_K_M      980 MB     39.7t      133t    8    38.9    72%
  qwen3b-q4km               Q4_K_M     1.92 GB     21.9t       76t    8    42.1    78%

  THE ROOFLINE
------------------------------------------------------------------------
    decode tok/s  =  37.18 GB/s  /  bytes per token
    fitted across 3 models, R^2 = 0.9960,
    taking each model at its own best thread count

That is a real run on a real laptop, trimmed of its caveats section, which is printed and is not optional.

The line worth staring at is the thread one. Setting -t 20 on a 20-thread machine looks free. It cost a quarter of the decode throughput.

diagnose also writes diagnosis.md, which is the same thing shaped for pasting into an issue or a forum thread when you want someone to look at your numbers.

Use Ollama or LM Studio and have no llama.cpp build?

You can still look inside the models you already have. inspect reads only the file header, needs no llama-bench, and takes about a second:

pip install llama-roofline
llama-roofline inspect ollama:qwen2.5:7b          # any model you have pulled
llama-roofline inspect ~/path/to/model.gguf         # LM Studio and every other GGUF

It prints what the file actually contains under its format label, how many bytes decode reads per token, and how much of that is the output head. Divide your memory bandwidth by that per-token figure and you have the ceiling on your generation speed before you run anything.

Check a model before you download it

inspect also reads a file on the Hugging Face Hub, or on any server that honours HTTP range requests, without downloading the weights. Only the header is fetched, usually a few MB of a file that may be tens of GB:

This is the beginning of the README. Read the rest at the original. Read the full README ↗