guidellm

Overview

GuideLLM is a platform for evaluating how language models perform under real workloads and configurations. It simulates end-to-end interactions with OpenAI-compatible and vLLM-native servers, generates workload patterns that reflect production usage, and produces detailed reports that help teams understand system behavior, resource needs, and operational limits. GuideLLM supports real and synthetic datasets, multimodal inputs, and flexible execution profiles, giving engineering and ML teams a consistent framework for assessing model behavior, tuning deployments, and planning capacity as their systems evolve.

Why GuideLLM?

GuideLLM gives teams a clear picture of performance, efficiency, and reliability when deploying LLMs in production-like environments.

• Captures complete latency and token-level statistics for SLO-driven evaluation, including full distributions for TTFT, ITL, and end-to-end behavior.
• Generates realistic, configurable traffic patterns across synchronous, concurrent, and rate-based modes, including reproducible sweeps to identify safe operating ranges.
• Supports both real and synthetic multimodal datasets, enabling controlled experiments and production-style evaluations in one framework.
• Produces standardized, exportable reports for dashboards, analysis, and regression tracking, ensuring consistency across teams and workflows.
• Delivers high-throughput, extensible benchmarking with multiprocessing, threading, async execution, and a flexible CLI/API for customization or quickstarts.

Quick Start

The Quick Start shows how to install GuideLLM, launch a server, and run your first benchmark in a few minutes.

Install GuideLLM

Before installing, ensure you have the following prerequisites:

• OS: Linux or MacOS
• Python: 3.11 - 3.15

Install the latest GuideLLM release from PyPi using pip :

pip install guidellm[recommended]

Or install from source:

pip install git+https://github.com/vllm-project/guidellm.git

Or run the latest container from ghcr.io/vllm-project/guidellm:

podman run \
  --rm -it \
  -v "./results:/results:rw" \
  -e GUIDELLM__SPEC__BACKEND='{"kind": "openai_http", "target": "http://localhost:8000"}' \
  -e GUIDELLM__SPEC__PROFILE='{"kind": "sweep"}' \
  -e GUIDELLM__SPEC__CONSTRAINTS='[{"kind": "max_duration", "seconds": 30}]' \
  -e GUIDELLM__SPEC__DATA='[{"kind": "synthetic_text", "prompt_tokens": 256, "output_tokens": 128}]' \
  ghcr.io/vllm-project/guidellm:latest

Published images are multi-arch (linux/amd64 and linux/arm64) for Docker, Podman, Kubernetes, and OpenShift. Prefer pinning a release tag in production:

Tag	Meaning
vX.Y.Z	Immutable release (multi-arch from v0.7.0+)
stable	Newest full release (vX.Y.Z)
latest	Newest release tag (may include pre-releases)
nightly	Tip of main

Architecture-specific tags such as vX.Y.Z-amd64 / vX.Y.Z-arm64 are build artifacts — do not use them as the image reference.

Launch an Inference Server

Start any OpenAI-compatible endpoint. For vLLM:

vllm serve "neuralmagic/Meta-Llama-3.1-8B-Instruct-quantized.w4a16"

Verify the server is running at http://localhost:8000.

Run Your First Benchmark

Run a sweep that identifies the maximum performance and maximum rates for the model:

guidellm run \
  --backend kind=openai_http,target=http://localhost:8000 \
  --profile kind=sweep \
  --constraint kind=max_duration,seconds=30 \
  --data kind=synthetic_text,prompt_tokens=256,output_tokens=128

You will see progress updates and per-benchmark summaries during the run, as given below:

Inspect Outputs

After the benchmark completes, GuideLLM writes benchmarks.json and benchmarks.csv by default. The files are saved in the directory configured by GUIDELLM__DEFAULT_RESULTS_DIR, or in the current directory when the variable is not set. GuideLLM also prints a summary and the generated file locations to the console.

Specifying --output replaces the default JSON and CSV outputs. See output configuration for examples of selecting formats, including generating HTML alongside JSON and CSV.

Output Files and Reports

Use JSON or YAML for detailed analysis, CSV for spreadsheet comparisons, HTML for self-contained visual reports, and PLOT for static performance graphs. See supported file formats for their contents and default filenames, and configuring file outputs to choose output paths.

The console provides a summary of each benchmark. Its tables can be copied into spreadsheet software using | as the delimiter. See console output for progress and display controls.

以上为 README 的开头部分。其余内容请阅读原文。 查看完整 README ↗