Blackbox Inference Simulator (BLIS)

BLIS is a discrete-event simulator of an LLM serving stack. Its goal is llm-d stack-level parity: simulating an llm-d deployment end to end — the router (EPP), admission and flow control, prefill/decode disaggregation, KV caching and offload, and autoscaling — initially with vLLM as the engine, and eventually a wider range of engines.

It is built for three things, done robustly and with high fidelity:

• Configuration search and optimization — which parallelism, engine limits, router configuration and P/D split serve a workload best.
• Capacity planning — how many GPUs and instances a target rate and SLO need.
• Policy discovery — comparing llm-d router configurations, scheduling and autoscaling policies before they ship.

BLIS is CPU-only and deterministic: the same seed produces byte-identical results, and no GPU is needed.

---

How BLIS fits together

A BLIS run draws on six repositories. Each owns one kind of thing.

Repository	Role	Holds
inference-sim (this one)	simulates	The end-to-end simulation: the clock and event loop, arrivals and workloads, the cluster and placement, the embedded llm-d router (admission, flow control, routing), metrics and reports. It holds no vLLM code.
blis-vllm	runs the engine	The vLLM engine: vLLM v0.32.0's scheduler, KV cache, MTP, DP and connectors, translated to Go and diff-tested against real vLLM; the vLLM engine configuration and rules; the kernel adapter.
blis-latency-kernel	prices	Step time, KV and fixed memory, P/D KV transfer, offload tier transfer, host overheads. The only latency backend.
blis-catalog	states the facts	Model graphs, chips, fabrics, storage devices, workload presets — declared facts, nothing fitted.
blis-registry	holds the fitted numbers	Coefficient sets the kernel prices with, each with its provenance and scope.
blis-schemas	defines the formats	The source of truth for every shared format: scenario, deployment, engine and catalog files.

The key relationships:

• A scenario (a blis-schemas Scenario + Deployment YAML) names a model and a chip from the catalog, coefficient sets from the registry, and the deployment: cluster nodes and fabric, and each pool's role, parallelism and engine settings.
• inference-sim reads the scenario and simulates traffic through the stack. It drives each instance's engine only through blis-schemas' engine.Engine interface, and asks the engine for every step: what runs, how long it takes, and what it emits.
• blis-vllm is that engine. It decides each batch as vLLM does and asks the kernel what the step costs.
• blis-latency-kernel computes that cost from the catalog's facts and the registry's coefficients. It holds no data of its own.

The dependencies run one way: blis-schemas ← blis-latency-kernel ← blis-vllm ← inference-sim. The kernel imports only blis-schemas. Non-test code under sim/ and cmd/ imports blis-vllm at one build site, cmd/engine_build.go.

Deployment choices live in the scenario; facts live in the catalog; fitted numbers live in the registry. Nothing is fetched at run time.

---

Install and run

Requires Go 1.27.2+.

git clone https://github.com/inference-sim/inference-sim.git
cd inference-sim
go build -o blis main.go

git clone --branch 0.3.0 --depth 1 https://github.com/inference-sim/blis-catalog.git
git clone --branch v0.2.0 --depth 1 https://github.com/inference-sim/blis-registry.git
export BLIS_CATALOG=$PWD/blis-catalog

./blis run --scenarios examplescenarios \
  --scenario deepseek-v4.1-flash-h100-tp8-dspark5.yaml \
  --registry $PWD/blis-registry \
  --rate 10 --num-requests 100

Every blis run and blis replay needs four inputs:

• the catalog, via --catalog <clone root> or BLIS_CATALOG (no default; the flag wins);
• the registry, via --registry <clone root>;
• a scenario directory, via --scenarios <dir> — any directory of scenario YAMLs. This repository ships ready-to-run ones in examplescenarios/;
• a scenario, via --scenario <file name> within that directory.

The clones are pinned to the releases BLIS is tested against; see Catalog compatibility.

vLLM versions. Every scenario states engine_version: "0.32.0"; one that omits it is refused. It runs blis-vllm's translated vLLM v0.32.0 engine (the latency kernel's 0.31.0 tuning prices it, and the run says so on stderr). Older engine versions are being retired. Priority scheduling (scheduling_policy: priority, --scheduler priority-fcfs or reverse-priority, --preemption-policy priority, SLO priority overrides) is refused for now. The engine block must state max_model_len as an explicit number (unstated or auto is refused), plus max_num_seqs and max_num_batched_tokens.

The quickstart's --rate 10 --num-requests 100 is a synthetic workload for a quick first run. examplescenarios/ also ships MoE deployments to try the same commands against — deepseek-v4-pro-b300-fp4-vllm-tp4-mtp2.yaml (B300, MTP) and the multi-node kimi-k3-h200-tp8dp4ep32-balanced.yaml; each file's header records its provenance.

The run prints JSON metrics on stdout — TTFT, ITL and E2E distributions (ttft_mean_ms, itl_p99_ms, e2e_p99_ms, ...), throughput (responses_per_sec, tokens_per_sec), completed_requests and preemption_count. See Interpreting Results.

The examples below use examplescenarios/gpt-oss-120b-h100-fp4-vllm-tp2-2node.yaml, a two-node deployment (2 nodes × 8 GPUs ÷ TP2 = 8 ranks), so they can run up to eight instances.

---

Features

Latency pricing

The scenario fixes the model, hardware, TP/PP/DP/EP, block size, batch limits, max model length, prefix caching, cache dtype and speculative decoding; the kernel derives the KV block budget from them. There are no --model, --hardware or --tp flags. → Latency model guide

Workloads

Named presets from the catalog, token distributions, multi-client YAML specs, closed-loop sessions.

./blis run --scenarios examplescenarios --scenario gpt-oss-120b-h100-fp4-vllm-tp2-2node.yaml \
  --registry $PWD/blis-registry \
  --workload chatbot --rate 20 --num-requests 500
./blis run --scenarios examplescenarios --scenario gpt-oss-120b-h100-fp4-vllm-tp2-2node.yaml \
  --registry $PWD/blis-registry \
  --workload-spec examples/servegen-language.yaml

→ Workloads

Routing

Every multi-instance cluster routes with the real llm-d router (the Endpoint Picker), embedded in-process. It reads a deployment's own EndpointPickerConfig, EPP flags and InferencePool / InferenceObjective manifests, named by the scenario's router: block (and pod labels from pools[].labels); with none, it uses the router's built-in default configuration. Kernel-backed scenarios expose host and declared-rack identity to llm-d topology plugins. Explicit simulated Node manifests can add zone and region; the routing guide explains the stamp handler and limits.

./blis run --scenarios examples/llm-d/scenarios \
  --scenario llama-3.1-70b-instruct-h200-tp4-4node-prefix-affinity.yaml --registry $PWD/blis-registry \
  --num-instances 8 --rate 20 --num-requests 80

--num-instances cannot exceed the scenario pool's rank capacity (pool nodes × gpus_per_node / (pp × tp × pcp)); the fixture above is four H200 nodes at TP4, so up to eight. For other shapes, copy a scenario, raise cluster.nodes and the pool's nodes, and add a cluster.fabric.

→ Routing, Cluster simulation

Admission and flow control

The llm-d router's admission and flow control: its legacy admission controller sheds sheddable requests from a saturated pool, and with featureGates: [flowControl] in the EndpointPickerConfig requests wait in priority bands with capacity limits and queue-wait budgets. SLO classes reach the router as InferenceObjective names. SLO goodput is reported per class.

./blis run --scenarios examplescenarios --scenario gpt-oss-120b-h100-fp4-vllm-tp2-2node.yaml \
  --registry $PWD/blis-registry \
  --num-instances 4 --rate 200 --num-requests 1000

→ Admission

Prefill/decode disaggregation

A scenario with prefill and decode pools gets one kernel per pool; the kernel prices each KV handoff over the scenario's fabric.

./blis run --scenarios examplescenarios --scenario glm-5-h200-3p1d-ib.yaml \
  --registry $PWD/blis-registry \
  --num-instances 4 --prefill-instances 3 --decode-instances 1 \
  --rate 10 --num-requests 200

Which requests are disaggregated is the router's decision: the EndpointPickerConfig's disagg-profile-handler and its P/D decider. This scenario's router: block names one that disaggregates every request.

→ Cluster simulation

KV caching and offload

Block-level KV accounting with prefix caching, and tiered offload whose tiers name catalog storage devices (--kv-offload-config). → KV cache, KV offload

Autoscaling

Horizontal autoscaling over node pools with provisioning delays, configured in a --policy-config YAML (each pool's gpu_type must equal the scenario's hardware). → Cluster simulation

Trace replay

Replay any TraceV2 — exported by blis run --trace-output, or converted with blis convert weka|otel|inference-perf|servegen — through the same kernel and deployment path. A run's exported trace replayed with identical flags (including --horizon) gives byte-identical stdout.

./blis replay --trace-header t.yaml --trace-data d.csv \
  --scenarios examplescenarios --scenario gpt-oss-120b-h100-fp4-vllm-tp2-2node.yaml \
  --registry $PWD/blis-registry

→ Observe / Replay / Calibrate. blis observe and blis calibrate are deprecated (#1901).

Analysis

Saturation detection (--detectors), decision tracing with counterfactual regret, per-SLO-class metrics and fitness scoring. → Interpreting Results

---

Documentation

Section	Description
Getting Started	Installation, quick start, capacity planning tutorial
User Guide	Task-oriented guides for every feature above
Concepts	Architecture, core engine, glossary
Reference	CLI flags, supported models, workload spec schema, project structure
Methodology	Strategy Evolution methodology, discovered principles
Contributing	Extension recipes, PR workflow, design process, standards
Roadmap	Where BLIS is going: llm-d parity, day-0 models, multi-node, networking, accuracy, config search

---

Roadmap

We plan to bring BLIS to llm-d-perf-simulator as the community-maintained llm-d performance simulator. The direction, guided by llm-d parity by design, spans six themes: llm-d parity, day-0 model support, multi-node accuracy, networking configs, accuracy transparency, and config search. See the full Roadmap.

---

Contributing

See CONTRIBUTING.md for the engineering standards, development workflow, and guides for adding components.

Citation

If you use BLIS in your research, please cite our paper (accepted at ACM SIGMETRICS, to appear). Preprint: arXiv:2610.14255.

@article{guhathakurta2026blis,
  title     = {Simulate Before You Scale: High-Fidelity LLM Inference Planning Without GPUs},
  author    = {Guhathakurta, Dipanwita and Toslali, Mert and Parthasarathy, Srinivasan and Chen, Jing and Kalantar, Michael and Masluk, Nicholas A. and Ramani, Vishakha and Tantawi, Asser and Oliveira, Fabio},
  journal   = {Proc. ACM Meas. Anal. Comput. Syst.},
  publisher = {Association for Computing Machinery},
  year      = {2026},
  note      = {To appear},
  eprint    = {2610.14255},
  archivePrefix = {arXiv},
  primaryClass  = {cs.PF}
}

License

Apache License, Version 2.0. See LICENSE.