vLLM Router

A high-performance and light-weight request forwarding system for vLLM large scale deployments, providing advanced load balancing methods and prefill/decode disaggregation support.

Key Features

• Core Architecture: Request routing framework and async processing patterns
• Load Balancing: Multiple algorithms (cache-aware, power of two, consistent hashing, random, round robin)
• Program Scheduling: Optional Program identity, RequestPool admission, Progress-TTL, and Global Queue placement for agent workloads
• Prefill-Decode Disaggregation: Specialized routing for separated processing phases
• Service Discovery: Kubernetes-native worker management and health monitoring
• Enterprise Features: Circuit breakers, retry logic, metrics collection
• gRPC workers: Route to vLLM Rust Inference workers via grpc:// URLs

Quick Start

Prerequisites

Rust and Cargo:

# Install rustup (Rust installer and version manager)
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

# Follow the installation prompts, then reload your shell
source $HOME/.cargo/env

# Verify installation
rustc --version
cargo --version

Python with pip installed

Installation & Basic Usage

Rust Binary

# Build Rust components
cargo build --release

Python Package

Install from PyPI

pip install vllm-router                                                                                                                                                        ```

To build from source:
```bash
pip install setuptools-rust wheel build
python -m build
pip install dist/*.whl

# Rebuild & reinstall in one step during development
python -m build && pip install --force-reinstall dist/*.whl

Usage Examples

Standard Data Parallelism Routing

# Launch router with data parallelism (8 replicas per worker URL)
# When data-parallel-size > 1, the router automatically creates DP-aware workers
./target/release/vllm-router \
    --worker-urls http://worker1:8000 http://worker2:8000 \
    --policy consistent_hash \
    --intra-node-data-parallel-size 8

# Alternative: using cargo run
cargo run --release -- \
    --worker-urls http://worker1:8000 http://worker2:8000 \
    --policy consistent_hash \
    --intra-node-data-parallel-size 8

# Alternative: using python launcher
vllm-router \
  --worker-urls http://worker1:8000 http://worker2:8000 \
    --policy consistent_hash \
    --intra-node-data-parallel-size 8

Optional WASM OnRequest middleware

Load an independently built WASM Component plugin (see examples/wasm_middleware/ and RFC #236). By default it attaches only to POST /v1/chat/completions and fails closed on plugin errors:

./examples/wasm_middleware/build.sh

./target/release/vllm-router \
    --worker-urls http://localhost:8000 \
    --wasm-middleware ./examples/wasm_middleware/wasm_middleware_example.component.wasm \
    --wasm-middleware-route /v1/chat/completions

Additional paths can be attached with repeated --wasm-middleware-route flags (must be one of the protected inference routes). Without --wasm-middleware, the Router does not initialize Wasmtime.

v0.1 resource / fail-closed defaults on attached routes (not configurable via CLI yet):

• Input body cap: min(10 MiB, --max-payload-size). Requests larger than this get 413 before the plugin runs, even if the plugin would only Continue. This is intentionally tighter than the Router's default 512 MiB payload limit.
• Execution deadline: 100 ms per invocation (Wasmtime epoch interruption). Deadline / trap failures fail closed with 500.
• Queue full: when the bounded worker queue is saturated, matching requests get 503.

Prometheus metrics for the WASM runtime are deferred to a later revision.

Prefill-Decode Disaggregation

# When vLLM runs the NIXL connector, prefill/decode URLs are required.
# See a working example in scripts/llama3.1/ folder.
cargo run --release -- \
    --policy consistent_hash \
    --vllm-pd-disaggregation \
    --prefill http://127.0.0.1:8081 \
    --prefill http://127.0.0.1:8082 \
    --decode http://127.0.0.1:8083 \
    --decode http://127.0.0.1:8084 \
    --decode http://127.0.0.1:8085 \
    --decode http://127.0.0.1:8086 \
    --host 127.0.0.1 \
    --port 8090 \
    --intra-node-data-parallel-size 1 \


# When vLLM runs the NCCL connector, ZMQ based discovery is supported.
# See a working example in scripts/install.sh
cargo run --release -- \
    --policy consistent_hash \
    --vllm-pd-disaggregation \
    --vllm-discovery-address 0.0.0.0:30001 \
    --host 0.0.0.0 \
    --port 10001 \
    --prefill-policy consistent_hash \
    --decode-policy consistent_hash

# When vLLM runs the Mooncake connector, pass --kv-connector mooncake.
# The router queries each prefill node's Mooncake bootstrap server at startup
# to learn engine_id per DP rank, and injects transfer_id / remote_bootstrap_addr /
# remote_engine_id into each request's kv_transfer_params for P/D coordination.
cargo run --release -- \
    --policy consistent_hash \
    --vllm-pd-disaggregation \
    --kv-connector mooncake \
    --prefill http://127.0.0.1:8081 \
    --prefill http://127.0.0.1:8082 \
    --decode http://127.0.0.1:8083 \
    --decode http://127.0.0.1:8084 \
    --host 127.0.0.1 \
    --port 8090 \
    --intra-node-data-parallel-size 1

This is the beginning of the README. Read the rest at the original. Read the full README ↗