AIBrix
AIBrix is a Kubernetes control plane for production LLM inference. You bring the engine — vLLM, SGLang, or TensorRT-LLM — and AIBrix routes each request from live engine state, scales on inference metrics, and manages models, LoRA adapters, and KV cache.
Use it when you run many models or replicas, mix GPU types, or need to hold cost and latency targets.
Features
| Capability | What it does |
|---|
| Gateway and routing | OpenAI-compatible API. Picks a pod from live engine state, including prefix-cache, load, and prefill/decode placement, with session affinity for agentic workloads. |
| Multi-engine | Serve vLLM, SGLang, and TensorRT-LLM behind one gateway. |
| LoRA | Pack many low-rank adapters onto shared base-model pods. |
| Model lifecycle | Declare models with ModelClaim, and preload images and weights with ModelWarmup so scale-up is faster. |
| Autoscaling | Scale replicas on engine metrics such as queue depth, KV cache, and tokens. |
| Distributed inference | Run multi-node and multi-role deployments, including prefill/decode, with StormService. |
| Distributed KV cache | Offload KV cache and reuse it across engines. |
| Heterogeneous GPUs | Mix GPU types and still target cost and SLO. Experimental. |
| Runtime | Sidecar that downloads weights, loads and unloads adapters, and normalizes engine metrics. |
Architecture
Requests hit an OpenAI-compatible gateway and go straight to an engine pod. Kubernetes controllers decide what should be running: model lifecycle, warmup, multi-node topology, autoscaling, and observability. The architecture guide walks through both planes.
Quick start
These commands install AIBrix, including its Envoy Gateway dependency, into the cluster your current kubectl context points at. KubeRay is included and is only required for Ray-based multi-node inference.
The snippet installs the latest release, v0.8.0. Change VERSION to install a different release.
Install a release
VERSION=v0.8.0
# Dependencies (Envoy Gateway, KubeRay, and friends)
kubectl apply -f "https://github.com/vllm-project/aibrix/releases/download/${VERSION}/aibrix-dependency-${VERSION}.yaml" --server-side
# CRDs ship separately so uninstalling the operator does not delete user resources
kubectl apply -f "https://github.com/vllm-project/aibrix/releases/download/${VERSION}/aibrix-core-crds-${VERSION}.yaml" --server-side
# Controllers, gateway plugins, and supporting services
kubectl apply -f "https://github.com/vllm-project/aibrix/releases/download/${VERSION}/aibrix-core-${VERSION}.yaml"
kubectl get pods -n aibrix-system
When the pods are ready, deploy a model and call the gateway. The quickstart has a base-model manifest and the OpenAI-compatible curl commands. Helm install steps are in the installation guide.
Install from this checkout
Images in config/default are tagged nightly.
git clone https://github.com/vllm-project/aibrix.git
cd aibrix
kubectl apply -k config/dependency --server-side
kubectl apply -k config/crd --server-side
kubectl apply -k config/default
Documentation
Full guides live at aibrix.readthedocs.io.
News
Releases
Talks
Community
Contributions are welcome. Read the contributing guide, talk in #aibrix on the vLLM Slack, or email [email protected].
Bugs and feature requests go on GitHub Issues. AIBrix is licensed under Apache 2.0.
