GPUStack

Overview

GPUStack is an open-source GPU cluster manager for AI model serving and GPU instance provisioning. It configures and orchestrates inference engines — vLLM, SGLang, TensorRT-LLM, or your own — and lets you launch SSH-accessible GPU instances on demand. Its core features include:

• Multi-Cluster GPU Management. Manages GPU clusters across on-premises servers, Kubernetes clusters, and cloud providers. Supports GPU slicing for model deployments and GPU instances, with configurable VRAM and compute allocations.
• Pluggable Inference Engines. Automatically configures high-performance inference engines such as vLLM, SGLang, and TensorRT-LLM. You can also add custom inference engines as needed.
• Day 0 Model Support. GPUStack's pluggable engine architecture enables you to deploy new models on the day they are released.
• Performance-Optimized Configurations. Offers pre-tuned modes for low latency or high throughput. Supports prefill/decode disaggregation with independently configurable replicas and resources for each phase. Platform-managed shared KV cache lets compatible deployments and replicas reuse cached prefixes, with L2 storage extending capacity and reuse across nodes. It also includes built-in support for speculative decoding methods such as EAGLE3, MTP, and N-grams.
• Built-in Benchmarking. Auto Tune automatically explores concurrency or request rates to find peak throughput or the highest concurrency that meets your service-level objectives (SLOs).
• GPU Instances. Launches SSH-accessible GPU instances on demand for development, fine-tuning, and interactive workloads.
• Enterprise-Grade Operations. Offers support for automated failure recovery, load balancing, monitoring, authentication, and access control.

Architecture

GPUStack enables development teams, IT organizations, and service providers to deliver Model-as-a-Service at scale. It supports industry-standard APIs for LLM, voice, image, and video models. The platform includes built-in user authentication and access control, real-time monitoring of GPU performance and utilization, and detailed metering of token usage and API request rates.

The figure below illustrates how a single GPUStack server can manage multiple GPU clusters across both on-premises and cloud environments. The GPUStack scheduler allocates GPUs to maximize resource utilization and selects the appropriate inference engines for optimal performance. Administrators also gain full visibility into system health and metrics through integrated Grafana and Prometheus dashboards.

Cluster Visibility at a Glance

The GPU Cluster Topology view in GPUStack Enterprise provides a live, bird's-eye view of your entire fleet — every worker, GPU, and model deployment in one place, with real-time utilization, allocation, and health status.

Optimized Inference Performance

GPUStack's automated engine selection and parameter optimization deliver strong inference performance out of the box. The following figure shows throughput improvements over default vLLM configurations:

For detailed benchmarking methods and results, visit our Inference Performance Lab.

Discover what your deployment can handle with GPUStack's built-in benchmarking. Auto Tune adaptively adjusts load to uncover peak throughput or find the highest concurrency that meets your service-level objectives (SLOs), with detailed throughput and latency charts to guide capacity planning.

Supported Accelerators

GPUStack supports a wide range of accelerators for AI inference:

• NVIDIA GPU
• AMD GPU
• Ascend NPU
• Hygon DCU
• MThreads GPU
• Iluvatar GPU
• MetaX GPU
• Cambricon MLU
• T-Head PPU

For detailed requirements and setup instructions, see the Installation Requirements documentation.

Quick Start

Prerequisites

1. A node with at least one NVIDIA GPU. For other GPU types, please check the guidelines in the GPUStack UI when adding a worker, or refer to the Installation documentation for more details.
2. Ensure the NVIDIA driver, Docker and NVIDIA Container Toolkit are installed on the worker node.
3. (Optional) A CPU node for hosting the GPUStack server. The GPUStack server does not require a GPU and can run on a CPU-only machine. Docker must be installed. Docker Desktop (for Windows and macOS) is also supported. If no dedicated CPU node is available, the GPUStack server can be installed on the same machine as a GPU worker node.
4. Only Linux is supported for GPUStack worker nodes. If you use Windows, consider using WSL2 and avoid using Docker Desktop. macOS is not supported for GPUStack worker nodes.

Install GPUStack

Run the following command to install and start the GPUStack server using Docker:

sudo docker run -d --name gpustack \
    --restart unless-stopped \
    -p 80:80 \
    --volume gpustack-data:/var/lib/gpustack \
    gpustack/gpustack

If you cannot pull images from Docker Hub or the download is very slow, you can use our Quay.io mirror by pointing your registry to quay.io:

sudo docker run -d --name gpustack \
    --restart unless-stopped \
    -p 80:80 \
    --volume gpustack-data:/var/lib/gpustack \
    quay.io/gpustack/gpustack \
    --system-default-container-registry quay.io

Check the GPUStack startup logs:

sudo docker logs -f gpustack

This is the beginning of the README. Read the rest at the original. Read the full README ↗