llm-d is a high-performance distributed inference serving stack optimized for production deployments on Kubernetes. We help you achieve the fastest "time to state-of-the-art (SOTA) performance" for key OSS large language models across most hardware accelerators and infrastructure providers, and sustain it in production operations with multi-tenant quality of service, zero-downtime upgrades, and agent-aware serving, all backed by well-tested guides and real-world benchmarks.
llm-d is a Cloud Native Computing Foundation (CNCF) sandbox project, founded by Red Hat, Google Cloud, IBM Research, CoreWeave, and NVIDIA.
What does llm-d offer to production inference?
Model servers like vLLM and SGLang handle efficiently running large language models on accelerators. llm-d provides state-of-the-art orchestration and optimizations above model servers to serve high-scale real-world traffic efficiently and reliably. Our offerings are organized into the following themes:
- Intelligent Routing: Maximize performance with prefix-cache and load-aware balancing, including experimental predicted latency-based scheduling to decrease latency and increase throughput.
- Advanced KV-Cache Management: Increase the effective "working set size" for multi-turn requests with tiered offloading to CPU or disk and precise global indexing of the KV cache state.
- Serving Large Models: Optimize massive models (e.g., DeepSeek-V4, Kimi K3, gpt-oss-120b) using prefill/decode disaggregation and wide expert-parallelism over fast accelerator interconnects.
- Agent-Aware Serving: Serve multi-turn agent workloads natively with session-aware routing, turn-priority scheduling, and native Responses API state, reusing context across turns instead of recomputing it.
- Operational Excellence: Run inference as a Day-2 service with multi-tenant quality of service (priority- and fairness-aware flow control), zero-downtime operations (router high availability and lockstep prefill/decode rollouts), SLO-aware autoscaling, and end-to-end observability.
- Batch Processing: Efficiently manage large-scale offline inference with OpenAI-compatible Batch APIs and asynchronous processing to maximize hardware utilization.
For a complete list of tested recipes and architectural patterns, see our well-lit path guides. These guides provide benchmarked recipes and Helm charts to start serving quickly with best practices common to production deployments. Our intent is to eliminate the heavy lifting common in tuning and deploying generative AI inference on modern accelerators.
Performance and Adoption Highlights
Validated performance gains from production deployments and partner benchmarks:
- 3x higher output throughput and 2x faster TTFT with prefix-cache-aware routing vs round-robin, on Llama 3.1 70B with 4× AMD MI300X, Tesla / Red Hat (blog)
- 40% reduction in TTFT and ITL with predicted-latency scheduling vs heuristics on NVIDIA GPUs, Google (blog)
- Up to 70% higher tokens/sec with prefill/decode disaggregation vs standard vLLM, on GPT-OSS with NVIDIA B200 (p6-b200), AWS (blog)
- 10–30% throughput improvement with disaggregated serving on identical infrastructure, for GPT-OSS-120B and Llama 3.3 70B on AMD MI300X, Oracle (blog)
- 50k tokens/sec cluster throughput with Wide Expert-Parallelism on 16×16 NVIDIA B200, ~3.1k tok/s per GPU (blog)
- 13.9x throughput improvement with hierarchical KV offloading at 250 concurrent users vs GPU-only, on 4× NVIDIA H100 (blog)
Explore detailed, reproducible benchmarks on Prism.
A growing roster of adopters runs llm-d in production across clouds, accelerator vendors, and enterprises:
This is the beginning of the README. Read the rest at the original. Read the full README ↗
