FlashInfer is a library and kernel generator for inference that delivers state-of-the-art performance across diverse GPU architectures. It provides unified APIs for attention, GEMM, and MoE operations with multiple backend implementations including FlashAttention-2/3, cuDNN, CUTLASS, and TensorRT-LLM.
Why FlashInfer?
- State-of-the-art Performance: Optimized kernels for prefill, decode, and mixed batching scenarios
- Multiple Backends: Automatically selects the best backend for your hardware and workload
- Modern Architecture Support: Support for SM75 (Turing) and later (through Blackwell)
- Low-Precision Compute: FP8 and FP4 quantization for attention, GEMM, and MoE operations
- Production-Ready: CUDAGraph and torch.compile compatible for low-latency serving
Core Features
Attention Kernels
- Paged and Ragged KV-Cache: Efficient memory management for dynamic batch serving
- Decode, Prefill, and Append: Optimized kernels for all attention phases
- MLA Attention: Native support for DeepSeek's Multi-Latent Attention
- Cascade Attention: Memory-efficient hierarchical KV-Cache for shared prefixes
- Sparse Attention: Block-sparse and variable block-sparse patterns
- POD-Attention: Fused prefill+decode for mixed batching
GEMM & Linear Operations
- BF16 GEMM: BF16 matrix multiplication for SM10.0+ GPUs.
- FP8 GEMM: Per-tensor and groupwise scaling
- FP4 GEMM: NVFP4 and MXFP4 matrix multiplication for Blackwell GPUs
- Grouped GEMM: Efficient batched matrix operations for LoRA and multi-expert routing
Mixture of Experts (MoE)
- Fused MoE Kernels
- Multiple Routing Methods: DeepSeek-V3, Llama-4, and standard top-k routing
- Quantized MoE: FP8 and FP4 expert weights with block-wise scaling
Sampling & Decoding
- Sorting-Free Sampling: Efficient Top-K, Top-P, and Min-P without sorting
- Speculative Decoding: Chain speculative sampling support
Communication
- AllReduce: Custom implementations
- Multi-Node NVLink: MNNVL support for multi-node inference
- NVSHMEM Integration: For distributed memory operations
Other Operators
- RoPE: LLaMA-style rotary position embeddings (including LLaMA 3.1)
- Normalization: RMSNorm, LayerNorm, Gemma-style fused operations
- Activations: SiLU, GELU with fused gating
GPU Support
| Architecture | Compute Capability | Example GPUs |
|---|
| Turing | SM 7.5 | T4, RTX 20 series |
| Ampere | SM 8.0, 8.6 | A100, A10, RTX 30 series |
| Ada Lovelace | SM 8.9 | L4, L40, RTX 40 series |
| Hopper | SM 9.0 | H100, H200 |
| Blackwell | SM 10.0, 10.3 | B200, B300 |
| Blackwell | SM 11.0 | Jetson Thor |
| Blackwell | SM 12.0, 12.1 | RTX 50 series, DGX Spark |
Note: Not all features are supported across all compute capabilities.
News
Latest:
Notable updates:
- [2025-10-08] Blackwell support added in v0.4.0
- [2025-03-10] Blog Post Sorting-Free GPU Kernels for LLM Sampling, which explains the design of sampling kernels in FlashInfer.
Getting Started
Installation
Quickstart:
pip install flashinfer-python
Package Options:
- flashinfer-python: Core package that compiles/downloads kernels on first use
- flashinfer-cubin: Pre-compiled kernel binaries for all supported GPU architectures
- flashinfer-jit-cache: CUDA-specific shim that installs architecture-specific pre-built kernel providers
For faster initialization and offline usage, install the optional packages to have most kernels pre-compiled:
pip install flashinfer-python
flashinfer download-kernels
flashinfer download-kernels installs the flashinfer-cubin and flashinfer-jit-cache wheels that match the installed flashinfer-python and CUDA versions. By default it installs pre-built kernels for every supported GPU architecture (several GB); pass --mode minimal to install the jit-cache provider only for the visible GPUs, or --sm (e.g. --sm sm90a, repeatable) to choose architectures explicitly.
Upgrading: the kernel wheels must match flashinfer-python exactly and are not upgraded with it. Run flashinfer download-kernels again after every pip install -U flashinfer-python; until then FlashInfer warns at import and ignores the old wheels (falling back to on-demand downloads and JIT compilation). Also note that flashinfer-cubin is no longer published to PyPI (0.6.13 was the last release there): a plain pip install -U flashinfer-cubin installs that stale wheel, so install it through flashinfer download-kernels or from --index-url https://flashinfer.ai/whl. See Upgrading.
For Blackwell (SM100+) CuTe DSL kernels, install with the CUDA 13 extra to enable Blackwell-optimized kernels:
pip install flashinfer-python[cu13]
Verify Installation
flashinfer show-config
Basic Usage
import torch
import flashinfer
# Single decode attention
q = torch.randn(32, 128, device="cuda", dtype=torch.float16) # [num_qo_heads, head_dim]
k = torch.randn(2048, 32, 128, device="cuda", dtype=torch.float16) # [kv_len, num_kv_heads, head_dim]
v = torch.randn(2048, 32, 128, device="cuda", dtype=torch.float16)
output = flashinfer.single_decode_with_kv_cache(q, k, v)
See documentation for comprehensive API reference and tutorials.
This is the beginning of the README. Read the rest at the original. Read the full README ↗
