llmcompressor is the fast, efficient, and easy-to-use library for optimizing models for deployment with vLLM, including:
- Comprehensive set of quantization algorithms and transforms for weight, activation, KV cache, and attention quantization
- Seamless integration with Hugging Face models and repositories
- Models saved in the
compressed-tensors format, compatible with vLLM
- DDP and disk offloading support for compressing very large models with hardware efficiency
✨ Read the announcement blog here! ✨
💬 Join us on the vLLM Community Slack and share your questions, thoughts, or ideas in:
#sig-quantization#llm-compressor
🚀 What's New!
Big updates have landed in LLM Compressor! To get a more in-depth look, check out the LLM Compressor overview.
Since the v0.14.0 release, a number of meaningful improvements have landed:
- HIGGS mixed-precision quantization: HIGGS collects layer-wise MSE sensitivity data and formulates an integer linear program to select an optimal mixed-precision configuration for a target average bitwidth. The resulting config can be passed to
oneshot or model_free_ptq.
- Block-wise QAD: A new
QADModifier runs quantization-aware distillation after a weight-quantization modifier in the same recipe. It caches calibration inputs and the unquantized subgraph output as the teacher, then trains the quantized weights through compressed-tensors fake quantization, without loading a separate teacher model.
- Layerwise compression and sequential offloading: The sequential pipeline can now decompress and compress one layer at a time, avoiding upfront whole-model conversion for large checkpoints. Onload/offload is managed manually with optional CPU staging and pinned memory.
- REAP reports and non-uniform pruning: REAP now writes per-expert saliency reports, supports calibration-only report generation with
sparsity=0.0, and accepts per-layer non-uniform expert sparsity. n_routed_experts is now recorded in config names, and GPT-OSS is skipped.
- MoNe expert pruning: MoNe collapses low-saliency experts into constant "novice" vectors instead of removing them, keeping the router at full width while reducing expert compute.
- Faster observers for E8M0 formats: The MSE and iMatrix observers gained exponential grid search and magnitude normalization for E8M0 scales, and the iMatrix observer now shares the MSE Triton kernel, with roughly 1.2-3.4x end-to-end speedups on tested models.
- MFPTQ SpinQuant rotations:
SpinQuantConverter applies offline R1/R2 Hadamard rotations directly to safetensors before model-free quantization, and model_free_ptq now accepts converter chains.
- MXFP6 and MXFP4xFP8 support: Added MXFP6A8/MXFP6A16 schemes and examples, plus an MXFP4xFP8 example and refreshed quantization documentation.
- Broader hardware and model support: The fused GPTQ Triton kernel now runs on Intel XPU, MTP layers load through Transformers
MtpModel, and new checkpoints and examples include Nemotron 3.5 Super VL FP8, Zyphra Zaya1, Muse Spark, and Qwen3.6 MXFP4xFP8.
- Docs and community: Added the Open Research Hub for community experiment results and a full Simplified Chinese README translation.
Model highlights
The Red Hat AI team has been using LLM Compressor to produce a fresh batch of production-ready quantized checkpoints:
This is the beginning of the README. Read the rest at the original. Read the full README ↗
