VeloxQuant-MLX

介绍

Fast KV-cache quantization for Apple Silicon (MLX) — 43 research-adapted compression methods with Metal kernels

翻译准备中

取自仓库 README

README

正在加载完整 README。 以下为已保存的摘录。

43 compression methods — quantizers, token-eviction caches, cross-layer merging — in MLX VeloxQuant-MLX shrinks the KV cache of any mlx_lm model on Apple Silicon, up to 16× smaller with near-lossless quality, in three lines of code. If you run models locally and keep hitting a context-length or memory wall, you swap in a compressed cache and change nothing else about the model.