Mooncake

Mooncake is the serving platform for Kimi, a leading LLM service provided by Moonshot AI. Under real workloads, Mooncake’s innovative architecture enables Kimi to handle 75% more requests while adhering to SLOs.

• Nov 8, 2026: AMD ATOM integrated Mooncake for shared KV cache pooling and efficient KV transfer. In collaboration with NeuroSpark AI, the integration achieved over 95% prefix reuse and up to 2× higher throughput.
• Oct 10, 2026: TensorRT-LLM merged a native Mooncake Store connector into KVCMv2, enabling content-addressed KV cache pooling and prefix reuse across serving instances.
• Oct 6, 2026: TokenSpeed v0.1.1 includes Mooncake Store as an L3 KV cache backend beneath its host cache, alongside Mooncake Transfer Engine support for PD and EPD disaggregation.
• Sept 17, 2026: vLLM added cross-encoder embedding reuse through Mooncake Store, complementing its encoder-cache transfer connector for multimodal disaggregated serving. Both capabilities are included in vLLM v0.31.0.
• Aug 28, 2026: LMDeploy merged Mooncake Store support, enabling shared KV cache pooling and prefix reuse across PyTorch inference instances. Included in LMDeploy v0.18.0.

This is the beginning of the README. Read the rest at the original. Read the full README ↗