llama-modchip (v2.5.4)

3.00–3.75 bpw Vector & Dyadic CPU Inference for 284B & 30B MoE Models

Continuous WikiText-2 & AtomicChat wiki-alt PPL · Logit KLD · 100-Pt Gate · Linux, macOS Metal & Windows 11

Quickstart · Signal Fidelity & Benchmarks · How 3D VQ & Fractal-64 Work · Quantization Specs · Supported Models · Leaderboard · v2.4 Paper Snapshot (paper/) · FAQ

Overview

Mixture-of-Experts (MoE) inference on CPU is bounded primarily by DRAM bandwidth per generated token. Standard 4.50-bpw scalar quantization (Q4_0) leaves significant memory-bandwidth headroom unused, while conventional 2–3 bit scalar K-quants (Q2_K, Q3_K) require multi-shift bit unpacking and fall back to Q4_0 whenever expert matrix dimensions are not multiples of 256 (such as 1856 × 2688 in Nemotron-3.5-30B-A3B).

llama-modchip v2.5.4 implements two block-64 quantization formats engineered for SIMD register lookup—Fractal-64 (GGML_TYPE_Q2_0 = 42, 30 bytes / 64 weights = 3.75 bpw) and LUT-VQ3-64 (GGML_TYPE_VQ3_0 = 43, 24 bytes / 64 weights = 3.00 bpw)—paired with cache-aware KV management (HARP-Cache), /dev/shm Slot-0 prefix snapshots, and RouteCrack + Stockfish-PVS v2 speculative decoding:

以上为 README 的开头部分。其余内容请阅读原文。 查看完整 README ↗